Poor man’s parallel computing on multiple computers in Matlab

Let’s say that you need to run the same analysis with multiple datasets; for instance, you need to do reverse correlation with multiple cells. This might take a while, so you would like to run the analysis on multiple computers. The computers might be a bit different from each other, as well as the recording lengths, so you’d like to distribute the tasks so that the computers finish at roughly the same time. Finally, admit that you don’t have the license for multi-computer parallel processing with the parallel processing toolbox.

Here’s an easy method to get license-less parallel computing on multiple computers in Matlab going:

  1. Set computers so that on each computer the files that are needed correspond to the same path. If you’re on Windows, you could use Windows Share for this purpose; if on Linux, you can use sshfs. Thus, the .m files as well as all the other files required for the analysis can be found in z:\myanalysis or ~/myanalysis
  2. Create an .m file called addincludes.m that consists of calls to addpath that can be run prior to an analysis to ensure that all required files are in Matlab’s path on each computer
  3. Create a subfolder of the myanalysis folder called lockfiles and another called results
  4. Set up an .m file so that a call to RunMyAnalysis(number) or RunMyAnalysis(filenames{number}) will run the analysis on one cell, and save the results in the results folder
  5. Put this file on your path
  6. Save this as a function:
    function dosave(fname,varargin)
        for ii = 1:length(varargin)/2
            theargs.(varargin{ii*2-1}) = varargin{ii*2};
        end
        save(fname,'-struct','theargs');
    end
    
  7. Create a script similar to the following:
    addincludes;
    for ii = [1:length(filenames)]
        thename = filenames{ii};
        lockfile = sprintf('lockfiles/lockfile_%s.mat',filenames);
    
        if exist(lockfile,'file')
            continue;
        else
            dosave(lockfile,'compname',getComputerName());
        end
        try
            RunMyAnalysis(filenames{ii});
            dosave(lockfile,'success',1,'compname',getComputerName());
        catch me
            dosave(lockfile,'success',0,'compname',getComputerName(),'theerror',me.getReport());
        end
    end
    

You can replace the for with a (local) parfor if you have enough RAM. Then run the script on every computer. You can diagnose the thing by looking at the variables saved in the lockfiles.

About these ads
Poor man’s parallel computing on multiple computers in Matlab

One thought on “Poor man’s parallel computing on multiple computers in Matlab

  1. b says:

    Hi,
    Tnx for the post it indeed helpful.
    I believe you have a typo on line 4 – it should be “thename” instead “filenames”

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s