Parallelizing FSL – without the pain¶
FSL is a very popular analysis suite for neuroimaging data. Unfortunately, right out of the box FSL’s routines do not make good use of today’s multi-core machines. Consequently, researchers using FSL on individual workstations have difficulties harnessing the computing power of their equipment.
However, it is not the case that FSL doesn’t support modern computing environments. It is actually one of very few packages that come with built-in support for running computations on a cluster back-end. Unfortunately, it relies on the GridEngine software (formerly Sun, now from Oracle) to be installed locally, and configured properly. This is a task that is known to be non-trivial and can cause serious headache in a neuroscientist. While FSL has been ported to use other batch systems, such as Torque, none of them is much easier to deploy.
What would be helpful is a batch system that can be easily installed without complicated configuration procedures. Preferably, such a system is powerful enough to scale across multiple machines, whenever the demand for computing power increases. At NeuroDebian we looked at a number of projects and found Condor to be the most versatile tools, yet still straightforward to use in a single-machine scenario.
Over the past two years we have been working on getting Condor packaged for Debian. We are now very proud to announce that, a few days ago, Condor has been accepted as an official Debian package (in Debian experimental for the time being). NeuroDebian, as usual, provides backports of Condor for numerous Debian and Ubuntu releases.
With the availability of Condor packages, parallelizing FSL on a local workstation is now possible by following a simple four-step procedure.
1. Configure your machine to use the NeuroDebian repository¶
[ if you are already a NeuroDebian user you can skip this step ]
Go to http://neuro.debian.net and follow the setup instructions given on the front-page. This means selecting your Debian or Ubuntu release, select a package repository mirror close to you, and run the little shell script snippet that is presented to you. After having updated you package lists you are ready to go.
If you are not running Debian or Ubuntu on your machine, you can also install the NeuroDebian virtual machine which is able to utilize multiple CPU-cores. The remaining setup steps apply to VM users as well.
2. Install FSL¶
[ if you already have the latest FSL package version, you can skip this too ]
Open a terminal and install FSL being running the command:
% sudo apt-get install fsl
Of course you can also use any graphical package manager front-end (e.g. software center). The terminal commands are simply used for clarity and simplicity.
Check what version of FSL got installed. Any version from 4.1.9-2 onwards is fine. If you got an older version, you either haven’t succeeded doing step 1, or you are running a version of Debian or Ubuntu that is too old, and you would need to upgrade it first.
3. Install Condor¶
Install Condor with:
% sudo apt-get install condor
Any Condor package version from 7.7.4-1 onwards comes with all necessary features. During installation a few question will be asked. You should enable the Debconf-based configuration and ask for a “Personal Condor” installation (see screenshots).
Now you have a fully functional Condor installation that advertises one compute
slot per detected CPU-core on your machine. You can display the current status
with the condor_status command. On a dual-core machine this could look like
this:
% condor_status
Name OpSys Arch State Activity LoadAv Mem ActvtyTime
slot1@meiner LINUX INTEL Owner Idle 0.320 1972 0+00:00:04
slot2@meiner LINUX INTEL Owner Idle 0.000 1972 0+00:00:05
Total Owner Claimed Unclaimed Matched Preempting Backfill
INTEL/LINUX 2 2 0 0 0 0 0
Total 2 2 0 0 0 0 0
By default, Condor will only run jobs when the machine is idle. If you want to run jobs immediately when they are submitted you need to change the configuration slightly. To do this run:
% sudo dpkg-reconfigure htcondor
Note
26/06/2015 updated for condor to htcondor package rename
This will again ask a number of questions. Leave everything on the respective defaults, but enable running jobs regardless of other machine activity (see screen shot below).
4. Enjoy parallelized FSL¶
Now you are ready to make use of Condor with FSL. Source the configuration
from /etc/fsl/fsl.sh as usual:
% . /etc/fsl/fsl.sh
Whenever you want FSL to parallelize jobs through Condor, simply set the
FSLPARALLEL environment variable to condor and run FSL as usual:
% export FSLPARALLEL=condor
You can check whether the setup is working by running condor_q to see the
list of submitted/running jobs. Running the FEAT part of the FSL-FEEDS test
suite could look like this:
% condor_q
-- Submitter: meiner : <127.0.0.1:52379> : meiner
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
1.0 michael 3/9 14:47 0+00:00:07 R 0 0.7 zsh -c /usr/share/
2.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/
3.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster2_sentinel.
4.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/
5.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster4_sentinel.
6.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/
7.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster6_sentinel.
8.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/
9.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster8_sentinel.
10.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/
11.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster10_sentinel
11 jobs; 0 completed, 0 removed, 0 idle, 6 running, 5 held, 0 suspended
Done¶
There is obviously much more that could be said about other exciting features
of Condor, but we will leave this for another blog post in the future. As a
little teaser we could mention that the Debian Condor package comes with an
emulator of SGE’s qsub. Through this adaptor many software that can make use
of SGE can also be used with Condor, for example NiPyPE…
If you are now keen on exploring the possibilities of Condor, take a look at the
manual that comes in the condor-doc package.