[Numpy-discussion] Followup on Python+MPI import performance
Thu Mar 15 12:38:14 CDT 2012
On Mon, Mar 5, 2012 at 10:17 AM, Asher Langton <email@example.com> wrote:
> This is a followup to my post from January
> and the panel discussion at PyData this weekend. As a few people have
> suggested, a better approach than the MPI-broadcasted lookups is to
> cache the locations of all the modules found in sys.path.
> I'll put an initial implementation of this importer on github sometime
> this week, and I'll follow up this post with some performance numbers
> when I have them.
Here are some numbers for the PEP302-based cached importer on an IBM
BlueGene/P machine. Numbers are wallclock measurements by the time
utility in minutes:seconds, one run for each test (not an average),
with no attempt to take into account other activity on the system or
fileservers. (With that said, I ran a variety of other tests, and the
results have been consistent.) I still need to run some larger tests,
particularly in the 16k-64k range, where Python imports start to scale
very poorly on this machine.
The tests use the code currently at github.com/langton/MPI_Import with
a script that simply imports 100 small C-extension modules.
With 1k cores/MPI processes:
- skip actual import : 13:37.77
- with checks : 27:09.60
- w/checks, no import: 26:23.63
cached_import.mpi4py_finder : 2:32.51
- skip actual import: 1:42.55
- with checks: 2:32.38
- w/checks, no import: 1:42.94
MPI_Import : 2:22.20
standard import : 15:43.63
- skip actual imports : 0:56.59
With 4k cores/MPI processes:
- skip actual import: 27:40.58
- with checks: 52:14.83
- w/checks, no import: 50:04.73
- skip actual import: 3:12.75
- with checks: 4:13.65
- w/checks, no import: 3:18.46
standard import : 35:24.77
- skip actual imports: 1:56.36
 Builds the cache, but omits the actual imports.
 Check whether modules in sys.path are readable while building the
cache. Because filesystem operations are expensive, these checks are
off by default.
 Only the rank 0 process builds the initial cache, which is then
broadcasted over MPI.
 The other import replacement.
 This is roughly the interpreter startup/initialization time.
More information about the NumPy-Discussion