[Numpy-discussion] create a numpy array of images

Francesc Alted faltet@pytables....
Tue Feb 1 13:57:57 CST 2011


A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigué:
> Den 01.02.2011 18:58, skrev Christopher Barker:
> > But if you really have big collections of images, you might try
> > memory mapped arrays -- as Sturla pointed out they wont' let you
> > create monster arrays on a 32 bit python,
> 
> But they will on 64 bit Python :D We can just memory map a temporary
> file, and fake as much "memory" as we need. The OS will optimize the
> disk access.

What do you mean by 'optimize the disk access'?  One of the drawbacks of 
memory mapped files is precisely that OS cannot distinguish between data 
that belongs to 'disk' and data that belongs to 'memory'.  This normally 
introduces extreme slowness in other programs when datasets in files 
exceeds physical memory but have to loaded by OS -- the reason being 
that OS swaps out most of programs/shared libraries that were in memory 
in order to be able to load new 'disk' data.

The other important drawback of the memory mapped files is that you need 
to have at very least an amount of virtual memory that is enough to keep 
all of these data files.  In general, you only have virtual memory that 
is between 1.5x and 2x the physical memory (having more than this is 
generally regarded as a waste of disk space).

This is why I prefer very much reading directly from a file: the OS in 
this case is able to distinguish between data belonging to 'disk' and 
data belonging to 'memory'.  Is in this case when the OS can really 
optimize disk access (unless you have complicated setups).

> Also consider that hardware is cheap compared to labour,
> at least in Europe. Is programming for memory limited 32 bit Python
> worth the effort?

-- 
Francesc Alted


More information about the NumPy-Discussion mailing list