[Numpy-discussion] create a numpy array of images

Christopher Barker Chris.Barker@noaa....
Wed Feb 2 11:12:47 CST 2011


> It seems that using 64 bit python is the solution.

It's certainly the easy way to access a lot of memory -- and memory is 
cheap these days.

> But the thing is i would
> compile my code and wanna distribute it to the clients..

I don't think 64 bit gets in the way of that -- except that it will only 
runon 64 bit systems, which may be an issue.

> only reason why i want to work on 32 bit system. Sturla, how I can make it
> sure that some part of the data is kept on the disk and only the necessary
> one in the memory; as this seems to be a solution to my problem.

You can "roll your own" and have a disk cache of some sort -- it would 
be pretty easy to store each image in a *.npz file and load the up as 
you need them.

But it would probably be even easier to use one of the hdf-based 
libarries, such as pytables -- I htink it will do it all for you.

One other option, that I've never tried, is carray, which is an array 
compressed in memory. Depending on your images, perhaps they would 
compress a lot (or not ....):

https://github.com/FrancescAlted/carray
http://mail.scipy.org/pipermail/numpy-discussion/2010-August/052378.html



> As i said i
> want a 3d visualization out of the numpy array. it works fine for the
> downsampled dataset. And to visualize, i have to convert the 16bit data into
> 8bit as PIL doesnt support 16 bit data.

It's unclear to me what you native data really is: 16 bitgreyscale? 8bit 
greyscale? either one should fit OK into 32 bit memory, and if 8bit is 
accurate enough foryour needs, then it should be pretty easy.


> stack = numpy.empty((120, 1024, 1024))

numpy defaults to double precision float, np.float64, i.e. 8 bytes per 
element -- you probably don't want that if you are concerned about 
memoery use, and have 8 or 16 bit greyscale images. Try:

stack = np.empty((120, 1024, 1024), dtype=np.uint8) # (or dtype=np.uint16)

> i = 0
>
> os.chdir(dirr)
> for f in os.listdir(dirr):
>
>     im = Image.open(f)
>     im = im.convert("L")

you ight want to try mode "I". That should give you a 32 bit integer 
grey scale, which should hold all the 16 bit data without loss -- then 
you can convert to 16 bit when you bring it into numpy.

>     a = numpy.asarray(im)
>     print a.dtype

what does this print? it should be np.uint8

>     stack[i] = a

here, if a is a uint8, numpy will convert it to a float64, to fit into 
array a -- that's why you want to set the dtype of array a when you 
create it.

> one more thing, it really doesnt work for tiff files at all, i have to
> convert them into jpgs as a prior step to this.

probably not the best choice either, jpeg is lossy -- is will smear 
things out a bit, which you may not want. It also only holds 24 bit RGB 
(I think), which is both a waste and will lose information from 16 bit 
greyscale. If you have to convert, try PNG, though I'm not sure if it 
handles 16 bit greyscale, either.

I'd look at a lib that can read tiff properly -- some have been 
suggested here, and you can also use GDAL, which is meant for 
geo-referenced data, but you can ignore the geo information and just get 
an image if you want.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov


More information about the NumPy-Discussion mailing list