[SciPy-user] non contiguous numpy arrays

Gary S. Thompson garyt at bmb.leeds.ac.uk
Thu Mar 16 05:57:42 CST 2006


>Hi
>    I have read the stuff on the numpy interface and understand most of 
>if(?) ;-) However I want to interface numpy to an array which is stored 
>as a series of data blocks which are not contiguous in memory... Help!
>
>so for example I hava 3d matrix of 1024*512*128 where data is stored in 
>blocks within memory which are 16x16x16 floats an read in from disk on 
>demand...
>
>regards
>gary
>  
>
>>i
>>>    I have read the stuff on the numpy interface and understand most of 
>>>if(?)  ;-)  However I want to interface numpy to an array which is stored 
>>>as a series of data blocks which are not contiguous in memory... Help!
>>>
>>>  
>>>
>>    
>>
>>>so for example I hava 3d matrix of 1024*512*128 where data is stored in 
>>>blocks within memory which are 16x16x16 floats an read in from disk on 
>>>demand...
>>>
>>>  
>>>
>>    
>>
> The memory model for numpy arrays requires that you can access the 
> next element in each dimension by striding a "fixed-number" of bytes. 
> In other words, to be a single ndarray, element i,j,k of the array, B, 
> must be at start_of_data + i*B.stride[0] + j*B.stride[1] + 
> k*B.stride[2] So, in your case, do these 16x16x16 blocks of floats all 
> exist at arbitrary memory locations? 


yep afraid so

> If so, then I don't see how you can map that to a single ndarray. You 
> could, however, write a class that wraps each block as a separate 
> sub-array and do the indexing calculations yourself to determine which 
> sub-block the data is stored in. But, I don't see a way to use a 
> single ndarray to access memory layed out like that. Perhaps I still 
> don't understand what you are doing. Are you using a memory map? -Travis 

no ;-) no memory maps I believe

So the problem I am trying to overcome is that the program I am using 
looks at large data matrices (100's of megabyte). Therefore it uses 
these submatrix formats to keep memory usage down if you are only 
lookiung at particular regions.

Mentally I was thinking of this as being a bit like a sparse matrix but 
with all the data present So is there sparse matrix supportfor numpy?  
Now this was something that wasn't clear do you access each value in the 
numpy array by a direct memory reference or by a function call when you 
want to use it...


regards
gary


-- 
-------------------------------------------------------------------
Dr Gary Thompson
Astbury Centre for Structural Molecular Biology,
University of Leeds, Astbury Building,
Leeds, LS2 9JT, West-Yorkshire, UK             Tel. +44-113-3433024
email: garyt at bmb.leeds.ac.uk                   Fax  +44-113-2331407
-------------------------------------------------------------------




More information about the SciPy-user mailing list