[SciPy-user] Real-time plotting and data storage format questions

Rick Wagner rwagner@physics.ucsd....
Thu Apr 3 03:19:13 CDT 2008


> I second the pytables recommendation, but if your simulations are in C
> and want to avoid additional dependencies, you might simply just  
> want to
> write binary data direct to disk in your own format. Numpy's memmap
> function can then use them directly if your data are in same-length
> rows. Note that the term "rows" does not hint at what is actually
> possible. For example, an n-dimensional array can be one "column" of
> each such "row".
>
> -Andrew
>


I have to disagree with using raw binary data, if data portability is  
at all a concern. I use binary formats often, but only as a quick and  
dirty way get data out of C. If you go with binary data, you will  
soon find yourself defining a custom format that includes array  
sizes, endianess, precision, etc. Which is why formats like HDF,  
NetCDF and FITS were invented. And HDF5's simple API makes it very  
convenient to write datasets from C. Plus, you get the HDF5 command  
line tools which allow you to inspect your data without needing to  
write your own custom tools.

Also, to clarify the PyTables aspect, I think HDF5 is useful, and  
PyTables provides a convenient Python API to HDF5 data. However, I  
would not suggest using PyTables for storing the data, unless you  
were writing everything in Python.

OK, that was longer and more opinionated than I intended. There must  
be a bad experience with raw binary data somewhere in my past (or  
several).

--Rick

>
> Gael Varoquaux wrote:
>> On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
>>
>>> For the data format I strongly recommend NetCDF and especially  
>>> NetCDF4
>>> (allows for compression)
>>>
>>
>> Actually, why not hdf5, which seems to be used by NetCDF4, but is  
>> also
>> largely used across many scientific comunities, and very well  
>> supported
>> under python (pytables)?
>>
>> My 2 cents,
>>
>> Gaël
>> _______________________________________________
>> SciPy-user mailing list
>> SciPy-user@scipy.org
>> http://projects.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>

------------------------------------------------------------------------ 
-
Rick Wagner, Graduate Student Researcher
UCSD Physics
9500 Gilman Drive
La Jolla, CA  92093-0424
Email:  rwagner@physics.ucsd.edu
WWW:    http://lca.ucsd.edu/projects/rpwagner
(858) 822-4784 Phone
------------------------------------------------------------------------ 
-
Measuring programming progress by lines of code is
like measuring aircraft building progress by weight.
--Bill Gates
------------------------------------------------------------------------ 
-



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-user/attachments/20080403/5885fe83/attachment.html 


More information about the SciPy-user mailing list