[SciPy-user] storing numpy arrays

Robert Kern robert.kern@gmail....
Fri Mar 13 13:22:40 CDT 2009


On Fri, Mar 13, 2009 at 12:36, Brian Granger <ellisonbg.net@gmail.com> wrote:
> Another option that could be better is to simply store the name of a
> file containing the array in the db and then just write the array to
> the file (use the array hash as a filename) using the NPY format.
> Depending on the size and number of arrays, there could be a
> performance benefit to this approach compared to storing the full
> array in the db.

That works well if the database is on the same system as the program.
If you are connecting to it remotely, you will have to write a server
to expose those files over the network. A simple HTTP server would
probably suffice. My use case was lots of smallish arrays (~1000
elements) rather than fewer big arrays, so BLOB storage is adequate, I
think.

It would be great if the database itself could do that transparently,
though. You would declare the column as a BLOBFILE, and it would only
store a reference to a file in its workspace. Ah well.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the SciPy-user mailing list