[Numpy-discussion] Thoughts on persistence/object tracking in scientific code
Wed Dec 24 07:21:16 CST 2008
On Tue, Dec 23, 2008 at 02:10:50AM +0100, Olivier Grisel wrote:
> Interesting topic indeed. I think I have been hit with similar problems on
> toy experimental scripts. So far the solution was always adhoc FS caches
> of numpy arrays with manual filename management. Maybe the first step for
> designing a generic solution would be to list some representative yet
> simple enough use cases with real sample python code so as to focus on
> concrete matters and avoid over engineering a general solution for
> philosophical problems.
Yes, that's clearly a first ste: list the usecases, and the way we would
like it solved: think about the API.
My internet connection is quite random currently, and I'll probably loose
it for a week any time soon. Do you want to start such a page on the
wiki. Mark it as a sratch page, and we'll delete it later.
I should point out that joblib (on PyPI and launchpad) was a first
attempt to solve this problem, so you could have a look at it. I have
already identified things that are wrong with joblib (more on the API
side than actual bugs), so I know it is not a final solution. Figuring
out what was wrong only came from using it heavily in my work. I thing
the only way forward it to start something, use it, figure out what's
wrong, and start again...
Looking forward to your input,
More information about the Numpy-discussion