[Numpy-discussion] lazy loading ndarray? (not from file, but from user function)

Robert Kern robert.kern@gmail....
Thu Aug 25 23:32:51 CDT 2011


On Thu, Aug 25, 2011 at 23:27, Kibeom Kim <kk1674@nyu.edu> wrote:
> Hello,
>
> Q1. Is lazy loading ndarray from user defined data supplying function possible?

No, not really.

> Q2. If possible, how can I implement it?
>
>
> The closest method I can think of is, (which requires c++ posix)
>
> 1. create a memory region using mmap and protect read operation by mprotect.
> 2. add SIGSEGV signal handler to trap read operation on the memory
> region, and the handler will provide appropriate user data and recover
> from SIGSEGV.
> 3. slightly modify memmap class to use the above mmap (memmap is
> already using mmap internally, so it's not a big deal)
>
> but obviously, recovering from SIGSEGV requires removing mprotect (see
> http://stackoverflow.com/questions/2663456/write-a-signal-handler-to-catch-sigsegv)
> and it's impossible to know when to lock the region by mprotect again.

Well, if you're willing to go *that* far, you might was well make a
userspace file system with fuse and mmap a file within that.

  http://fuse.sourceforge.net/

You can even implement it in Python!

  http://pypi.python.org/pypi/fuse-python
  http://code.google.com/p/fusepy/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the NumPy-Discussion mailing list