[Numpy-discussion] lazy loading ndarray? (not from file, but from user function)
Thu Aug 25 23:32:51 CDT 2011
On Thu, Aug 25, 2011 at 23:27, Kibeom Kim <firstname.lastname@example.org> wrote:
> Q1. Is lazy loading ndarray from user defined data supplying function possible?
No, not really.
> Q2. If possible, how can I implement it?
> The closest method I can think of is, (which requires c++ posix)
> 1. create a memory region using mmap and protect read operation by mprotect.
> 2. add SIGSEGV signal handler to trap read operation on the memory
> region, and the handler will provide appropriate user data and recover
> from SIGSEGV.
> 3. slightly modify memmap class to use the above mmap (memmap is
> already using mmap internally, so it's not a big deal)
> but obviously, recovering from SIGSEGV requires removing mprotect (see
> and it's impossible to know when to lock the region by mprotect again.
Well, if you're willing to go *that* far, you might was well make a
userspace file system with fuse and mmap a file within that.
You can even implement it in Python!
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
More information about the NumPy-Discussion