[Numpy-discussion] memmap from fd?

Robert Kern robert.kern@gmail....
Sat Jan 10 19:41:47 CST 2009

On Fri, Jan 9, 2009 at 22:15, Neal Becker <ndbecker2@gmail.com> wrote:
> Robert Kern wrote:
>> On Fri, Jan 9, 2009 at 08:08, Neal Becker <ndbecker2@gmail.com> wrote:
>>> Robert Kern wrote:
>>>> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2@gmail.com> wrote:
>>>>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>>>>> exposes the FPGA memory via mmap.
>>>>> It might be nice to use numpy memmap to read/write data.  One issue is
>>>>> that I think I will need to create the memmap array from a fd, not a
>>>>> file
>>>>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>>>>> and I already have it open for other reasons.  Any chance to create a
>>>>> memmap array from a fd?
>>>> Use os.fdopen(fd) to create a file object which can be passed to the
>>>> memmap constructor.
>>> Looks like this is not going to work without some change to memmap.  The
>>> problem is, I need read/write access.  The only choice in memmap is 'w+'.
>> 'r+' is for reading and writing.
>>> But this does:
>>>        if (mode == 'w+') and shape is None:
>>>            raise ValueError, "shape must be given"
>>>        fid.seek(0,2)
>>> My device has hijacked 'read' to mean something entirely different than
>>> you might expect.  The seek call invokes 'read'.
>>> It looks like the purpose of this code is to find the size of the
>>> mappable area.  The best solution I think is just throw it away.
>> We can't. We need it.
>>> Consistent with mmap semantics, attempting access outside the mappable
>>> area should cause and error - but I don't think there is any reliable way
>>> to know the length of the mappable area apriori.
>> For regular files, that seems to me to be fairly reliable. Why isn't it?
> Because I'm not mmapping a file.  I'm mmapping a device.  It exposes the memory of the FPGA board as seen on the PCI bus.  You just have to know the size.

That's a fair point. However, given the wider range of issues that you
had in trying to get memmap to work with your device, perhaps we
should just state that the memmap class is intended for the common use
case of regular files. The detection of the existing file size is
quite useful in that case. For special files where seek() et al. might
have different semantics, one should either subclass memmap to get the
behavior you need, or simply use frombuffer() or ndarray() on the mmap
object directly. The two reasons for the memmap class are the
auto-length-detection and tracking the lifetime of the open file. The
former you don't need (and can't do), and the latter isn't that hard
to do yourself.

Does that sound reasonable?

Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

More information about the Numpy-discussion mailing list