[Numpy-discussion] Loading a > GB file into array
Mon Dec 3 23:13:53 CST 2007
Martin Spacek wrote:
> Gael Varoquaux wrote:
>> Very interesting. Have you made measurements to see how many times you
>> lost one of your cycles. I made these kind of measurements on Linux using
>> the real-time clock with C and it was very interesting (
>> http://www.gael-varoquaux.info/computers/real-time ). I want to redo them
>> with Python, as I except to have similar results with Python. It would be
>> interesting to see how Windows fits in the picture (I know nothing about
>> Windows, so I really can't make measurements on Windows).
> Neat, thanks for that, I'll have a look. I'm very slowly transitioning
> my computing "life" over to linux, but I've been told by Andrew Straw (I
> think http://visionegg.org might have some details, see the mailing
> list) that it's harder to get close to a real-time OS with linux (while
> running high level stuff like opengl and python) than it is in windows.
My impression is that is is more like the contrary; linux implements
many posix facilities for more 'real-time' behaviour: it implements a
FIFO scheduler, you have mlock facilities to avoid paging, etc... and of
course, you can control your environment much more easily (one buggy
driver can kill the whole thing as far as latency is concerned, for
example). I did not find those info you are talking about on visionegg ?
Now, for python, this is a different matter. In you need to do things in
real-time, setting a high priority is not enough, and python has several
characteristics which make it less than suitable for real-time (heavy
usage of memory allocation, garbage collector, etc...). I guess that
when you do things every few ms, with enough memory (every ms gives you
millions of cycle on modern machines), you can hope that is it not too
much of a problem (at least for memory allocation; I could not find in
Andrew's slides whether he disabled the GC). But I doubt you can do much
I wonder if python can be compiled with a real time memory allocator,
and even whether it makes sense at all (I am thinking about something
like TLSF: http://rtportal.upv.es/rtmalloc/).
> I hope that's changed, or is changing. I'd love to switch over to 64-bit
> linux. As far as windows is considered, I'd like 32bit winxp to be my
> last iteration.
With recent kernels, you can get really good latency if you do it right
(around 1-2 ms worst case under high load, including high IO pressure).
I know nothing about video programming, but I would guess that as far as
the kernel is concerned, this does not change much. I have not tried
them myself, but ubuntu studio has its own kernel with 'real-time'
patched (voluntary preempt from Ingo, for example), and is available
both for 32 and 64 bits architectures. One problem I can think of for
video is that if you need binary-only drivers: those are generally
pretty bad as far as low latency is concerned (nvidia drivers always
cause some kind of problems with low latency and 'real-time' kernels).
More information about the Numpy-discussion