[Numpy-discussion] [ANN] carray: an in-memory compressed data container

[Zbyszek Szmek]

OK, I've got a case where carray really shines :|

zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py numpy 800000 1000 4 1
problem size: (800000) x 1000 = 10^8.90309
time for concat: 4.806s
size of the final container: 6103.516 MB
zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py concat 800000 1000 4 1
problem size: (800000) x 1000 = 10^8.90309
time for concat: 3.475s
size of the final container: 6103.516 MB
zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py carray 800000 1000 4 1
problem size: (800000) x 1000 = 10^8.90309
time for concat: 1.434s
size of the final container: 373.480 MB

Size is set to NOT hit the swap. This is still the easily compressible
arange... but still, the results are very nice.

Of course when the swap is hit, the ratio between carray and a normal array
can grow to infinity :)

zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py numpy 1000000 1000 3 1
problem size: (1000000) x 1000 = 10^9
time for concat: 35.700s
size of the final container: 7629.395 MB
zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py carray 1000000 1000 3 1
problem size: (1000000) x 1000 = 10^9
time for concat: 1.751s
size of the final container: 409.633 MB

Best,
Zbyszek