[SciPy-user] What is fastest load/save matrix methods?

Francesc Altet falted at pytables.org
Tue Dec 20 09:10:19 CST 2005


A Dimarts 20 Desembre 2005 05:27, Dave Kuhlman va escriure:
> Am I right that PyTables does not *directly* support scipy arrays?

Not currently. But after the implementation of an efficient array
protocol in all the basic numeric packages (numarray 1.5, Numeric 24.x
and scipy_core), supporting scipy_core should be easy. Stay tuned :-)

> That's not a big problem.  You can write a scipy array to an HDF5
> file using PyTables with something like the following::
>
> Is there a more efficient way?

As Andrew has explained, use the array interface/protocol:

# For scipy --> numarray
numarray_array = numarray.asarray(scipy_array)

# For numarray --> scipy
scipy_array = scipy.base.asarray(numarray_array)

I've taken the opportunity to run some elementary bencharmks to
compare the speed of PyTables vs scipy (and numarray) .tofile() method
(see the attachment). I've taken vector sizes from 10 elements up to
100 millions and run the benchmarks on a Opteron machine (BSD Unix),
with 8 GB of RAM and with IDE disks @ 7200 RPM. The results are:

% python speedcomp_pt_tofile.py scipy_core w
Python 2.4.2 (#2, Oct  3 2005, 09:13:40)
[GCC 3.4.2 [FreeBSD] 20040728]
Optimization flags: -DNDEBUG -O -pipe -march=opteron -D__wchar_t=wchar_t  
-DTHREAD_STACK_SIZE=0x20000
getNCPUs is_32bit
tables.__version__ --> 1.2
scipy_core.__version__--> 0.8.6.1687
---
PyTables, scipy_core, 10**1 time: 1.72 ms       46.514 KB/s
tofile(), scipy_core, 10**1 time: 0.051 ms      1559.387 KB/s
---
PyTables, scipy_core, 10**2 time: 1.727 ms      463.192 KB/s
tofile(), scipy_core, 10**2 time: 0.051 ms      15596.702 KB/s
---
PyTables, scipy_core, 10**3 time: 1.74 ms       4596.699 KB/s
tofile(), scipy_core, 10**3 time: 0.066 ms      121065.204 KB/s
---
PyTables, scipy_core, 10**4 time: 2.685 ms      29797.503 KB/s
tofile(), scipy_core, 10**4 time: 0.358 ms      223171.018 KB/s
---
PyTables, scipy_core, 10**5 time: 4.339 ms      184391.353 KB/s
tofile(), scipy_core, 10**5 time: 19.243 ms     41573.555 KB/s
---
PyTables, scipy_core, 10**6 time: 20.44 ms      391389.008 KB/s
tofile(), scipy_core, 10**6 time: 159.981 ms    50005.959 KB/s
---
PyTables, scipy_core, 10**7 time: 953.043 ms    83941.649 KB/s
tofile(), scipy_core, 10**7 time: 2101.34 ms    38070.951 KB/s
---
PyTables, scipy_core, 10**8 time: 16896.645 ms  47346.677 KB/s
tofile(), scipy_core, 10**8 time: 18389.829 ms  43502.307 KB/s

So, for writing, scipy.tofile() is faster for small arrays (as
expected). However, for medium to large arrays, PyTables wins, being
the crosspoint 10**5. For very large arrays, the disk becomes the
bottleneck (both .tofile() and PyTables can reach this speed for this
case).

For reading, we have similar figures:

% python speedcomp_pt_tofile.py scipy_core r
Python 2.4.2 (#2, Oct  3 2005, 09:13:40)
[GCC 3.4.2 [FreeBSD] 20040728]
Optimization flags: -DNDEBUG -O -pipe -march=opteron -D__wchar_t=wchar_t  
-DTHREAD_STACK_SIZE=0x20000
getNCPUs is_32bit
tables.__version__ --> 1.2
scipy_core.__version__--> 0.8.6.1687
---
PyTables, scipy_core, 10**1 time: 1.656 ms      48.3 KB/s
tofile(), scipy_core, 10**1 time: 0.028 ms      2889.435 KB/s
---
PyTables, scipy_core, 10**2 time: 1.583 ms      505.212 KB/s
tofile(), scipy_core, 10**2 time: 0.04 ms       19814.244 KB/s
---
PyTables, scipy_core, 10**3 time: 1.64 ms       4878.112 KB/s
tofile(), scipy_core, 10**3 time: 0.053 ms      150373.9 KB/s
---
PyTables, scipy_core, 10**4 time: 2.986 ms      26795.014 KB/s
tofile(), scipy_core, 10**4 time: 0.126 ms      637093.339 KB/s
---
PyTables, scipy_core, 10**5 time: 7.911 ms      101121.173 KB/s
tofile(), scipy_core, 10**5 time: 5.663 ms      141268.138 KB/s
---
PyTables, scipy_core, 10**6 time: 80.21 ms      99737.73 KB/s
tofile(), scipy_core, 10**6 time: 71.455 ms     111958.572 KB/s
---
PyTables, scipy_core, 10**7 time: 637.432 ms    125503.579 KB/s
tofile(), scipy_core, 10**7 time: 682.141 ms    117277.74 KB/s
---
PyTables, scipy_core, 10**8 time: 16455.255 ms  48616.688 KB/s
tofile(), scipy_core, 10**8 time: 40503.035 ms  19751.606 KB/s

Here the trends are similar. However, for very large arrays scipy_core
starts to perform quite badly (I've run the benchmark twice, just to
be sure). I don't know exactly the reasons for this, but it can be a
good point to optimise ;-) Also interesting is that, for array sizes
of 10**6 elements, PyTables can go faster writing (400 MB/s) than
reading (100 MB/s), which of course, also offers a good point to look
at it <wink>.

Just for the sake of comparison, here are the results for numarray:

% python speedcomp_pt_tofile.py numarray w
Python 2.4.2 (#2, Oct  3 2005, 09:13:40)
[GCC 3.4.2 [FreeBSD] 20040728]
Optimization flags: -DNDEBUG -O -pipe -march=opteron -D__wchar_t=wchar_t  
-DTHREAD_STACK_SIZE=0x20000
getNCPUs is_32bit
tables.__version__ --> 1.2
numarray.__version__--> 1.5.0
---
PyTables, numarray, 10**1 time: 1.697 ms        47.132 KB/s
tofile(), numarray, 10**1 time: 0.089 ms        898.443 KB/s
---
PyTables, numarray, 10**2 time: 1.715 ms        466.589 KB/s
tofile(), numarray, 10**2 time: 0.092 ms        8721.621 KB/s
---
PyTables, numarray, 10**3 time: 1.712 ms        4672.512 KB/s
tofile(), numarray, 10**3 time: 0.125 ms        64004.639 KB/s
---
PyTables, numarray, 10**4 time: 2.618 ms        30561.652 KB/s
tofile(), numarray, 10**4 time: 0.604 ms        132555.482 KB/s
---
PyTables, numarray, 10**5 time: 4.291 ms        186422.832 KB/s
tofile(), numarray, 10**5 time: 26.183 ms       30554.667 KB/s
---
PyTables, numarray, 10**6 time: 19.51 ms        410038.803 KB/s
tofile(), numarray, 10**6 time: 162.777 ms      49146.889 KB/s
---
PyTables, numarray, 10**7 time: 1131.729 ms     70688.323 KB/s
tofile(), numarray, 10**7 time: 2185.191 ms     36610.077 KB/s
---
PyTables, numarray, 10**8 time: 17065.664 ms    46877.755 KB/s
tofile(), numarray, 10**8 time: 18673.172 ms    42842.212 KB/s

The figures for PyTables are very similar to those of scipy, despite
the fact that numarray is at the core of PyTables (and not scipy).
This means that the array interface implementation is very efficient.
On the other hand, the figures for numarray.tofile() are generally
slower than scipy.tofile(), specially for small array sizes.

For reading:

% python speedcomp_pt_tofile.py numarray r
Python 2.4.2 (#2, Oct  3 2005, 09:13:40)
[GCC 3.4.2 [FreeBSD] 20040728]
Optimization flags: -DNDEBUG -O -pipe -march=opteron -D__wchar_t=wchar_t  
-DTHREAD_STACK_SIZE=0x20000
getNCPUs is_32bit
tables.__version__ --> 1.2
numarray.__version__--> 1.5.0
---
PyTables, numarray, 10**1 time: 1.598 ms        50.072 KB/s
tofile(), numarray, 10**1 time: 0.078 ms        1028.914 KB/s
---
PyTables, numarray, 10**2 time: 1.532 ms        522.14 KB/s
tofile(), numarray, 10**2 time: 0.078 ms        10285.167 KB/s
---
PyTables, numarray, 10**3 time: 1.63 ms         4909.243 KB/s
tofile(), numarray, 10**3 time: 0.144 ms        55448.124 KB/s
---
PyTables, numarray, 10**4 time: 3.003 ms        26641.708 KB/s
tofile(), numarray, 10**4 time: 0.458 ms        174787.246 KB/s
---
PyTables, numarray, 10**5 time: 6.208 ms        128863.699 KB/s
tofile(), numarray, 10**5 time: 7.264 ms        110128.993 KB/s
---
PyTables, numarray, 10**6 time: 68.284 ms       117157.148 KB/s
tofile(), numarray, 10**6 time: 84.222 ms       94987.418 KB/s
---
PyTables, numarray, 10**7 time: 640.721 ms      124859.399 KB/s
tofile(), numarray, 10**7 time: 795.182 ms      100605.9 KB/s
---
PyTables, numarray, 10**8 time: 17064.221 ms    46881.718 KB/s
tofile(), numarray, 10**8 time: 39898.904 ms    20050.676 KB/s

In this case, PyTables performs slightly better than before because it
does not have to perform the additional conversion to scipy objects.
However both performances (with numarray and scipy_core) are *very*
close. Regarding numarray.fromfile() performance, one can again see
that it is generally slower than scipy.fromfile(). Interestingly
enough, for very large arrays (10**8 entry), numarray.fromfile() is
also quite slow when compared with smaller sizes (10**7), i.e. it
exhibits the very same symptom than scipy.

In summary:

- PyTables cannot compete in speed for small arrays (<10000 elements).
  However, the latency for saving/reading these objects is quite low
  (< 5 ms).

- For larger arrays, PyTables (and hence, HDF5) always exposes better
  speed, even in the case of dealing with extremely large datasets
  (which is the field were it is supposed it has been designed for).
  

Regards,

-- 
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   Cárabos Coop. V.   Enjoy Data
 "-"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: speedcomp_pt_tofile.py
Type: application/x-python
Size: 4380 bytes
Desc: not available
Url : http://www.scipy.net/pipermail/scipy-user/attachments/20051220/dbf02dd1/speedcomp_pt_tofile-0001.bin


More information about the SciPy-user mailing list