[IPython-User] distributed arrays
Mon Mar 7 16:24:28 CST 2011
Thank you guys for the input. That should be enough to get me started.
Bill: I'm not sure I understand your second point:
"This means you can have distributed arrays that are locally recognized (in
python) as multidimensional, but not globally recognized as multidimensional
by either python or the underlying C++."
Does it mean that each process sees the bit of the array it owns as
multidmensional, but that somehow the global array is not recognized as
At the moment I haven't been able to transfer the several tera bytes of data
I need to work with to a cluster, so I'm only working on my workstation
(i.e. with a share memory). However, I might start playing around with
distributing arrays on the cluster I have access to. Distributed arrays are
definitely something I will need in the future, so I'm really grateful for
Hopefully I'll be able to contribute something useful at some point!
On 7 March 2011 21:48, Bill Spotz <firstname.lastname@example.org> wrote:
> Just a few notes off the top of my head:
> * Trilinos is developed largely for finite element approximations to PDEs,
> so there is a strong emphasis on unstructured grids (resulting in single
> index rather than multi-dimensional arrays), and no FFT support yet.
> * Linear algebra classes such as Vectors are in the Epetra package. I have
> created python wrapper classes that inherit, e.g., from both the Vector C++
> class and the numpy array python class. This means you can have distributed
> arrays that are locally recognized (in python) as multidimensional, but not
> globally recognized as multidimensional by either python or the underlying
> * I'm fairly certain that FFTPACK, e.g., supports distributed FFTs. So the
> hard part is done and what is left to do is an interface that works with
> On Mar 7, 2011, at 2:01 PM, Fernando Perez wrote:
> > Dear Samuel,
> > On Mon, Mar 7, 2011 at 10:32 AM, samuel sinayoko
> > <email@example.com> wrote:
> >> This is really useful. I didn't know about these projects. I'll see if I
> >> use ipythondistarray for my problem and will let you know how things go.
> > I'm CC'ing here Bill Spotz, the author of the pytrilinos code, who may
> > be able to provide you with some more info (Bill, for reference, this
> > is about distributed arrays in python, see
> > http://mail.scipy.org/pipermail/ipython-user/2011-March/007402.html
> > for the background).
> > Bill has continued to maintain pytrilinos, and the underlying Trilinos
> > machinery is complex but precisely designed for large-scale, really
> > high end computing, so it may fit your needs well.
> > Note that this isn't dissing Brian's work, simply that as he already
> > said, that code hasn't seen much recent development, while Trilinos
> > (and PetSc) both have high-end production-ready machinery under the
> > hood and are used in large-scale computing everyday, so they may be a
> > good choice.
> > Now, if you want to pitch in to improve a lightweight, pure-python set
> > of distributed arrays (since trilinos and petsc are pretty non-trivial
> > dependencies), that would be awesome and I'm sure Brian would love the
> > help :)
> > Best,
> > f
> ** Bill Spotz **
> ** Sandia National Laboratories Voice: (505)845-0170 **
> ** P.O. Box 5800 Fax: (505)284-0154 **
> ** Albuquerque, NM 87185-0370 Email: firstname.lastname@example.org **
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User