[IPython-User] distributed arrays

Bill Spotz wfspotz@sandia....
Mon Mar 7 16:55:35 CST 2011


Yes.  For a contrived example, consider 16 global array elements on two processors.  You can inform each local array that it is 2x4, but cannot inform the global array that it is 4x4.

Epetra domain decompositions are based upon a class called the Epetra_Map that stores a list of global IDs.  We could derive from this class and add the n-dimensional array decomposition information ... it just hasn't been done before.


On Mar 7, 2011, at 3:24 PM, samuel sinayoko wrote:

> Thank you guys for the input. That should be enough to get me started. 
> Bill: I'm not sure I understand your second point:
> "This means you can have distributed arrays that are locally recognized (in python) as multidimensional, but not globally recognized as multidimensional by either python or the underlying C++."
> Does it mean that each process sees the bit of the array it owns as multidmensional, but that somehow the global array is not recognized as multidimensional?
> At the moment I haven't been able to transfer the several tera bytes of data I need to work with to a cluster, so I'm only working on my workstation (i.e. with a share memory). However, I might start playing around with distributing arrays on the cluster I have access to. Distributed arrays are definitely something I will need in the future, so I'm really grateful for your help. 
> Hopefully I'll be able to contribute something useful at some point!
> Sam 
> On 7 March 2011 21:48, Bill Spotz <wfspotz@sandia.gov> wrote:
> Just a few notes off the top of my head:
> * Trilinos is developed largely for finite element approximations to PDEs, so there is a strong emphasis on unstructured grids (resulting in single index rather than multi-dimensional arrays), and no FFT support yet.
> * Linear algebra classes such as Vectors are in the Epetra package.  I have created python wrapper classes that inherit, e.g., from both the Vector C++ class and the numpy array python class.  This means you can have distributed arrays that are locally recognized (in python) as multidimensional, but not globally recognized as multidimensional by either python or the underlying C++.
> * I'm fairly certain that FFTPACK, e.g., supports distributed FFTs.  So the hard part is done and what is left to do is an interface that works with Epetra.
> -Bill
> On Mar 7, 2011, at 2:01 PM, Fernando Perez wrote:
> > Dear Samuel,
> >
> > On Mon, Mar 7, 2011 at 10:32 AM, samuel sinayoko
> > <samuel.sinayoko@gmail.com> wrote:
> >>
> >> This is really useful. I didn't know about these projects. I'll see if I can
> >> use ipythondistarray for my problem and will let you know how things go.
> >
> > I'm CC'ing here Bill Spotz, the author of the pytrilinos code, who may
> > be able to provide you with some more info (Bill, for reference, this
> > is about distributed arrays in python, see
> > http://mail.scipy.org/pipermail/ipython-user/2011-March/007402.html
> > for the background).
> >
> > Bill has continued to maintain pytrilinos, and the underlying Trilinos
> > machinery is complex but precisely designed for large-scale, really
> > high end computing, so it may fit your needs well.
> >
> > Note that this isn't dissing Brian's work, simply that as he already
> > said, that code hasn't seen much recent development, while Trilinos
> > (and PetSc) both have high-end production-ready machinery under the
> > hood and are used in large-scale computing everyday, so they may be a
> > good choice.
> >
> > Now, if you want to pitch in to improve a lightweight, pure-python set
> > of distributed arrays (since trilinos and petsc are pretty non-trivial
> > dependencies), that would be awesome and I'm sure Brian would love the
> > help :)
> >
> > Best,
> >
> > f

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-0154      **
** Albuquerque, NM 87185-0370    Email: wfspotz@sandia.gov **

More information about the IPython-User mailing list