[IPython-User] distributed arrays

samuel sinayoko samuel.sinayoko@gmail....
Mon Mar 7 16:24:28 CST 2011


Thank you guys for the input. That should be enough to get me started.

Bill: I'm not sure I understand your second point:
"This means you can have distributed arrays that are locally recognized (in
python) as multidimensional, but not globally recognized as multidimensional
by either python or the underlying C++."
Does it mean that each process sees the bit of the array it owns as
multidmensional, but that somehow the global array is not recognized as
multidimensional?

At the moment I haven't been able to transfer the several tera bytes of data
I need to work with to a cluster, so I'm only working on my workstation
(i.e. with a share memory). However, I might start playing around with
distributing arrays on the cluster I have access to. Distributed arrays are
definitely something I will need in the future, so I'm really grateful for
your help.

Hopefully I'll be able to contribute something useful at some point!

Sam



On 7 March 2011 21:48, Bill Spotz <wfspotz@sandia.gov> wrote:

> Just a few notes off the top of my head:
>
> * Trilinos is developed largely for finite element approximations to PDEs,
> so there is a strong emphasis on unstructured grids (resulting in single
> index rather than multi-dimensional arrays), and no FFT support yet.
>
> * Linear algebra classes such as Vectors are in the Epetra package.  I have
> created python wrapper classes that inherit, e.g., from both the Vector C++
> class and the numpy array python class.  This means you can have distributed
> arrays that are locally recognized (in python) as multidimensional, but not
> globally recognized as multidimensional by either python or the underlying
> C++.
>
> * I'm fairly certain that FFTPACK, e.g., supports distributed FFTs.  So the
> hard part is done and what is left to do is an interface that works with
> Epetra.
>
> -Bill
>
> On Mar 7, 2011, at 2:01 PM, Fernando Perez wrote:
>
> > Dear Samuel,
> >
> > On Mon, Mar 7, 2011 at 10:32 AM, samuel sinayoko
> > <samuel.sinayoko@gmail.com> wrote:
> >>
> >> This is really useful. I didn't know about these projects. I'll see if I
> can
> >> use ipythondistarray for my problem and will let you know how things go.
> >
> > I'm CC'ing here Bill Spotz, the author of the pytrilinos code, who may
> > be able to provide you with some more info (Bill, for reference, this
> > is about distributed arrays in python, see
> > http://mail.scipy.org/pipermail/ipython-user/2011-March/007402.html
> > for the background).
> >
> > Bill has continued to maintain pytrilinos, and the underlying Trilinos
> > machinery is complex but precisely designed for large-scale, really
> > high end computing, so it may fit your needs well.
> >
> > Note that this isn't dissing Brian's work, simply that as he already
> > said, that code hasn't seen much recent development, while Trilinos
> > (and PetSc) both have high-end production-ready machinery under the
> > hood and are used in large-scale computing everyday, so they may be a
> > good choice.
> >
> > Now, if you want to pitch in to improve a lightweight, pure-python set
> > of distributed arrays (since trilinos and petsc are pretty non-trivial
> > dependencies), that would be awesome and I'm sure Brian would love the
> > help :)
> >
> > Best,
> >
> > f
> >
>
> ** Bill Spotz                                              **
> ** Sandia National Laboratories  Voice: (505)845-0170      **
> ** P.O. Box 5800                 Fax:   (505)284-0154      **
> ** Albuquerque, NM 87185-0370    Email: wfspotz@sandia.gov **
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20110307/35bf8018/attachment.html 


More information about the IPython-User mailing list