[AstroPy] Pandas vs Astropy tables.

Wolfgang Kerzendorf wkerzendorf@gmail....
Thu Jun 20 13:09:17 CDT 2013


I think we're all forgetting one of the messages of astropy: foster interoperability between Python astronomy (and science) packages.

I use pandas a lot in my radiative transfer code as I need (among other things)  to do group by functionality as fast as possible (and sqlite is too slow). However when I read in the tables they first go through astropy tables because it can handle units and converts them all to cgs. I think we should not try to compete with PANDAS but foster interoperability (similarly to us not trying to compete with numpy or scipy) . PANDAS will never care about writing tables to a deluxe table and that's where astropy shines (for example). 

So if you need to read in CSV files (and speed is important, which it is often not) I have no problem recommending pandas, however if you need to deal with DAOPhot, CDS, etc. files there's no alternative to astropy.tables. 

For now I don't see a reason for including PANDAS in astropy core, as we don't have a need for its functionality there. But including this in affiliated packages (if needed) I think is a good idea.

My 2 cents,
   Wolfgang
On 2013-06-20, at 1:47 PM, Éric Depagne <eric@depagne.org> wrote:

> Just a quick question:
> 
> Since the discussion is about what pandas, wouldn't it be possible to add in 
> cc some pandas devs, to let them know?
> 
> Éric.
>> On Thu, Jun 20, 2013 at 12:50 PM, Chris Beaumont <beaumont@hawaii.edu>wrote:
>>> I thought I'd chime in on the pandas discussion :)
>>> 
>>> I'm starting to use pandas a bit more in my day-to-day work. The two
>>> features most useful to me are:
>>> 
>>> 1) Its file parsers are pretty robust and fast. I always try parsing CSV
>>> with pandas first
>> 
>> I've wondered how hard it would be to incorporate some of the pandas CSV
>> fast reading functions for the easy cases.  I'm assuming it is licensed so
>> that would be an option.
>> 
>>> 2) For tables tables with lots of categorical data, the grouping
>>> functionality is very nice. For example, calculations like "the mean age
>>> of each spectral type of star in my catalog" are usually one liners like
>>> df.groupby(['spectral_type']).age.mean. I spend a lot of time on the
>>> "split-apply-combine" page on the pandas docs (
>>> http://pandas.pydata.org/pandas-docs/stable/groupby.html).
>> 
>> Group-by and related functionality is top on my list of priorities for
>> astropy.table (in fact I see it every day on my google keep app...).  Join
>> and merging are in master now.  In my tests the astropy table join is
>> within a factor of 2 to 3 in speed relative to pandas, so in most use cases
>> it should be good enough.
>> 
>> It's probably worth pointing out to the community that it was not a
>> lightly-taken decision to reject pandas for use as the base data storage
>> container.  For the case of tables there is one show-stopper which is that
>> pandas DataFrame does not support arbitrary multi-dimensional columns, i.e.
>> column where each element is itself an N-d array.  These occur enough in
>> astronomy and are supported by FITS and VO standards, so the astropy Table
>> must be able to represent that.  The lack of support for table and column
>> metadata is a smaller but still important issue.
>> 
>> Having said that, there is no question pandas has a ton of highly-efficient
>> and useful machinery and we are working on ways to improve
>> inter-operability.  This includes being able convert between Table and
>> DataFrame easily.  Suggestions and (especially) pull requests welcome.
>> 
>>> I won't speculate about whether that's enough an asset to warrant a
>>> dependency in astropy. I do agree that lots of other pandas features
>>> don't translate as well into astronomy use.
>>> 
>>> On Thu, Jun 20, 2013 at 12:34 PM, Erik Tollerud 
> <erik.tollerud@gmail.com>wrote:
>>>> I'm of mixed minds about traits UI because once you know it you can make
>>>> great GUIs with it, but I've spent a lot of time troubleshooting
>>>> people's python installations to get traits to work.  That is, in
>>>> general it can be tricky to get installed because of all the
>>>> dependencies.  Maybe this has improved recently with Enthought's Canopy
>>>> (or other new python distros), but that's been my past experience.
>>>> 
>>>> More generally, the view in the astropy core package is that we don't
>>>> want to put GUIs in the core because GUIs always carry lots of
>>>> dependencies, which we don't want to be forced to deal with.  But part
>>>> of the whole reason for affiliated packages was to get around this, so
>>>> we're happy to see GUI-based affiliated packages.
>>>> 
>>>> 
>>>> As for Pandas, to be totally honest, I don't see a huge amount to be
>>>> gained from adding a Pandas dependency Astropy.  It's honestly not clear
>>>> what it gives the astronomy community that numpy does not already have.
>>>> 
>>>> The following quote from the Pandas web site has guided me to that
>>>> 
>>>> conclusion: "*pandas* helps fill this gap, enabling you to carry out
>>>> your entire data analysis workflow in Python without having to switch to
>>>> a more domain specific language like R."
>>>> 
>>>> I have been carrying out my entire data analysis workflow for some time
>>>> now in python without using Pandas.  It looks to me like Pandas is a
>>>> tool that was written by and for statisticians who use R.  While we can
>>>> take lessons from this, it's not clear we get much out of it in an
>>>> astronomy context. For example, how would it make astropy's NDData,
>>>> Quantity, or Table better to use a Pandas DataFrame vs. a numpy array?
>>>> Most of what we are doing is building astronomy-convenient interfaces,
>>>> and I'm not sure what Pandas adds there, at the cost of a pretty
>>>> heavy-weight dependency.
>>>> 
>>>> It could just be that I don't know enough about Pandas, though.  So if
>>>> someone who knows Pandas better can speak to this, I'm all ears.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Tue, Jun 18, 2013 at 3:35 PM, Thøger Rivera-Thorsen
>>>> <trive@astro.su.se
>>>> 
>>>>> wrote:
>>>>> Pandas is a part of the newly-defined SciPy stack, after all, so that
>>>>> 
>>>>> would be part of any science-oriented distribution worth its salt. In
>>>>> fact, I think it could be a good idea for astropy in general to use
>>>>> under the hood, but again, could clash with the philosophy of the
>>>>> project and possibly also maintainabillity.
>>>>> 
>>>>> As for offering my code or just my experience, I'll have to square it
>>>>> with my supervisor first, and I also think it depends on what direction
>>>>> the project in question will take. I'm positive about the idea (which
>>>>> is why I wrote in the first place), but supervisor might think it is a
>>>>> better idea to actually get my paper in the project wrapped up before
>>>>> sending the code out there. Will get back about that one!
>>>>> 
>>>>> /Emil
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 2013-06-18 20:53, Slavin, Jonathan wrote:
>>>>> 
>>>>> Hi Emil,
>>>>> 
>>>>> That looks very nice!  I don't see Pandas as a big issue in terms of
>>>>> 
>>>>> dependencies.  I don't know that much about traits, etc.  My thought
>>>>> about the gui was just based on my experience with matplotlib, and the
>>>>> fact that it is widely used -- though I would agree that too many
>>>>> dependencies can be a deterrent to people using something.  Are you
>>>>> offering your code as a starting point for the project?  It strikes me
>>>>> that many have gotten some sort of fitting package to a point of
>>>>> personal usability but no one has the time/interest/motivation to make
>>>>> a more generally usable package.
>>>>> 
>>>>> Jon
>>>>> 
>>>>> On Tue, Jun 18, 2013 at 2:34 PM, <astropy-request@scipy.org> wrote:
>>>>>> Date: Tue, 18 Jun 2013 20:39:55 +0200
>>>>>> From: Th?ger Rivera-Thorsen <thoger.emil@gmail.com>
>>>>>> Subject: Re: [AstroPy] ESA Summer of Code in Space 2013
>>>>>> To: astropy@scipy.org
>>>>>> Message-ID: <51C0A97B.8090703@gmail.com>
>>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>> 
>>>>>> I have been working on a fitting GUI for a while, although it is made
>>>>>> with a specific task in mind.
>>>>>> However, it is not based on Matplotlib but on Traits/Traitsui/Chaco
>>>>>> and Pandas. It is made for a specific projhect I'm working and as
>>>>>> such not yet usable for more general cases, but it could be a
>>>>>> starting point, if the dependencies don't conflict with astropy
>>>>>> politics.
>>>>>> 
>>>>>> Especially, I am happy about the choice of Pandas for managing a quite
>>>>>> complex data structure (the fitted and/or guessed values of an
>>>>>> arbitrary number of transitions for an arbitrary number of rows or
>>>>>> collapsed rows of a spatially resolved spectrum) of a), but also with
>>>>>> the Traits-based interactive interface to build complex line profiles
>>>>>> from single gaussians, good for fitting-by-eye and giving good
>>>>>> initial guesses for fitting of complex line profiles. It hooks
>>>>>> directly up to a wrapper I've made for lmfit, but given the
>>>>>> modularity, it should be relatively easy to change to other backends.
>>>>>> 
>>>>>> It's still a work-in-progress, but there are some screenshots here:
>>>>>> http://flic.kr/s/aHsjGaEMGg .
>>>>>> I know the choice and number of dependencies may be prohibitive but it
>>>>>> saved a lot of work on the GUI, and Pandas means the difference
>>>>>> between sanity and madness when it comes to keeping track of so many
>>>>>> parameters.
>>>>>> 
>>>>>> Cheers,
>>>>>> Emil
>>>>>> 
>>>>> ________________________________________________________
>>>>> 
>>>>> Jonathan D. Slavin                 Harvard-Smithsonian CfA
>>>>> jslavin@cfa.harvard.edu       60 Garden Street, MS 83
>>>>> phone: (617) 496-7981       Cambridge, MA 02138-1516
>>>>> fax: (617) 496-7577            USA
>>>>> ________________________________________________________
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> AstroPy mailing
>>>>> listAstroPy@scipy.orghttp://mail.scipy.org/mailman/listinfo/astropy
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> AstroPy mailing list
>>>>> AstroPy@scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>> 
>>>> --
>>>> Erik
>>>> 
>>>> _______________________________________________
>>>> AstroPy mailing list
>>>> AstroPy@scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/astropy
>>> 
>>> --
>>> ************************************
>>> Chris Beaumont
>>> Graduate Student
>>> Institute for Astronomy
>>> University of Hawaii at Manoa
>>> 2680 Woodlawn Drive
>>> Honolulu, HI 96822
>>> www.ifa.hawaii.edu/~beaumont
>>> ************************************
>>> 
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
> Un clavier azerty en vaut deux
> ----------------------------------------------------------
> Éric Depagne                            eric@depagne.org
> _______________________________________________
> AstroPy mailing list
> AstroPy@scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/astropy/attachments/20130620/08672be5/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/astropy/attachments/20130620/08672be5/attachment-0001.bin 


More information about the AstroPy mailing list