[Numpy-discussion] question about future support for python-3

Dag Sverre Seljebotn dagss@student.matnat.uio...
Wed Sep 9 11:03:52 CDT 2009

Darren Dale wrote:
> On Tue, Sep 8, 2009 at 9:02 PM, David Cournapeau<cournape@gmail.com> wrote:
>> On Wed, Sep 9, 2009 at 9:37 AM, Darren Dale<dsdale24@gmail.com> wrote:
>>> Hi David,
>>>> I already gave my own opinion on py3k, which can be summarized as:
>>>>  - it is a huge effort, and no core numpy/scipy developer has
>>>> expressed the urge to transition to py3k, since py3k does not bring
>>>> much for scientific computing.
>>>>  - very few packages with a significant portion of C have been ported
>>>> to my knowledge, hence very little experience on how to do it. AFAIK,
>>>> only small packages have been ported. Even big, pure python projects
>>>> have not been ported. The only big C project to have been ported is
>>>> python itself, and it broke compatibility and used a different source
>>>> tree than python 2.
>>>>  - it remains to be seen whether we can do the py3k support in the
>>>> same source tree as the one use for python >= 2.4. Having two source
>>>> trees would make the effort even much bigger, well over the current
>>>> developers capacity IMHO.
>>>> The only area where I could see the PSF helping is the point 2: more
>>>> documentation, more stories about 2->3 transition.
>>> I'm surprised to hear you say that. I would think additional developer
>>> and/or financial resources would be useful, for all of the reasons you
>>> listed.
>> If there was enough resources to pay someone very familiar with numpy
>> codebase for a long time, then yes, it could be useful - but I assume
>> that's out of the question. This would be very expensive as it would
>> requires several full months IMO.
>> The PSF could help for the point 3, by porting other projects to py3k
>> and documenting it. The only example I know so far is pycog2
>> (http://mail.python.org/pipermail/python-porting/2008-December/000010.html).
>> Paying people to do documentation about porting C code seems like a
>> good way to spend money: it would be useful outside numpy community,
>> and would presumably be less costly.
> Another topic concerning documentation is API compatibility. The
> python devs have requested projects not use the 2-3 transition as an
> excuse to change their APIs, but numpy is maybe a special case. I'm
> thinking about PEP3118. Is numpy going to transition to python 3 and
> then down the road transition again to the new buffer protocol? What
> is the strategy here? My underinformed impression is that there isn't
> one, since every time PEP3118 is considered in the context of the 2-3
> transition somebody helpfully reminds the list that we aren't supposed
> to break APIs. Numpy is a critical python library, perhaps the

I'd be surprised if this is the case and if there are any issues.

What Robert said applies, plus: In Python 2.6 the ndarray type would 
support *both* the old and the new buffer protocols, which can be usedin 
parallel on Python 2.6.

There's no real issue on the PEP 3118 at all as I can see, it just needs 
to be done. I'll try hard to give this a small start (ndarray export its 
buffer) in November (though when the time comes I might feel that I 
really should be studying instead...).

> transition presents an opportunity, if the community can yield a
> little on numpy's C api. For example, in the long run, what would it
> take to get numpy (or the core thereof) into the standard library, and
> can we take steps now in that direction? Would the numpy devs be
> receptive to comments from the python devs on the existing numpy
> codebase?

I think this one is likely a question of semantics. My feeling is that 
for instance the slice-returns-a-view on an array type would be hard to 
swallow on a standard library component? (Seeing as list returns a copy.)

Python 3 kind of solved this by calling the type "memoryview", which 
implies that slicing returns another view.

I have a feeling the the best start in this direction might be for 
somebody to give the memoryview type in Python 3 some love, perhaps set 
it up as a light-weight ndarray replacement in the standard library.

(If anybody implemented fancy indexing on a memoryview I suppose it 
should return a new view though (through a pointer table), meaning 
incompatability with NumPy's fancy indexing...)

> I'm willing to pitch in and work on the transition, not because I need
> python-3 right now, but because the transition needs to happen and it
> would benefit everyone in the long run. But I would like to know that
> we are making the most of the opportunity, and have considered our
> options.

Well, something that may belong here: There's been some talk now and 
then on whether one should to port parts of the NumPy C codebase to 
Cython (which gives automatic Python 3 compatability, up to string/bytes 
issues etc.). That could probably take somewhat longer, but perhaps 
result in a better maintainable code base in the end which more people 
could work on.

Dag Sverre

More information about the NumPy-Discussion mailing list