[IPython-dev] Status of pre-0.10 reviews and merges

Brian Granger ellisonbg.net@gmail....
Thu Apr 2 22:04:09 CDT 2009

Getting caught up after a few days teaching...

> Thanks for pinging back in: here we had a massive sprint for
> neuroimaging.scipy.org (a single merge was over 92000 lines long, to
> give you an idea of how much code we moved around :) that I'm only now
> recovering again from, as grant deadlines pile up.

Wow, that is a pretty big merge.  Sounds like good progress though.

> Over the next two weeks I'll plug back into this so we can put 0.10
> out.  I've been thinking about the twisted bug, and I suspect it has
> to do with global state.  The ipdoctest plugin insantiates a real
> ipython, and that makes some global changes.  I wonder if those are
> confusing twisted somehow.  Can you think of anything in this
> direction?

The big things we need to look at are:

* If ipdoctest is picking up things that should really be run as tests
that are somehow using twisted, but not in an officially "good" way
* If there are globals that both twisted.trial and ipdoctest is
mucking with that would be a problem.
* The reactor is being started twice.  That is very odd, but could
help us debug this.

I think this will take both of us to tackle.  My idea is to start
disabling parts of ipdoctest, or run the test suite on small parts of
the codebase to try to isolate where the problem is coming from.

>> I want to get these things resolved quickly as I am being paid to work
>> on IPython during April these pending merges are currently holding me
>> up.  This is because I am going to start making MAJOR changes that
>> will make it extremely difficult if not impossible to merge branches
>> that are children of our current "trunk."  We don't need to release
>> 0.10 for me to get started with this work, but we do need to get these
>> branches merged before I branch.
> I know we've talked a bit about this, but before we get started on
> this major reorganization, let's make sure we do plan it carefully.
> For better or worse, while ipython is formally 'pre 1.0' and as such,
> all API changes are in principle fair game, we need to ensure we find
> a good balance of practicality vs purity.  Just like numpy kept some
> Numeric-inspired baggage that wasn't ideal so that users could
> transition without unnecessary pain, we should keep the same ideas in
> mind.  We should obviously fix the major messes we have (ipmaker is
> ridiculous, 3 config systems is crazy, etc), but we should also be
> willing to tolerate small warts if they make existing user's lives
> easier.

I am trying to distinguish between users and developer_users.  For
users, I strongly agree that IPython shouldn't really change, or if it
does, in a very limited manner that improves things drastically (like
the config system).  For developers though, we need to allow much more
freedom in changing APIs.  We won't be able to do the work without it.
 But, even there, I agree that we should only change an API when we
really need to.

But...,the first things I will be working on will be pretty different
from the types of changes you are talking about here.  Basically, I am
going to do a massive module/package reorganization.  A draft of the
idea is this...

I am going to reorganize the top-level modules into logical
subpackages.  I am also going to use this as an opportunity to:

* Identify rotting and broken code and move it to a directory (I won't
delete it....yet) where everyone can look at the code and decide if we
want to revive it (with tests+docs) or simply kill it for good.

* Renaming modules and packages to pep 8 style lowercase.py

* Adding tests to make sure that every moved/renamed module and
package is importable (yeh, tests!!!) and working.

* Where it is appropriate, begin to partition large modules (iplib,
genutils, etc.) into smaller more logic submodules to make them easier
to refactor and work with.

* For really public facing developer APIs, I will put in a
compatibility layer with deprecation warnings.

The only thing that will change in the actual code though will be
import statements and all of this will be tested.  No API changes will
be combined with this reorganization work.  This reorganization is
needed to help us really identify and isolate the things that need to
be refactored moving forward

But, because this work will move *everything* around, it will be
difficult to merge branches back in in an automatic way.  By hand
though it might not be too bad to merge things.

> Part of that is communicating a plan for the breakage to everyone in
> advance, so that feedback can be provided by all affected before
> anything irreversible has happened.

Yes, I will absolutely ask for feedback about these plans before
starting.  I am working on an IPEP (IPython Enhancement Proposal) that
describes my plans.  More details to follow soon...

> Furthermore, I'd like to suggest that post 0.10 we hold a 'bug and
> test weeks' (2-4 max) to see if we could do an 0.11 release where we
> basically try to *only* fix existing tickets and add as many tests as
> we can, before starting to do major API breakage.

In theory this is a great idea.  In practice this timescale will cause
problems with the work I need to do.  I currently have funding that
ends May 1 to work on this and I need to get started immediately.  It
is so rare to have dedicated time and $ for this, I don't want to
waste the opportunity.  That is basically 4 weeks away.  In reality, I
think I should soon just "go for it" and help people to do their
merges into my branch by hand.

> I'd love to do a lot, but my next 12 weeks are
> insane: *3* grants to write, another full-week sprint to host here,
> and one major conference coming up.  I'm sure others have also hectic
> schedules.

I completely understand if other people are really busy and can't
participate.  At a minimum, I will probably need code review help and
participation in discussing the details of the plans.



More information about the IPython-dev mailing list