[Numpy-discussion] automatic differentiation with PyAutoDiff

srean srean.list@gmail....
Thu Jun 14 17:59:50 CDT 2012


> Of course, maybe you were pointing out that if your derivative
> calculation depends in some intrinsic way on the topology of some
> graph, then your best bet is to have an automatic way to recompute it
> from scratch for each new graph you see. In that case, fair enough!

That is indeed what I had in mind. In neural networks, Markov random
fields, Bayesian networks, graph regularization etc it is something
that has to be dealt with all the time.

> Right, and what I want is to do those correctness checks once, and
> then save the validated derivative function somewhere and know that it
> won't break the next time I upgrade some library or make some
> seemingly-irrelevant change to the original code.

Exactly.
What I was getting at is: even if it is not feasible to get a pretty
printed python output, the byte code can still be validated (somewhat)
with a few numeric sanity checks. So, yes the derivatives
needn't/shouldn't be re-computed in runtime all the time and an API
that that returns even some opaque but computable representation of
the derivative that can be validated and then "frozen" would be
helpful.

I think one can go further and formally prove the correctness of the
derivative computing engine. I dont know if anyone has done it. Maybe
Theano does it. Should be possible for a statically typed sublanguage.


More information about the NumPy-Discussion mailing list