[SciPy-dev] Proposal for more generic optimizers (posted before on scipy-user)
Michael McNeil Forbes
Wed Apr 18 01:04:22 CDT 2007
Okay, I think we are thinking similar things with different terminology:
I think you are saying that only one object should maintain state
(your "optimizer") (I was originally sharing the state which I agree
can cause problems). If so, I agree, but to me it seems that object
should be called a "partially optimized function". I think of an
"optimizer" as something which modifies state rather than something
that maintains state. I am thinking of code like:
roughly_locate_minimum = Optimizer
find_precise_minimum = Optimizer
f = Rosenbrock(...)
x0 = ...
f_min = OptimizedFunction(f,x0)
#OR (this reads better to me, but the functions should return copies
of f_min, so may not be desirable for performance reasons)
f_min = roughly_locate_minimum(f_min)
f_min = find_precise_minimum(f_min)
# Then one can query f_min for results:
print f_min.x # Best current approximation to optimum
print f_min.err #Estimate error
The f_min object keeps track of all state, can be passed from one
optimizer to another, etc. In my mind, it is simply an object that
has accumulated information about a function. The idea I have in
mind is that f is extremely expensive to compute, thus the object
with state f_min accumulates more and more information as it goes
along. Ultimately this information could be used in many ways, for
- f_min could keep track of roughly how long it takes to compute f
(x), thus providing estimates of the time required to complete a
- f_min could keep track of values and use interpolation to provide
fast guesses etc.
Does this mesh with your idea of an "optimizer"? I think it is
strictly equivalent, but looking at the line of code
"optimizer.optimize()" is much less useful to me than "f_min.optimize
What would your ideal "user" code look like for the above use-case?
I will try to flesh out a more detailed structure for the
On 16 Apr 2007, at 12:27 PM, Matthieu Brucher wrote:
> OK, that is not what a function is :)
> A function is the set of f, df, ddf but not with the options. What
> you are exposing is the construction of an optimizer ;)
> Did you see the code in my different proposals ?
> In fact, you have a function class - for instance Rosenbrock class
> - that defines several methods, like gradient, hessian, ... without
> a real state - a real state being something other than for instead
> the number of dimension for the rosenbrock function, the points
> that need to be approximated, ... a real state is something that is
> dependent of the subsequent calls to the functor, the gradient, ...
> - so that this function can be reused efficiently.
> Then you use an instance of this class to be optimized.
> You choose your step mode with its parameters like gradient,
> conjugate gradient, Newton, Quasi-Newton, ...
> You choose your line search with its own parameters -
> tolerance, ... - like section methods, interpolation methods, ...
> Actually, you choose your stopping criterion.
> Then you make something like :
> optimizer = StandardOptimizer(function = myFunction, step =
> myStep, .......)
> That is a modular design, and that is why some basic functions must
> be provided so that people that don't care about the underlying
> design really do not have to care. Then if someone wants a
> specific, non-standard optimizer, one just have to select the
> wanted modules - for instance, a conjugate-gradient with a golden-
> section line search and a relative value criterion onstead of a
> fibonacci search and an absolute criterion -.
> It can be more cumbersome at the start, but once some modules are
> made, assembling them will be more easy, and tests will be more fun :)
More information about the Scipy-dev