[SciPy-user] maxentropy

Ed Schofield schofield at ftw.at
Tue Mar 14 01:58:18 CST 2006

Hi Matt,

I've just read a paper on maxent text classification (by Nigam,  
Lafferty and McCallam, 1999) to try to understand how you'd do this.   
In this paper the model was p(y|x), like you were intending.  So the  
sample space is {0,1}.  They express features as functions f_i(x, y)  
of both the class and the text.  Fine ... but now I can understand  
why you don't want to explicitly train models for each x, which might  
not even be possible; instead you want a model with one set of  
features {f_i, i=1,...,m} and one set of corresponding parameters.   
Hmmm ... so a conditional framework is trickier than I thought...

... And I've now just re-read Malouf's (2002) paper.  I think fitting  
a conditional model just requires proper handling of the  
normalization constant.  So I'll thrash out some code in the next few  
days and get back to you :)

-- Ed

On 14/03/2006, at 2:46 AM, Matthew Cooper wrote:

> Hi Ed,
> Thanks very much for your reply.  I think it helped a lot, but I may
> be a bit confused about conditional versus unconditional modeling.
> What I'm doing is similar to text categorization.  I observe some text
> (vector X) and want to determine if a binary label (scalar y) should
> be applied (y=1) or not (y=0).  So, I look at this as using maxent to
> estimate P(y|X).  In this case, is my sample space simply {0,1} or is
> it the space from which X is sampled?  I had thought this was
> conditional modelling, but I don't want to explicitly train models for
> every different X, rather I want to select and weight features from
> the whole corpus that in turn imply, for each X', some P(y|X').  I
> assumed it was conditional because I'm not modelling P(y',X').
> I have been trying to build models using a sample space with tuples as
> elements like (X_n,y_n).  I then am greedily building a set of feature
> functions using information gain to select features.  It's not quite
> working, and I'm worried I'm not defining the sample space properly.
> I will try to figure out how to define the sample space as {0,1} and
> then redefine the features, but I'd appreciate any advice.  Also, if I
> get this working, I'd be happy to try and help with adding some
> feature selection examples to your documentation.  At the moment, I'm
> still, as you can tell, figuring out what I'm doing.
> Thanks,
> Matt
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.net
> http://www.scipy.net/mailman/listinfo/scipy-user

More information about the SciPy-user mailing list