[Numpy-discussion] Histograms of extremely large data sets

[eric jones]

Glad to here it worked for you.

see ya,
eric

Cameron Walsh wrote:
> Thanks very much, Eric.  That line fixed it for me, although I'm still
> not sure why it broke with the last line.
>
> Your weave_histogram works a charm and is around 16 times faster than
> any of the other options I've tried.  On my laptop it took 30 seconds
> to generate a histogram from 500 million numbers, which is fine.
>
> Thanks and best regards,
>
> Cameron.
>
>
> On 14/12/06, eric jones <eric at enthought.com> wrote:
>   
>> Hmmm.
>>
>> ?  Not sure. ?
>>
>> Change that line to this instead which should work as well.
>>
>>         code = array_converter.declaration_code(self, templatize, inline)
>>
>> Both work for me.
>>
>> eric
>>
>> Cameron Walsh wrote:
>>     
>>> On 13/12/06, Cameron Walsh <cameron.walsh at gmail.com> wrote:
>>>
>>>       
>>>> On 13/12/06, eric jones <eric at enthought.com> wrote 290 lines of
>>>> awesome code and a fantastic explanation:
>>>>
>>>>
>>>>         
>>>>> Hey Cameron,
>>>>>
>>>>> I wrote a simple weave based histogram function that should work for
>>>>> your problem.  It should work for any array input data type.  The needed
>>>>> files (and a few tests and examples) are attached.
>>>>>
>>>>>           
>>> [...]
>>>
>>> Hi Eric,
>>>
>>> I've ran test_weave_histogram.py and histogram_speed.py, but each one
>>> seems to fail on calling typed_array_converter.declaration_code() as
>>> follows:
>>>
>>> Traceback (most recent call last):
>>>   File "histogram_speed.py", line 26, in <module>
>>>     res2 = histogram(data, bins)
>>>   File "/home/cameron/repos/wavesmaker/trunk/code/gui/process_modules/eric_histo/weave_histogram.py",
>>> line 67, in histogram
>>>     compiler='gcc')
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
>>> line 339, in inline
>>>     **kw)
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
>>> line 447, in compile_function
>>>     verbose=verbose, **kw)
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
>>> line 353, in compile
>>>     kw,file = self.build_kw_and_file(location,kw)
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
>>> line 334, in build_kw_and_file
>>>     file = self.generate_file(location=location)
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
>>> line 295, in generate_file
>>>     code = self.module_code()
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
>>> line 203, in module_code
>>>     self.function_code(),
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
>>> line 269, in function_code
>>>     all_function_code += func.function_code()
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
>>> line 83, in function_code
>>>     decl_code = indent(self.arg_declaration_code(),4)
>>>   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
>>> line 62, in arg_declaration_code
>>>     arg_strings.append(arg.declaration_code(inline=1))
>>>   File "/home/cameron/repos/wavesmaker/trunk/code/gui/process_modules/eric_histo/typed_array_converter.py",
>>> line 18, in declaration_code
>>>     code = super(typed_array_converter, self).declaration_code(templatize,
>>> TypeError: super() argument 1 must be type, not classobj
>>>
>>> I've tried this with Python2.4 and Python2.5 with the same results.
>>>
>>> What do I need to change, since it seems to have worked for you but not for me.
>>>
>>> Thanks and best regards,
>>>
>>> Cameron.
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>>       
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>