[SciPy-dev] huge speed regression in loadmat from 0.6.0 to 0.7.0

Ryan May rmay31@gmail....
Wed Feb 11 14:09:30 CST 2009


On Wed, Feb 11, 2009 at 2:03 PM, Scott David Daniels
<Scott.Daniels@acm.org>wrote:

> Ryan May wrote:
> > ... Well, here's a patch against gzipstreams.py that changes to add the
> > chunks to a list and only add to the string at the very end. See if it
> > helps your case.  If not, is there somewhere you can put the datafile so
> > that we can test with it?
> Well, in your patch, instead of:
> @@ -95,11 +100,12 @@
>               data = self.fileobj.read(n_to_fetch)
>               self._bytes_read += len(data)
>               if data:
> -                self.data += self._unzipper.decompress(data)
> +                self_data += self._unzipper.decompress(data)
>              if len(data) < n_to_fetch: # hit end of file
> -                self.data += self._unzipper.flush()
> +                self_data += self._unzipper.flush()
>                  self.exhausted = True
>                  break
> +        self.data += ''.join(self_data)
>
> Use:
> @@ -95,11 +100,12 @@
>               data = self.fileobj.read(n_to_fetch)
>               self._bytes_read += len(data)
>               if data:
> -                self.data += self._unzipper.decompress(data)
> +                self_data.append(self._unzipper.decompress(data))
>              if len(data) < n_to_fetch: # hit end of file
> -                self.data += self._unzipper.flush()
> +                self_data.append(self._unzipper.flush())
>                  self.exhausted = True
>                  break
> +        self.data += ''.join(self_data)
>
>
Yeah, you're right.  I thought += for lists just mapped to append, but
apparently it appends other lists, but extends the list by other sequences.
Weird.

But if you do make that change, it solves your performance problem?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
Sent from: Norman Oklahoma United States.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-dev/attachments/20090211/93af041f/attachment.html 


More information about the Scipy-dev mailing list