[SciPy-User] synchronizing timestamps from different systems; unpaired linear regression
Thu Apr 12 10:25:51 CDT 2012
I had a similar problem with a 16 channel recording setup I used for
The ADC clock rates were slightly different, but after some minutes of
recording the error was two orders of magnitude larger than the
waveforms I was recording.
I solved the problem by measuring the relative clock rates very
accurately (by time-staming TTL signals from the parallel port a
computer running MS-DOS), upsampling to some biological ridiculous
sampling rate (with an FIR least squares interpolation filter and FFTs),
and then downsampling to approximate synchrony.
All in all it worked well, but I nevertheless modified the system by
connecting all ADCs to a single oscillator.
The problem had gone undetected by the manufacturer because each
oscillator served four ADCs, and synchrony was only tested between ADCs
served by the same oscillator.
And I learned an important lesson:
It can take less time to redo the whole experiment than fix corrupted
data. But my anger at wasting time with a faulty recording system made
me waste even more time correcting the corrupted data files.
On 10.04.2012 11:27, Chris Rodgers wrote:
> I have what seems like a straightforward problem but it is becoming
> more difficult than I thought. I have two different computers
> recording timestamps from the same stream of events. I get lists X and
> Y from each computer and the question is how to figure out which entry
> in X corresponds to which entry in Y.
> 1) There are an unknown number of missing or spurious events in each
> list. I do not know which events in X match up to which in Y.
> 2) The temporal offset between the two lists is unknown, because each
> timer begins at a different time.
> 3) The clocks seem to run at slightly different speeds (~0.3%
> difference adds up to about 10 seconds over my 1hr recording time).
> I know this problem is solvable because once you find the temporal
> offset and clock-speed ratio, the matching timestamps agree to within
> 10ms. That is, there is a strong linear relationship between some
> unknown X->Y mapping.
> Basically, the problem is: given list X and list Y, and specifying a
> certain minimum R**2 value, what is the largest set of matched points
> from X and Y that satisfy this R**2 value? I have tried googling
> "unmatched linear regression" but this must not be the right search
> One approach that I've tried is to create an analog trace for X and Y
> with a Gaussian centered at each timestamp, then finding the lag that
> optimizes the cross-correlation between the two. This is good for
> finding the temporal offset but can't handle the clock-speed
> difference. (Also it takes a really long time because the series are
> 1hr of data sampled at 10Hz.) Then I can choose the closest matches
> between X and Y and fit them with a line, which gives me the
> clock-difference parameter. The problem is that there are a ton of
> local minima created by how I choose to match up the points in X and
> Y, so it gets stuck on the wrong answer.
> Any tips?
> PS: my current code and test data is here:
> Chris Rodgers
> Graduate Student
> Helen Wills Neuroscience Institute
> University of California - Berkeley
> SciPy-User mailing list
More information about the SciPy-User