Batch reading accelerometer gets non-increasing timestamp ?

[zfzhong]

Hello,

I developed a fitbit app to batch reading accelerometer data. I configured he Accelerometer to {frequency: 50, batch: 50}.  When I examined the timestamps of each batch (of 50 records), I found that some timestamps (and also the data for this timestamp) are duplicated.  

Specifically, I got the following data for a single batch of 50 readings:

#, timestamp, x, y, z

... 

0, 508723056, x, y, z

1, 508723076, x, y, z

2, 508723096, x, y, z

3, 508723116, x, y, z

4, 508723135, x, y, z

5, 508723056, x, y, z  <---- same as row 0

6, 508723076, x, y, z  <---- same as row 1

7, 508723096, x, y, z  <---- same as row 2

8, 508723116, x, y, z  <---- same as row 3

9, 508723135, x, y, z  <---- same as row 4

I thought the timestamps for the 50 readings from a batch should be evenly distributed (one record per 20 ms). But somehow the above data shows that the 50 readings contain duplicates, and therefore we don't get 50 records after removing these duplicates.

Is this an expected behavior of the batch reading? Or was it because that I didn't configure it properly?

Please comment if you have similar experience and I welcome your input on this. I appreciate your help with this problem.

Thanks,

David 

[Gondwana]

I've looked at batched accel readings quite extensively, and have never noticed this. It would be interesting to see if the x,y,z were identical.

console.log can drop lines if too much output is logged, but that shouldn't cause what you're seeing.

[zfzhong]

Hi Peter,

Thank you very much for providing comments! Yes, the x, y, z values are identical for entries with duplicated timestamp.

I used the following code to catch the timestamp duplicates:

let currentTimestamp = 0;

for (let i = 0; i < accel.readings.timestamp.length; i++) {
    let ts = accel.readings.timestamp[i];
    if (currentTimestamp < ts) {
        currentTimestamp = ts;
    } else {
        console.log(`*** Timestamp Discrepency: ${currentTimestamp}, ${ts}, ${i}`);
    }

    accelRecordTimeView[i] = ts;
    accelRecordXView[i] = Math.round(x[i]);
    accelRecordYView[i] = Math.round(y[i]);
    accelRecordZView[i] = Math.round(z[i]);
}

Please let me know if you find anything wrong in my code. Thanks again!

-- David

[zfzhong]

Peter,

I cloned your code from https://github.com/gondwanasoft/fitbit-accel-fetcher. 

I put some simple logic to check the timestamps reading from batches. 

Unfortunately, I found the timestamp gaps and duplicates as well. 

I put my code (slightly modified from yours) and described the problem here:

https://github.com/zfzhong/fitbit-accel-fetcher

I would appreciate it very much if you can help look into it.

Thanks,

David

[Gondwana]

Thanks. Yes, I know of that problem. I started to look at it a while ago. My theory is that files may be occasionally sent out of sequence, or even sent more than once.

Unfortunately, it will be a while before I get time to resume work on it.

[zfzhong]

Peter,

Thank you for your response!

The way I check timestamps is to check them soon after we read them from the batch (before they get written into files). 

I suspect they are not in strictly increasing order in the batch. Your code of writing them into files should be fine.

Thanks,

David

[Gondwana]

You could be right. I never got time to finish my analysis.

I did import a transfer of 401 files (each of 240 readings) into a single spreadsheet, and checked for timestamp discontinuities in there. There are about 60 anomalies, and all but one of them occur between the end of one file and the start of the next one. To further investigate, better logging is required, but that's hard to do without producing too much of it. Plus, I don't know whether the problem is in the Fitbit or Android side.

One way to simplify the analysis would be to transfer the whole timestamp, rather than just the lower 32 bits of it. I wouldn't rely on that as a solution, though; if files are indeed duplicated or transferred out of order, that needs to be corrected.

[zfzhong]

Peter,

I did check on the fly (even before we write the timestamps into array buffer). 

There is one small thing that you probably should notice: before you transfer the files from the companion to the server, you add 0x8000 for any occurrence of timestamp decreasing. But some of the occurrences are not because of rolled-around, they are simply the timestamp gaps which come from batch reading.  

I am directing this problem to Fitbit developers. Hopefully, they will have someone to look at this issue.

Thanks,

David

[Gondwana]

I tried to attach my spreadsheet here. I hope you can make some sense of it.

You may have observed a different phenomenon to me. I don't think I've ever seen a raw timestamp value go backwards.

[Gondwana]

I (eventually) stand corrected. I've just been looking at some more accel batch readings, and in a file containing 48,360 readings, discovered a sequence of readings with these differential timestamps:

28
32
168
32
30
30
30
-110
30
30

The readings should all be spaced at about 30 ms.

Resequencing the readings to put them in strict order is probably the wrong answer because that would create non-standard timestamp differences with adjacent readings.

Unfortunately, because the watch wasn't in motion at the time, it's not possible to assess the continuity of readings by looking at the x,y,z values.

Subtracting 139 (or thereabouts) from the third to the seventh reading results in a smoother sequence. I don't know why that should be necessary.

This happened near the middle of one file (out of 404), so I don't think the error is caused by my (new) way of transferring timestamps.

[zfzhong]

This is exactly what I have observed. It would be great if the Fitbit team starts to investigate this problem.

[Gondwana]

I've pushed some changes to fitbit-accel-fetcher and android-fitbit-fetcher to improve handling of some sorts of unexpected comms behaviour. fitbit-accel-fetcher should no longer give incorrect timestamps if files are received out-of-sequence, and files reported to be corrupt by android-fitbit-fetcher will be resent.

Neither app tries to adjust strange timestamps reported by the Fitbit API. I think that would be best attempted downstream; the aim of these repos is only to demonstrate how to make sensor data accessible.