We are trying to use fitbit data for a research project and have noticed something strange with the timestamps.
We are often getting pairs or multiples of rows that are duplicates of each other (same time stamp and same values in all measurement and data source fields) apart from sometimes a differing value for ‘time received’. For some data types (breathing_rate, intraday_heart_rate_variability, skin_temperature, sleep_classic, sleep_stages) this happens for over half of the timestamps and there are often many repeated rows for the same timestamp. We are currently assuming that these duplicates are all from the same raw record, rather than different records with errors in the timestamp, and are planning on deleting the duplicates. However, for data types where we are interested in the cumulative values over a length of time (e.g steps, calories), deleting records could have quite a big effect on daily/hourly totals. Therefore we are wondering whether our assumption that these duplicates are all from the same raw record is likely to be correct?
Another issue is that we also sometimes see different measured values (e.g different skin temperature) for the same time stamp. This is more problematic because in this case we do not know which value to use. This issue is quite rare in most data types, however in breathing_rate there are different values for about 12% of timestamps, which is quite a big proportion of the data that we are not able to trust. Does anyone have any idea what might be causing this or what the best approach is for dealing with this?
