Working with JSON data export files

[SunsetRunner]

@MrMarv

@iccir919

@eglons

@craigvan

There has been a lengthy discussion about the use of the JSON file format over on this thread, which is now closed.  https://community.fitbit.com/t5/Fitbit-com-Dashboard/Resolved-Changes-to-Data-Export-tool/td-p/30687.... The tread has been marked as resolved but only because after a huge outcry the original CSV data export was reinstated. There will no doubt be more so here is another thread so we can learn more about the content, provide further feedback to the Fitbit team and hopefully get things fixed.

The JSON format is no doubt here to stay and if you want heart rate data it is the only way to get it.  However, the current export file is not without its problems, here are some and there will no doubt be more.

• You have to download everything from day one each time you export. For some that is a huge file.
• Inconsistent data formats and types and standards
• Missing data
• No explanation of some of the data e.g “confidence” in the Heart Rate data
• Conversion from JSON to CSV format

Tools

The tools problem can be addressed 2 ways. If you have Excel in Windows there is a JSON to CSV add in. It only does one file at a time.

@ICCIR919 has produced an online tool that converts and merges data files of the same type. It also fixes the timezone problem making time local.

https://iccir919.github.io/fitbit-json-to-csv/index.html

It runs in the web browser on your PC so none of data goes anywhere.

Data Content

@mrmarv has also provided an overview of the data I’ve copied it here so you can see some of the issues for yourself.

Time zones:

     GMT - Distance,  Exercise, Heart Rate, Steps

     Local Time -  Altitude, Calories

     No standard - and a company's reputation often lives or dies on its standards.

Start Dates of the monthly bucketed files - looks like a holdover from the 30 days in a Fitbit month. Maybe starts with the day of incorporation, I don't know. I could be missing something here. Could be a serialized date formula on the system they work on.

Files with a sequential number as part of the key:

     The exercise logs have the oldest logs with a zero and the latest, in my case with 500.

     The food logs have the newest with a zero and the oldest with, in my case 3300

     Standards, people.

Calorie logs - no comment, other than it is in local time zone.

Distance logs:

     GMT

     No clue how they decide to book an entry - the time differential between the zero distance entries is sporadic

     The value appears to be in centimeters - no biggie but interesting that other logs are not in centimeters.

Exercise logs:

     GMT

     My treadmill entries show distance in miles NOT centimeters

     My device initiated walk shows distance in miles NOT centimeters

     Duration is in milliseconds - really?

Food logs:

     ZERO nutritional values for customized foods. And let's face it people, the only way to track nutritional values and get them right is to customize them. Fitbit is not even close to being perfect and neither is MyFitnessPal or FatSecret so don't try and point me there. Especially after the "Nutritional Label Fiasco" a while back.

     Fitbit has already shown weakness in breaking down nutritional values for display on the Web and on their apps. This just shows their developer and project leader's blindness once again.

     How are people going to get data to present to their doctors when Fitbit is no longer reporting on these very important values? Or even evaluate their own food plan? Can you say "Disservice"?

Heart Rate logs:

     GMT

     Confidence level of 1 - 3 - I'd like a definition.

[LanuzaFitbit]

Hello @SunsetRunner.

Thanks for creating this thread and sharing all of those details.

Please do know that we pay attention to all the feedback and that we will be keeping an eye on this thread. Hopefully other users find this useful.

Please be sure to keep this civil, constructive and to keep the Community Guidelines in mind.

As always, feel free to reach out with any further questions.

[SunsetRunner]

@LanuzaFitbit

Thanks; let's make a start.

There is no GPS activity data in the JSON export. GPS data has to be exported separately for each activity. I can see why a separate export may be necessary as there could well be a lot of data, however, the current GPS export format is a TCX file. A similar XML format to JSON but not the same (there may well be good reason for this and happy to hear it)

As the JSON export is a response to Fitbit meeting GDPR rules there should probably be on option for a separate export of the full GPS data on the same page as the full data export.

I'm suggesting two separate exports to make the data more manageable for those who aren't interested in GPS data

[Craig]

@SunsetRunner   Thank for the ICCIR919 location but it apparently does not like over 4k files at once 

[Craig]

@LanuzaFitbit    I may be missing it but I also do not see anything for my glucose reading in the JSON files

[Colinm39]

Here is my reply from Support today about the confidence levels of the Heart Rate in json files.

"Confidence scores indicate confidence in the accuracy of the captured data with “3” being the highest and “0” being the lowest; a score of 0 indicates no Heart Rate is detected by the Fitbit tracker and no Heart Rate is displayed on the tracker device".

[Colinm39]

I have just analyzed a Sleep Workout of  5h:46m to analyze the HR Confidence level..on my Ionic. 

I do this occasionally because my sleeping HR has periods where it gets down to 38bpm. This Workout test, the average for the time period was 49 bpm.  Yes, I set my Ionic to Workout to get a better analysis, and in this case I started it at 2:54am on my first bathroom break...

With 0 being no detected HR, there were 2 at about 1.5 minutes burst and 19 at about 1 minutes and the majority 5-15 secs.. A couple of them coincided with bathroom breaks and probably when I rollover in my sleep.

The breakdown was from  8923 pulse points over 346 minutes  average 26/minute..

Confidence #
          0      12.69%   = 43 minutes out of 346 minutes
          1       1.47%
          2       2.49%
          3     83.36%

Total        100.00%

For those interested, my RHR is 52, Cardio Score is 42-46 and I'm 80 next year.  According to the specialists I'm genetically built this way and no bradycardia.

---

@Colinm39 wrote:

Here is my reply from Support today about the confidence levels of the Heart Rate in json files.

"Confidence scores indicate confidence in the accuracy of the captured data with “3” being the highest and “0” being the lowest; a score of 0 indicates no Heart Rate is detected by the Fitbit tracker and no Heart Rate is displayed on the tracker device".

---

[SunsetRunner]

Colin

This is very interesting. In a file of 3031 Heart Rate entries I have 93 with a confidence of 0. However, each of the entries has a HR value and they aren't the last known value and block of 4 consecutive entries don't have the same value. So if 0 is no detected heart rate value how are these values determined?

[Colinm39]

@SunsetRunnerThanks for the post... I have the same question, and to double check what is happening I downloaded the PulseWatch data which gives you the Seconds.. and I can't find a pattern..We have 1,2 and 3 second intervals

I have a missing heart beat and that wasn't detectable this time because when that happens my heart does about 3 faster catchups and there was no evidence of that...

Where I had nearly a minute of "0" it coincided with a bathroom break which also coincided with being awake on the sleep graph.. Here is a segment of that with the PulseWatch data.

My heart rate averaged 49 for this sleep period and you can see the HR pickup. it eventually got to 93 after 3 minutes, then back to 47 after another 2 and finally 44 for a total of 7 minutes

My Sleep graph had me awake for 6 minutes... So near enough

[SunsetRunner]

Colin

@Colinm39

The intervals between your data is 2-4 secs if you have a heart rate of say 60 that's 1 beat per second so if your heart is doing "triples" the extra beats could be missed or at least lost in any smoothing that may be going on.

Sleep is another issue I'm struggling with. Not my sleeping but the results Fitbit are providing. I go to bed around the same time most nights. However, the records show that my sleep commences after my first wake up and bathroom visit. I had hoped that the detailed data would provide some clues but we don't get the accelerometer data and have no idea what the code does to determine sleep start time.

[Colinm39]

@SunsetRunner  We can be lucky some times.. A few days ago I had just woken up and was lying on my left side in bed waiting for the 7:30am news on the radio. I felt the miss, and then the sudden quick catch up pulses, twice in 3 minutes .. I made a note of the times and this is the result. 

I accept that at 1 second intervals it will be difficult to pickup.

My Omron BP monitor always picks up the change in rhythm when it occurs when I test my BP...

Sleep... I'm fortunate because each of my Fitbits that were designed for sleep, work correctly. Recently I was watching a Jason Bourne movie on TV and the Ionic had me asleep for about an hour.... I must admit I was relaxed. That is very rare happening and I wasn't asleep.   The sleep graph always picks up the bathroom breaks..

@SunsetRunner wrote:

Colin

@Colinm39

The intervals between your data is 2-4 secs if you have a heart rate of say 60 that's 1 beat per second so if your heart is doing "triples" the extra beats could be missed or at least lost in any smoothing that may be going on.

Sleep is another issue I'm struggling with. Not my sleeping but the results Fitbit are providing. I go to bed around the same time most nights. However, the records show that my sleep commences after my first wake up and bathroom visit. I had hoped that the detailed data would provide some clues but we don't get the accelerometer data and have no idea what the code does to determine sleep start time.

---

[jdeluyck]

I've been playing with the export too. 

I've got some questions though as to what the content is of certain json files, as I have a fitbit one..

• altitude - this just doesn't make sense whatsoever. 
• calories - is this calories burned, calories eaten, activity related calories, ... 

[LanuzaFitbit]

Hello everyone.

Again, thank you very much for all of your feedback. 

As mentioned before, we're always keeping an eye on all feedback. I'm sure that other users will also find the findings and details posted in here to be very useful.

Feel free to reach out if there is anything I can assist you with.

[jdeluyck]

@LanuzaFitbit

can I find anywhere what data is actually in the files?

[LanuzaFitbit]

Hello @jdeluyck.

I apologize for the delayed answer.

You can read more about the export function and all of the data that can be exported in this help article. 

Please let me know if there is anything else that I can help you with.

[RaloAttis]

Hi @LanuzaFitbit,

The link you've posted to the help page describing the included data in the export is far from complete. I've been trying to process the JSON data creating and applying custom filters and combination techniques, however there are many data fields which contain unclear data. We can make an educated guess however I would really appreciate a complete disclosed documentation describing all fields in all separate files that users can download. As with the data, this can of course be rather technical documentation since the data itself is also more geared towards developers.

In my data dump (spanning 7 months) I have about 4500 'altitude' measurements of which 4350 have the value '10' and the only other values (150 or so) are either 20, 30 or 40. Your link hints that it might have something to do with 'floors', but it's most definitely not clear. I can easily name 10 other fields I've come across that contain unexpected data values.

Please let us know if there is any other documentation available or please pass this information to the people who might be able to provide it. Thanks! 

[jdeluyck]

Hello @LanuzaFitbit,

Thanks. This info helps. but there's not enough information in there. To be honest, it's actually not describing stuff at all, that is in the data export.

Is there more information available except this limited page?

[LanuzaFitbit]

Hello @jdeluyck and @RaloAttis.

I apologize for the delay in my response. 

At this time this is all of the information we have available to share. Please do know that as soon as there is anything else to share with everyone we'll be letting everyone know about it.

Let me know if there is anything else I can help you with.

[OudjesEric]

I stumbled on this thread trying to make sense of the JSON data. 

Let me compress this thread by saying the following: Fitbit enables us to download JSON data without telling us what it means. That's what I understand from the very sparing replies to any questions.

Does that sound at all weird to anyone? If you. Fitbit, as a company decide to give us our data and will not tell us what it means, what is the point then? Someone designed and wrote the format for that export and based it on actual data we generated. So, I can only hope, having been around software design for a lot of years, there must be a logic and it should be public, since again, it's our data - or so I've heard.

It is really totally unexpected for a company like Fitbit to say the information is not available. 

[OudjesEric]

Aha, being less active these past months I missed the outcry back in November. Just skimmed through 20 pages of forum outcry and it seems 2 months further there has been little or no development.

The default export is still the option 99% of the users don't want to have and makes absolutely no sense to have as the default choice. Certainly while being stuck with downloading everything always.

There don't appear to be tools or even information around to actually translate/interpret/let alone use the JSON data intelligently.

Last and certainly not least: there doesn't seem to be anybody at Fitbit able or willing to defend the choice at all. Wow, talk about a real stinker. Good job of alienating your userbase...