Resolved: Changes to Data Export tool

[KateFitbit]

Fitbit Update 11/30/2018: This issue has been resolved and I'll be closing it from further responses. 

---

Fitbit Update 11/28/2018: The old data export tool is now back and available for use. You can now choose which way you would like to export your data going forward. If you would like to access the old tool, please go here and select "if you would like to use the old data export tool click here" under Export My Fitbit Data. 

If you have any issues, please let me know. Thanks for your patience and reports! 

---

Fitbit Update 11/27/2018: Happy Tuesday! Just a heads up that we hope to have an update soon on the Data Export tool. I'll continue to update the thread this week. 

---

Fitbit Update 11/20/2018: While I don't have an update today, I can assure you that your feedback is being heard. One note, if you contact our support team, they will not be able to supply additional details, but I will continue to update this thread as I know more. We do appreciate the feedback.

---

Fitbit Update 11/16/2018:  Hi Everyone, here's an update on the recent changes to the export tool:

With the recent improvements to our tool, you can now export your Fitbit data, starting from your first step with Fitbit.

You can also access new data for download, such as heart-rate data, exercise logs, and your activity in the Fitbit community. For more information, see How do I export my data?

We understand that some customers are having difficulty interpreting their data in the new download format. We’ll continue to add features and make improvements based on your feedback.

Thank you all for your input in the thread. Our team will continue to monitor this thread and I'll let you know when I have further updates. 

[Colinm39]

@Rich_Laue I have just analysed another HR download and it shows a confidence level of 3.

What does this mean.. So I have now a level of 1 and 3.

[SunsetRunner]

---

@Colinm39 wrote:

@Rich_LaueI have just analysed another HR download and it shows a confidence level of 3.

What does this mean.. So I have now a level of 1 and 3.

---

Its an error I think we all agree that the current confidence level is zero

Pete

[stvding]

My goodness. I am a programmer. Even I am furious about change. I hate to write more code to parse these json files!!!!!

I just want simple csv file containing my specified days of data.

I don't want to write more code after work! come on!

just bring the old stuff back as another option as most of us, even me as a developer want that!

[larryliulcs]

Thanks Kate for the feedback. The biggest reason why I'm using my fitbit is because all the data can be exported and it syncs to my weight scale which can ALSO exported in excel formate. PLEASE PLEASE PLEASE PLEASE PLEASE bring the CSV/Excel functionality back so that EVERYDAY people can actually use it. Thanks,

[Colinm39]

@SunsetRunnerBeing a very retired IT person I can feel for the Fitbit programmers, probably given a vague scope for the change.. . I'm just hoping it's not a last gasp dash to give us our data at any cost and leave us with the unscrambling. @stvding I feel like you about this change.. Not very professional from Fitbit and with no explanation to support he change.

I'm sure we will finish up with a workable layman's solution.... With the number of posts here @KateFitbit will be keeping us up to date..

---

@SunsetRunner wrote:

---

@Colinm39 wrote:

@Rich_LaueI have just analysed another HR download and it shows a confidence level of 3.

What does this mean.. So I have now a level of 1 and 3.

---

Its an error I think we all agree that the current confidence level is zero

Pete

---

[SunsetRunner]

---

@Colinm39 wrote:

@SunsetRunnerBeing a very retired IT person I can feel for the Fitbit programmers, probably given a vague scope for the change.. . ..

---

Me too. The programmers weren't the problem it's their managers that need revving up. We know how these things should go and this has clearly been mismanaged by somebody. By now that person should have already done something about this fiasco and taken action. Why it has taken so long is incomprehensible.

I don't think it is a vague scope I think they've mixed up to completely different requirements and thrown the baby out with the bath water. Poor analysis and confirmation of the scope. Why nobody questioned what would happen with the existing data download when the new one went live beats me. All they had to do was looks at the logs and they’d have a good idea how regularly people used it. My guess would be that programmers probably did that but nobody listened to them.

Pete

Pete

[Colinm39]

Maybe our frustration this week is part of Fitbit's integration via Google to the Health Sector in this link

[SunsetRunner]

---

@Colinm39 wrote:

Maybe our frustration this week is part of Fitbit's integration via Google to the Health Sector in this link

---

Interesting, it remains to be seen where this goes. Mergers and partnerships are not always what they may seem. 

Pete

[rubie218]

It's not that we need more features... we need the old feature back!  Why fix what wasn't broken?  Having my data in an easily-digestible daily format is incredibly important and the only value I get out of having a fitbit.  With this feature missing, I don't plan to continue to purchase fitbit devices.

[syphus]

• Maybe our frustration this week is part of Fitbit's integration via Google to the Health Sector in this link

@Colinm39I checked that link and it is interesting.    Kinda explains why this change has occurred (without anyone being aware).  Fitbit trying to get in bed with Google.  And of course, without any acknowledgement from the owners of the data.

The data from my Fitbit is private and I believe as long as it is tied to me, it falls under HIPAA laws.  In short Fitbit cannot give my data away to anyone without my permission (and i don't mean by "use of this app...".

This also explains why the change in format  . . . I believe it is not meant for the user, it is meant for Google. (IMHO)

Syphus

[slakis]

I've been a Fitbit user for 4 years. If the previous export to excel csv isn't restored by Christmas, I hope a lot of people switch to Garmin. This is horse **ahem**.

[ares952]

I have created a script to generate basic csv data which can be imported by e.g. garmin connect.

If somebody gets it usefull, feel free to use it.

https://www.dropbox.com/s/edr2u9ww1bime0k/convert-to-old-csv.sh?dl=0

https://www.dropbox.com/s/pt59tyl5rm2q15h/convert-to-old-csv.php?dl=0

However, it is tested on linux systems only with php cli installed. But it should be used on any php server.

The only thing, you need to parametrize ./convert-to-old-csv.sh:

FILENAME="20181117-MyFitbitData.zip"
USERPROFILE="<tobedefined>"

i.e. set .zip file you have downloaded and set the profilename (it is basically the subfolder within the downloaded .zip file).

and then to execute ./convert-to-old-csv.sh script.

It will generate the file "fitbit-export.csv".

[SunsetRunner]

The new data export dumps a huge zip file which then splits into files for EVERYTHING..  The files are JSON format which may be useful to API programmers but are useless to an end user like me.

I bought my Charge 3 solely for getting sleep data only for my doctor.  I used to be able to select sleep and date ranges that produced CSV or XLS spreadsheet formats that we could easily read and run statistics on.  No we can't use the data at all.

My Charge 3 is still in the 45 day return period.  I am looking at returning it.

[PaulGreatrex]

This is my first login to website based Fitbit. I have used the android app till now.

I came on here to download my data. I want to overlay my sleep, calories and food log - i was hoping this was available but will add to the features request. I want to correlate if/any issues in my plateau.

Anyway my point knowing a little about data formats and how they differ.

What aggravated me was the first post 'i understand people are having issues...' and then another 'what is/are the issues... we are looking for ways to improve...'

As im sure there are enough followers and clearly someone made this decision - what was the drive for it.

Not to say that CSV is a far more primitive format that can be read and managed manually - what was the drive to JSON? There must have been a positive reason for change? Duplicate data so one copy only for the website ...how about a converter for us?

I write software and work in embedded and PC plus significant test equipment data and analysis - i have never used JSON and to me it is there with XML and standardised data formats for web use - thats my opinion on it.

As for the comments and followings i have just read - please bring back CSV - and also please explain how you expected the non technical community including myself on how you intended Joe Bloggs to use that data.

My first impression is that there is a piece of software for it? or it can be imported into the website-but its already there so i am new and a little confused.

CSV would certainly help me achieve what i want.

Also (for when the download is requested) one of your software guys could knock up a JSON to CSV converter that collates and concatenates the data files. I wouldn't be annoyed if the steps, food, exercise etc.. were in different files. As long as they had a common index of time for scatter graphs or manual manipulation.

Apologies i didn't get to the end of this forum - i got the point on page one after i downloaded the data and thought - huh!

[misia06]

KateFitbit,

When will you have the export format restored to the old format, or offer the old format alongside of the new format??????

[MrMarv]

Ahhh @Colinm39 -

---

@Colinm39 wrote:

Maybe our frustration this week is part of Fitbit's integration via Google to the Health Sector in this link

---

Maybe so, but my initial review of the files generated still leaves much to be desired. Maybe @KateFitbit can try and explain it away, but I doubt it. Good luck with that, Kate.

First off, for those who don't already know, I am an application developer e.g. programmer, among other things, and have been for nearly 40 years. And when I was an IT Director or VP of Development, if someone presented this zip file to me as an example of their work, I would show them the door. Politely, of course, but rest assured they would not be invited for a second interview.

Yes, JSON is one of the newest "things", allegedly an upgrade from xml, which in my mind still cannot replace a simple tab-delimited or .csv file that emulates an sql query when it comes to the exchange of data. And I don't say that just because I'm an IT Dinosaur. I've written one or two (probably more) JSON parsers for my people when a vendor just gushes over how it is just the best thing. And while there is a JSON standard, it seems that there are many different interpretations of that standard out there.

But let's forget my prejudice over people trying to get too cute over their presentation of the data and have a look at the data, itself, and see if it is even usable.

Time zones (I'm in PST):

     GMT - Distance,  Exercise, Heart Rate, Steps

     PST -  Altitude, Calories

     No standard - and a company's reputation often lives or dies on its standards.

Start Dates of the monthly bucketed files - looks like a holdover from the 30 days in a Fitbit month. Maybe starts with the day of incorporation, I don't know. I could be missing something here. Could be a serialized date formula on the system they work on.

Files with a sequential number as part of the key:

     The exercise logs have the oldest logs with a zero and the latest, in my case with 500.

     The food logs have the newest with a zero and the oldest with, in my case 3300

     Standards, people.

Calorie logs - no comment, other than it is in local time zone.

Distance logs:

     GMT

     No clue how they decide to book an entry - the time differential between the zero distance entries is sporadic

     The value appears to be in centimeters - no biggie but interesting that other logs are not in centimeters.

Exercise logs:

     GMT

     My treadmill entries show distance in miles NOT centimeters

     My device initiated walk shows distance in miles NOT centimeters

     Duration is in milliseconds - really?

Food logs:

     ZERO nutritional values for customized foods. And let's face it people, the only way to track nutritional values and get them right is to customize them. Fitbit is not even close to being perfect and neither is MyFitnessPal or FatSecret so don't try and point me there. Especially after the "Nutritional Label Fiasco" a while back.

     Fitbit has already shown weakness in breaking down nutritional values for display on the Web and on their apps. This just shows their developer and project leader's blindness once again.

     How are people going to get data to present to their doctors when Fitbit is no longer reporting on these very important values? Or even evaluate their own food plan? Can you say "Disservice"?

Heart Rate logs:

     GMT

     Confidence level of 1 - 3 - I'd like a definition.

BOTTOM LINE: The data export was already an aberration - they have just now made it pretty much useless. And I am talking about the data. The output formats for the old and new version are a topic into itself - they are both lacking in thought.

[janifarrell]

I'm also protesting about this retrograde step. I need to share my weekly data quickly and simply on an excel spreadsheet with my personal trainer and now it's impossible. I'm just thinking of upgrading my device this coming weekend - it was going to be a more expensive FitBit but now that's looking unlikely. Be aware, this will hit your sales.

[davidgm1]

Why do you insist on calling the changes an Improvement?

Nothing could be less accurate.

It is difficult to decide which is the most offensive, the decision to inflict this clumsy and cumbersome process on the user or the mendacity with which it is defended

[pcostigan]

I just logged in to see this new data download format. I assume that there is more detail in this new format than in the previous but it appears that I have to download all my data - all the time! How does this make any sense? What is the use case for this? Clearly you should be able to select the time frame and possibly a particular data type. I only want my sleep data and only for the last week.  There is a huge saving in time and bandwidth if the data could be restricted by the user. 

Actually, I just requested a download. It did not take long to finish, but when it did I could not find the data. I re-requested it and of course you can only make 1 download request in 24 hours. 

I read in one of the posts that the data format is of type JSON, which is a text format and similar to HTML. There are many JSON web readers available. Google 'JSON reader' to find some. Typically you paste your JSON text into the reader and it formats it for you. It will be much easier to read and may be a good place to start until someone develops a Fitbit specific reader. 

[SelmaCat]

Please bring back the old data export tool - or at least provide the data in a user-friendly format.

Thank you.