Resolved: Changes to Data Export tool

[KateFitbit]

Fitbit Update 11/30/2018: This issue has been resolved and I'll be closing it from further responses. 

---

Fitbit Update 11/28/2018: The old data export tool is now back and available for use. You can now choose which way you would like to export your data going forward. If you would like to access the old tool, please go here and select "if you would like to use the old data export tool click here" under Export My Fitbit Data. 

If you have any issues, please let me know. Thanks for your patience and reports! 

---

Fitbit Update 11/27/2018: Happy Tuesday! Just a heads up that we hope to have an update soon on the Data Export tool. I'll continue to update the thread this week. 

---

Fitbit Update 11/20/2018: While I don't have an update today, I can assure you that your feedback is being heard. One note, if you contact our support team, they will not be able to supply additional details, but I will continue to update this thread as I know more. We do appreciate the feedback.

---

Fitbit Update 11/16/2018:  Hi Everyone, here's an update on the recent changes to the export tool:

With the recent improvements to our tool, you can now export your Fitbit data, starting from your first step with Fitbit.

You can also access new data for download, such as heart-rate data, exercise logs, and your activity in the Fitbit community. For more information, see How do I export my data?

We understand that some customers are having difficulty interpreting their data in the new download format. We’ll continue to add features and make improvements based on your feedback.

Thank you all for your input in the thread. Our team will continue to monitor this thread and I'll let you know when I have further updates. 

[NotThatBob]

They need to know using Excel is not really a valid response. .. so Fitbits official line is that we need to pay an extra £120 to get excel so we can open files they create that used to be given in a universal and easy to view format of csv. 

I brought this product so I could easily track my changes over the years. Will they be giving us all free copies of Excel in compensation? 

[PeterWombat]

Back in Aug/Sep of 2017 they had a long period of a few weeks where downloads were broken. Many of you may remember that. I never found an explanation for why it happened. But at least when it came back it offered the same level of support as before. July 2017 every weekly update email we received said we'd done "10,281 steps more (or less)" than the prior week, regardless of what the actual total was. It took weeks for this to get addressed too. (Look for postings by me from Aug, Sep 2017 if anyone wants to see these discussions.)

Your points are all perfectly valid. WHY they would make such a major change and try to tell us (and force poor @KateFitbit to keep repeating useless tripe) they've "improved" things is utterly beyond me. If they offered a tool suite for download that we could get to turn the pile of json into something useful then it might be sort of an improvement. But apparently we're supposed to write that. It's NOT simply that it's json, it's also that there are no explanations for each file's date/timezones, units, etc. to even allow me to decipher them. I know how to read json but interpreting the data is another thing. Here is an extract from my steps-2018-08-28.json file:

},{

"dateTime" : "08/27/18 22:53:00",

"value" : "0"

},{

"dateTime" : "08/28/18 10:29:00",

"value" : "31"

},{

"dateTime" : "08/28/18 10:30:00",

"value" : "37"

},{

What are they telling me? The time gap makes absolutely no sense. Is it telling me that in the apparent minutes I did "0", "31" and "37" steps? Ok. But, those times make no sense.

I always thought the "maximum 1 month" of data download was annoying but wasn't fatal. I used to download every two to three weeks and I keep all of my data in my own PostgreSQL database. All I want is what we already had. If they want to add this new method IN ADDITION, fine. But why remove an easy to use and understand feature and add crap in its place?

[Reydoog]

The granularity of the data they are now providing is amazing. What you are
looking at is the minute by minute step data. There are equivalent sets of
files that give minutes by minute calories burnt and minute by minute heart
rate data. I think the issue is that we are swimming in too much data


Having said that I searched " json to csv" on Google and found all my
historic VO2 data in two files which I am very happy about.


The best solution seems to be to keep providing this level of data to
comply with GPCR and for those who find them useful, but also to bring back
the summary csv download facility for those who just want a selective
overview of their data.


Ian


Ian Goodyer
+44 787 6542722






Sent from my mobile so please excuse my brevity and any spelling mistakes

[BarbKM]

JSON file converter at
http://www.convertcsv.com/json-to-csv.htm
worked for a monthly step file. It converts a month worth of steps into
two columns 1) date time and 2) steps at that time. Open this in Excel.
As someone mentioned earlier, it's a lot of data, steps for every minute of
every day. You can sum each day individually or be adventurous and try a
pivot table (details below) Hope this helps until Fitbit gets our data
back into an easier format.

Here's a quick and dirty step-by-step for the steps pivot table
1. Open the csv file in Excel, you'll have 2 columns (A and B).
2. Select columns A&B
3. Click "Insert" from the menu at the top
4. Select Pivot Table and a new window pops up
5. Data should default to columns A&B
6. Choose where to put the data - I always pick new worksheet
7. Click ok and a new tab opens
8. (Be patient here). On the right is a column Labeled "PivotTable
Fields". You should see 3 fields - "DateTime", "value" and "Days". Below
that are 4 small blocks labelled "Filters", "Columns", "rows", and (Greek
letter E - sum of) Values.
9. Drag "Days" and "Date Time" fields to the "Rows" area.  Make sure Days is listed first.
10. Drag "Value" to the "E values"
11. In the table you should now see the total steps for each day. Mine
defaulted to each day's total. Note that each day as a PLUS sign that you
can click to expand and then get hourly totals! (This is actually a big
plus for me and is data that wasn't available in the downloaded file).

[snthompson]

Have you hears from anybody who thins the change is a step forward or thinks the change is a good idea?  If so, I would like to hear how they are handling their data.

[PeterWombat]

I hope Fitbit sends you a paycheck for doing their work for them. As to

"Hope this helps until Fitbit gets our data back into an easier format."

They will NEVER do that. They DO NOT CARE ABOUT US. If they did @KateFitbit would have a real answer for us. Not tripe.

All of this minute-by-minute data was already available via their REST APIs. You could code up GET calls and retrieve it as json files and have what they have now forced all of us to live with. It was even better because you could request specific date ranges and choose what data you wanted. Not simply a blind dump of everything every time. I just didn't and don't want that level of data. I didn't want to bother with converting it. The REST APIs are still there according to a quick look at the developer side of their website, so if you find having that level of data is useful go look at those, you can just pull down in increments (i.e., once have baseline, just get most recent days).

But I'm guessing they'll not bother to ever return us the csv/xls export. I just remain livid NOT that they simply took the data always available through REST and dumped it on us, but that they REMOVED functionality and then told us they'd improved our experience. I cannot use the language I want to use here. Apparently their widdle engineers can't handle both types of export.

[mgreess]

I cannot tell you how annoyed I am with this change.  This completely undermines my ability to track my data without making an unacceptable investment of time and energy.

I WOULD ASK FITBIT TO RESTORE THE OLD MECHANISM FOR DOWNLOADING MY DATA.

[BarbKM]

The problem is most of us don't know anything at all about APIs and GET and
really don't want to know!
I'm fiddling around with stuff I know nothing about. I tried using the
Excel query tool and since I *never* do queries I'm missing some kind of
fundamental step and I couldn't get them to work...so now I use the
conversion website, import into Excel and make the Pivot Table. I use
very simple Pivot Tables at work and thought they would be useful instead
of summing by day the files with 36,000 rows! All of this to get 1 months
worth of data that took less than a minute before this insane change.

[syphus]

@KateFitbit

This will most likely be my last post in this thread - I had to unsubscribe due to the volume of replys and posts.

But I do have one last statement.  Per some of the posts, people have said they were told (thru different channels) that this change can not be set back to how it was . . .   That is complete crap!  It is software and a good development team would be able to do this (they have backed up what they had).  The truth is Fitbit does not want to do the change.

Whatever "data format" they changed to, they could have "still" provided an interface for the users to get their data just as they had before.  That leaves me only to conclude that Fitbit does not wants the users to have their data (for what ever reason).

I would encourage those with social accounts (twitter etc) or are members of other health/activity sites to spread the word that as long as Fitbit stands firm on this crappy-a$$ change that no one buy their products  This the only way Fitbit is going pay attention to the customer and "What they want"  and not a bunch of "money-hungry" overly ambitious management fools who try to bend us to their needs and not the customers who pay their salary.  They need to feel it where it hurts (i.e. the bottom line).

I was looking into a "Charge 3" - I am no longer. 

Thanks for listening  . . .

[IamBob]

I urge you all to get on twitter. That will make it visible to the rest of the world and maybe push Fitbit to take some action. Obviously, they intend on ignoring this message board until we go away. If they intended on correcting this defect, it would have been already done. They are blowing smoke up our you know where...

Fitbit will respond to tweets.
Use these hashtags and add these twitter accounts:
#FitbitExportBroke #FitbitIgnoringUserBase #fitbit @FitbitSupport @parkjames
They claim they are looking at the forums! lol
If you tweet to them they can not delete your tweets.

btw It is very easy to restore the old screen and add a link for those who wish to take the extra 15-20 minutes to export and then jump through the json conversion scrip hoops... But what do I know after 40 years in IT....."

[LanuzaFitbit]

Hello everyone.

Thank you very much for sharing your thoughts and feedback! 

As it has been mentioned before we are reviewing this thread and will update everyone as soon as we have anything to share.

Thanks for your patience and understanding.

[GershonSurge]

Truthfully, I don't think there is a single post expressing patience or understanding.

[MrMarv]

@LanuzaFitbit -

With an average of approximately 30 posts a day, a total of over 200 posts, I'm not going to review again each and every post - - - but I suspect, as @GershonSurge said, there are very few, if any, posts expressing patience.

How about this -

If your development team can:

• start to act like a PROFESSIONAL development TEAM
• come up with some develpoment standards and adhere to them
• can start to look at their work as a reflection of themselves as professionals
• realize that incomplete data is just as bad, if not worse than no data
• understand that the Fitbit customers are the true source of their salaries and their actions are causing many of them to rethink their status and where to spend their money

And if your development management team can:

• stop resting on their laurels - 2001 is a lifetime ago in this business
• realize that "shooting from the hip" is great for prototyping and proof of concept displays, but not for production
• understand that letting the current development methodology continue is doing your team and your customers a major disservice
• take off their blinders

And if they deliver a quality software product...

I will gladly retroactively apply patience and a modicum of understanding to this whole fiasco.

Until then, there is no way I can condone your company's development team's lack of standards, from Oakes on down. And let's face it, Park and Friedman are just as much responsible.

As I learned a long, long time ago as a United States Marine:

You can delegate authority, but you cannot delegate responsibility.

[SunsetRunner]

Well the Thought Police have been in action and my post has been deleted again. I'm trying to find out exactly what I've said that is pressing their buttons becuse the explanation given doesn't fit with what was written. Anybody else had their postings purged? Pete

[MrMarv]

@SunsetRunner -

---

@SunsetRunner wrote:
Well the Thought Police have been in action and my post has been deleted again. I'm trying to find out exactly what I've said that is pressing their buttons becuse the explanation given doesn't fit with what was written. Anybody else had their postings purged? Pete

---

Not me, and I will have to admit that I have been, what some would think, extremely critical in my posts as they pertain to this subject.

[BarbKM]

Does anyone know of a thread for exchanging ideas and expertise  on working with these insane json files?  My daily total step data from json files do not correspond to values posted on the dashboard.  Maybe it's a dateTime issue, but how the heck would I know?   

Please do not send me to the developers website, I'm neither a developer, programmer nor IT person.  I shouldn't have to be one to get the data in a usable form.  I've looked at the posts in the developer section and it's like a foreign language to me.  All I can do is use a website to convert from json to a 30,000-some row csv file.  

[MrMarv]

@BarbKM - 

From my post 96 (if you sort your threads by first-to-last posted:

"

Time zones (I'm in PST):

     GMT - Distance,  Exercise, Heart Rate, Steps

     PST -  Altitude, Calories

     No standard - and a company's reputation often lives or dies on its standards."

In that post I detail out my initial analysis of what is missing and what is not standardized. I've found a few issues since, but most of what I've found is in that post.

Good luck.

[BarbKM]

@MrMarv - Thanks much.  What a pain the time zone change is.  No clue how to reset it in the Excel file.

sigh

[MrMarv]

@BarbKM -

I am in the PST which is 8 hours behind GMT.

So if the GMT time is in cell A1 and I want the local time in A2 then in A2 I would place:'

=+A1-TIME(8,0,0)  

Which means subtract 8 hours and 0 minutes and 0 seconds from the GMT time.

So 19:13:00 becomes 11:13:00 

The above doesn't take care of what happens when the GMT time is before 8:00 am and the day shifts backwards - let Google be your friend on that one. I'm personally waiting for Fitbit to get their act together. I have never been impressed with their data extracts as it is and the current one is worthless to me with all of the missing food data.

Good luck.

[PeterWombat]

@MrMarv @BarbKM

No easy way to automagically adjust Excel for DST changes. Your time zone conversion formula is pretty well what there is. Unless you want to write some macros... And I agree, the use of REST APIs is only useful for certain people. And even knowing how I could use them I still had zero interest.

Just another issue that Fitbit has decided they can dump on their users. The lack of consistency in the data that MrMarv has pointed out and the lack of any sort of data dictionaries or descriptions just highlights their lack of professionalism and care.

One last time because I'm a broken record. They chose to throw us all into the REST/json world for reasons they've never explained and then just leave their reps (like Kate and Lanusz) twisting in the breeze. In a way, that would be fine to me, IF THEY HAD KEPT THE csv/xls OPTION (lame though it was, it was immediately useful to many more people).

I've tried Twitter too. Just got the same pablum as they put here.