07-31-2015 10:27
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 10:27
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Fitbit team, starting yesterday we started receiving a lot of errors when trying to retrieve the Access Token and Secret.
The error we are seeing is "The request was aborted: Could not create SSL/TLS secure channel".
Can anyone in Fitbit confirm or explain why this is happening?
Answered! Go to the Best Answer.

- Labels:
-
OAuth 1.0a
Accepted Solutions
01-12-2016 16:26
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



01-12-2016 16:26
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Today, Microsoft released an update to fix this issue ( https://support.microsoft.com/en-us/kb/3109853) and issued a security advisory ( https://technet.microsoft.com/library/security/3109853.aspx ).
Thank you to everyone for their help in investigating this issue!

07-31-2015 10:44 - edited 07-31-2015 10:53
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 10:44 - edited 07-31-2015 10:53
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
We added CloudFlare in front of api.fitbit.com on Thursday, July 30, 2015 at 00:10 PDT. I am investigating another report of these errors.
Are you using Azure by chance? There seems to be a longstanding issue with Azure and CloudFlare: https://social.msdn.microsoft.com/Forums/azure/en-US/ca6372be-3169-4fb5-870f-bfbea605faf6/azure-weba...

07-31-2015 11:14
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 11:14
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
No, we are not using any Azure service.
Could you keep me updated about this issue? or post an incident in the status.fitbit.com?
Federico

07-31-2015 11:17 - edited 07-31-2015 11:21
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 11:17 - edited 07-31-2015 11:21
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg: Are you using .Net? If not, what are you using and where are you hosting?
Are any of your requests succeeding? What percentage would you estimate?
Do you have any additional debugging information?
Because this is still rather isolated and we have a confirmed issue with .Net/Azure, I'm not creating an incident on status.fitbit.com yet.

07-31-2015 11:22
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 11:22
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
We are using .Net 4.5 which supports up to TLS 1.2. Our web sites are using IIS on Windows Server 2012.

07-31-2015 11:28
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 11:28
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Are you noticing that the errors are at the top of the hour, last about 5 minutes, afterwhich requests succeed?

07-31-2015 11:31 - edited 07-31-2015 11:32
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 11:31 - edited 07-31-2015 11:32
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Answering your additional questions:
Some of the request are succeeding but I don't have enough information to provide you with a percentage.
Requests seem to be failing either when requesting the temprary token (https://api.fitbit.com/oauth/request_token) or when requesting the access token (https://api.fitbit.com/oauth/access_token).
Yesterday we encountered 71 errors and today we have encountered 28 errors. The last one was 11 minutes ago.

07-31-2015 11:50
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 11:50
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
It seems that the error is affecting all others endpoints as well.
The error count is even bigger:
- Today 90 errors
- Yesterday: 256 errors
We currently have almost 20,000 Fitbit accounts connected so probably that error count will get bigger.

07-31-2015 11:57
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 11:57
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Appears to be .NET based and calls fail on the first call at the top of the hour.
We're .NET on Azure.
--Aaron


07-31-2015 11:59
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 11:59
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg wrote:
It seems that the error is affecting all others endpoints as well.
I think all endpoints will be affected, since it's a TLS connection issue.

07-31-2015 12:01
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 12:01
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg: Do you have any timing information on the failures? We believe the issue might be related to how CloudFlare rotates session encryption keys hourly and Windows or .Net not correctly expiring them.

07-31-2015 12:03
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 12:03
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Fitbit turned off CloudFlare at 11:50 AM PDT while we investigate this issue with CloudFlare and Microsoft.

07-31-2015 12:11
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 12:11
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
Hey Jeremiah, here are the timings of today's errors (Pacific Time):
12:00 PM
12:00 PM
12:00 PM
12:00 PM
11:59 AM
11:25 AM
11:19 AM
11:18 AM
11:08 AM
11:05 AM
11:03 AM
11:03 AM
11:03 AM
11:03 AM
11:01 AM
11:01 AM
11:00 AM
11:00 AM
10:33 AM
10:19 AM
10:18 AM
10:09 AM
10:07 AM
10:07 AM
10:06 AM
10:06 AM
10:05 AM
10:04 AM
10:00 AM
10:00 AM
9:46 AM
9:46 AM
9:32 AM
9:29 AM
9:29 AM
9:13 AM
9:12 AM
9:12 AM
9:11 AM
9:10 AM
9:09 AM
9:06 AM
9:06 AM
9:01 AM
9:01 AM
9:00 AM
9:00 AM
8:49 AM
8:31 AM
8:28 AM
8:08 AM
8:06 AM
8:02 AM
8:01 AM
8:01 AM
8:01 AM
8:00 AM
8:00 AM
8:00 AM
7:52 AM
7:36 AM
7:19 AM
7:18 AM
7:17 AM
7:17 AM
7:15 AM
7:09 AM
7:08 AM
7:08 AM
7:01 AM
7:00 AM
6:57 AM
6:52 AM
6:51 AM
6:34 AM
6:26 AM
6:26 AM
6:21 AM
6:17 AM
6:14 AM
6:11 AM
6:11 AM
6:11 AM
6:03 AM
6:01 AM
5:22 AM
5:09 AM
5:09 AM
5:05 AM
5:01 AM
4:54 AM
4:53 AM
4:51 AM
4:43 AM
4:43 AM
4:27 AM
4:17 AM
4:03 AM
3:37 AM
3:01 AM
2:02 AM
1:02 AM
12:11 AM

07-31-2015 12:23
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



07-31-2015 12:23
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg: Thank you for the timing information. Are you still seeing the errors now that CloudFlare has been turned off?

07-31-2015 12:28 - edited 07-31-2015 12:28
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

07-31-2015 12:28 - edited 07-31-2015 12:28
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
- Who Voted for this post?
Jeremiah, I am not seeing any further error being logged so far (12:28 PM). The last 5 errors that were logged between 11:59 AM and 12:00 PM said "Unable to connect to the remote server"
I am imagining those were when CloudFare was turned off.
Federico
10-14-2015 11:19
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

10-14-2015 11:19
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@JeremiahFitbit Why did Fitbit re-enable CloudFare without any notification. Our Fitbit users are currenlty experiencing issues.

10-14-2015 11:30
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

10-14-2015 11:30
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
- Who Voted for this post?
For those affected please see (and upvote) this .NET Framework issue that we believe is the culprit:
In short, the hourly rotation of keys by CloudFlare isn't properly handled by HttpWebRequest.
Our hack solution is to schedule a quick first API call at the top of every hour that we known will fail, such that subsequent (real) Fitbit calls succeed but looking for a team effort here to get to a real solution.
--Aaron

10-14-2015 12:00
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



10-14-2015 12:00
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg: The disabling of CloudFlare was temporary while we investigated the issue. Fitbit, CloudFlare, and Microsoft believe this is an issue with .Net. The workaround is to retry the failed request.

10-14-2015 12:12
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post

10-14-2015 12:12
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@JeremiahFitbit I would recommend being more proactive about major decisions like this. I believe it is in our best and shared interest to maintain our Fitbit population happuy.
I would recommend working on improving this type of communications going forward. When Fitbit originally enabled CloudFare we did not receive any notification, after reporting these issues, Fitbit decided to disable it temporarily. So, why it was decided to re-enable CloudFare without any proper notification knowing the impact of this? Why Fitbit is not following the same apporach as they are doing with OAuth 2.0 or deprecated API endpoints?
This would be an excellent topic to discuss in your team's next retrospective.
Thanks!

10-14-2015 13:40 - edited 10-14-2015 13:43
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post



10-14-2015 13:40 - edited 10-14-2015 13:43
- Mark as New
- Bookmark
- Subscribe
- Permalink
- Report this post
@FedericoArg: Fitbit puts forth significant effort to not break implementations. Backwards incompatible changes follow a 30-day change policy, frequently with significantly more time.
In this particular case, Fitbit did not make a backwards incompatible change. The change exposed a bug in your application runtime regarding a fundamental security protocol.
Fitbit has been working with CloudFlare and Microsoft to identify and resolve this bug. I can't share all of the communication between the three companies, but please trust that Fitbit (with help from @aarondcoleman) has worked hard to get the issue resolved. Fitbit made the decision to re-enable CloudFlare because the security benefits are more significant than waiting on a resolution to this issue.
Please let Microsoft know that this issue is important to you by voting and commenting here.
The workaround for this bug is straightforward: retrying failed requests will cause .Net to renegotiate a TLS session and the requests will succeed again until the next hour.

