Error: "The request was aborted: Could not create SSL/TLS secure channel"

[FedericoArg]

Fitbit team, starting yesterday we started receiving a lot of errors when trying to retrieve the Access Token and Secret.

The error we are seeing is "The request was aborted: Could not create SSL/TLS secure channel".

Can anyone in Fitbit confirm or explain why this is happening?

[JeremiahFitbit]

We added CloudFlare in front of api.fitbit.com on Thursday, July 30, 2015 at 00:10 PDT. I am investigating another report of these errors.

Are you using Azure by chance? There seems to be a longstanding issue with Azure and CloudFlare: https://social.msdn.microsoft.com/Forums/azure/en-US/ca6372be-3169-4fb5-870f-bfbea605faf6/azure-weba...

[FedericoArg]

No, we are not using any Azure service. 

Could you keep me updated about this issue? or post an incident in the status.fitbit.com?

Federico

[JeremiahFitbit]

@FedericoArg: Are you using .Net? If not, what are you using and where are you hosting?

Are any of your requests succeeding? What percentage would you estimate?

Do you have any additional debugging information?

Because this is still rather isolated and we have a confirmed issue with .Net/Azure, I'm not creating an incident on status.fitbit.com yet.

[FedericoArg]

We are using .Net 4.5 which supports up to TLS 1.2. Our web sites are using IIS on Windows Server 2012. 

[JeremiahFitbit]

Are you noticing that the errors are at the top of the hour, last about 5 minutes, afterwhich requests succeed?

[FedericoArg]

Answering your additional questions:

Some of the request are succeeding but I don't have enough information to provide you with a percentage.  

Requests seem to be failing either when requesting the temprary token (https://api.fitbit.com/oauth/request_token) or when requesting the access token (https://api.fitbit.com/oauth/access_token).

Yesterday we encountered 71 errors and today we have encountered 28 errors. The last one was 11 minutes ago.

[FedericoArg]

It seems that the error is affecting all others endpoints as well.

The error count is even bigger:
- Today 90 errors
- Yesterday: 256 errors

We currently have almost 20,000 Fitbit accounts connected so probably that error count will get bigger.

[aarondcoleman]

Appears to be .NET based and calls fail on the first call at the top of the hour.

We're .NET on Azure.

--Aaron

[JeremiahFitbit]

---

@FedericoArg wrote:

It seems that the error is affecting all others endpoints as well.

---

I think all endpoints will be affected, since it's a TLS connection issue.

[JeremiahFitbit]

@FedericoArg: Do you have any timing information on the failures? We believe the issue might be related to how CloudFlare rotates session encryption keys hourly and Windows or .Net not correctly expiring them.

[JeremiahFitbit]

Fitbit turned off CloudFlare at 11:50 AM PDT while we investigate this issue with CloudFlare and Microsoft.

[FedericoArg]

Hey Jeremiah, here are the timings of today's errors (Pacific Time):

12:00 PM
12:00 PM
12:00 PM
12:00 PM
11:59 AM
11:25 AM
11:19 AM
11:18 AM
11:08 AM
11:05 AM
11:03 AM
11:03 AM
11:03 AM
11:03 AM
11:01 AM
11:01 AM
11:00 AM
11:00 AM
10:33 AM
10:19 AM
10:18 AM
10:09 AM
10:07 AM
10:07 AM
10:06 AM
10:06 AM
10:05 AM
10:04 AM
10:00 AM
10:00 AM
9:46 AM
9:46 AM
9:32 AM
9:29 AM
9:29 AM
9:13 AM
9:12 AM
9:12 AM
9:11 AM
9:10 AM
9:09 AM
9:06 AM
9:06 AM
9:01 AM
9:01 AM
9:00 AM
9:00 AM
8:49 AM
8:31 AM
8:28 AM
8:08 AM
8:06 AM
8:02 AM
8:01 AM
8:01 AM
8:01 AM
8:00 AM
8:00 AM
8:00 AM
7:52 AM
7:36 AM
7:19 AM
7:18 AM
7:17 AM
7:17 AM
7:15 AM
7:09 AM
7:08 AM
7:08 AM
7:01 AM
7:00 AM
6:57 AM
6:52 AM
6:51 AM
6:34 AM
6:26 AM
6:26 AM
6:21 AM
6:17 AM
6:14 AM
6:11 AM
6:11 AM
6:11 AM
6:03 AM
6:01 AM
5:22 AM
5:09 AM
5:09 AM
5:05 AM
5:01 AM
4:54 AM
4:53 AM
4:51 AM
4:43 AM
4:43 AM
4:27 AM
4:17 AM
4:03 AM
3:37 AM
3:01 AM
2:02 AM
1:02 AM
12:11 AM

[JeremiahFitbit]

@FedericoArg: Thank you for the timing information. Are you still seeing the errors now that CloudFlare has been turned off?

[FedericoArg]

Jeremiah, I am not seeing any further error being logged so far (12:28 PM). The last 5 errors that were logged between 11:59 AM and 12:00 PM said "Unable to connect to the remote server"

I am imagining those were when CloudFare was turned off.

Federico

[FedericoArg]

@JeremiahFitbit Why did Fitbit re-enable CloudFare without any notification. Our Fitbit users are currenlty experiencing issues.  

[aarondcoleman]

For those affected please see (and upvote) this .NET Framework issue that we believe is the culprit:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/eb999cd4-b43b-4779-b01b-4a04017714e3/httpwebr...

In short, the hourly rotation of keys by CloudFlare isn't properly handled by HttpWebRequest.

Our hack solution is to schedule a quick first API call at the top of every hour that we known will fail, such that subsequent (real) Fitbit calls succeed but looking for a team effort here to get to a real solution.

--Aaron

[JeremiahFitbit]

@FedericoArg: The disabling of CloudFlare was temporary while we investigated the issue. Fitbit, CloudFlare, and Microsoft believe this is an issue with .Net. The workaround is to retry the failed request.

[FedericoArg]

@JeremiahFitbit I would recommend being more proactive about major decisions like this. I believe it is in our best and shared interest to maintain our Fitbit population happuy. 

I would recommend working on improving this type of communications going forward. When Fitbit originally enabled CloudFare we did not receive any notification, after reporting these issues, Fitbit decided to disable it temporarily. So, why it was decided to re-enable CloudFare without any proper notification knowing the impact of this? Why Fitbit is not following the same apporach as they are doing with OAuth 2.0 or deprecated API endpoints?

This would be an excellent topic to discuss in your team's next retrospective. 

Thanks!

[JeremiahFitbit]

@FedericoArg: Fitbit puts forth significant effort to not break implementations. Backwards incompatible changes follow a 30-day change policy, frequently with significantly more time.

In this particular case, Fitbit did not make a backwards incompatible change. The change exposed a bug in your application runtime regarding a fundamental security protocol.

Fitbit has been working with CloudFlare and Microsoft to identify and resolve this bug. I can't share all of the communication between the three companies, but please trust that Fitbit (with help from @aarondcoleman) has worked hard to get the issue resolved. Fitbit made the decision to re-enable CloudFlare because the security benefits are more significant than waiting on a resolution to this issue.

Please let Microsoft know that this issue is important to you by voting and commenting here.

The workaround for this bug is straightforward: retrying failed requests will cause .Net to renegotiate a TLS session and the requests will succeed again until the next hour.