Seeing high volume of 500 errors on Refresh Token endpoint

[aarondcoleman]

Starting at about 5:50pm Pacific time we've started to see a high volume of 500 errors to the refresh token endpoint and still ongoing now. Subsequent retries usually succeed, so far though we've seen about 150 in the last 20 minutes.

Here's one error id towards the beginning: 

error id: C71B80E2:2487_A92D900A:01BB_57BF926C_C663551:0C6E

Just reporting in case this is a middle tier issue not showing up in your logs.

Thanks,

--Aaron

[DavidSFitbit]

The errors you're seeing are coming from you're application making concurrent requests to the refresh token endpoint with the same user and refresh token. Some of these requests had the same timestamp on our end.

This should be returning a 409 instead of a 500. We're working to improve this. Thanks for reporting this @aarondcoleman!

[aarondcoleman]

Hi @DavidSFitbit, ok, thanks for that helpful debug.

We see a few edge cases within edge cases in our logic where that might happen and are deploying some more checks to mitigate making mirrored requests from our multiple worker services.

Is this the same scenario as the previous thread here?

https://community.fitbit.com/t5/Web-API/quot-Concurrent-refresh-token-requests-were-made-by-the-same... I'm seeing different error messages. What's the message we should be on the lookout for once the 500 is fixed?

Thanks,

--Aaron