Over the last 6 hours work API call is now timing out

[Stipend]

We have been using the Fitbit web api with great success for a few years. In the last 5 hours the API calls have been failing due to timeout in the call to great activities 

900,000

100.00%

Net::HTTP[https://api.fitbit.com/oauth2/token]: POST

900,000

99.97%

Net::HTTP[https://api.fitbit.com/1/user/23PHN6/activities.json]: GET

This call lasts 900 seconds. Code not changed in awhile.  We did push new code, but no changes in this area.

Anyone seen these issues, or can suggest things to check?

I tried manual calls in the rails console without issue

Confused

Geoff

[Stipend]

I should say this is using Rails and running a delayed_job

Geoff

[Gordon-C]

Hi @Stipend.  Would you you please post the complete error message that you are receiving?

[Stipend]

There is no real error message - just a delayed_job  timeout exception for maximum job duration

2019-01-14 22:06:55.728495 #21548 ERROR update_activityexecution expired (Delayed::Worker.max_run_time is only 900 seconds) user:#<User:0x007f28736e5148>

Also this is not consistent - a few calls will go though , then one blocks.

Same "run"

230

0.03%

Net::HTTP[https://api.fitbit.com/1/user/23PHN6/activities/date/2019-01-14.json]: GET

Then the next call

900,000

99.97%

Net::HTTP[https://api.fitbit.com/1/user/23PHN6/activities.json]: GET

Geoff

[Stipend]

The environment is AWS with a VPC and EC2 with Ubuntu

[Gordon-C]

Also, would you please private message me your client ID.  I'll do some investigating on my side.

[Gordon-C]

I've checked our systems and everything is looking good.  Cloudflare did report a network issue today, but it seems to be isolated to the EU/Paris.   We don't think this would affect you.  Have you checked with AWS to see if they are having any delays that might affect your application?

Is the timeout only occuring for this user?  I am seeing the activities endpoint being executed for other users.

[Gordon-C]

Also, our timeout period is short, so you should never be waiting 900 seconds for a response to come back from Fitbit.  I'm thinking your API call is localized to your application/environment.

[Stipend]

Hey Gordon,

I understand , but I have seen issues,  not  with your service, where a connection can be kept alive for much longer than a standard timeout. But not an expert.

 After leaving the subscriptions off overnight, receiving/processing was taking major parts of our service down, I started testing this morning and it appears to have just gone away.

Could you check one thing, the subscription service was disabled sending to our servers  around 2019-01-03 15:00:00 -0600 , normally I can see why we were not responding and you disabled the send. But not at this time , we seemed to be operating normally.   Do you know the issue on Saturday?

Best

Geoff

[Gordon-C · Accepted solution]

Hi @Stipend

Unfortunately, we only keep our log data for 7 days.  The timing between when your last post occurred and the data the subscription service was disabled is too great. 

Have you had any problems recently?   If not, continue to monitor the situation.  If the problem occurs again, please let us know.

Gordon

[Stipend]

My bad for not responding.   It was our issue not yours.  Thanks for checking in