Rate Limiter seems to be Double Counting

[jputney]

I have implemented strict logging, call counts, and ensured no redirect calls are being made. I have been retroactively pulling data from authorized users of my application; going back to their enrollment in a research study, 100 days at a time, for various endpoints. During my testing, I keep getting the rate limit exceeded 429 error. After my token refresh call and 1 call per day for 100 days, I expect to be well under the 150 calls per user per hour limit. My count on calls when the error throws is either 74 or 75. Which led me to believe that the fitbit rate limiting logic is double counting somehow. I have log files that have unique header ids, counts, and reset urls or each call to rule out duplicate/redirect calls. The rate limit count reports a drop of 1 or 2 each call, sometimes 3 or more. After about 75 calls it hits the 0 or -1 count and throws the error. I've even tried slowing down the call rates to no avail. I need someone on the fitbit side to look into why this is happening.

[GordonFitbit]

Hi @jputney 

Have you checked the values that we return in the response headers to see if the numbers match what you counted?   There are 3 response headers you can use to checking the rate limit.

Keep in mind, the rate limit is 150 API requests per hour and per user.  We reset the rate limit, approximately, at the top of each hour.

Gordon

[jputney]

I use the below two headers to compare to my count:

              'Fitbit-Rate-Limit-Remaining'

              'Fitbit-Rate-Limit-Reset'

As I stated before, my count is significantly less than what the rate limiter is reporting. On average, it seems to be double counting although the limit remaining sometimes reports a decrement of 1 per call, while other times its upwards of 2 or 3 count reduction at a time, but tends to average out to half of the actual 150 limit before throwing the 429 error. Recently it seems to have gotten worse, and is even less than usual before the error is thrown. I just ran a batch today that errored out after 47 calls, 48 counting the auth token call. If I provide my application ID, is there logs on the back end that could reveal if unintended calls are being made or if there indeed is a counting issue with the rate limiter?

[GordonFitbit]

Yes, please provide me with your client ID.   I'll see if I can find anything in our logs.

[GordonFitbit]

Hi @jputney 

Also, would you tell me which endpoints your application is calling?

[agent_never]

My client_id is: 22CTJN
I am seeing the same thing in our app. 

I am seeing the rate limit decrease by an average of 2 for these endpoints:
   /1/user/:UID/activities/minutesSedentary/date/DATE/DATE/15min/time/23:07/23:59.json
   /1/user/:UID/activities/minutesSedentary/date/DATE/1d/15min.json
And the same thing with:
minutesLightlyActive
minutesFairlyActive
minutesVeryActive
calories
distance
elevation
floors
steps

However, these endpoints showed response header "fitbit-rate-limit-remaining" to be decreasing by about 1:
/1/user/:UID/profile.json
/1/user/:UID/devices.json
/1/user/:UID/activities/heart/date/DATE/1d.json
/1/user/:UID/hrv/date/DATE1/DATE2/all.json

Can you look into why this may be occurring?

[GordonFitbit]

Thank you for sharing the endpoints.  When I tested it, I wasn't seeing a decrease by 2 API calls.  I wasn't testing the endpoints you are using.  Let me see if I can reproduce it.

[GordonFitbit]

Hi @agent_never 

I tested the endpoints you provided and I am able to reproduce the problem.  The rate limit counter adjusts by 2 when using the intraday resources minutesSedentary, minutesLightlyActive, minutesFairlyActive, and minutesVeryActive.   Calories, distance, elevation, floors and steps adjust the rate limit counter by 1.

minutesSedentary, minutesLightlyActive, minutesFairlyActive, and minutesVeryActive resources are not supported with the Intraday Activity endpoints.  See Get Activity Intraday by Date.  There is a bug that allows developers to call these resources.  I've reported the problem and your application will not be able to call them in the future.  My recommendation is to stop using them.  When we disable the resources for the Intraday endpoints, your application will receive errors.

Gordon

[jputney]

My client ID is 23QD3H

The endpoints are activity intraday, activity zone mintue intraday, daily activity summary , and heart rate intraday.

[jputney]

"/1/user/".urlencode($user->auth_fitbit_id)."/activities/heart/date/".urlencode($date)."/".urlencode($date)."/1min.json"

"/1/user/".urlencode($user->auth_fitbit_id)."/activities/date/".urlencode($date).".json"

"/1/user/".urlencode($user->auth_fitbit_id)."/activities/active-zone-minutes/date/".urlencode($date)."/".urlencode($date)."/15min.json"

This endpoint I cycle through 

'calories', 'distance', 'floors' and , 'steps' which count as 4 api calls

"/1/user/".urlencode($user->auth_fitbit_id)."/activities/".urlencode($resource)."/date/".urlencode($date)."/".urlencode($date)."/15min.json";

Currently hitting these endpoints for a 24hr time frame with the intention of going to a 6hr pull schedule for the intraday endpoints. 

[GordonFitbit]

Thank you!   Let me see if I can reproduce the problem with these endpoints.

[jputney]

Bumping for update

[agent_never]

From my logs, it looks like they fixed the problem on October 16th.