Rate limiting

[smashrun]

Our application, Smashrun, provides data analysis for runners. Apps like ours can provide a lot of value to the Fitbit user base by supporting the type of technical analysis specific to runners that Fitibit's site doesn't (and shouldn't). If a user has been running for years they may have thousands of runs on your system. Each of those runs requires an individual request, and so the time to acquire that data from the Web API at 150 runs per hour can be many hours, making our service much less valuable. 

I understand that 150 req/hour seems like a reasonable rate limit. 2-3 requests per minute for an hour. However, no other fitness APIs we've worked with have such a small rate limit. Now that your watches have evolved to have GPS and better support runners perhaps your API can as well. Can you consider either raising the rate limit or even keeping the limit the same but expanding the window? For example 1500 requests/day? This would allow users to aquire and analyze their activity without returning the next day. The effective load, at least from our use case, would remain similar because this traffic is distributed over many user accounts.

[JohnFitbit]

Hi @smashrun,

The vast majority of applications (including Fitbit's own mobile applications) do not exceed the rate limit of 150 per hour. However, I will be happy to go over your use case with you and see if we can implement a more efficient way to return this data to you without exceeding the rate limit.

Can you tell me more about your integration and the type of data you are trying to extract? Also, can you PM me your client ID? Perhaps I can suggest strategies to reduce the number of API calls used.

Looking forward to hearing from you!

[smashrun]

So, we have a generalized import routine which works against multiple API's. The goal is to syncronize a user's complete running activities and provide features that allow them to gain insight into that data.

Most API's we make use of (including Fitbits) work as a summary/detail import. In Fitbit's case we call the activity list endpoint indexed by date then we filter the output for the running type activities. Once those are imported, a separate multi-threaded process begins to import the details. Rate limiting is challenging in this environmnent, and of course each API has unique rate limit types (per second, per client, per user). 

I've just rearchitected the details import to handle Fitbit's particular rate limits, and will be making this live to a small group of beta testers tomorrow. In this process I've done a more detailed analysis of how we're utilizing your rate limits, and it's a bit discouraging.

First, the summary import requires iterating over the activity-list endpoint in batches of 20 (this is the smallest limit of any of the APIs we access). This in itself would not be a major issue, except for the fact that there appears to be no way to filter those activities beforehand. Since my concern is only runs, I need to skip past walks, bike rides, stair climbs, etc. A typical active runner seems to have about 20% of their activities as runs. So if a user has 1,000 runs, that means it would take 5 X 1000 / 20 = 250 requests = 2 hours. Then to import the details we need 1000 / 150 = 7 hours for a total of 9 hours.

The details are understandable, it is a lot of data, and this is especially true for longer runs. However, the rate limit is more onerous for the summary details. These are light weight, and unfortunately contain mostly data we are disregarding.

Honestly, I'm not sure how our process will need to look, since we will need to queue the summary import for at least an hour, the user will need to receive a message that their summaries are being imported, and then a queue would be processed, which will trigger an email back to the user in say 2 - 9 hours that they now can see their Fitbit data.

Aside from expanding the rate limit, expanding the limit option of the run summary list endpoint beyond 20, or allowing activity types to be filtered before they are returned could all help to make the API more useful.

[JohnFitbit · Accepted solution]

Hi @smashrun,

Thanks for providing me with that information.

I may be able to suggest a way to make this process more efficient. I noticed that when you execute the Get Activity Logs List endpoint, that you are only calling in batches of 20, which is expected since our documentation does state a maximum of 20 returned entries:

limit

The max of the number of entries returned (maximum: 20). Required.

However, our documentation is currently in the process of being updated and will reflect the actual max return entries of 100. With that in mind, I suggest increasing your batch entries to 100, which in turn should reduce the number of calls you make per user.

Additionally, I just wanted to clarify that our rate limit of 150 is per user, not per client ID.

I hope this helps. Let me know if you have any additional questions.

[smashrun]

The limit of 100 is a lot more reasonable and helps a lot actually. Perhaps, you can consider also adding a filter by activity type in the future. Thanks for you help!