Webhook notifications disabled without failures

[richhatch]

Hi there,

The last 2 days we've had several instances where our webhook notifications were turned off. Usually this is because we have some kind of web outage that causes the errors to spike over 10%, but in the last couple days, we've not had any failures and were turned off.

Has something changed recently that caused this? See the timestamp difference from line 67 -> 68 in the attached image:

https://www.dropbox.com/s/5in5r8gx5msno2d/data.png?dl=0

https://www.dropbox.com/s/sj6vhbapbj2bx80/stats.png?dl=0

Thanks,

Rich

[JohnFitbit]

Hi @richhatch,

I'm sorry to hear that your webhook notifications have been disabled recently. So that I can investigate further, can you please PM me your Client ID?

Let me know if you have any additional questions in the meantime as well. Thanks!

[jeaglemc]

We just had this happen a 2nd time now, too.

Both times I received an email that our endpoint was disabled: once on Tuesday at 10:03 PM CST, and now once last night (Thursday) also at 10:03 PM CST.  Both times the delivery success rate reported by Fitbit was 100% before disabling our endpoint and after I re-enabled it -- I see no reason for it to have been disabled.

When it was disabled on Tuesday, I was available to re-enable the endpoint right away.  When it was disabled last night, I was not available -- however, the endpoint remained disabled for 3.5 hours then somehow re-enabled itself around 1:30 AM CST.  It has been running at 100% since.

With a 100% delivery success rate, what other factors would contribute to Fitbit disabling our endpoint?

[richhatch]

We had outages at exactly the same times as @jeaglemc until we were turned back on

EDIT: our timestamps were around 10:06pm central for both days and 1:11am when we got turned back on

[ErikPW]

I would like to hop on this bandwagon, we have been having the same issue this week. 

We had incidents on:
- Monday 2019-07-08 starting at 18:30 MST
- Wednesday 2019-07-10 starting at 21:12 MST
- Thursday 2019-0711 at 21:11 MST

No increased error rate or logged slow responses. Any additional information to help trouble shoot this would be awesome. 

Thanks!

[JohnFitbit]

Hi @richhatch @ErikPW @jeaglemc,

Thanks for reporting these incidents. Our team is currently investigating a few reports of subscribers being disabled without notice.

Once I have more information, I will promptly let you all know. 

Thanks for your patience everyone! Please let me know if you have any other questions in the meantime.

[JohnFitbit]

Hi Everyone,

Thanks for your patience as I investigated this issue.

We’ve had some problems with our network last week which has resulted in our user’s subscribers being disabled. On Friday, we implemented a fix which should not penalize our customers if future network issues happen.

I apologize for any inconvenience this has caused. Please let me know if you continue to see this behavior and I'll follow up with the team.

[richhatch]

Hi @JohnFitbit it looks like we were turned off again at around 2019-08-09 14:48:54 (38 minutes ago). The odd thing is when this happens, you'll see in the 2nd screenshot that we're still enabled. In order for us to turn it back on, usually we need to disable it, and then re-enable it. This time that is not fixing the issue.

[DSGTeam]

We are seeing the same thing right now. We stopped receiving notifications about an hour ago. 

We tried disabling the subscribers and re-enabling them and that had no effect. I am assuming the services are down but the status page says its operational. 

[jeaglemc]

We are seeing the same thing.

Normally I receive 5 to 20 notifications every second or two -- for the last 20 minutes, haven't received anything. 

Looking at Fitbit's delivery log, we've been 100% successful receiving notifications for the entire log.  I've tried disabling/re-enabling and still nothing is coming in and they're showing no attempt to deliver notifications ... it's like Fitbit's notification system is down.

[Gordon-C]

Thank you for notifying us of the outage.  We are aware of this problem and have posted an update on https://www.fitbitstatus.com/

[Gordon-C]

In case you're not signed up for fitbitstatus.com, the webhook issue has been resolved and you should start receiving notifications again.