Subscription service messages count spiked and messages were larger than normal

[DSGTeam]

At approximately 6:00 AM EST this morning we started experiencing large spikes in the messages and size of those messages and this lasted about 45 minutes. This caused our queues to back up to almost 6 million individual user alerts (after parsing the batched messages).

Was there an issue?

[JohnFitbit]

Hi @DSGTeam,

Thank you for reporting the spike in webhook messages to your subscribers this week.  The recent large batch of webhook notifications you received was a result of a problem with our messaging queue causing a backup of messages.  Once the service was back online, we sent the stored messages to you. 

In an effort to minimize network traffic, we do batch messages together.  At the moment, our system is configured to batch messages that pile up in 0.5 seconds up to 2000 records.  Reaching this batch size limit happens occasionally when we’ve had a system lag and we’re trying to catch up. 

When processing webhook notifications, we require your application to respond with an HTTP Status 204 within 3 seconds of receiving the webhook notification.  Therefore, your system can receive and store the messages, reply with a 204, then process them on your own time. 

We understand not all systems are designed this way.  We are investigating options to provide our customers to restrict the messages batch size, including adjusting our maximum batch size limits.   If we adjusted the batch size on our side, what is the maximum range of messages your system can process at once effectively?  We’re collecting requirements from various customers and will try to accommodate as many customers as possible.

Hope to hear from you soon!