High increase in 504 and 502 errors across all API endpoint

[aarondcoleman]

Seeing a sharp increase today and yesterday as well of 502 & 504 errors which on retry succeed. Anyone else seeing this?

[Gordon-C]

Hi Aaron,

From the title of the message, it sounds like you are getting the 502 and 504 messages for all Web API endpoints.  Would you let me know when this behavior started, and how long it has been going on?

Thanks!

Gordon

[Gordon-C]

@aarondcoleman After further research, I found out we are aware of this problem.  Once the problem was observed, we backed out the changes generating the errors, and you should see a decrease in 502 and 504 errors.  We are currently investigating the cause.  

if you continue to have problems, please let me know.

Gordon

[aarondcoleman]

thanks @Gordon-C -- still seeing them at the moment but glad there's a root cause you can see on your end.

Cheers,

--Aaron

[Gordon-C]

@aarondcoleman Are you seeing the same volume of errors or has the number decreased?

[aarondcoleman]

Hey @Gordon-C, we're back down to "normal" levels, a couple hundred or so per 12 hours. Would be awesome for our logging to get this down even further, though not a showstopper.