Elevated 500 errors/disabling feed

[kgajula]

Hi all,

So we've been getting a bunch of HTTP 500 errors all day today. It started happening intermittently yesterday and now all incoming notifications are failing as 500. Previously it was working without any issues for a while. With all of these HTTP 500 errors, are you disabling the feed as a result too? 

Particularly it is failing when the notification is being identified as a MultiPart form. 

The error from our logs is reproduced :

java.io.IOException: Corrupt form data: premature ending
	at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:205)
	at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:119)
	at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:100)

We're just wondering if anything has changed from your side? We expect all notifications to be coming in normally.  

[JeremiahFitbit]

I will start an investigation to verify the XML files sent by the Fitbit Subscriptions API to your subscriber endpoint are valid. Fitbit has not yet received any other reports and does not observe an elevated error rate across the system.

If you can capture a failed file, please email it to api at fitbit dot com for review.

Quickly doing a Google search for this error, I found this: http://stackoverflow.com/questions/240163/corrupt-form-data-premature-ending-resolved . Perhaps you need to configure the library you're using to accept uploads larger than 1 MB. You can see the size of the request being sent to your subscriber endpoint by clicking on 'details' link beside your subscriber in your app settings on https://dev.fitbit.com/apps .

I also found http://stackoverflow.com/questions/10292993/java-io-ioexception-corrupt-form-data-premature-ending that determined it was a bug in the oreilly library and suggested using the Apache Commons FileUpload or the new Servlet 3.0 built in request.getPart() method.

[JeremiahFitbit]

Also, I would strongly recommend using the "JSON body" option for subscription notifications instead of the "XML file".

[kgajula]

Thank you for the suggestions. It was initially our hunch too and we're looking deeply into it. 

[bkevern]

We are seeing the exact same thing on our site....  What we are seeing is that the "updates" is missing in the data being received from FitBit.  The notification is making it to our handler, but when the handler looks for the "updates" data, it is not being found in the request.  This began occurring this morning it looks like from our logs....

[bkevern]

Note that we do receive the data as JSON data.

[bkevern]

Sorry, what I meant to say is "we are set up to receive the data as JSON, not XML"..

[JeremiahFitbit]

I just confirmed "JSON body" notifications are working.

@bkevern: I assume you are using "JSON file"?

[GajulaKranthi]

Jeremiah, to add, we have setup our subscription API to recieve json file. We have started to see issues the day before yesterday evening around 5.30 PM CST and not before that. We also were able to replicate the issue in stage.

We did change the library from oreilly to Apache and things started to work, in stage.We plan to move to production soon.

Now what we are not able to understand is that why all of a sudden we saw those issues! The oreilly library thows this error when the json file does not have any content, that is an empty file.

Is there any chance of fitbit sending a json file with empty content?

[bkevern]

So, we've been continuing to dig into this.  In our application, we use Microsoft's MVC.NET as the technology for developing the web service that FitBit calls to notify us of new data being available.  As of Tuesday this week, the MVC.NET library is no longer able to parse the multipart form data that is being posted by the FitBit service.  This service has literally been running for months without issue and began to fail Tuesday morning.  So to figure this out, we've dumped the data being posted by FitBit and we compared it to the specifications for Multi-part form data to see if we could find what might be causing MVC.NET to be unable to parse the data from FitBit.  What we think we are seeing is missing CRLF characters in the FitBit data.  Specifically, the spec indicates that there should be a CRLF after the boundary key which breaks apart the multipart form data.  We believe the FitBit data is not providing the CRLF according to spec.  As such, and because there are other characters then on the line with the boundary key (like content-disposition... etc) , the Microsoft MVC.NET system does not see a match for the boundary key specified in the header and the boundary key in the multipart form data.  We believe this is the root of the problem.  We have a copy of the posted data and I'd be happy to send it to you outside of the forum if you'd like to see it. ( It has user identifiable data in it and I'm not comfortable posting that on an open forum).  We'd be really interested in hearing what you think and whether or not you believe we are on to something that will help us all resolve the problem....

[JeremiahFitbit]

We haven't knowingly made a change to the Subscriptions API. We are continuing to investigate.

For now, I recommend using the "JSON body" method, which is not affected by this issue.

[bkevern]

Jeremiah,

Thanks for looking in to this.  I'm pretty sure it is a problem with the multi-part form data header missing it's CRLF as we were able to get things to go by hacking in the CRLF.

I have also re-written our Notification parser to handle getting the data from the body now instead of the multipart form data and data is flowing correctly to us using this method.

Brent

[AMushal]

Recently I started getting this error on incoming notifications:

Index was out of range. Must be non-negative and less than the size of the collection.

It is fixed now by changing the parsing method of the Post we get from the Fitbit server, but was there any change to the subscription based notification format.  After all it was running for over 3 years using the earlier process which suddenly stopped Monday morning 10/3/2016.

Best,

Ala

[JeremiahFitbit]

@AMushal: I merged your post into this topic. We have determined there was a change, but are still investigating a fix. The change is that a CLRF is no longer appended to the end of the request, which some client libraries do not accomodate.

Three interim solutions:

1. Switch to a multipart parsing library that interprets EOF the same as a CLRF.
2. Append a CLRF to the response before parsing.
3. Switch to "JSON body" subscription notification types. The notification type can be changed in your app settings at https://dev.fitbit.com/apps . You will also need to update your app to accept a JSON request body.