File System Issue

[Thomy13]

Hi,

I'm developing an application that reads, stores and transfers sensor data via the messaging protocol from the fitbit watch to the companion app. Sensor data is stored locally as files on the watch until a bluetooth connection is established. Later on this data is transferred to a server via https.

When reading and transferring 500+ files the system starts to throw me 'out of memory' errors:

[3:10:11 PM]mem: 31416/65528
[3:10:10 PM]Error 12 Critical glue error
[3:10:11 PM]mem: 31416/65528
[3:10:10 PM]Error 12 Critical glue errorapp/sensor-io.js:252,5
[3:10:10 PM]file processing err: RangeError: Out of memory: Couldn't create buffer
[3:10:17 PM]App msg queue full, see https://dev.fitbit.com/kb/message-queue-full for recommendations.
[3:10:17 PM]Error 12 Critical glue errorapp/sensor-io.js:13,5
[3:10:17 PM]data store[8]: Out of memory: Failed to allocate buffer for cbor allocationapp/sensor-io.js:86,3
[3:10:21 PM]mem: 30128/65528

I am reading files with readFileSync(file, 'cbor'). I think there may be open files in background that aren't correctly closed? I do delete these files once successfully read and transferred. I've also only found a way to forcibly close binary files so I assume reading cbor encoded files automatically does that for me?

What I also thought of would be faulty files that are somehow really large thus causing this issue? Any ideas what could be happening here. I know it's difficult without the full code. I can provide a git repo if necessary.

Thanks in in advance!

Thomy

[Thomy13]

I'm also getting these when trying to save a file:

Out of memory: Failed to allocate buffer for cbor allocation

The JavaScript memory seems fine though: 31408/65528

[JonFitbit]

How large are the files? There is a limit of 15mb total storage on the device, including your app code. Do you think you're hitting this limit?

[Thomy13]

I am not hitting that limit, because each file is roughly 1KB and I don't allow storing more than 4,000. I doubt that's the reason because the app works just fine when being restarted. I do think I've found the reason though.

In my app I use the messaging protocol to transfer stored files. I've found out that the send function does not guarantee the arrival of data.

messaging.peerSocket.send(...)

I could sometimes wait a minute for a message on my Companion app that would never arrive. I relied on the messaging protocol flag. That if it was true the transfer would always work:

messaging.peerSocket.readyState === messaging.peerSocket.OPEN

That doesn't seem to be the case though, because some messages just never arrive on the Companion app. No errors thrown. My theory is that out of memory crashes came from more and more messages being queued up that could not be sent. I am using quite an aggressive approach to transfer files once a bluetooth connection is established.

What I've done now is confirm every message sent through the messaging protocol and only queue up a maximum of 8 messages. That seems to have solved memory issues. I have also read about the onbufferedamountdecrease event (https://dev.fitbit.com/build/reference/device-api/messaging/) but I think confirming messages separately and re-sending if necessary seems the safer approach.

[JonFitbit]

Nice. We also have the File Transfer API which manages the process for you.

[Thomy13]

Yeah, thanks! I am using the File Transfer API now, that works much better and the memory issues are gone completely. However I do need to still manually set a maximum amount of queued files to ensure I don't run out of memory.

Maybe a warning or maximum amount of queued data could be added to Messaging and File Transfer Protocol? It isn't very intuitive to find out that over-using the send / queue function can result in a out of memory error. It could return a boolean telling if data was able to be queued or not.

[JonFitbit]

Doesn't that already happen with the file transfer?

  // Queue the file for transfer
  outbox.enqueue(destFilename, data).then(function (ft) {
    // Queued successfully
    console.log("Transfer of '" + destFilename + "' successfully queued.");
  }).catch(function (error) {
    // Failed to queue
    throw new Error("Failed to queue '" + destFilename + "'. Error: " + error);
  });

[Thomy13]

Theoretically yes, but if I put like 10+ files into this I get memory problems and the app crashes. My files are about 8KB each and I allow a maximum of 4 to ensure it runs smoothly. Also, I feel like the file-transfer / messaging stuff just isn't working right. I left my watch on during the night (it collects and stores data in files) and transfers it whenever the companion app can be reached.

In the morning when I connect my phone to the watch (got confirmation from companion) the file-transfer just won't happen. I queue 4 files and they never arrive, I give each file a maximum of 2 minutes before I time it out. The files are enqueued successfully. I tried restarting the companion app and watch app but no change. I remember having the exact same problem yesterday in the morning but I cannot remember what or how it fixed itself.

I also can't say if this is only my Versa but overall messaging between the watch and companion is unreliable. I can't tell when it will work and when not. The messaging and file-transfer functions are unsafe to use. They accept anything and the application will just crash at some point unless you take care of memory usage yourself.

Is there additional debugging tools for messaging / file-transfer? 

[Gondwana]

Have you investigated whether the file transfer queue gets emptied? I've known it to clog up, and nothing less than clearing the Fitbit app's data (Android) would clear it up. (This was a few months ago; subsequent patches may have relieved this constipation.)

You can also monitor state changes as each file progresses through the transfer process.

If you'd like rough code for these things, let me know.

[Thomy13]

Thanks! I clear the file transfer queue on every start of my app and only allow 4 concurrent pending transfers at a time:

// cancel all pending files when we start the app
cancelAllFileTransfers();
function cancelAllFileTransfers(__removals) {
    let removals = __removals || "";

    outbox.enumerate().then(function (files) {
        console.log("files: "+files.length+", "+removals);
        //console.log("removals: "+removals+", "+removals.length);

        // iterate over all queued files and see if we need to delete them
        for (let k = 0; k < files.length; k++) {
            // Idk why the **ahem** removals becomes a string here
            if (!removals || removals.indexOf(files[k].name) !== -1) {
                removePendingFileTransfer(files[k].name);
                files[k].cancel();
                console.log("cancelling "+files[k].name);
            }
        }
    })
}

Rebooting the watch is the only way to fix this. The transfer started immediately and continued flawlessly once the watch came back up.

I am also monitoring file-transfer:

                        outbox.enqueueFile(filename).then((ft) => {
                            util.fileTransferSuccess();
                            console.log(`${ft.name} queued`);

                            ft.onchange = function(e) {
                                console.log(`${this.name}: readyState=${this.readyState}`);

                                if (this.readyState === "transferred") {
                                    util.fileTransferSuccess();
                                    util.addFileCount(-1);

                                    removePendingFileTransfer(this.name, function (file) {
                                        console.log("deleting "+file);
                                        try {
                                            fs.unlinkSync(file);
                                            processFiles(util.currentTimeMillis(), 1);
                                        } catch (err) {
                                            console.error(err.message);
                                            console.error(err.stack);
                                        }
                                    })
                                }
                            };
                        }).catch((error) => {
                            removePendingFileTransfer(filename);
                        });

I cannot confirm but it looks like a bug in the underlying file-transfer code. There won't be any progress nor transfer state changes when the watch is in that buggy state.

[Gondwana]

I hoped these problems had been resolved by now. In particular, I thought I'd been told that the cancel() method now actually cancelled the transfer; when I tried it previously, a subsequent dump of the transfer queue showed that it didn't.

At a quick glance, you seem to be using the same API calls as me. For what it's worth ($0), here's some test code of mine:

function queueFile(filename) {
  outbox.enqueue(filename, payload)
    .then(fileTransfer => {
      console.log(filename+" queued: readyState="+fileTransfer.readyState);
      fileTransfer.onchange = onFileTransferEvent;
      dumpQueue();
    }).catch(function (error) {
      throw new Error("Failed to queue "+filename+"; error: " + error);
    });
}

function dumpQueue() {
  outbox.enumerate()
    .then(fileTransferArray => {
      console.log('dumpQueue(): length='+fileTransferArray.length);
      fileTransferArray.forEach(function(transfer) {
        console.log(`  ${transfer.name}: ${transfer.readyState}`);
      });
    });
}

function onFileTransferEvent(e) {
  console.log(`onFileTransferEvent(): name=${this.name} readyState=${this.readyState}`);
  dumpQueue();
}

function purgeFileTransferOutbox() {
  console.log(`purgeFileTransferOutbox()`);
  outbox.enumerate()
    .then(fileTransferArray => {
      console.log('  queue length='+fileTransferArray.length);
      fileTransferArray.forEach(function(transfer) {
        console.log(`    cancelling ${transfer.name}`);
        transfer.cancel();
      });
      dumpQueue();
    });
}

[Thomy13]

Thanks a lot! The code is similar to what I already have. I have identified in my tests yesterday that certain files just won't transfer: their state never changes. New files are immediately transferred.

I am now trying to find out how such files are generated and why they aren't transferred. My guess would be corrupted cbor files that are saved when the application is exiting. I have seen the creation of such faulty files in the past before. It's not that big of a deal when they can be detected.

I am saving a file with existing data on the exit method of the app... so that could be a possible problem.

[JonFitbit]

If you can reproduce it and can provide a code sample, please let me know.

[Thomy13]

After some more testing and development I couldn't find the problem. Files sometimes just aren't transferring and get stuck. I cannot reliably reproduce it. It seems to happen when I let my app run for a few days and generate a lot of files. The maximum amount of files I allow to be stored is 1,500 with a maximum file size of 8-10 kilobytes. That should not be reaching the maximum file size limit per app of 15MB? I also wasn't getting any errors regarding saving new files.

I was only able to observe a few things that may assist in finding the bug:

• The app / companion can get into a "buggy state" where file transfers no longer work
• After enqueueing new files it is no longer possible to run:

• outbox.enumerate().then(...)

  It just never finishes. No exception thrown, the code is basically stuck in there forever.
• I have seen files with size 0 while being in the "buggy state".
• Restarting device nor phone fixes the problem.
• Successful file transfers are not getting the "transferred" state event within the app while the companion could read the file successfully.

Something within the file-transfer protocol gets stuck. Maybe a file size of 0 or a very large file size? I don't know it and am also not able to debug or investigate any further without seeing the underlying code. Maybe the bluetooth code responsible for transfers is suffering from errors?

I have also come across a really weird issue that always crashes my app when adding the following lines:

let bodyPresenceSensorModel = new BodyPresenceSensor();
bodyPresenceSensorModel.addEventListener("reading", () => {
    console.log(`The device is ${bodyPresenceSensorModel.present ? '' : 'not'} on the user's body.`);
});
bodyPresenceSensorModel.start();

Something seems to be wrong there too, but I have absolutely no idea why. The code runs fine in another test project. So possibly a combination of imports / methods causes this?

I can add you to my gitlab project if that helps? I would require a username or email for that.

Maybe Fitbit isn't the right device for my project? I am writing an app that collects and stores raw sensor data for later comparison and enrichment. I am really suffering from such random encounters that will stop my watch from working as expected.

[Thomy13]

The only way to fix this problem was reinstalling the app. However at the cost of all user data being deleted, which isn't ideal when I have a lot of measurements stored on the watch. 

[JonFitbit]

That's weird. Are you definitely using the latest mobile app? There were some issues relating to stuck file transfers in the queue, but it was fixed.

[Thomy13]

Yes, it's on the latest version.

[JonFitbit]

If you can provide a minimal working sample app and instructions, that would be great. PM me.

[Gondwana]

Same issue as here, I think.

[Thomy13]

Yes, looks like the same issue. My app does a little bit more cleanup & remembers files in the queue to not crash the application. File-transfer protocol blindly accepts anything and crashes the entire application if it's too much data.

I've run my app for about 1-2 days and am stuck with the same issue: files no longer transferring.