SDK for Accelerometer Data in Esense2 for a project on Sleep at the University of Essex

[vitodefeo]

Dear Developers Community and Fitbit Official Support,

I am a researcher at the University of Essex. We recently secured £1.1M funds for a project about sleep and mental health.

I wrote this post for the SDK community but a user commented: “This forum isn't monitored by Fitbit (Google) staff. However, the Web API forum is monitored by a Fitbit (Google) person. Given the scale of your proposal, he may be able to put you in contact with staff who could provide more official help.” – So I reposted the post here.

For our project, we will recruit about 600 participants and they will wear a health band to keep track of their sleep quality and mental health issues. We are evaluating the possibility of buying 200 Fitbit Esense2.

Our main goal is to perform sleep stages classification. We know that from Esense2 we can have some metrics to infer a person's activity level through the Web API such as activeMinutes, activityCalories, etc.  During sleep, we can have sleep stages (e.g. wake, light, rem, deep, restless, asleep) but we want to implement a more complex algorithm based on actigraphy (x,y,z accelerometer signals) and heart rate variability (HRV), recorded by means of photoplethysmography. Therefore we need to record also the x,y,z accelerometer signals.

Furthermore, we need to record the data continuously for 8 weeks and we will send the heath band to each participant via post.
The devices only store data for 5-7 days.  So, the participants of our study will need to sync their devices with the Fitbit mobile application on a regular basis so Fitbit can store the data.   Then we can pull the data at the end of the 8 week period using the Fitbit Web API.  We are aware that to pull the data through the Web API, we will need to have each participant consent to share their data with your application.

Our problem is that, unfortunately, the Fitbit Web API does not store the accelerometer x,y,z data.  That data is only available through the Device SDK which requires building a clockface for the Sense 2 device (https://dev.fitbit.com/build/reference/device-api/accelerometer/)
Is that correct?

All of the data you asked about, except for the accelerometer data, will come from the servers using the Web API.   The accelerometer data comes directly from the Fitbit device and you'll use the Device SDK to get that data.   

Given that we are going to buy a lot of Esense2 (about 200), we need to be super sure about all the SDK functionalities (e.g. accelerometer data from the clockface sync).
In particular, we need to understand how the clock face interface work.
Can we program the clock face interface in order to save the data to our server? If so, should we use an ftp protocol?
Can we, instead, save the data on Fitbit server and download the accelerometer data together with the other data (e.g. HRV) that we download through the Web API? I mean that we use the Web API to download all the data except the accelerometer data and something else (what?) to download the accelerometer data.
Can be the sync performed automatically? Should the user intervene, instead?

When you request the accelerometer data from the clockface, I think this is done at sync time.  The data is transferred to your server, but also transferred to our servers.   We use the accelerometer data to calculate the sleep stage information.  However, the raw data is not available through the Web API. 

A user from the other forum commented: “To capture accelerometer data on Sense 2, you'd need to develop and install a custom clockface/app on the watches (so yes, you're right about the Device SDK). Fitbit hasn't released a SDK for Sense 2 that would allow this. Some developers have managed to get a beta SDK to work, or you could use an earlier Fitbit OS watch type.”

And another user advised to use Sense rather than Sense2.

Another user suggested to  SimpleSleep app monitors sleeping position during the night on a Sense using the Gyroscope which may be useful information for your study. He also suggested to use a universal chest strap watch band extension.
However, I don’t think I can integrate SimpleSleep app into my app because I don’t think that it’s open source.

Another user commented: “Your clockface/app would need to run continually, since Fitbit OS doesn't support background processes. It would need to transfer data from the watch fairly frequently since accelerometer data is voluminous and watch storage is limited. Keeping this up for eight weeks is theoretically possible but, I fear, ambitious in practice. Basically, you have to write your own sync system for accelerometer data.”

So I understand that all this is feasible but I have to program my own sync system and ask each participant to keep the clockface always active. Furthermore, I would need an "alarm" system that warns me when a participant is not syncing. 
However, I would need an official opinion from someone from Fitbit (Google).

Happy to have a Zoom / Teams  meeting with someone from the support.

Kind regards,
Vito

Dr. Vito De Feo
Lecturer in Health Artificial Intelligence for Brain and Mental Health

Artificial Intelligence Team and Consumer Neuroscience Research Cluster

Future Health Technology and Brain-Computer Interfaces laboratories, 

School of Computer Science and Electronic Engineering, 

University of Essex, Wivenhoe Park, Colchester CO4 3SQ, UK ►https://www.essex.ac.uk/people/defeo60201/vito-de-feo

[Gordon-C]

Hi @vitodefeo 

I've contacted someone on the Device SDK team to respond to your post in the Device SDK room.

Gordon

[vitodefeo]

Hi Gordon,

Many thanks.
Vito

[vitodefeo]

Hi again @Gordon-C 
No-one replied... 
Can you please help me?
Many thanks,
Vito

[vitodefeo]

Hi @Gordon-C,
No one replied... could you please chase someone to reply?
Many thanks!
Vito

[Gordon-C]

Hi @vitodefeo 

I've been on holiday the last few business days.   I will see if I can get answers for you.

[Gordon-C]

Hi @vitodefeo 

What I understand from our engineering team is what you're trying to achieve is not simple. it's recommend you find a 3rd party developer with existing experience to build their solution.

Anyway, in relation to the specific questions:

1. Can we program the clock face interface in order to save the data to our server? Yes
2. If so, should we use an ftp protocol? No, FTP is not supported, use HTTP POST.
3. Can we, instead, save the data on Fitbit server and download the accelerometer data together with the other data (e.g. HRV) that we download through the Web API? I mean that we use the Web API to download all the data except the accelerometer data and something else (what?) to download the accelerometer data. No, it has to be your own backend.
4. Can the sync performed automatically? Should the user intervene, instead? The device automatically syncs every 15 minutes if the Fitbit mobile application is running on the mobile device, but this excludes accelerometer data.

For the clock face specifically, it would need to do the following:

1. Use OAuth to make the user login with their Fitbit account. This will give the clock face the Fitbit User ID. Example 😞https://github.com/Fitbit/sdk-oauth) << doesn't include handling of refresh tokens, but that probably won't be an issue once they have the User ID.
2. Read accelerometer data Example: (https://github.com/Fitbit/sdk-scientific/blob/master/app/index.js#L40)
3. Periodically send data from the device to the companion, then from the companion to their backend, keyed by the Fitbit User ID. Examples: (https://github.com/dillpixel/fitbit-asap, https://github.com/sammachin/Fitbit-MessageDemo/blob/master/companion/index.js)
4. The clock face would need to check that data is being sent to the backend correctly, and if not, display a message to the user.
   On the backend, you can combine data from the Web API with data from the clock face.

Hope this helps,

[vitodefeo]

Hi @Gordon-C ,

Many thanks for your reply and sorry if I pushed you. I didn’t know you were on holiday.
Your answers are very helpful!

About the engineering team, what I understand that what I am trying to achieve is not simple but possible with Sense2. From previous comments, I had understood that the SDK was not enough developed for Sense2.
Can you please ask your engineering team:

1. To confirm that this is possible also with Sense2 (from what follows it seems possible) because Sense1 is not anymore on stock. I think that the answer is that it's possible but we must download the Beta SDK version (https://drive.google.com/file/d/1ABEREa0ii46VZTKT4jdRvMx2mgM0A3hb/view?usp=share_link). Am I right?
2. If they know a 3rd party developer. We contacted both Connected Life and Fitabase but they don’t record accelerometer data.
3. If one of them can have a Zoom chat with me.

In relation to my specific questions and your answers:

1. OK.
2. OK to use HTTP POST.
3. OK, fine.
4. If the accelerometer data are excluded from the 15-minute automatic sync, can we create an automatic sync method for the accelerometer clock face (contacting our server and using HTTP POST)? Or should the user do something manually from time to time? From 3 and 4 below, it seems that the only thing that the user must do is to check whether there is an error message on the clock face.

In relation to the clock face specifically and your answers:

1. Thanks. Very helpful.
2. Thanks. I had already found this but one member of the community commented that this is not possible with Sense2 (only with Sense). Can you confirm that this works also with Sense2 (see my initial questions)?
3. That’s super helpful! So, we can do it periodically without the user's intervention.
4. Perfect! Thanks a lot! So, the user should just check time to time the clock face. Right?

Kind regards,
Vito

Dr. Vito De Feo
Lecturer in Health Artificial Intelligence for Brain and Mental Health
Artificial Intelligence Team and Consumer Neuroscience Research Cluster
Future Health Technology and Brain-Computer Interfaces laboratories, 
School of Computer Science and Electronic Engineering, 
University of Essex, Wivenhoe Park, Colchester CO4 3SQ, UK ►https://www.essex.ac.uk/people/defeo60201/vito-de-feo

[Gordon-C · Accepted solution]

Hi @vitodefeo 

You are correct that you will need to download the provided Beta SDK to use the Sense 2.  We typically don't recommend a developer for projects.  My recommendation is to ask in the SDK community forum to see if someone is willing to help you, or you can check with your IT team to see if you can acquire developer resources.

The Accelerometer API documentation, https://dev.fitbit.com/build/reference/device-api/accelerometer/, states there is an "onreading" property that let's you know when there is new data to download.   Also, on that page is a link to the Accelerometer Sensor Guide, https://dev.fitbit.com/build/guides/sensors/accelerometer/, with some information on how to pull the data along with some sample code and best practice.   I would read these 2 pages for best practices on reading accelerometer data.

[vitodefeo]

Hi @Gordon-C  ,

Thank you very much for the valuable information.
We will definitely hire a developer and I will also ask in the community.
My biggest concern is the data transmission. Surely the raw data from the accelerometer is much heavier than the aggregated data that is normally downloaded via Bluetooth. I think that is why this transfer is not done.
For this reason, we will do an initial feature extraction directly in the device via the SDK.
Any advice is welcome!
Thank you very much!
Vito

[vitodefeo]

Hi @Gordon-C ,
Do you think that Google is going to release the SDK full version or is the beta version not going to be updated?
Many thanks,
Vito

[Gordon-C]

If this is the release that I'm thinking of, you should treat the beta as code final and had been run through our full test suite.

[vitodefeo]

Thanks @Gordon-C 
Then, maybe, you (Google / Fitbit) can stop to call it "Beta".
If you tested it, then IMHO, it would been a better marketing strategy to call it "Final".
I mean: it's a great thing that it has been tested!
Many thanks again,
Vito

[Guy_]

HI @vitodefeo - in light of the Google restrictions for EEA users who will no longer be able to download clocks or apps from the Gallery as of June 2024 it might be wise to look at other brands which have better support for your project in the long term.

The restrictions may not currently apply to the UK [as yet unknown, as no list of countries was specified, so it's open to interpretation] but it may not be long before they too do.

I've add this comment here too as it may be helpful for users only following this thread.

[GeekBit]

Anyone knows if it possible after June 2024 to download private apps through a private link?

[Guy_]

Hi @GeekBit - thanks, I have already escalated questions such as this and also will developers have full GAM access to maintain their work for non Europeans and how will already downloaded items be maintained in Europe if download is not possible, etc. We will see if we get more information and will post it here.

[Guy_]

@GeekBit-  see more information here Removal of third party products from the Fitbit Online Gallery for EEA users 

[vitodefeo]

Hi all ,
Actually, I don't understand this. Why should we download third-party apps?
We need to design our own clock face.
Sorry, I don't understand.
Can you please explain to me more?
Sorry if I don't understand and many thanks,
Vito

[Guy_]

@vitodefeo - see this post for explanation.