Hello, on June 24, the Sleep Timeline on my Fitbit app showed a low value for Deep Sleep. Surprisingly, on June 25, the Sleep Timeline Deep Sleep for June 24 was changed to 1h 32m (and the corresponding graph also changed). In fact, the Deep Sleep in the Timeline for June 25 was uncharacteristically high (2h 14m). I have lost confidence in the sleep data since June 24.
On June 25, I found the sleep.json file, which contains the data for June 24: It sill shows the >original< June 24 data for Deep Sleep. However, the UserSleepStages csv file shows the >new< June 24 data (1h 32m) for Deep Sleep.
On June 27, I called the 1-800 number and was directed to change the Sleep Tracking to Sensitive; however, the UserSleepStages CSV data for Jun 28 still is different from the Sleep JSON data. It looks like the two files are using different algorithms to determine the sleep stages.
Also, as mentioned below, starting on June 24, the overal score in the sleep_score.csv file in the exported data also diverges from the Sleep score in the Fitbit app; prior to that their values are identical.
More info:
The UserSleepStages.csv file data up to June 23 matches the data in the Sleep.json file; however, starting on June 24, you see discrepancies such as:
"dateTime" : "02:46:00.000",
"level" : "deep",
"seconds" : 870 ← This is 14 minutes and 30 seconds, i.e., a Deep Sleep instance occured between 2:46:00 AM and 3:00:30 AM (compare with json below).
"dateTime" : "07:40:30.000",
"level" : "deep",
"seconds" : 510 ← This is 8 minutes and 30 seconds, i.e., Deep Sleep between 7:40:30 AM and 7:49:00 AM.
"dateTime" : "08:31:00.000",
"level" : "deep",
"seconds" : 30
for Sleep.json
versus
| sleep_stage_type | sleep_stage_start | sleep_stage_end |
| DEEP | 01:25:00+0000 | 01:25:30+0000 |
| DEEP | 01:26:00+0000 | 01:55:30+0000 |
| DEEP | 02:28:30+0000 | 03:01:30+0000 |
| DEEP | 04:14:00+0000 | 04:22:00+0000 |
| DEEP | 05:44:00+0000 | 05:48:30+0000 |
for UserSleepStages.csv
Using Python to show the first set of data (i.e., UserSleepStages csv file = Graph under Fitbit app), we get the graph below. Notice the sleep stages:
Now, below notice the sleep stages for the second set of data (i.e., the Sleep.json file):
Notice, for instance, that time intervals for Deep Sleep are different! It appears that your monthly sleep profile uses the data in the second set rather than one displayed in the Fitbit app (the first set, namely, the first graph above).
Please help!
Best Answer
Using Python to show the first set of data (i.e., UserSleepStages csv file = Graph under Fitbit app), we get the graph below. Notice the sleep stages:
Now, below notice the sleep stages for the second set of data (i.e., the Sleep.json file):
Notice, for instance, that time intervals for Deep Sleep are different! It appears that your monthly sleep profile uses the data in the second set rather than one displayed in the Fitbit app (the first set, namely, the first graph above).
Best AnswerRegarding sleep last night (which ended this moring), the graph for the first set of data (i.e., UserSleepStages csv file = Graph under Fitbit app) is as follows. Notice the sleep stages.
Notice (above) the long AWAKE red segment starting at around 350 minutes: I woke up at that time but stayed in bed without much moving; I stayed still inhaling, holding, exhaling, holding (in 5.5 second cycles or around 11 brpm). Also notice the two instances of DEEP sleep in the first 100 minutes.
Now, below is the graph for the for the second set of data (i.e., the Sleep.json file). As you can see, there are no instances of DEEP sleep in the first 100 minutes. Then, after 350 minutes, there are instances of LIGHT sleep and REM sleep when I am actually awake (it is hard to say if I am fully awake because all I was doing is thinking). On other days, I have had instances of DEEP sleep after I woke up (but I stayed still in bed doing the 11-brpm breathing hoping to fall asleep again). The problem is that these second set of data (Sleep.json file) is used to generate the monthly sleep profile. Notice that the REM sleep instances are different for the two graphs (although sometimes they overlap). Which set of data is right?! Neither? One is right for one sleep stage, the other is right for another sleep stage?
Best AnswerBelow is the Sleep timeline in the Fitbit app for today August 7. I woke out of vivid dream at around 5:25 am and didn't fall asleep after that. However, this graph shows period of REM sleep and Light sleep. I cannot avow for REM sleep as I was in a sleepy pensive mode similar to the final minutes of one's night before fully getting out bed. I finally got up from bed at 7:50 am.
I used the data in the UserSleepStages csv file to produce the same graph (as above) below. Notice that time is counted in minutes since going to bed (at 12:00 am).
Compare that graph, with the one below, obtained from Sleep.json file. This data in the Sleep.json file appears to be the data used to come up with "Your sleep profile" (Bear, Dophin, etc., along with montly averages for Deep Sleep, percentage for REM sleep, restorative sleep, etc.). Notice for the graph below that after I woke up at 5:25 am, there are periods of REM sleep and Deep Sleep (while I stayed in bed). Also, in general, notice that the Deep Sleep periods are different compared with the graphs above (although they sometimes overlap). When they do overlap, they seem to be shorter in the graph below. I have added another updated graph further down Another important difference is that the graph below does not have those with the sharp brief notches of "detected movement" that are presumably used to measure restleness. Incidentally, this "detected movement" is not necessarily observed every time a step is counted. I see it is detected via the Oxymeter (when you move a bit during a minute, the Oxymeter cannot properly provide a reliable measurement for that minute, and Fitbit appears to use this Oxymeter failure as "detected movement").
@StephanieFitbit
Best AnswerHello All and @StephanieFitbit ,
As you understood from my previous posts, there are two different sets (and algorithms) for Sleep Stage data produced by the device.
Below is the Sleep Timeline for this past Sunday, August 8. I got up from bed sometime between 6:00 and 6:15 am and then went back to bed at around 6:55 am after getting some food and spending some time seated on a sofa. In fact, between 6:22 and 6:40 am, I have the timestamps of the social media videos I watched seated on my sofa; starting at 6:42 am, I watched a video on YouTube. However, between 6:31 and 6:43 am, it is marked as REM sleep in the Sleep Timeline. I mean, seated on a sofa, watching videos on social media and YouTube was interpreted as REM sleep:
I have the same graph plotted via Python using the UserSleepstage csv file:
Now, compare this with the graph plotted via Python using the Sleep JSON file (this graph is not available in the Fitbit app but its data seems to be used for the monthly profile). Notice that that particular REM instance (correctly) is not there. However, the first two instances of Deep Sleep (seen in the previous graph) are not seen below. Which data set/algorithm is correct? My experience is that the first set (which is shown in the Fitbit app) generally overestimates the total durations for Deep Sleep and for REM (this overestimation phenomenon started on June 24); the second algorithm either underestimates these two stages (or it is just stricter, more accurate?), but its values oftentimes seem to be low when I feel I actually had a good sleep (e.g., maybe the two deep sleep instances above, at the beginning of the night, should also be below). Incidentally, for August 7, the Sleep Timeline showed 53 minutes for Deep Sleep, while the Sleep JSON file, on the contrary, now showed a higher value, at 70 minutes.
In my view, the long and the short of it is that you maybe one cannot expect to properly/perfectly calculate the sleep stages indirectly by leveraging the HR/HRV/Mobility rather than measuring them directly using electrodes/sensors connected to your head. However, perhaps you can use the values obtained one night as a benchmark, that is a reference, to determine the relative quality on other nights.
Best Answer