Heart rate accuracy test - cycling

[SunsetRunner]

I've been thinking to make another comparison of HR in Ionic and chest strap. A few days ago one of the moderators claimed that PurePulse ( tech used in Ionic ) is more accurate than chest straps. A very bold statement without any evidence.

I always record my cycling trips both with Ionic and my chest strap. Usually, I end up with logging the activity manually in the Fitbit app because after what I see in the HR readings of Ionic is PureMess ( that's a good name for that tech 😉 ). Well, less talking more showing. First two graphs from the same period of time:

Purple is Ionic, turquoise is Strava which using PolarH10 chest strap. If the heart rate monitor is accurate ( and heart is healthy of course ) then it means there must be a certain pattern followed. Slowing down and stopping - HR goes down. Speeding up - HR goes up, maintaining speed - HR more or less the same with small fluctuations. Simple, but makes sense. Following this simple rule, I decided to pick a few areas from the graph where my HR goes down significantly. Since there are less of such areas in the Strava graph I chose this one, verified declines whether the circumstances match and checked what Ionic did at same time:

Now, I looked into GPS coordinates at each place and tried to figure out whether there is a reason why my HR drops. My assumption was simple - junctions, traffic lights, roundabouts - anything I may need to stop and wait or at least significantly slow down.

Area 1:

Traffic lights. I had to wait here a while. Strap shows drop in HR, Ionic something opposite.

Area 2:

This is roundabout. I believe some traffic from the right slowed me down, however, HR drop isn't significant. Ionic HR raises while I'm waiting and it should be dropping. Also, it seems to be a lot lower than captured by the strap.

Area 3:

Again junction. In here I was approaching from a visible cycling path and had to stop for a while due to traffic. Ionic HR is quite low and when the strap registers drop, Ionic registers something opposite.

Area 4:

Traffic lights but you can see the drop is very significant. This is because currently roadworks take place in this area waiting time for the light to change is ages! My heart could rest quite well here.

In this case, Ionic goes crazy. Drops a little bit but while I'm still waiting it raises to some crazy level. When I'm starting to pedal and the chest strap registers increasing HR, Ionic sees the drop. Then it takes about 3 minutes to catch up with the strap.

Area 5:

I don't have a picture of it. It's just a long almost straight road where I could maintain speed. For the time my HR seems to be quite stable.

Area 6:

Just waiting for the light to turn so I could turn right. It always takes a while here.

Ionic here almost shows correct behaviour. The HR seems to be dropping and raising. Overall, it stays higher than registered by the chest strap.

There are a few interesting things about Ionic I learned this way. It's far from being accurate, especially during intense exercise ( but this is something I already knew ). Another thing is that very often the trends are totally opposite. PolarH10 shows increase, so Ionic shows drop in HR. Strap shows a declining trend, so Ionic shows incline. This tells me there may be a lag and it takes time for Ionic to actually catch up. Another thing is that the overall HR registered by Ionic is significantly lower than registered by the chest strap. Note that the strap is using ECG to monitor HR and it's strapped around the actual heart, not the wrist so it's hard to question its accuracy.

What concerns me is that HR is a very important factor in calories computation. At least this is what Fitbit claims. While I understand that different apps use different algorithms, however still I'm more likely going to trust the app/device that gives more accurate readings. On this ride, the difference is about 120kcal ( for some past workouts even more ). It's not really a lot but in my case, when I'm trying not only to maintain weight but also gain some muscle mass, then wrong readings may actually get me into slight deficit and slow down expected progress ( which in fact happened, so I started to log activities manually based on the results captured by the strap ).

Bottom line is that I don't think Ionic can get an update that will fix this issue and it's sad that all they can come up with is just copy/paste answer. The reason why I did this test is that I have enough of being treated by moderators as a dumb person. It is insulting and upsetting. I read the same advice of "fixing" the problem all over again while the problem isn't fixable. There is a solution which, at least in my case, would make me happy - let Fitbit sync with external HR monitors ( like I can do with Strava ) but this won't happen unless one day Fitbit Chest Strap will be announced.

[NellyG]

I am actually impressed at how close the two different technologies are! Yes, I agree with you that the chest strap is probably a lot more accurate than a wrist monitor though I have no facts to back up that belief. I know the fitbit heart rate is sampled every few minutes rather than being recorded exactly as it happens which probably accounts for some of the lagging in your graphs (I have no idea how the strap measures HR though)

A very interesting comparison - thanks for doing it!

[SunsetRunner]

@NellyG I'm actually thinking we are looking at different data 🙂 It's possible that "accuracy" means something else for different people. Let me then explain what it means to me. For most of the time, the HR should stay within 10bpm from ECG reading. Here I don't really define "most of the time". Chest strap uses ECG. It's the same technology that is used in the medical field and the same that Apple introduces now ( although, as far as I'm aware it's not going to be possible to use it during workouts, I don't know how they're planning to utilize it ). Knowing that we may assume that the chest strap is rather on the more accurate side.

Let's start more detailed data breakdown:

There are few things to notice:

- 50% of readings stay within "acceptably accurate" range (0-10bpm).

- It's worrying that all together 11-30bpm is 35% of the whole workout. These two ranges I find less acceptable not reliable but still, we can say it's a medium level of inaccuracy ( arguably )

- Very high inaccuracy altogether is little over 8,5%. This is really a lot! It's a difference between fat-burn, cardio and peak zones. Fitbit uses a very simplistic representation of HR zones. Other software usually defines more of them. 30-40bpm can go easily across two different zones.

- Full accuracy is only 6.37%, however, this is something I'm not going to complain as perfect accuracy isn't really possible with an optical sensor.

That's not all. The workout took 35.5 minutes in total. How much time then I spent within different "error zones"?

Reliable readings are about 19 minutes ( little more than half of the total time ). The rest is just a garbage. For some people, it may be a dangerous garbage. In this case, I took an absolute difference and ignored signs. Most of the people say that Ionic overestimate their HR but my case is different. Most of the time it's too low. If I relied on it trying to stay within a particular HR zone, with difference reaching 20, 30 or even 40+ not only I would be training wrong but in some circumstances, I could cause a danger to myself. 

Bottom line here is that the overall accuracy of HR readings in this particular workout is roughly 50%. Is this really something to be impressed? Data don't lie. Unfortunately, during cycling, I'm not going to be checking whether the watch is placed two fingers above the wrist and how tight it is. I don't want to end up on somebody's car bonnet 🙂 

[lenny4d]

@SunsetRunnerman: your heartrate must have peaked at putting this post together!! (JOKING)

Seriously:

>> sad that all they can come up with is just copy/paste answer.

Are you refering to fitibit official customer support [probably not], or the people answering on these forums? They are 'community volunteers' I believe, not necessarily having that level of knowledge...

>> Note that the strap is using ECG to monitor HR and it's strapped around the actual heart, not the wrist so it's hard to question its accuracy.

That's true. The polar is probably using some form of filtering too. In your graph there are only a few points per minute. That being said, it is highly accurate, better than a few bpm at any time I would think.

>> This tells me there may be a lag and it takes time for Ionic to actually catch up.

That's 100% certain, it has quite a heavy filter. I see that there is about a 1-1.5minute time constant on the filter that is used on the heartbeat data in the exercise report. The HR data is logged at ~5s intervals, the calories at ~1minute interval. I don't know where you get your data from in the plot because it does not have the same time resolution as fitbit uses. You should have 12 pts/minute but have much fewer it seems.

I think you are taking it way beyond claims to try to match fitbits HR data to physical events, it is not THAT accurate. If you need that type of accuracy, yes you need a cheststrap (which unfortunately fitbit doesn't support, which is silly as they easily could!).

The main thing is that the HR is underreported, which is important if you are chasing your peak heartrate zone. In addition your calories are underreported.

>> What concerns me is that HR is a very important factor in calories computation. At least this is what Fitbit claims. While I understand that different apps use different algorithms,

Yeah, the formulas used for calculating calories from HR + personaldata is a bit obscure, but see here and scroll down+discussion. Wether fitbit uses exactly these formulas or small adjusted ones I don't know. But in the end it is a formula and it doesn't matter as long as the sensor input is substantially off.

In this case it is the HR sensor. As long as there are pulsed light wrist HR sensors, this issue has come up. What I can suggest is - using a cheststrap or similar (e.g. EKG handles on treadmill) - try to figure out what conditions give the best and worst reading matches. Be especially aware of sweat under your watch/tracker!!, the dryer the better. Experiment with tight fit/loose fit, closer and further from wrist, experiment with the way you have your wrist. I did that a lot for my Charge 2 and in the end was very satisfied. The Versa and Ionic have a different physical sensor and do not seem to work as well (as you have also figured out). I still have to start playing around and figuring out how to get it better. It probably also depends on the type of wrist you have.

[SunsetRunner]

In fact, my heart rate was at rest 😛

I realize that my expectations in regard to the fitness tracking might have changed over time. I mentioned once that I bought Fitbit Charge 2 to support my weight loss and despite several complaints I had it was a help. Then I decided to go beyond that and simply it wasn't enough anymore. That's why I bought Ionic ( I admit, it was just an impulse when I've seen the box in the store at the airport ). Maybe I just have grown out of Fitbit due to my expectations? Can't really tell. All I can tell is that if something is called "fitness tracker" then I expect it to do at least this right - track the fitness progress ( or regress 😉 at my age anything is possible ). Also, looking back at my posts I notice that I never really encounter other problems like for example sync issues. Ok, last Friday after my ride I couldn't sync the app with the watch but 30 minutes later magically it happened and I can't complain because it was just an episode. For many weeks nothing like that happened to me and I'm hoping it won't change.

I'm going to put one more comparison for simple walking as the results are different here. My HR registered by Ionic is a few points higher than the chest strap. It's quite interesting that both readings usually meet in my case when I run on the treadmill with pace 5:00 ( which is little more than jog less than resting pace ). My HR then oscillates around 130-140bpm and both Ionic and strap are quite close. I started wondering even whether I could turn it into a function and simply apply a correction to the HR read by Ionic based on data coming from the strap because I see a certain pattern here:

- Very low intensity: Ionic too high

- Moderate intensity: Ionic very often equal

- High intensity: Ionic too low

Collecting data with the treadmill is harder because there is no TCX file coming out of non-GPS activities but nothing stops me from using actual GPS walk on the treadmill ( this will probably trash my stride but that's not an issue ) and get HR out of it ( which is another plan of mine ).

>>In this case it is the HR sensor. As long as there are pulsed light wrist HR sensors, this issue has come up. What I can suggest is - using a cheststrap or similar (e.g. EKG handles on treadmill) - try to figure out what conditions give the best and worst reading matches. Be especially aware of sweat under your watch/tracker!!, the dryer the better. Experiment with tight fit/loose fit, closer and further from wrist, experiment with the way you have your wrist. I did that a lot for my Charge 2 and in the end was very satisfied. The Versa and Ionic have a different physical sensor and do not seem to work as well (as you have also figured out). I still have to start playing around and figuring out how to get it better. It probably also depends on the type of wrist you have.

The conditions giving the worst readings I already know:

- high and very intensity

- changing intensity

- certain motions during activity ( push-ups are an example I always give because then Ionic can drop with readings almost to my resting HR like 60BPM )

- positioning and sweat - true, but this is hard to control when you are in the middle of an exercise, maybe instead of making users to constantly fix the watch, Fitbit should work on better fitting. I would love to have Fitbit wide sweatband for a wrist made of a soft stretchable material that can embed Ionic in, Hmm... maybe I should just get a regular one and just wear it over Fitbit and see the results. It should stop it from moving and keep it pressed to my wrist.

About the lag, this is a problem. I very often plan my rest according to the heart rate when exercising for endurance. I elevate it to certain BPM, then try to keep it that way and let it plummet before next set. Because Ionic's readings are not reliable, it's not even possible to use it for that purpose ( but again, maybe I have too high expectations? ).

I'm going to investigate it little more ( and I think, once I mentioned it, I like the idea with the sweatband on the wrist, so I'm going to test such solution ). I don't have many hopes left and I'm afraid this Christmas is going to be my last with Ionic but it's all just for my own curiosity as I want to know how things work ( or don't ) and understand.

[lenny4d]

Thanks for all your tests and contributions @SunsetRunner ! In the end what fitbit could and should have done is to provide a chestwrap that talks to the watch/tracker via bluetooth for intensity workouts where accurate HR matter (f.e. around peak-zone).  Just googling shows many people like to see this. It's a bit like using gps to get accurate distance instead of stride.

I haven't checked in detail the bluetooth API for fitbit to see if it has sufficient possibilities to hook up a 3rd party bluetooth chestwrap. Plus one need to hack the dataprotocol from the strap because these are all 3rd party product without open standard. Regardless one would not be able to replace the optical HR monitor. That's something fitbit needs to create and they are presumably not interested in it, rather develop & fix purePulse. So.... stuck we are. 

[DavideFitbit]

Hello @SunsetRunner and @lenny4d, thank you for all the details and information you've shared about heart rate accuracy, I'm sure it will be useful for other users as well. 

I'm not aware of any plans to have a chest strap from Fitbit developed, but it'd be interested to see how many people are interested in something like that, I found this idea posted in the Community, and it's been changed to the state of Not currently planned, but it's still open to votes.

However, if you have any more specific ideas you could also post the suggestion in the Feature Suggestion board of the Fitbit Community, this section is checked more regularly by the team of developers for possible updates in the future. 

It's good to see you around in the forums.