Comparison of different devices using photoplethysmography (PPG) to extract RR intervals: can we use them for HRV analysis?

Blog post by Marco Altini

The aim of this post is to provide an update on the current status of optical devices / PPG sensors (most sensors that are not chest straps) and determine if they can be used for HRV analysis. All data and code used in this post is available on Github here.

As I've explained elsewhere, while there are a million PPG sensors right now, we cannot use most of them because they either do not comply to standard Bluetooth heart rate profiles or do not provide any standard HRV feature.

This is a pity as such Bluetooth heart rate profiles were designed so that any app could communicate with any sensor, but most sensors manufacturers still do not comply to the protocol. There might be different reasons for this, first being that providing accurate RR intervals is not really their business (sometimes getting reliable heart rate is good enough if your application is targeting measuring during sports), or maybe they prefer to lock their device to their proprietary app. This means the choice is actually quite limited when we need to pick sensors that implement the profile (so that we can record and export data) or sensors that report well known features so that they can at least be compared in terms of features if not at the RR interval level. In my opinion, implementing the profile and providing high quality RR intervals should be also in the interest of these companies, as they could sell more sensors because anyone could use them with other apps and enable applications they haven't thought of or are not interested in (for example HRV4Training or Elite HRV in the context of HRV analysis to optimize performance). Hopefully with time we will see some more sensors taking the time to go through the process.

This being said, some sensors do comply with the protocol, for example the Mio alpha or Schosche Rhythm+. These sensors work exactly like a Polar H7 in terms of data transmission, so they compute RR intervals and send them to any app that implements the Bluetooth heart rate profile, for example the HRV Logger I initially developed exactly for this reason. However, as we've shown in the past, the fact that these sensors send RR intervals, does not mean that the RR intervals are accurate and can be used for HRV analysis. As a matter of fact, most of the times the data are heavily filtered to make sure heart rate is measured correctly (dealing with movement and artifacts is tough), and therefore we loose the accurate beat to beat information that we need to compute HRV reliably. It's simply a different target application.

Finally, a few users emailed us recently asking more questions related to the Schosche Rhythm+ as well as informing us about the availability of a ear-clip (Kyto) that implements the correct protocols. A ear-clip already sounds better than a chest-strap when it's early morning, hence we grabbed one and ran additional tests.

In particular, in this post we compare the following sensors:

• Polar H7 (our trusted reference)
• Kyto HRM-2931 (ear-clip)
• Schosche Rhythm+ (arm/wrist)
• Mio alpha (wrist)
• HRV4Training camera based algorithms (discussed many times on this blog).

​
I included HRV4Training PPG algorithms mainly to highlight once again that it is definitely possible to get high quality HRV data using optical measurements, but the device needs to be developed with this goal in mind.

This post does not aim to be a comprehensive validation, however a small dataset collected under the proper conditions can be sufficient to understand if a device can do the job or not. If a device works well, then additional measurements would be a good idea, to make sure a broader range of values and individuals result in the same outcome, as we did last spring when we validated HRV4Training algorithms at the Sports Performance Research Institute in New Zealand.

If you don't wanna go through the analysis, feel free to jump to the takeaways at the bottom for the main outcomes.

Data acquisition setup & measurement protocol

Data was acquired using the sensors mentioned above all connected to a different device running the HRV Logger app, which is an app that simply records everything coming from the sensor plus additional features. Polar data and HRV4Training PPG data were acquired using a custom app that I developed for our clinical validations at the Sports Performance Research Institute in New Zealand. This app is not commercially available and can record simultaneously data from the two sensing devices, making it easier to synchronize data.

During data acquisition, we collected data a few minutes while breathing freely, and a few minutes while deep breathing, to elicitate higher HRV due to RSA. You will see in the plots below visually the effect of deep breathing as we get greater swings in RR intervals.

A final note on data synchronization: data cannot be perfectly synchronized because it is not timestamped by the sensors. What we can do is either to log real time and then to split data in windows based on when data was collected, then compute HRV features on these windows or to sum up RR intervals over time. For this analysis we went with the second option, and as a result you will see that sensors that are not able to capture all RR intervals typically end up with less data.

Data analysis

RR intervals

We will start by looking at RR intervals, the basic unit we need to compute HRV features. RR intervals (peak to peak differences in consecutive heart beats) are provided by all these sensors, so we don't really need to do much to collect them, apart from linking the sensor to the HRV Logger app. For the camera based algorithms, RR intervals are also computed directly in the app and can be exported as csv files (you can do this using the Camera HRV app, in HRV4Training we do not store RR intervals but only features). A higher resolution image of the plot below can be found on Github.

What can we derive from these data? You can see clearly the first 3 minutes involve regular/free breathing while the 4th and 5th deep breathing. The Mio alpha and Schosche sensor heavily filter the signal and as a result we have an almost flat RR intervals time series. This means these devices cannot be used for HRV analysis as they would provide extremely low values artificially (we will see later the actual HRV data). Note that these sensors can still be used reliably for heart rate tracking, just not for HRV.

On the other hand, we see almost perfect correlation between Polar H7 and HRV4Training for all conditions (first two rows), while we see that the Kyto sensors performs rather well for the first 3 minutes, but gets in trouble during the paced breathing part, where we have bigger swings in RR intervals.

Let's see another subject's data:

Here we can see much higher variability (same range as before on the y axis, but more variation). We can see again how the Kyto ear-clip seems to work well for the first 2 minutes but gets more in trouble with minutes 3 and 4, where we have deep breathing (see the effect of RSA, breathing in reduces RR intervals and breathing out increases RR intervals). 

HRV features (rMSSD)

As features, we will look only at rMSSD, the only feature we really care about. rMSSD is a clear marker of parasympathetic activity and the main feature we use for our analysis in HRV4Training, similarly to what other apps do as well. Additionally, the sports science community seems to have settled on this feature for several reasons (practical as well as it is easy to acquire, compute and reliable over short time windows and less controlled conditions), and therefore we'll stick to it.

What we expect given the data above is to see extremely close values between the Polar H7 chest strap and HRV4Training PPG data, as well as similar values for some windows for the Kyto sensor, but lower variability reported during paced breathing or higher HRV in general. We also expect the Mio alpha and Schosche sensor to provide extremely low rMSSD given the heavy filtering shown above.

For the plot below, I computed rMSSD for each time window and subject:

Results are exactly as expected after seeing RR intervals data. I think it is particularly interesting to see that the Kyto sensor does not seem to suffer higher HRV as much as it is suffering swings in RR intervals due to deeper breathing. For example, data is well matched for the first 3 minutes of subject 001, regardless of higher HRV compared to subject 002. However, for minute 3 of subject 002, rMSSD is poorly detected (this is even more clear looking at the RR intervals plot above), even though the absolute rMSSD value is not really much higher than in the first 3 minutes of subject 001. This is not much data and it is probably not the case to extrapolate too much, however it seems that the Kyto sensor is having trouble with quick successive variations in RR intervals, maybe some heavier filtering on expected beat to beat differences that end up clipping the signals as shown in the RR intervals plot. For these reasons, I would not recommend using this sensor at this stage, regardless of quite good results on most time windows.

Takeaways

In this post we looked at different sensors using PPG / optical measurements to derive RR intervals. We chose the devices above specifically because they are some of the few that implement the Bluetooth heart rate profile and therefore you could use them with any HRV app such as HRV4Training, Elite HRV or ithlete. From what shown above, we derive that:

• You cannot trust a sensor simply because it provides RR intervals and "seems to work" with your HRV app. You need to make sure the sensor is actually sending accurate RR intervals (I know, the sensor manufacturer should do that, but that's what we get).
• HRV4Training camera based algorithms are as good as a chest strap, as we validated already multiple times. We have a paper on this coming up soon, so we will be able to share more data (see some data here or this plot showing agreement with a full ECG). 
• The Kyto sensor is not bad, but not good enough either. We found limitations when HRV was a bit higher or during paced breathing. At this stage I would not recommend using this sensor as a replacement for a chest strap or for the camera based measurements. However, this sensor is by far the best we ever tried among the PPG sensors, so it's definitely going in the right direction and HRV was captured correctly in many of the instances above. We will try to run additional tests to see if we can better highlight limitations.
• Mio alpha and Scosche Rhythm+ are not usable for HRV analysis.

There are other PPG devices that have been validated for HRV analysis, in particular the finger sensor provided by ithlete. I believe that sensor is also communicating to the app via a non-standard protocol, hence the ithlete app is required to be able to use it. However, the sensor has been clinically validated and therefore it's something you can trust.

I hope this post provides some clarity on some of the devices out there as well as on the limitations around many other sensors that cannot be tested simply because they do not comply to standard protocols or do not compute standard features (e.g. Fitbit, Garmin, etc.). 

For now, keep relying on the camera measurement or the chest strap. If you come across another sensor that complies to the protocol and sends RR intervals, let us know and we might be able to add it to this analysis.