Heart rate variability using the phone camera: Android edition

Blog post by Marco Altini

[HRV4Training will be available on Android by the end of March, 2017. Please follow this page for updates]

Back in 2013 I've developed our first camera-based heart rate variability algorithms, which I first documented here. We went a long way in the past 3-4 years, providing multiple validations and finally publishing a comparison with respect to chest straps and electrocardiography on a very broad range of HRV values last month. While recently we've focused much of our efforts in going beyond the daily measurement and providing meaningful and actionable insights by combining physiological data over the medium and long term with more context (your morning tags, related to sleep, training lifestyle, etc.), there is no doubt one of the main features of HRV4Training is the camera based measurement. 

This is also the reason why we hesitated for so long to move to Android. Back when we started development in 2012, it was nearly impossible to have even a Bluetooth link working correctly across phones, let alone hacking the camera. As we decided to finally get busy with Android development again this year, we gave it another shot. It took some time and a few extra optimizations but I'm happy to announce that we have a reliable camera based measurement on Android as well, and in this post I will go over the algorithm and show some data as well as the usual comparisons with respect to chest straps.

Fragmentation is real

I'd like to start this with a bit of a disclaimer. While we ported our validated algorithms on Android and tested them with respect to chest straps, fragmentation might cause issues. We can test all iPhones, but we cannot test all Android devices and guarantee accuracy on the different phones, cameras and operating system versions. However, we tried to develop methods to detect possible issues with your specific setup and inform you, so that you can take action. 

In particular, during the Practice mode as well as during each measurement, we check that the camera is providing frames at the required frequency and with very limited variations, which is key to provide accurate data. We also check signal quality as we do on iPhones and estimate the validity of the HRV measurement, but on top of this we take special care to check how your hardware is behaving. If your phone is unable to sustain the required frequency or lags some frames, we can detect it and inform you. Please check this post for more practical details on using the camera mode on Android and how to provide the required permissions, as we will focus only on the algorithms and comparisons with reference systems in this post.

Algorithm overview

The technique used to extract heart rate and HRV is called photoplethysmography (PPG for short) and consists in detecting changes in blood volume during a cardiac cycle, by illuminating the skin and measuring changes in light absorption.

PPG has become quite a popular non-invasive method for extracting physiological measurements such as heart rate and oxygen saturation. However, most applications today focus simply on heart rate, and while we've shown that an iPhone camera can be misused to acquire reliable HRV data, no app to date can do the same on Android. Let's go over our current implementation and some data.

The signal processing pipeline on Android is very similar to what we covered here for the iPhone version, and also explained in this paper. 

Phone camera initialization: Android phone cameras can also be configured to acquire videos at certain frame rates, and while this process is a bit more challenging and required several optimizations to get a stable frame rate on older phones, we could acquire our 30 frames per second, which we have shown previously to be sufficient after cubic spline interpolation (which effectively brings the signal up to 180 Hz, before feature extraction). The main challenge here was in getting Android to provide frames at this exact frequency, as by default there are huge lags between frames.

Frame processing: after initialization of the whole system (frame rate, flash, etc.), we start getting frames and extract the RGB values  from the camera, which we then average over the whole frame, and finally keep only the red value to construct our PPG signal. This is the main difference with the iPhone version where we were converting data to the HSV color space. We found that on Android keeping only the red component gives better results across different phones and cameras and provides a pretty good signal.

Filtering and smoothing: the acquired PPG signal needs some further processing. More specifically, we applied a fourth order Butterworth band pass filter. The band pass filter helps in both removing the DC component, due to finger movement or changes in venous pressure, and also high frequency noise.

Resampling and interpolation: one of the biggest issues for HRV analysis is the low resolution of the signal, due to the limited frame rate we can obtain from the phone's camera (i.e. 30 Hz). Using a cubic spline interpolation at 180 Hz, we can solve this problem. The interpolation seems fast enough for older Android phones to compute, and significantly improves the resolution of the signal.

Peak detection: peak detection is performed using a simple slope inversion algorithm, plus a local peak search to avoid detecting double peaks due to the nature of the PPG signal and some additional tricks we learnt in the past few years. You can see the result of this algorithm below.

RR interval correction and features extraction: peak to peak intervals are corrected for artifacts, before features extraction. Intervals correction prevents artifacts due to ectopic beats or motion from affecting features computation, as often reported in literature for HRV analysis. Especially for the PPG signal, intervals correction is very important since the signal is very sensitive to motion artifacts, and HRV features risk to get completely unreliable even when heart rate is quite stable. We correct in different ways, based on the distribution of the RR intervals in a given window as well as consecutive beat to beat differences.

Here is a one minute sample of our upsampled PPG at 180 Hz, including detected beats (in green) and discarded ones (in red):

These data were acquired during deep breathing, as you can clearly see from the modulation of the signal, going up and down when inhaling and exhaling. All beats are detected correctly, while the first one is discarded as part of your initialization phase (the first 10 seconds are potentially discarded based on how stable the signal is).

Comparison with chest straps (Polar H7)

We've acquired data under different conditions and used different phones so that we could validate the algorithms reliability across a broader set of heart rate and heart rate variability values. As HRV metric, we focused on rMSSD, for reasons explained elsewhere. 

Phones

Phones tested that were able to provide accurate HRV data:

• Samsung S5 and S6 (2014, 2015)
• HTC A9 (2015)
• Alcatel Pixi (2016)
• LG Nexus 5 (2013)
• LG Nexus 5X (2015)
• Motorola Nexus 6 (2014)

Phones tested that were not able to keep up with the algorithm (in all cases the app was able to detect it and inform the user, at that point you can either get a chest strap or a refund)

• Samsung S3 (2012, 2013)

My guess is that any phone bought in the past few years should have no problems as algorithms are optimized to use limited resources. However, 4-5 years old phones might struggle and be unable to keep up. We had perfect results on quite old phones (e.g. Nexus 5), but other phones released around the same period could not keep up (Samsung S3 and S4). Special thanks to our friend Thomas for lending us all his phones. 

Protocol and data synchronization

We collected data a few minutes while breathing freely, and a few minutes while deep breathing, to elicitate higher HRV due to RSA. You will see in the plots below visually the effect of deep breathing as we get greater swings in RR intervals.

A final note on data synchronization: data cannot be perfectly synchronized because it is not timestamped by the reference system (bluetooth sensors). What we can do is either to log real time and then to split data in windows based on when data was collected, then compute HRV features on these windows or to sum up RR intervals over time. For this analysis we went with the first option, meaning that for each minute we grouped all RR intervals coming from a sensor in a certain time window, and used that data to compute HRV (rMSSD). We also tried to visually synchronize data streams by moving one of them a few seconds ahead or behind in time, as visually it's quite easy to re-sync data given the oscillations in RR intervals due to breathing (see some pictures below).

RR-intervals

Let's first look at RR intervals and some individual recordings, then analyze heart rate and rMSSD values for all our recordings. Here is an example at rest, where we can see RR intervals extracted by the algorithms above using an Android phone, and simultaneously acquired using a Polar H7 connected to an iPhone using the HRV Logger app:

Here is another one under similar resting conditions:

We can see quite clearly the high correlation between the two RR-intervals time series, we report also rMSSD and heart rate values directly in the plot, as correlation is not really the most relevant metric here. We need to make sure RR intervals are basically identical so that rMSSD can be computed correctly. A difference of a few milliseconds is in line with our previous comparisons on iPhones. Below is another example taken under different conditions, paced breathing, and we can see the bigger variations and changes in RR intervals due to inhaling and exhaling just a few times:

Again, very good agreement between the two signals and extracted features. We repeated this procedure for different phones and the aggregated heart rate and rMSSD results can be found in the section below.

Heart rate and HRV (rMSSD)

Below you can see heart rate and rMSSD for all 60 seconds segments included in this analysis, showing very good agreement between the Polar H7 and Android camera over a decent range of values:

Both heart rate and rMSSD were computed on 60 seconds windows, after discarding the first few seconds which we typically need to stabilize the filters and make sure PPG data is of good quality. We have 25 data points above, with a Root Mean Square Error of 5 ms for rMSSD and 1 beat for heart rate.

That's all for now

In this post we went over our latest camera based algorithms, the Android implementation and showed data comparing these algorithms to a chest strap. Results have been positive and in line with what we obtained using iPhones, which we validated in a recent paper.

For obvious reasons linked to fragmentation on Android, we cannot test more than a few devices. However, data shown above were obtained using quite old phones (a Nexus 5 for example) and therefore a reasonable assumption is that newer phones can operate at faster speeds and should be able to keep up with the current algorithms rather well.

Additionally, when you use the app, we check that the camera is providing frames at the required frequency and with very limited variations, which is key to provide accurate data. We also check signal quality as we do on iPhones and estimate the validity of the HRV measurement, but on top of this we take special care to check how your hardware is behaving.

​If your phone is unable to sustain the required frequency or lags some frames, we can detect it and inform you. Please check this post for more practical details on using the camera mode on Android, as we will focus only on the algorithms and comparisons with reference systems in this post.

​We will keep working on these models and extending our validations on Android similarly to what we've been doing on iPhones in the past few years. 

Hopefully this is a good step forward in making physiological data accessible and easy to acquire (and interpret) for more people.