Heart Rate tracking is a feature which the majority of smartwatches provide nowadays. How can these small hand-worn devices record the heart activity and calculate the heart rate?
Humans can feel the heart pulse just by touching in the wrist. Similarly a smartwatch can record or measure the heart’s pulse. But the mechanism behind the smartwatch sensor is different than the one humans use.
Smartwatches are equipped with photoplethysmography (PPG) sensors. These are optical sensors observing the blood flow in the skin underneath the sensor’s location (Figure 1). In its simplest form, a PPG sensor is comprised of a light emitter, an LED, and a receiver, i.e. a light sensor. The LED is emitting light into the skin, which travels through the hand’s soft tissues. A portion of the light is absorbed by the body and another is reflected back to the sensor.
Figure 1: Photoplethysmography sensor on skin. The sensor’s LED is emitting light into the skin and a proportion of it is reflected back to the light sensor. The amount of light detected by the sensor is propotional to the arterial flow, hence the PPG sensor can sense the Blood Volume Pulse.
The amount of reflected light that is captured by the optical sensor is proportional to the artieral pulsating flow, forming the Blood Volume Pulse (BVP) signal. This is the part of the PPG signal which can be attributed to the heart’s operation. Let’s see an example of a PPG signal in Figure 2 taken from the PPGDalia dataset. 8-second durations of PPG data are quite common for extracting heart rate.
Figure 2: An example of clean photoplethysmography signal. Left: The Blood Volume Pulse signal recorded by the PPG sensor is clearly visible as it progresses through time. Right: Using the Fourier transform we can estimate the periodicity of the signal. The frequency where the majority of energy is concentrated corresponds to the sample’s frequency and by extension to the heart rate.
Similarly to how we, as humans, can manually count pulses per minute to extract the heart rate, the smartwatch can also track the heart’s pulses. One simple way is to count the number of peaks in the PPG signal. For this example, we count 11 peaks over the duration of 8 seconds, corresponding to 82.5 Beats per Minute (BPM).
In essence, extracting the heart rate means estimating the periodicity of the signal. A more robust, but still straightforward, approach is to plot the energy of the signal as a function of frequency, using the Fourier transform. In our example, 82.5 BPM is the frequency with the highest energy (Figure 2 – Right), indicating this example’s heart rate. We can also notice a peak at 2 • Heart Rate, a behavior which is expected in good quality PPG signals.
Unfortunately, the PPG sensor’s output can be severly damaged by the hand’s movement. Movement can affect the sensor’s readings in two ways:
- It causes the sensor to physically slide on top of the skin, changing the light’s pathway and generating additional oscillations.
- Venus and Capillary flow, along with soft tissues get affected by the accelerations. Although under normal situations their contribution to the PPG signal is considered constant, during motion it can be varying over time. This results in the PPG recording additional hand-motion information.
Let’s check an example of a PPG signal affected by motion artifacts, Figure 3. We also indicate the ground truth HR as measured from gold-standard electrocardiogram (ECG). Multiple components (parts of the signal) are present which do not align with the HeartRate or 2 • HeartRate, like in Figure 2. In this example we cannot be sure of the heart rate by looking at this ppg signal.
Figure 3: An example of photoplethysmography signal affected by motion artifacts. Left: We can no longer clearly see the periodic Blood Volume Pulse signal as in Figure 3. Right: In this case the Fourier transform also cannot provide much insight. The true ground truth heart rate, around 100 BPM as indicated by the orange circle, is not captured by the PPG sensor. Rather, the majority of the signal’s energy is concentrated in lower frequencies.
In some cases, e.g. during periodic hand motion, measuring the smartwatch acceleration can help us determine the part of the PPG which is generated by movement. Accelerometers are sensors which are often found in smartwatches, allowing the watch to measure acceleration.
Figure 4 presents one such example. Check how the oscillations found in PPG are almost the same as the ones in the acceleration. The majority of information in this sample is just motion. This is why several algorithms extracting heart rate from PPG, also use the acceleration to get an idea of what motion artifacts to expect in the PPG. In these cases, the acceleration acts as a reference signal to the motion artifacts captured by the PPG sensor.
Figure 4: Example of PPG signal (blue) where the acceleration (orange) is providing information about the motion artifacts in the PPG. We can see how the acceleration (almost) completely matches the PPG signal.
A plethora of methods have been proposed to robustly estimate the periodicity / heart rate of PPG signals while minimizing the effect of motion artfacts. Such methods process the PPG and/or the acceleration signals to calculate the heart rate. In the available literature, we can distinguish two main categories of available approaches:
- Signal Processing Methods. Here researchers try to model the behavior of the PPG signal and the motion artifacts. Then they use their model to estimate the heart rate, while also trying to reduce the effect of motion.
- Deep Learning Models. In this case researchers are building generic deep models, e.g. Convolutional Neural Networks, which are then trained on PPG/heart rate data. This way, the researchers do not need to meticulously hand-craft a model, like in the Signal Processing case. Instead, they feed to the model a large amount of inputs: the PPG (occasionally acceleration as well) and outputs: the corresponding heart rate. The model then learns how to extract the heart rate from the input data.
The Deep Learning models are currently the state of the art achieving good results at heart rate tracking. However, one major disadvantage found in the current literature is that the design and training of the Deep Learning models does not utilize prior information about PPG, Heart Rate and Motion Artifacts.
In our work we have identified three such key-prior insights which are crucial for robust heart rate estimation. Along with these key insights we have also devised mechanisms to integrate them into our model, which we call KID-PPG: Knowledge Informed Deep Learning – PPG. Through our exploration we demonstrate that taking into account prior, problem-specific, information not only boosts the model’s performance but also its robustness.