Tim Cook is placing a big bet on health. He wants Apple to be known for making people healthier as much as making people buy AirPods. Researchers from the Stanford School of Medicine launched the Apple Heart Study using the Apple Watch in 2017 and enrolled over 400,000 people. Apple released initial results in March, showing that 2,161 people got an alert about an irregular pulse— 0.52% of the total.

Stanford researchers just published a paper in the New England Journal of Medicine about the Apple project, “Large-Scale Assessment of a Smartwatch to Identify Atrial Fibrillation.”

Researchers concluded that the Watch is pretty good at detecting irregular heart beat, which is a significant advance in medical technology. That’s great for people who have their own Apple Watch as well as the people who can get a watch through their Medicare plan.

The study also highlights the limitations of using the Watch for clinical trials. Researchers relying on the Watch to notice health problems face some of the same limitations traditional researchers. Here are three flaws found in the virtual health study that Apple should fix next time around.

Wrong demographic

About half of the people in the study were under 40. Only 6% were over 65. Atrial fibrillation is most common in people over 65, meaning this study may be “the opposite of a desirable age profile for a study of atrial fibrillation,” as Edward W. Campion, M.D., and John A. Jarcho, M.D. wrote in an editorial about the research paper.

In “Watched by Apple,” they also noted, “It’s difficult to draw any conclusions about the true frequency of atrial fibrillation, since only 21% of those with irregular pulse notifications based on monitoring by the smartwatch subsequently returned the ECG patch for analysis.”

It’s easy to see why Apple and Stanford let so many people join such a novel project. People were excited to use their personal tech to contribute data to a first-of-its-kind study. Now that the initial thrill has worn off, it’s time to be more selective with enrollment. The data will be worthless if half the participants in the Apple/NIH women’s health study are men.

High drop-out rate

As soon as the heart health study got real, people started dropping out. When an individual got an alert about an irregular heart rhythm, he or she had to do more than wear the Apple Watch. The person was supposed to attend a telemedicine visit and wear an electrocardiography (ECG) patch for up to seven days. These two additional tasks were still convenient but even those two responsibilities were too much.

According to Business Insider, about 79% of people who got an initial notice of an irregular heartbeat didn’t complete the study or were later excluded from it. People either dropped out or didn’t do the telemedicine visit.

Only 450 people among the 2,161 who got alerts returned the patches. Researchers determined that in this group, atrial fibrillation was present in 34% overall and in 35% of participants 65 years of age or older. The confidence interval was 97.5% for both groups, which is the score researchers want to get.

Some of those alerts were undoubtedly false and could be dismissed. The watches in the Stanford study used an optical sensor to detect irregular rhythms but the newer watches use a special sensor designed to monitor heart beats. That should reduce the number of false positives. However, there should be some mechanism in place to bug people who got the alert and did nothing.

No follow-ups

There are dozens of digital health companies using reminders, alerts, online groups, and text-based coaching to get people to take better care of themselves. There are apps and nudges to get people to do everything from taking their meds as prescribed to completing physical therapy exercises to documenting their feelings.

In the Apple Watch study, people who got an alert about an irregular heart rhythm didn’t get a second alert or any kind of follow-up. Undoubtedly, some of those alerts were false alarms, and rightly dismissed but not all of them.

This is the perfect moment to bug a person with a second reminder, or if the person is in a clinical trial, trigger a notification to a nurse. Certainly, clinical trials that are fully digital with no human monitoring are cheaper than trials that have nurses or physician assistants or case workers tracking participants. However, coaching from a trained medical professional may be better for changing an individual’s behavior in the long term.

Of course, for the people who got an alert and completed all the follow-up tasks, this is a game-changing technology, just like all the people Apple finds for its marketing videos. However, Apple should use its power and business priorities to improve the current standards of clinical trials, not prolong the same weaknesses. At the same time, Apple and researchers working with them should incorporate the effective techniques of the more traditional system. Personal devices like the Apple Watch can be an early warning sign that we don’t have now, but they can’t replace human interactions entirely.

Apple just launched three new health studies, so the Health team has the perfect opportunity to fix these flaws from the heart study.

This Apple Watch close-up shows the heart monitor, displaying a heart rate of 83 beats per minute.
Image:TraceyAPhotos / Getty Images