The Limits Hidden in the Device

A wearable result can be correct inside its original study and still fail as a portable claim. The subject here is transportability: whether a signal measured by one device, in one population, with one set of labels and missing-data rules, means the same thing elsewhere. The practical task is to diagnose the failure before applying the finding—not to dismiss every limitation as a ritual disclaimer, but to identify which new validation would change the conclusion.

A named metric can hide a device-specific measurement

The first transport question is deceptively simple: did the new study measure the same thing? “Sleep stage,” “heart rate,” “steps,” and “temperature” sound like stable biological objects, but a wearable reports an estimate produced by a particular sensor and algorithm. A Fitbit sleep-stage value is calculated from heart rate and movement. Oura supplies its own algorithmic measures of heart rate, heart-rate variability, and sleep. A study built on either vendor’s output inherits the vendor’s definition and error pattern.

The All of Us resource makes this dependence visible. Researchers received Fitbit summary metrics rather than raw accelerometry. That choice can help standardize analyses within the resource, but it prevents researchers from reconstructing the original signal or directly comparing the underlying sensor behavior with another cohort. The resource also contains 41 Fitbit models, and device information was not available for every participant. A measure averaged across those records may therefore combine devices with different sensors, algorithms, and measurement limits.

The sleep study’s validation discussion shows why the name of the construct is not enough. Cited comparisons with polysomnography found no significant difference for total sleep time or deep sleep in the largest referenced validation study, but Fitbit underestimated REM sleep by 11.4 minutes. That supports a bounded statement about particular measures and a particular device family. It does not establish that another brand’s REM estimate has the same bias, or that a Fitbit-derived association can be reproduced with a different sleep algorithm.

Reference methods do not erase the problem; they define the comparison being made. If a device estimate agrees with a laboratory method for total sleep time, researchers can be more confident that the estimate captures that construct under the tested conditions. They still need to ask whether the agreement holds in the population and setting of interest, whether the study uses a different feature such as irregularity or stage proportion, and whether the remaining error changes the health association. Agreement for measurement is not automatic agreement for a downstream clinical claim.

The same issue appears when the sensor interacts differently with different bodies or movements. The resource paper describes evidence that heart-rate measurement may be less reliable for people with darker skin tones and that step estimation may be less accurate for people with irregular gait patterns. These are not reasons to declare the device universally inaccurate. They are reasons to ask whether the measurement error is evenly distributed across the groups being compared. If it is not, a model may learn a mixture of physiology, sensor performance, and participation patterns.

A proprietary algorithm creates an additional reproducibility boundary. A researcher can preserve the input, code, and analysis while still being unable to inspect the manufacturer’s stage classifier or rebuild its raw signal. When the algorithm changes, or when another vendor’s algorithm uses a different definition, the visible variable name may stay the same while the measured construct shifts.