Overview

What is the problem being addressed?

Commonly used sleep-wake algorithms for wrist actigraphy tend to overclassify sleep, relative to wake. This overclassification is most evident when wrist actigraphy includes data from outside of an ‘in-bed’ (or ‘rest’) interval, as is the case for longitudinal studies of participants in free-living scenarios. For this reason, sleep-wake algorithms are often used in conjunction with estimates of participant in-bed periods, obtained from sleep diary, visual inspection of actigraphy counts, or a supplemental in-bed detection algorithm. These methods are often used to filter the sleep-wake classifications, setting any epochs outside of an estimated in-bed interval to wake. However, each of these methods carry limitations.

What was the approach to solving the problem?

Here we explored an alternative approach. We sought to develop a new epoch-by-epoch sleep-wake classifier that could be used on data over 24-hour periods without the previously observed bias towards sleep classification. We supposed that commonly used sleep-wake algorithms overclassify sleep due to a focus on in-bed intervals during algorithm development. To this end, we trained and evaluated a temporal convolutional network (TCN) neural network sleep-wake classification model, critically using data with ground-truth sleep-wake labels from 24-hour periods. These ground-truth labels were derived from either 24-hour ambulatory PSG, or nocturnal PSG in combination with laboratory observation of wakefulness during the daytime. Several datasets from our lab were pooled for model training and evaluation.

What NSRR data were used?

The Multi-Ethnic Study of Atherosclerosis (MESA) study was used for supplemental evaluation of the trained classifier. Only participants who consented to allow their data to be used for commercial purposes (2,068 of the 2,237 in the full sample) were used. Of this group, only participants with both PSG staging and concurrent actigraphy, and a way to link the two using the record of PSG/actigraphy overlap within the MESA dataset were used, resulting in 1,698 potentially usable participants, each with 1 night of data. 8 participants were subsequently removed due to having 25% of more of their actigraphy missing or off-wrist during the staged interval, resulting in 1,690 usable participants.

What were the results?

The performance of the TCN classifier was compared to the Oakley actigraphy algorithm as a point of reference. When comparing the data over ∼24 hours without first identifying a candidate in-bed interval, the TCN classifier performed statistically equivalent or better than the Oakley classifier on all measures, including for example improved epoch-by-epoch accuracy, specificity to detecting wake epochs, as well as bias and absolute error in the estimation of total sleep time across the full day. When restricting analysis to only the in-bed interval as derived from PSG, the TCN was still favorable on several metrics, though the Oakley classifier was favorable in terms of bias in sleep efficiency and wake after sleep onset. While the MESA dataset available in the NSRR does not have PSG staged actigraphy over 24-hours, the staged data that was available was used as a supplemental evaluation of the trained classifier; the TCN was favorable on some metrics, but the Oakley classifier was favorable on others.

What were the conclusions and implications of this work?

The work suggests that it is feasible to classify wrist actigraphy into sleep-wake at 30-second epochs throughout a full day, without first identifying candidate sleep intervals. Application of the trained model to the MESA dataset suggests that there is room for improvement in the model in terms of generalizing to other datasets. Gathering a larger amount of data with ground truth over 24-hours may facilitate models with improved generalization.

Are there any tools available?

Preprocessing code, model development code, and trained model files are available here

Resources/multimedia

The manuscript is open access here

I would like to point out that there is also a Corrigendum with some value corrections

Paper Summary

Commonly used sleep-wake algorithms for wrist actigraphy tend to overclassify sleep, relative to wake. This overclassification is most evident when wrist actigraphy includes data from outside of an ‘in-bed’ (or ‘rest’) interval, as is the case for longitudinal studies of participants in free-living scenarios. For this reason, sleep-wake algorithms are often used in conjunction with estimates of participant in-bed periods, obtained from sleep diary, visual inspection of actigraphy counts, or a supplemental in-bed detection algorithm. These methods are often used to filter the sleep-wake classifications, setting any epochs outside of an estimated in-bed interval to wake.

Here we explored an alternative approach. We sought to develop a new epoch-by-epoch sleep-wake classifier that could be used on data over 24-hour periods without the previously observed bias towards sleep classification. We supposed that commonly used sleep-wake algorithms overclassify sleep due to a focus on in-bed intervals during algorithm development. To this end, we trained and evaluated a temporal convolutional network (TCN) neural network sleep-wake classification model, critically using data with ground-truth sleep-wake labels from 24-hour periods. These ground-truth labels were derived from either 24-hour ambulatory PSG, or nocturnal PSG in combination with laboratory observation of wakefulness during the daytime. Several datasets from our lab were pooled for model training and evaluation. Although not used for model training as it doesn’t include PSG staging over 24-hour intervals, the Multi-Ethnic Study of Atherosclerosis (MESA) study from the NSRR was used for supplemental evaluation of the trained classifier.

The performance of the TCN classifier was compared to the Oakley actigraphy algorithm as a point of reference. The classifiers were compared over both 24-hour intervals, and within the limits of the in-bed period, with the in-bed periods defined by the concurrent PSG staging. The classifiers were compared using both epoch-by-epoch measure of performance such as accuracy, sensitivity, specificity, etc., and in their ability to reproduce sleep metrics such as total sleep time, wake after sleep onset, etc. When comparing the data over ∼24 hours without first identifying a candidate in-bed interval, the TCN classifier performed statistically equivalent or better than the Oakley classifier on all measures, including for example improved epoch-by-epoch accuracy, specificity to detecting wake epochs, as well as bias and absolute error in the estimation of total sleep time across the full day. When restricting analysis to only the in-bed interval as derived from PSG, the TCN was still favorable on several metrics, though the Oakley classifier was favorable in terms of bias in sleep efficiency and wake after sleep onset. While the MESA dataset available in the NSRR does not have PSG staged actigraphy over 24-hours, the staged data that was available (the in-bed interval and sometimes a few additional hours before and after) was used as a supplemental evaluation of the trained classifier. Within the MESA dataset, the TCN was favorable on some metrics, but the Oakley classifier was favorable on others.

The work suggests that it is feasible to classify wrist actigraphy into sleep-wake at 30-second epochs throughout a full day, without first identifying candidate sleep intervals. Application of the trained model to the MESA dataset suggests that there is room for improvement in the model in terms of generalizing to other datasets. Gathering a larger amount of data with ground truth over 24-hours may facilitate models with improved generalization.

Fig 1. Polysomnography (PSG)-derived total sleep time (TST) over the ∼24-hour interval, compared to the Oakley or temporal convolutional network classifiers. Left column shows PSG vs. classification for each classification type, and for visual reference includes a linear regression of predicted TST on true TST across days (without accounting for clustering within participants; black line). Right column shows Bland-Altman PSG vs. classification error (classification TST - PSG TST), computed using a mixed-effects approach accounting for clustering of days within participants. Proportional bias was evaluated but did not reach statistical significance for either classifier on ∼24-hour TST. Indicated are bias (black line) and the upper and lower limits of the 95% limits of agreement (dashed black lines) with 95% confidence intervals for each shaded in gray.

Fig. 2. Polysomnography (PSG)-derived sleep metrics restricted to the in-bed interval (as defined by registered polysomnography technicians or algorithm in the case of EcoSleep), compared to values from the Spectrum classification outputs or our temporal convolutional network classifier restricted to the same interval. Left column shows the scatterplot of PSG vs. classification derived metric for each classification type, and for visual reference includes the linear regression of predicted metric on true metric (without accounting for clustering within participants; black line). Right column shows a Bland-Altman plot of PSG metric vs. classification metric error (classification metric - PSG metric), computed using a mixed-effects approach accounting for clustering of days within participants. Plotted are sleep onset latency (SOL), total sleep time (TST), sleep efficiency (SE), and wake after sleep onset (WASO). Proportional bias was evaluated and reached significance for all statistics except for TST, which did not reach significance for either classifier. Indicated are bias (black line) and the upper and lower limits of the 95% limits of agreement (dashed black lines), with 95% confidence intervals for each shaded in gray. The EcoSleep dataset was excluded from comparisons of SOL or SE, as the method of imputing in-bed labels for the EcoSleep data cannot identify pre-sleep wakefulness within the in-bed interval.

Citation: Roberts, D. M., Schade, M. M., Master, L., Honavar, V. G., Nahmod, N. G., Chang, A.-M., Gartenberg, D., & Buxton, O. M. (2023). Performance of an open machine learning model to classify sleep/wake from actigraphy across ∼24-hour intervals without knowledge of rest timing. Sleep Health, 9(5), 596–610. https://doi.org/10.1016/j.sleh.2023.07.001

Authors

Paper Authors: Daniel M. Roberts PhD^a,b , Margeaux M. Schade PhD^a , Lindsay Master MAS^a , Vasant G. Honavar PhD^c , Nicole G. Nahmod MPH, MMS, PA-C^a , Anne Marie Chang PhD^a , Daniel Gartenberg PhD^b , Orfeu M. Buxton PhD^a

^a Department of Biobehavioral Health, The Pennsylvania State University, University Park, Pennsylvania, USA

^b Proactive Life, Inc, New York, New York, USA

^c Faculty of Data Sciences, College of Information Science and Technology, The Pennsylvania State University, University Park, Pennsylvania, USA

Guest Blogger: Dr. Daniel Roberts, PhD