mrueschman

Kelsey,

Sure, if you could pass along the syntax that would be helpful. You can send it to me at mrueschman@bwh.harvard.edu.

Did you do any manual checks to see which of the weekend sleep duration values (ones generated with code above or ones with your code) seemed objectively "right" (based on the component variables)?

I should have a chance in the next week or so to dive in deeper and explore the issue myself. Thanks for keeping us updated!

I guess I didn't remove the correction from the code. It was this snippet (I corrected the identifier link to use our obfuscated ID):

*manual fix for this one (otherwise end_dur_hr is negative);
         if  obf_pptid = 802529 then ENDWAKE_time2 = dhms(date()+1,hour(ENDWAKE_time),minute(ENDWAKE_time), 0 );

I didn't attempt to run everything without the manual fix to see if the result is indeed negative for this subject.

I dug through the archives and found some SAS code from long ago. I went ahead and tried the below against the CFS dataset here on the NSRR and was able to replicate dayslp_dur_hr and other related variables. At a glance, in comparing with daybed and daywake, the results look correct.

There was a manual correction as well to one of the weekend times -- I left that out of the code snippet.

If you continue to encounter differences and find that the dayslp_dur_hr values do not correctly line up with the component variables, please let us know and we can look to make dataset corrections. Thanks!

SAS macros:

* macro to create time variables from 24-hour times collected as integers;
%macro fixtimes(varin,varout);
        if &varin >=0 then do;
                 &varin.c = trim(left(put(&varin, 8. )));
                 &varout = input(substr(trim(left(reverse(substr(trim(left(reverse(&varin.c)!! '0000' )), 1 , 4 )))), 1 , 2 )
                                   !! ':' !! substr(trim(left(reverse(substr(trim(left(reverse(&varin.c)!! '0000' )), 1 , 4 )))), 3 , 2 ), time5. );
        end;
        format &varout time5. ;
        drop &varin.c;
%mend ;

Transformation code:

*******************************************************************************;
    * SELF-REPORTED SLEEP TIME
    *******************************************************************************;
        * FIX SELF-REPORTED BED AND WAKE TIMES (WEEKDAYS);
        %fixtimes(DAYBED ,DAYSLP_time);
        %fixtimes(DAYWAKE ,DAYWAKE_time);

         * if slptime before midnight and waketime after midnight set date as previous day;
         if hour(DAYSLP_time) > 12 and hour(DAYWAKE_time) <=12 then DAYSLP_time2 = dhms(date()- 1 ,hour(DAYSLP_time),minute(DAYSLP_time), 0 );
         * if both slptime  and waketime before midnight then use current day;
         else if hour(DAYSLP_time) > 12 and hour(DAYWAKE_time) >12 then DAYSLP_time2 = dhms(date() ,hour(DAYSLP_time),minute(DAYSLP_time), 0 );
         * otherwise use current day;
         else DAYSLP_time2 = dhms(date(),hour(DAYSLP_time),minute(DAYSLP_time), 0 );

         * if   waketime at midnight then waketime use next day;
         if  hour(DAYWAKE_time) =0 then DAYSLP_time2 = dhms(date()+1 ,hour(DAYSLP_time),minute(DAYSLP_time), 0 );
         * otherwise use current day for waketime;
         else DAYWAKE_time2 = dhms(date(),hour(DAYWAKE_time),minute(DAYWAKE_time), 0 );

        * calculate midpoint of bed and wake times;
         DAYMID_time2 = DAYSLP_time2 + (DAYWAKE_time2 - DAYSLP_time2)/2;
         DAYMID_time = timepart(DAYMID_time2);

        format daymid_time time5. DAYSLP_time2 DAYWAKE_time2 DAYMID_time2 datetime16.;

         * calculate time in bed as difference between wake and bed time (accounting for change in midnight);
         DAYSLP_dur_mn  = (DAYWAKE_time2 - DAYSLP_time2)/ 60;
         DAYSLP_dur_hr2  = (DAYWAKE_time2 - DAYSLP_time2)/ 3600;
         Label    DAYSLP_dur_mn  ='calculated sleep duration (minutes) during weekday-by DAYWAKE -DAYSLP '
                  DAYSLP_dur_hr  ='calculated sleep duration (hourss) during weekday -by DAYWAKE -DAYSLP ';

        drop daywake_time2 dayslp_time2 daymid_time2;
        *****************************************************************************************************************;

        * FIX SELF-REPORTED BED AND WAKE TIMES (WEEKENDS);
        %fixtimes(ENDBED ,ENDSLP_time);
        %fixtimes(ENDWAKE ,ENDWAKE_time);

         * if slptime before midnight and waketime after midnight, set date as previous day;
         if hour(ENDSLP_time) > 12 and hour(ENDWAKE_time) <=12 then ENDSLP_time2 = dhms(date()- 1 ,hour(ENDSLP_time),minute(ENDSLP_time), 0 );
         * if both slptime  and waketime before midnight then use current day;
         else if hour(ENDSLP_time) > 12 and hour(ENDWAKE_time) >12 then ENDSLP_time2 = dhms(date() ,hour(ENDSLP_time),minute(ENDSLP_time), 0 );
         * otherwise use current day;
         else ENDSLP_time2 = dhms(date(),hour(ENDSLP_time),minute(ENDSLP_time), 0 );

         * if waketime at midnight then waketime use next day;
         if  hour(ENDWAKE_time) =0 then ENDWAKE_time2 = dhms(date()+1,hour(ENDWAKE_time),minute(ENDWAKE_time), 0 );
         * otherwise use current day for  wake time;
         else ENDWAKE_time2 = dhms(date(),hour(ENDWAKE_time),minute(ENDWAKE_time), 0 );
         *manual fix for this one (otherwise end_dur_hr is negative);
         if  obf_pptid = 802529 then ENDWAKE_time2 = dhms(date()+1,hour(ENDWAKE_time),minute(ENDWAKE_time), 0 );

        * calculate midpoint of bed and wake times;
         ENDMID_time2 = ENDSLP_time2 + (ENDWAKE_time2 - ENDSLP_time2)/2;
         ENDMID_time = timepart(ENDMID_time2);

         format ENDmid_time time5. ENDSLP_time2 ENDWAKE_time2 ENDMID_time2 datetime16.;

         * calculate time in bed as difference between wake and bed time (accounting for change in midnight);
         END_dur_mn  = (ENDWAKE_time2 - ENDSLP_time2)/ 60;
         END_dur_hr  = (ENDWAKE_time2 - ENDSLP_time2)/ 3600;
         Label    END_dur_mn  ='calculated sleep duration (minutes) during weekend -by ENDWAKE  - ENDSLP '
                  END_dur_hr  ='calculated sleep duration (hourss) during weekend-by ENDWAKE  - ENDSLP ';

        drop endwake_time2 endslp_time2 endmid_time2;
        *****************************************************************************************************************;

Stephany and Mathias,

Thanks for the heads-up. We are going to begin more thorough testing of these download cutoff issues this week. We may reach out to you to test any potential solutions we devise.

Stephany: I asked one of our resident EDF experts to take a look at this particular issue you raise. I'll report back with the findings.

It looks like these midsleep times were likely calculated from questions on the Baseline Lab Adult-Child Questionnaire (page 7, question 43). Those questions asked for average times (over the past month) that the participant went to bed and woke up on both weekdays and weekends.

Are you asking about the Cleveland Family Study (CFS) dataset? I was looking through the CFS variables and did not see a school start time variable.

As for family health history, some of this could be ascertained from the medical history variables and a variable like familyi, which is an identifier for family units.

Let me know if I can help with any other questions!

The series of ndes*ph variables may be tricky because they (can) include desaturations that occurred in wake time.

As to your assumptions:

1. You could use these to calculate ODI (at 3%, 4%, etc.) if the desaturation events were scrubbed from wake time.
2. Typically, the desaturation events will be output to the XML annotation files even when they are independent of apneas/hypopneas/arousals.

That said, SHHS is a bit tricky and presents difficulties with the above. We have lost data from the SaO2 channel over time due to the studies being converted from Compumedics Replay (original MS DOS scoring program used 20 years ago) to Profusion at some point. Only desaturations that were associated with other events were retained, for whatever reason. In newer datasets (e.g. CHAT) the XML files will contain desaturation events that occurred independent of other events.

Let me know if you'd like more clarification -- I can call upon some of my colleagues who know more about SHHS' history and the utility of these variables. Thanks for inquiring!

@fsann I updated shhs1-204871-profusion.xml to remove that invalid character, as suggested by the resident expert here. Were you doing some sort of batch processing across all the files when you encountered this one that wouldn't parse? Hopefully there aren't any others that sneaked through our processing steps!

Thanks again for bringing it to our attention.

Thanks for raising this issue! I can confirm what you are seeing in that particular file. It looks like this for me:

I know we have seen this before in our XML exports from the Profusion software. I will talk it over with my colleague who has worked extensively with the EDF/XML files we have posted. If the XML files otherwise appears intact/correct, I believe our crude "fix" will simply be to remove the invalid character and re-save the file. You could probably go ahead and do that in the meantime.

I'll post an update when we clear things up!