Over the past two days we've updated EDFs across the site due to an issue identified with our original approach in scrubbing the EDF start date of recording.
In order to have EDFs deidentified before being posted on the NSRR, the EDFs had their header start date set to "00.00.00". This seemed reasonable as it clearly indicated that the date had been removed, while adhering to the "dd.mm.yy" format for writing a date in the signal header.
As it turns out, this "zero date" is not valid EDF specification1 and caused many EDF viewers to fail to load the EDFs posted on the NSRR.
First, we had to identify a new date for EDFs. We settled on the oldest date possible based on the original EDF specification. Due to the limitation of only being able to enter two digits for years, EDFs have a "clipping year" of 1985. This means that a date that is "31.12.84" represents the date 31 Dec 2084, while the date "01.01.85" represents the date 1 Jan 1985. We like to refer to the date "01.01.85" as the "clipping date".
Next, we created a script using
edfize2 to update the EDF header field from the "zero date" to the new "clipping date".
If you downloaded EDFs from the NSRR in one of the following datasets before December 8, 2016, you will need to fix the dates locally using our script, or alternatively redownload the EDFs. The script is identical to what we used to fix the EDFs posted on the NSRR, and is much quicker to run than redownloading the EDFs.
EDFs Date Fixed
ccshsCleveland Children's Sleep and Health Study
cfsCleveland Family Study
shhsSleep Heart Health Study
heartbeatHeart Biomarker Evaluation in Apnea Treatment
sofStudy of Osteoporotic Fractures
The following newer datasets are unaffected by this and have NOT had their EDFs changed:
haassaHonolulu-Asia Aging Study of Sleep Apnea
mesaMulti-Ethnic Study of Atherosclerosis
This script3 is the same for Windows or Mac/Linux and requires Ruby 2.0 or higher and
edfize to be installed. (If you've used the NSRR download gem, you already have Ruby installed). Open Command Prompt or Terminal, and navigate to your EDF folder. This script will modify any EDF date that is set to "00.00.00" to "01.01.85".
gem install edfize --no-document
Enter the following command to check that the script ran correctly.
Here's sample output when running the above command in the
Started shhs1-200001.edf PASS Expected File Size PASS Reserved Area Blank PASS Valid Date shhs1-200002.edf PASS Expected File Size PASS Reserved Area Blank PASS Valid Date ... 5793 EDFs, 17379 tests, 0 failures
EDFs downloaded before December 8, 2016 contained invalid dates in their header causing issues when opening these EDFs in viewers. We've provided a script so that you can fix these locally without redownloading all of the EDFs. Moving forward, all EDFs will have the valid "clipping date" set to "01.01.85" to avoid problems in the future.