Fixing Invalid EDF Dates

Posted by remomueller on December 8, 2016 in Releases

Over the past two days we've updated EDFs across the site due to an issue identified with our original approach in scrubbing the EDF start date of recording.

What Happened

In order to have EDFs deidentified before being posted on the NSRR, the EDFs had their header start date set to "00.00.00". This seemed reasonable as it clearly indicated that the date had been removed, while adhering to the "dd.mm.yy" format for writing a date in the signal header.

As it turns out, this "zero date" is not valid EDF specification1 and caused many EDF viewers to fail to load the EDFs posted on the NSRR.

How We Fixed This

First, we had to identify a new date for EDFs. We settled on the oldest date possible based on the original EDF specification. Due to the limitation of only being able to enter two digits for years, EDFs have a "clipping year" of 1985. This means that a date that is "31.12.84" represents the date 31 Dec 2084, while the date "01.01.85" represents the date 1 Jan 1985. We like to refer to the date "01.01.85" as the "clipping date".

Next, we created a script using edfize2 to update the EDF header field from the "zero date" to the new "clipping date".

How This Affects You

If you downloaded EDFs from the NSRR in one of the following datasets before December 8, 2016, you will need to fix the dates locally using our script, or alternatively redownload the EDFs. The script is identical to what we used to fix the EDFs posted on the NSRR, and is much quicker to run then redownloading the EDFs.

EDFs Date Fixed

  • ccshs Cleveland Children's Sleep and Health Study
  • cfs Cleveland Family Study
  • shhs Sleep Heart Health Study
  • heartbeat Heart Biomarker Evaluation in Apnea Treatment
  • sof Study of Osteoporotic Fractures

The following newer datasets are unaffected by this and have NOT had their EDFs changed:

EDFs Unaffected

  • haassa Honolulu-Asia Aging Study of Sleep Apnea
  • mesa Multi-Ethnic Study of Atherosclerosis

Running the Fix Locally

This script3 is the same for Windows or Mac/Linux and requires Ruby 2.0 or higher and edfize to be installed. (If you've used the NSRR download gem, you already have Ruby installed). Open Command Prompt or Terminal, and navigate to your EDF folder. This script will modify any EDF date that is set to "00.00.00" to "01.01.85".

  1. Right click and "Save As" the script to <my-edf-folder>/rewrite_signal_date.rb
  2. Install edfize
    • gem install edfize --no-document
  3. Navigate to EDF folder
    • cd <my-edf-folder>
  4. Run the date rewrite script
    • ruby rewrite_signal_date.rb

Enter the following command to check that the script ran correctly.

edfize test

Here's sample output when running the above command in the shhs1 folder.

Started

shhs1-200001.edf
  PASS Expected File Size
  PASS Reserved Area Blank
  PASS Valid Date

shhs1-200002.edf
  PASS Expected File Size
  PASS Reserved Area Blank
  PASS Valid Date

...

5793 EDFs, 17379 tests, 0 failures

Summing Up

EDFs downloaded before December 8, 2016 contained invalid dates in their header causing issues when opening these EDFs in viewers. We've provided a script so that you can fix these locally without redownloading all of the EDFs. Moving forward, all EDFs will have the valid "clipping date" set to "01.01.85" to avoid problems in the future.

Thank you to everyone who helped us identify the issue. Please let us know if you encounter any problems, either via email, support@sleepdata.org, or on the forum.

References

  1. EDF specifications are well documented here: http://www.edfplus.info/specs/edf.html
  2. Edfize is a Ruby gem built by the NSRR team used to check for errors in EDF files, including computing "Expected File Size", making sure the header "Reserved Area is Blank", and checking for "Valid Date".
  3. Rewrite EDF Dates, https://gist.github.com/remomueller/9c0f5da531fde34da91a9c013f895fe2
Author Posts
19
all 2 comments
· sorted by
Write a Reply
5 posts
Was this reply useful? Learn more...
   
[-] fsann +1 point · 7 months ago

Thank you for this! Much appreciated!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · 7 months ago

You're welcome, thanks for bringing it to our attention!