Invalid/erroneous EDF files in SHHS

5 posts
Was this reply useful? Learn more...
   
[-] fsann +2 points · 9 months ago

Hi,

I'm using the latest (1.58) EDFBrowser (https://github.com/Teuniz/EDFbrowser) to view the EDF-files in SHHS. The files won't open because the start dates are invalid (00.00.00). When I use the EDF Editor and Translator (https://sleepdata.org/tools/edf-editor-and-translator) and manually change the start date to, for example, 01.01.01, the files open just fine. Seems like I'm not the first one to run into this problem: https://sleepdata.org/forum/getting-started-with-edf-files/1#comment-139

I understand why these dates have been set (de-identification) and looking at the EDFs specs, I could not see anything that prohibits day and month values to be set to 00 per se. However, I do feel that this date is a poor choice as it is confusing, makes little sense, and makes further analysis unnecessarily difficult (for example, reading the EDF-files with Python and using the datetime-package to handle the start date is not possible).

I would suggest that you use January 1st 1970 00:00:00 (Epoch time 0) as the start point for the dates.

PS. checking for errors in the EDF-files using EDF Editor and Translator shows some errors. For example, the physical max and min are switched for channel THOR RES in file shhs1-200001.edf.

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · 9 months ago

Hello fasann,

Thanks for reaching out on the forum! We currently know about the issue with the invalid dates, and are going to work our way back through some of the originally posted datasets to set the time to 01.01.85. We have already started using this new approach in the MESA EDFs that are posted here. This date is chosen specifically to adhere to the two digit year clipping present in regular EDFs (not EDF+), as 1985 is the "clipping year", and as such the earliest date we can represent in a regular EDF.

We're also releasing an update to our EDF ruby tool edfize that we use internally to check that validity of EDFs tomorrow, that will include checking for invalid dates. I'll put together an example Ruby script that could demonstrate how to rewrite the date in the header as well, I'll post it in this thread in the next couple of days. Example gists using edfize: Check EDFs for unusual Physical Dimensions Check EDFs for signal labels that include a dash

I'll pass along to the team the issue with the switched physical min and max, however, I'm not sure there are any current plans to address that, as I believe it does not go against the EDF specifications, and represents the leads being reversed when collecting the signal initially. I think most EDF viewers provide the ability to flip a signal for cases like these.

Hope that helps, and let us know if you have any other questions! Thanks!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +1 point · 9 months ago

Hello fasann,

I've put together a simple script that will update the start_date_of_recording for EDFs.

The script is located here: https://gist.github.com/remomueller/9c0f5da531fde34da91a9c013f895fe2

You will need to have edfize v0.4.0 or higher installed as well: gem install edfize --no-document

To run navigate in your terminal or command prompt to the folder of EDFs, and then run ruby rewrite_signal_date.rb

It will rewrite the EDFs in place, so make sure you run this on a copy of your data.

We'll also be updating the EDFs on the SHHS dataset, however this may take some time as we will run other tests against the EDFs before reposting as well.

Hope this helps!

5 posts
Was this reply useful? Learn more...
   
[-] fsann +0 points · 9 months ago

Amazing!

Thanks for the quick update and the tool! This will save me a lot of time!

About the physical dimensions: I realised after posting that it is correct to have them switched (negative gain), but in this case the EDF Editor and Translator is giving a false positive. So the data is good, but the tool has a bug.

Keep up the good work, really appreciate it!

2 posts
Was this reply useful? Learn more...
   
[-] Shiqiang Tao +1 point · 8 months ago

Hello fasann,

Thank you very much for your valuable feedback on the EDF Editor. We have removed that false positive warning and include this in our new release for the tool EDF-Editor/Translator. Here is the link to the new release is: https://github.com/nsrr/edf-editor-translator/releases/tag/2.1.0 . Thank you for helping improve the quality of our tools and don't hesitate to let us know of anything else you may run into.

Thanks!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · 9 months ago

Glad to hear it! We're also working on rechecking dates throughout our other datasets and reposting EDFs that had the issue, and I'll be putting together a general post of details soon that includes which datasets were updated. Thanks for letting us know, and don't hesitate to let us know of anything else you may come across. Thanks!

Edit: Blog Post now Live

Write a Reply