A new CHAT release is now available on sleepdata.org. There were not a significant number of changes made to the dataset, although we importantly removed a small number of subjects whose data were struck from the CHAT analytic datasets. Read more below.
Shouldn't it read v0.3.0?
That is correct, we released a newer version (v0.3.0) on September 4, after this post was initially made. Here are some of the changes for the v0.3.0 dataset:
Dear NSRR team,
I have started to work on a subset of edf's from the CHAT study and noticed that for those edf's with a sampling rate of 512 Hz (e.g. 300721 and 300990), the duration stated in the edf header does not correspond to the actual duration of the edf file, which is considerably shorter and with ending times of the edf's frequently around 2:00 in the night. I was wondering if you would have any longer, more complete versions of those files somewhere?
With kind regards
Stephany: I asked one of our resident EDF experts to take a look at this particular issue you raise. I'll report back with the findings.
I just checked the EDFs for 300721 and am failing to reproduce this issue. Both the baseline and followup EDFs for 300721 end at the expected times (5:56:38 and 6:03:13 respectively) and both show a "duration of data record" of 1 when I look at the headers. Can you clarify what you are seeing as the stated duration in the data records? Were the EDFs causing the problem downloaded before Sept 4? Some EDFs were reprocessed for the Sept 4 posting so there is a possibility that the current versions (that I downloaded and checked) are free from this duration bug.
Dear Michael, thanks very much for the fast and helpful reply. It made me realize that my files must have been truncated during the download and indeed 1 GB is my maximum file size for any NSRR EDF. I then tried several ways to manually download a single larger CHAT EDF file but all got truncated at 1 GB. I realize that this is most likely a problem at my end, however, I can and did today download without any problems larger files (> 1.5 Gb) from other internet sites. I tried the usual tricks including limiting the download speed, but in no case could I download a complete EDF file > 1 Gb. I verified that this is specific to the NSRR files and does not happen with other sites. Since you did download the complete file, I am a bit at a loss what to try next. Would you have any idea/advice for me?
With kind regards, steph
I have had issues downloading >1GB files using the Ruby Gem. I used a web browser for those few files instead and it worked fie.
Thanks Mathias! That is what I did too in the end, but when I downloaded them "manually", although it seemed to work without problems, I have now seen that they got truncated to around 1 Gb and the file size did no longer match the length of the recording. What web browser did you use? With kind regards, steph
Steph, It would have been Waterfox or Chrome. Mathias
Stephany and Mathias,
Thanks for the heads-up. We are going to begin more thorough testing of these download cutoff issues this week. We may reach out to you to test any potential solutions we devise.
Hi Stephany and Mathias, thanks for reporting the issue! We updated the configuration for the servers to better handle files over 1GB, and have hopefully resolved the issues. If you run into the same problem going forward, please send us a message and we'll investigate further. Thanks again! Remo
Thanks very much Remo! I tried it right away and the NSRR gem now downloads also the larger files without problems. Thanks as always for the impressively fast and efficient help! Steph
Great to hear! Thanks for following up!
I am working with a random selection from the CHAT data sets and noticed that I did not find any "mixed apnea" annotations in the nsrr or profusion xml files. As an example, I took case 300998 which according to the dataset variable HMANRBP [Number of mixed apnea per hour (NREM, Back, all desaturations)] has a value of 13.3, so there should be at least 1. I also "handsearched" the annotation files but could not locate any mixed apnea annotations. I hope I am not missing something obvious, but from the documentation it seems that mixed apneas were scored.
Thanks and with kind regards
Good catch. This appears to be something we overlooked, but the root cause is fairly obvious in retrospect. What happened is that Mixed Apneas were scored in Compumedics Profusion with an "Unsure Hypopnea" tag. This was a case of the Sleep Reading Center "monkey patching" the software and scoring process to work for CHAT. This distinction in the tags was handled in the data processing steps to output the result datasets, but the XML annotation exports came straight from Profusion. As such, I'm guessing all the Mixed Apneas have been output as Hypopneas in the XML annotation files.
I will add this as an item of discussion for the team. Hopefully the scoring/tools group can come up with a straightforward solution to re-generate the XML with correct annotations. In the meantime I will ask that they write up some documentation about this issue and its cause.
Thanks for raising the issue, and sorry for this oversight!
Edit: These events should actually show up as "Unsure" in the XML files, so luckily they are distinguishable from other respiratory events. We may be able to simply do a find-and-replace to substitute "Mixed Apnea" for "Unsure". In the meantime, though, this should help you identify the Mixed Apnea events in your version of the data.
Dear Mike, thanks much for the fast reply! Glad to hear that I can just re-label the "unsure" events. With kind regards, Steph