How to download full datasets

2 posts
Was this reply useful? Learn more...
   
[-] Robert Thomas +0 points · almost 3 years ago

What is the process to download a full dataset (e.g., all CHAT edf's)

Robert Thomas

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · almost 3 years ago

Hi Robert,

We provide an official download tool called nsrr which is a Ruby gem.

Installation instructions for the gem are here: https://github.com/nsrr/nsrr-gem

In short, once you have Ruby installed, you can do the following two steps:

Install nsrr Ruby Gem

gem install nsrr --no-document

Finally, to download only the EDFs for chat, you can type:

nsrr download chat/polysomnography/edfs

The downloader supports pause and resume, so that it will pick up where it last left off.

NSRR Gem Console Output

The above is a screenshot of the app running through the CHAT folder

Additionally, each page in a dataset's download folder contains the specific nsrr command needed to download that particular folder once you have signed in and filled out a Data Access and Use Agreement for that particular dataset.

2 posts
Was this reply useful? Learn more...
   
[-] Robert Thomas +0 points · almost 3 years ago

Looks like I need a download token. Robert

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · almost 3 years ago

After signing in, you can access your download token here: https://sleepdata.org/token

Additionally, the download token will only work for datasets for which you have an approved DAUA.

2 posts
Was this reply useful? Learn more...
   
[-] Zhu Wang +0 points · over 2 years ago

hello , i followed your instructions as you said,and i type the website in the bowser,which returned the download token. however ,i could type the download token in the console, the only thing i can do is to press the enter button.unfortunately, it can create the folder but any contens.what should i do?

2 posts
Was this reply useful? Learn more...
   
[-] Zhu Wang +0 points · over 2 years ago

sorry , i could not type the download token in the console.....

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · over 2 years ago

Hi Zhu,

The easiest way is to copy and paste the download token from the website directly into the console when using the NSRR gem. Make sure to include the number before the dash - as well as the rest of the string. The NSRR gem hides the input on the screen, but will still register the token if entered correctly. The gem still allows you to download publicly available files (like forms and data dictionaries) without entering a password which is why you sometimes see the folder structure generated without any files in it. A future version of the gem (v0.2.0+) should provide better feedback if the password was entered correctly or not. Hope that helps and thanks for the feedback!

For others interested in using the NSRR gem for downloading large numbers of files easily, you can view a short tutorial here:

https://sleepdata.org/demo#download-data

10 posts
Was this reply useful? Learn more...
   
[-] Stephany Fulda +0 points · about 2 years ago

Dear NSRR team,

I just got my DAUA approved and a mere 20 minutes later I already downloaded the first data set with the nsrr Ruby Gem without any problems. So, thanks very much for the absolutely clear and easy to follow tutorial! The only question I would like to ask is whether there is any way to specify a different download path? Thanks very much!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · about 2 years ago

Dear Stephany,

Glad to hear that you were able to use the nsrr gem!

If you want to only download a portion of a dataset, you can do so by navigating to the appropriate subfolder under the dataset files.

For example, if you only want to download datasets (and not EDFs) for SHHS, you can go to the datasets subfolder. Near the bottom of the page you will see a command listed that will only download that subfolder, in this case nsrr download shhs/datasets.

There are also other parameters you can set that you can read about here on our NSRR gem Documentation page.

If you need any further questions, I'd be happy to clarify further, thanks!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +1 point · about 2 years ago

Just as another quick reference, here are a couple of nsrr gem commands to download different datasets:

SHHS Files

  • nsrr download shhs - Download ALL files for SHHS
  • nsrr download shhs/polysomnography/edfs - Download EDFs for SHHS
  • nsrr download shhs/datasets - Download all datasets for SHHS

CHAT Files

  • nsrr download chat - Download ALL files for CHAT
  • nsrr download chat/polysomnography/edfs - Download EDFs for CHAT
  • nsrr download chat/datasets - Download all datasets for CHAT

Also remember to run the command in the same location on your computer, that way the nsrr gem can resume if you already downloaded files from a dataset.

If your current directory in the command prompt or terminal is "~/Desktop" for example, running nsrr download chat/polysomnography/edfs will create a folder in "~/Desktop/chat/polysomnography/edfs". If you then run the full CHAT download command (nsrr download chat) from the "~/Desktop", then it would already see that it has downloaded the EDFs, and continue with "~/Desktop/chat/datasets" and other subfolders.

10 posts
Was this reply useful? Learn more...
   
[-] Stephany Fulda +0 points · about 2 years ago

Dear Remo,

got it! As you explained, the files will be downloaded to the current directory in the command prompt at the time of execution. Thanks very much, again. If I may follow up with another question - could you also only download selected edf files with the nsrr gem? I tried some variations of the nsrr download command but got nowhere. Thanks!

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · about 2 years ago

Hi Stephany, currently there is no method for single file downloads, outside of directly downloading them through the browser. However we have had this requested in the past, and may be adding this in a v0.3.0 of the nsrr gem. I'll send you a message when the functionality has been added. Thanks for the feedback!

edit: v0.3.0 is now available!

4 posts
Was this reply useful? Learn more...
   
[-] Hardik Madhu +0 points · almost 2 years ago

Hi,

My access for shhs database has been approved. But when i try to download with nsrr command as per the posts above, and copy paste my token, it always gives me this error. "UNAUTHORIZED Public Access Only".

Is there anything that I am missing ?

Thanks in Advance, Hardik

36 posts
Was this reply useful? Learn more...
   
[-] remomueller +0 points · almost 2 years ago

Dear Hardik,

Sorry for the inconvenience, I noticed an issue with your access token and have reset the token. Can you give the NSRR gem another try, it should now correctly download the files from SHHS. You token has also changed, so you'll need to visit https://sleepdata.org/token to get the new one.

Thanks!

Remo

4 posts
Was this reply useful? Learn more...
   
[-] Hardik Madhu +0 points · almost 2 years ago

Now it works. Many thanks Remo.

Hardik

[+] [deleted] +0 points · over 1 year ago
Write a Reply