Professional Documents
Culture Documents
Overview:
Note: this is extracted from >> http://disc.sci.gsfc.nasa.gov/recipes/?q=recipes/How-toDownload-Data-Files-from-HTTP-Service-with-wget
This data recipe shows an example for downloading data files from an HTTP service at GES
DISC with GNU wget commend. The GNU wget is a free software for non-interactive
downloading of files from the Web. It is a Unix-based command-line tool, but is also available
for other operating system, such as Linux, Windows, Mac OS X, etc.
Best When:
Want to script data downloading of multiple files.
Task:
Obtaining Data
Example:
Download MERRA-2 Monthly data files for 1981.
Time to complete the following procedures: 10 minutes
This data recipe has been tested on: Linux (wget version 1.12), and Mac OS X (wget
version 1.17.1 ).
Procedure:
1. Install wget
Skip this step if you already have wget installed.
Download wget: https://www.gnu.org/software/wget/
2. Authorize NASA GESDISC DATA ARCHIVE Data Access and setup cookies
Starting August 1 2016, access to GES DISC data requires all users to be registered with
Earthdata Login and then authorize NASA GESDISC DATA ARCHIVE Data Access by
following the instructions:
How to Register a New User in Earthdata Login
How to Authorize NASA GESDISC DATA ARCHIVE Data Access in Earthdata
Login
To run wget, you need to set up .netrc and create a cookie file:
Create a cookie file. This file will be used to persist sessions across calls to Wget or Curl.
For example:
a. cd ~ or cd $HOME
b. touch .urs_cookies
Please read more regarding user registration and data access at:
http://disc.sci.gsfc.nasa.gov/registration
Preview the list of data files by opening the URL with any Web browser.
Note: On Mac OS X (or any Unix system which has the "curl" command available), list data files
can be done via curl by substituting 'curl -s' for 'wget -q -nH -nd', and omitting '-O -'. For
example,
curl
-s http://goldsmr4.sci.gsfc.nasa.gov/data/MERRA2_MONTHLY/M2TMNXSLV.5.12.4/1981/ |
grep MERRA2_100 | cut -f4 -d\"
Since curl does not have the ability to do recursive download. wget or a download manager may
work better for multi-file downloads.
For example,
To download all data and metadata files in the directory:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on
--keep-session-cookies -r -c -nH -nd -np -A nc4,xml
"http://goldsmr4.sci.gsfc.nasa.gov/data/MERRA2_MONTHLY/M2TMNXSLV.5.12.4/1981
/"
To download only data files in the directory:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on
--keep-session-cookies -r -c -nH -nd -np -A nc4
"http://goldsmr4.sci.gsfc.nasa.gov/data/MERRA2_MONTHLY/M2TMNXSLV.5.12.4/1981
/"
To download part of data files in the directory (from Oct 1981 to Dec 1981):
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on
--keep-session-cookies -r -c -nH -nd -np -A '*19811*nc4'
"http://goldsmr4.sci.gsfc.nasa.gov/data/MERRA2_MONTHLY/M2TMNXSLV.5.12.4/1981
/"
or
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on
--keep-session-cookies -r -c -nH -nd -np -A '*19811*nc4'
"http://goldsmr4.sci.gsfc.nasa.gov/data/MERRA2_MONTHLY/M2TMNXSLV.5.12.4/"