Help: dataselect v.1

Description

from obspy.clients.fdsn import Client
client = Client("EARTHSCOPE")

from obspy.clients.fdsn import Client
client = Client("https://service.earthscope.org/")

To retrieve raw waveform data, submit a request by either of two methods:

• via HTTP GET : Provide a series of parameter-value pairs in the URL that specify the start-time and end-time, along with the desired network(s), station(s), location(s) and channel(s) . Wildcards are supported. Please visit the fdsnws-dataselect service interface for parameter usage details.
• via HTTP POST: Submit a pre-formatted request (e.g. a file) to the service containing a list of the desired networks, stations, locations, channels, start-times and end-times. The POSTing method is described in more detail on this page.

This service is designed for broad usability and can easily be used with command line programs such as curl (include by default on most systems) or similar utilities.

Data selection

A data selection is composed of a list of network, station, location, channel, start time and end time entries. Channel codes follow the conventions documented in the FDSN Source Identifier specification.

An example selection, submitted using HTTP POST, might look like:

TA A25A -- BHZ 2010-03-25T00:00:00 2010-04-01T00:00:00
IU ANMO * BH? 2010-03-25T00:00:00 2010-04-01T00:00:00
IU ANMO 10 HHZ 2010-03-25T00:00:00 2010-04-01T00:00:00
II KURK 00 BHN 2010-03-25T00:00:00 2010-04-01T00:00:00

• When the selected SEED quality is “best” (the default) all qualities available are merged together and overlap data is removed in order of increasing quality preference.
• Glob expressions (wildcards) are allowed in all fields except date fields.

cURL example

Requests can be made with a selection file and the curl command line utility.

$ cat waveform.request
format=miniseed
TA A25A -- BHZ 2010-03-25T00:00:00 2010-04-01T00:00:00
TA A25A -- BHE 2010-03-25T00:00:00 2010-04-01T00:00:00

$ curl -L --data-binary @waveform.request -o TA.miniseed https://service.earthscope.org/fdsnws/dataselect/1/query

Here is the equivalent request using query parameters instead of a selection file…

$ curl -L -o TA.miniseed "https://service.earthscope.org/fdsnws/dataselect/1/query?net=TA&sta=A25A&cha=BHZ,BHE&loc=--&starttime=2010-03-25&endtime=2010-04-01"

This will send the request to the server and save the results in a file name TA.miniseed

We recommend always using the -L option to allow curl to follow HTTP redirections specified by our systems. The service uses HTTP redirection during maintenance to keep servicing requests.

You may wish to use the -f option. This will cause curl to return an exit code of 22 if data is not found or the request is improperly formatted.

See the curl manual for more information.

Working with miniSEED

A variety of software tools are available to assist with organizing and viewing miniSEED data or converting it to another format. Detailed descriptions and usage examples for each piece of software can be found by clicking the links below.

mseed2sac – for converting miniSEED to SAC format
mseed2ascii – for converting miniSEED to ASCII formats
dataselect – for selecting and sorting miniSEED
miniSEED Inspector – for quickly parsing and summarizing miniSEED data

Accessing restricted data

Requesting restricted data via this web service requires authentication.

If you are allowed to access restricted data but do not yet have credentials for this service visit https://www.earthscope.org/user, create an account (if you do not already have one), then:

• navigate to the Credentials tab
• click “REVEAL MY CREDENTIALS,” and “CREATE FDSNWS CREDENTIALS” if none exist

The authentication is done using a standard HTTP mechanism called digest access authentication, a sort of 3-way handshake. To submit a request with authentication credentials you would use the queryauth method of the service in place of the query method. You must use a client that supports digest authentication.

For example, submitting a request and subsequently initiating the authentication handshake would be done by requesting this URL:

https://service.earthscope.org/fdsnws/dataselect/1/queryauth?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00&end=2010-02-27T10:30:00

This request could be submitted, along with authentication credentials, using a command line tool like curl:

$ curl -L --digest --user USERNAME:PASSWORD -o data.mseed 'https://service.earthscope.org/fdsnws/dataselect/1/queryauth?net=IU&sta=ANMO&loc=0Z&start=2010-02-27T06:30:00&end=2010-02-27T10:30:00'

where you replace USERNAME and PASSWORD with your own credentials. If you are submitting this request from the command line, then for security purposes, you may consider not including PASSWORD in your request, as it is an optional parameter. If only USERNAME is specified, then curl will prompt you for your password when the request is submitted.

Accessing restricted data with ObsPy

ObsPy’s FDSN Client (obspy.clients.fdsn.client.Client)
supports the user and password parameters.

For ObsPy version 1.4.2 or earlier, the combination of the default data center or the explicit use of “IRIS” with credentials does not work.
Instead the Client must be created with a base_url of "https://service.earthscope.org", for example

from obspy.clients.fdsn import Client
client_earthscope = Client("https://service.earthscope.org", user=USERNAME, password=PASSWORD)

where USERNAME and PASSWORD are your own username and password.

The same client_earthscope is usable with ObsPy’s Mass Downloader as documented at: https://docs.obspy.org/packages/autogen/obspy.clients.fdsn.mass_downloader.html#authentication

Considerations

¹ In general, it is preferable to not ask for too much data in a single request. Large requests take longer to complete. If a large request fails due to any networking issue, it will have to be resubmitted to be completed. This will cause the entire request to be completely reprocessed and re-transmitted. By breaking large requests into smaller requests, only the smaller pieces will need to be resubmitted and re-transmitted if there is a networking problem. Web service network connections will break after 5 to 10 minutes if no data is transmitted. For large requests, the fdsnws-dataselect web service can take several minutes before it starts returning data. When this happens, the web service may “flush” the HTTP headers with an “optimistic” success (200) code to the client in order to keep the network connection alive. This gives about 10 minutes to the underlying data retrieval mechanism to start pulling data out of the IRIS archive. Thus for larger requests, the HTTP return code can be unreliable. As data is streamed back to the client, the fdsnws-dataselect service partially buffers the returned data. During time periods when the underlying retrieval mechanism stalls, the web service will dribble the partial buffer to the client in an effort to keep the network connection alive.

It is less efficient to ask for too little data in each request. Each time a request is made, a network connection must be established and a request processing unit started. For performance reasons, it is better to group together selections from the same stations and place them in the same request. This is especially true of selections that cover the same time periods.

Usage guidelines & real time data

All usage should follow the usage guidelines, specifically do not make too many concurrent requests or requests too quickly.

All open data arriving in (near) real time are available from the SeedLink server.

This web service should not be used to retrieve continuous, real time data via repeated polling. Instead, the SeedLink server should be used when continuous data streams are needed.

Problems with this service?