Overview
This shef-parser package provides the capability to:
Parse SHEF text
Load parsed text into data stores as time series
Generate SHEF text from time series stored in data stores
What is SHEF?
SHEF is an acronym for Standard Hydrometeorologic Exhchange Format, created in the early 1980s by NOAA/NWS, with major participation from USACE, USGS, and other agencies, to provide a standard format for exchanging hydromet observations and forecasts that is both human readable and machine parsable.
SHEF provides multiple ways of encoding information and has been extended over time to provide more capabilities. The current standard is version 2.2, dated July 5, 2012, available here.
Although SHEF is no longer actively supported by NWS and was intended to be replaced by newer standards such as WaterML, it is still widely used.
Installing
To install the latest version of the pacakge, execute pip install --upgrade shef-parser
Parsing
Parsing SHEF text includes the following steps:
Reading the SHEF-formatted text
Identifying individual messages within the text
Validating messages and logging errors
Outputting the parsed text using one of two rigid output formats.
So the SHEF parser is really just a fancy text format converter. However, it is a necessary one because while SHEF text must adhere to a standard, in order to meet the goal of human readability the standard is very flexible with regard to:
unit system
time zone
use of whitespace
order of information
assumed defaults
intermingling of SHEF and non-SHEF content
Having a dedicated parser to convert SHEF text into a not-so-human-readable format that is much simpler to parse by machine allows downstream programs to employ much simpler parsing code.
In this package SHEF parser is the shef.shef_parser module; it is assumed you will execute it using run_shef_parser.
Format Examples
SHEF Text:
.E KEYO2 20251107 Z DH1400/HT/DIH01/637.74/637.73/637.72/637.71/637.71/637.7/637.7/637.38
.E00 641.01/641.11/641.15/641.11/638.93/638.32/638.06/637.93/637.85/637.81/637.78/637.76
.E01 637.74/637.74/637.35/640.99/641.08/641.13/638.9/638.31/637.7/641.01/641.11/641.15
Output Format 1:
KEYO2 2025-11-07 14:00:00 0000-00-00 00:00:00 HTIRZZ 637.7400 Z -1.000 0000 0 1 " "
KEYO2 2025-11-07 15:00:00 0000-00-00 00:00:00 HTIRZZ 637.7300 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 16:00:00 0000-00-00 00:00:00 HTIRZZ 637.7200 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 17:00:00 0000-00-00 00:00:00 HTIRZZ 637.7100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 18:00:00 0000-00-00 00:00:00 HTIRZZ 637.7100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 19:00:00 0000-00-00 00:00:00 HTIRZZ 637.7000 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 20:00:00 0000-00-00 00:00:00 HTIRZZ 637.7000 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 21:00:00 0000-00-00 00:00:00 HTIRZZ 637.3800 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 22:00:00 0000-00-00 00:00:00 HTIRZZ 641.0100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-07 23:00:00 0000-00-00 00:00:00 HTIRZZ 641.1100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 00:00:00 0000-00-00 00:00:00 HTIRZZ 641.1500 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 01:00:00 0000-00-00 00:00:00 HTIRZZ 641.1100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 02:00:00 0000-00-00 00:00:00 HTIRZZ 638.9300 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 03:00:00 0000-00-00 00:00:00 HTIRZZ 638.3200 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 04:00:00 0000-00-00 00:00:00 HTIRZZ 638.0600 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 05:00:00 0000-00-00 00:00:00 HTIRZZ 637.9300 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 06:00:00 0000-00-00 00:00:00 HTIRZZ 637.8500 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 07:00:00 0000-00-00 00:00:00 HTIRZZ 637.8100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 08:00:00 0000-00-00 00:00:00 HTIRZZ 637.7800 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 09:00:00 0000-00-00 00:00:00 HTIRZZ 637.7600 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 10:00:00 0000-00-00 00:00:00 HTIRZZ 637.7400 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 11:00:00 0000-00-00 00:00:00 HTIRZZ 637.7400 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 12:00:00 0000-00-00 00:00:00 HTIRZZ 637.3500 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 13:00:00 0000-00-00 00:00:00 HTIRZZ 640.9900 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 14:00:00 0000-00-00 00:00:00 HTIRZZ 641.0800 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 15:00:00 0000-00-00 00:00:00 HTIRZZ 641.1300 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 16:00:00 0000-00-00 00:00:00 HTIRZZ 638.9000 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 17:00:00 0000-00-00 00:00:00 HTIRZZ 638.3100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 18:00:00 0000-00-00 00:00:00 HTIRZZ 637.7000 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 19:00:00 0000-00-00 00:00:00 HTIRZZ 641.0100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 20:00:00 0000-00-00 00:00:00 HTIRZZ 641.1100 Z -1.000 0000 0 2 " "
KEYO2 2025-11-08 21:00:00 0000-00-00 00:00:00 HTIRZZ 641.1500 Z -1.000 0000 0 2 " "
Output Format 2:
KEYO2 202511 714 0 0 0 0 0 0 0 0 HT RZZ 637.740 Z -1.00 0 0 1
KEYO2 202511 715 0 0 0 0 0 0 0 0 HT RZZ 637.730 Z -1.00 0 0 2
KEYO2 202511 716 0 0 0 0 0 0 0 0 HT RZZ 637.720 Z -1.00 0 0 2
KEYO2 202511 717 0 0 0 0 0 0 0 0 HT RZZ 637.710 Z -1.00 0 0 2
KEYO2 202511 718 0 0 0 0 0 0 0 0 HT RZZ 637.710 Z -1.00 0 0 2
KEYO2 202511 719 0 0 0 0 0 0 0 0 HT RZZ 637.700 Z -1.00 0 0 2
KEYO2 202511 720 0 0 0 0 0 0 0 0 HT RZZ 637.700 Z -1.00 0 0 2
KEYO2 202511 721 0 0 0 0 0 0 0 0 HT RZZ 637.380 Z -1.00 0 0 2
KEYO2 202511 722 0 0 0 0 0 0 0 0 HT RZZ 641.010 Z -1.00 0 0 2
KEYO2 202511 723 0 0 0 0 0 0 0 0 HT RZZ 641.110 Z -1.00 0 0 2
KEYO2 202511 8 0 0 0 0 0 0 0 0 0 HT RZZ 641.150 Z -1.00 0 0 2
KEYO2 202511 8 1 0 0 0 0 0 0 0 0 HT RZZ 641.110 Z -1.00 0 0 2
KEYO2 202511 8 2 0 0 0 0 0 0 0 0 HT RZZ 638.930 Z -1.00 0 0 2
KEYO2 202511 8 3 0 0 0 0 0 0 0 0 HT RZZ 638.320 Z -1.00 0 0 2
KEYO2 202511 8 4 0 0 0 0 0 0 0 0 HT RZZ 638.060 Z -1.00 0 0 2
KEYO2 202511 8 5 0 0 0 0 0 0 0 0 HT RZZ 637.930 Z -1.00 0 0 2
KEYO2 202511 8 6 0 0 0 0 0 0 0 0 HT RZZ 637.850 Z -1.00 0 0 2
KEYO2 202511 8 7 0 0 0 0 0 0 0 0 HT RZZ 637.810 Z -1.00 0 0 2
KEYO2 202511 8 8 0 0 0 0 0 0 0 0 HT RZZ 637.780 Z -1.00 0 0 2
KEYO2 202511 8 9 0 0 0 0 0 0 0 0 HT RZZ 637.760 Z -1.00 0 0 2
KEYO2 202511 810 0 0 0 0 0 0 0 0 HT RZZ 637.740 Z -1.00 0 0 2
KEYO2 202511 811 0 0 0 0 0 0 0 0 HT RZZ 637.740 Z -1.00 0 0 2
KEYO2 202511 812 0 0 0 0 0 0 0 0 HT RZZ 637.350 Z -1.00 0 0 2
KEYO2 202511 813 0 0 0 0 0 0 0 0 HT RZZ 640.990 Z -1.00 0 0 2
KEYO2 202511 814 0 0 0 0 0 0 0 0 HT RZZ 641.080 Z -1.00 0 0 2
KEYO2 202511 815 0 0 0 0 0 0 0 0 HT RZZ 641.130 Z -1.00 0 0 2
KEYO2 202511 816 0 0 0 0 0 0 0 0 HT RZZ 638.900 Z -1.00 0 0 2
KEYO2 202511 817 0 0 0 0 0 0 0 0 HT RZZ 638.310 Z -1.00 0 0 2
KEYO2 202511 818 0 0 0 0 0 0 0 0 HT RZZ 637.700 Z -1.00 0 0 2
KEYO2 202511 819 0 0 0 0 0 0 0 0 HT RZZ 641.010 Z -1.00 0 0 2
KEYO2 202511 820 0 0 0 0 0 0 0 0 HT RZZ 641.110 Z -1.00 0 0 2
KEYO2 202511 821 0 0 0 0 0 0 0 0 HT RZZ 641.150 Z -1.00 0 0 2
Loading
Loading is the process of reading the output of the SHEF parser and loading the data into a data store. While the SHEF parser can be agnostic of any specific data store, a loading program must be intimately tied to the datastore’s structure, API, etc….
In this package, loaders are not stand-alone programs but are sub-modules of the shef.loaders module written to an API that allows
shef.shef_parser to pass its output directly to them to be loaded to the desired data store. To tell
shef.shef_parser to use a specific loader you use the --loader command line switch follwed by loader-specific
arguments. The arguments required by each loader are specified on the command line in one or more bracketed
text strings (e.g., [arg1][arg2]...) and can be displayed by running run_shef_parser --description.
In the event that one has on hand not the SHEF text but parsed text, shef.shef_parser can pass the processed text
through to an attached loader by using the --processed command line argument.
Information on how loaders interact with shef_parser and the API they are required to implement is detailed on the Loader Requirements page.
Command line information is shown in the Command Line Options page.
Exporting/Unloading
As described on the Loader Requirements page, each loader module is required to have a public boolean variable named
can_unload that specifies whether its loader class contains an unload() method. If provided, the unload()
method reads time series data from the loader’s input and generates one or more SHEF messages for each time series read.
Typically, one .E message or multiple .A messages for each time series depending on whether the time series
has a regular or irregular interval.
While a loader’s loading input (parsed SHEF text) can be generated by shef.shef_parser, there is no way for it to
generate a loader’s unloading input (loader-specific-formatted time series). This is the role of exporters. This
requires each loader to have its own exporter. [1]
Exporters are sub-modules of the shef.exporters module.
The role of an exporter is to retrieve time series from a data store and write them to the loader’s input in the correct format for unloading. This normally involves:
choosing a data store
setting a time window
selecting and retrieving time series
writing time series in the loader’s format
While exporters may define their own methods to allow them to be executed from the command line, there is no requirement for them to do so. The API they are required to implement is detailed on the Exporter Requirements page.