Tools

Notes on command-line interface

The “cli” in fre-cli derives from the shell “fre SUBCOMMAND COMMAND” structure inspired by git, cylc, and other modern Linux command-line tools. This enables discovery of the tooling capability, useful for complex tools with multiple options.

To discover subcommands, use --help, e.g.:

fre --help

fre make --help

fre pp --help

Commands that require arguments to run will alert user about missing arguments, and will also list the rest of the optional parameters if --help is executed. e.g.:

fre pp configure-yaml --help

Argument flags are not positional, can be specified in any order. Some arguments expect sub-arguments.

fre app

fre app tools are intended to be a collection of single-purpose tools.

fre catalog

builder

Generate a catalog

  • Builds json and csv format catalogs from user input directory path

  • Minimal Syntax: fre catalog builder -i [input path] -o [output path]

  • Module(s) needed: n/a

  • Example: fre catalog builder -i /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp -o ~/output --overwrite

validate

Validate the catalog

fre cmor

This set of tools leverages the external cmor python package within the fre ecosystem. cmor is an acronym for “climate model output rewriter”. The process of rewriting model-specific output files for model intercomparisons (MIPs) using the cmor module is, quite cleverly, referred to as “CMORizing”.

run

fre cmor run rewrites climate model output files in a target directory in a CMIP-compliant manner for downstream publishing. It accepts 6 arguments, only one being optional. A brief description of each:

arguments

  • (required) -d, --indir TEXT, input directory containing netCDF files to CMORize.

    • all netCDF files within indir will have their filename checked for local variables specified in varlist as keys, and ISO datetime strings extracted and kept in a list for later iteration over target files

    • a debugging-oriented boolean flag constant at the top of cmor_mixer.py, if True will process one file of all files found within indir and cease processing for that variable after succeeding on one file

  • (required) -l, --varlist TEXT, path to variable list dictionary.

    • each entry in the variable list dictionary corresponds to a key/value pair

    • the key (local variable) is used for ID’ing files within indir to be processed

    • associated with the key (local variable), is the value (target variable), which should be the label attached to the data within the targeted file(s)

  • (required) -r, --table_config TEXT, path to MIP json configuration holding variable metadata.

    • typically, this is to be provided by a data-request associated with the MIP and participating experiments

  • (required) -p, --exp_config TEXT, path to json configuration holding experiment/model metadata

    • contains e.g. grid_label, and points to other important configuration files associated with the MIP

    • the other configuration files are typically housing metadata associated with coordinates, formula_terms, and controlled-vocabulary (CV).

  • (required) -o, --outdir TEXT, path-prefix inwhich the output directory structure is created.

    • further output-directories and structure/template information is specified specified in exp_config

    • in addition to the output-structure/template used, an additional directory corresponding to the date the CMORizing was done is created near the end of the directory tree structure

  • (optional) -v, --opt_var_name TEXT, a specific variable to target for processing

    • largely a debugging convenience functionality, this can be helpful for targeting more specific input files as desired.

examples

with a local clone of fre-cli, the following call should work out-of-the-box from the root directory of the repository.

fre cmor run \
   -d fre/tests/test_files/ocean_sos_var_file \
   -l fre/tests/test_files/varlist \
   -r fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json \
   -p fre/tests/test_files/CMOR_input_example.json \
   -o fre/tests/test_files/outdir

background

The bulk of this routine is housed in fre/cmor/cmor_mixer.py, which is a rewritten version of Sergey Malyshev’s original CMORcommander.py script, utilized during GFDL’s CMIP6 publishing run.

This code is dependent on two primary json configuration files- a MIP variable table and another containing experiment (i.e. model) specific metdata (e.g. grid) to append to the output netCDF file headers, in addition to other configuration options such as output directory name specification, output path templates, and specification of other json configuration files containing controlled-vocabulary (CV), coordinate, and formula term conventions for rewriting the output metadata.

fre make

create-checkout

fre make create-checkout [options]

  • Purpose: Creates the checkout script and can check out source code (with execute option)

  • Options:

    • -y, –yamlfile [experiment yaml] (required)

    • -p, –platform [platform] (required)

    • -t, –target [target] (required)

    • -j, –jobs [number of jobs to run simultneously]

    • -npc, –no-parallel-checkout (for container build)

    • -e, –execute

create-makefile

fre make create-makefile [options]

  • Purpose: Creates the makefile

  • Options:

    • -y, –yamlfile [experiment yaml] (required)

    • -p, –platform [platform] (required)

    • -t, –target [target] (required)

create-compile

fre make create-compile [options]

  • Purpose: Creates the compile script and compiles the model (with execute option)

  • Options:

    • -y, –yamlfile [experiment yaml] (required)

    • -p, –platform [platform] (required)

    • -t, –target [target] (required)

    • -j, –jobs [number of jobs to run simultneously]

    • -n, –parallel [number of concurrent modile compiles]

    • -e, –execute

create-dockerfile

fre make create-dockerfile [options]

  • Purpose: Creates the dockerfile and creates the container (with execute option)

  • With the creation of the dockerfile, the Makefile, checkout script, and any other necessary script is copied into the container from a temporary location

  • Options:

    • -y, –yamlfile [experiment yaml] (required)

    • -p, –platform [platform] (required)

    • -t, –target [target] (required)

run-fremake

fre make run-fremake [options]

  • Purpose: Create the checkout script, Makefile, compile script, and dockerfile (platform dependent) for the compilation of the model

  • Options:

    • -y, –yamlfile [experiment yaml] (required)

    • -p, –platform [platform] (required)

    • -t, –target [target] (required)

    • -npc, –no-parallel-checkout (for container build)

    • -j, –jobs [number of jobs to run simultneously]

    • -n, –parallel [number of concurrent modile compiles]

fre pp

configure

  • Postprocessing yaml configuration

  • Minimal Syntax: fre pp configure -y [user-edit yaml file]

  • Module(s) needed: n/a

  • Example: fre pp configure -y /home/$user/pp/ue2/user-edits/edits.yaml

checkout

  • Checkout template file and clone gitlab.gfdl.noaa.gov/fre2/workflows/postprocessing.git repository

  • Minimal Syntax: fre pp checkout -e [experiment name] -p [platform name] -t [target name]

  • Module(s) needed: n/a

  • Example: fre pp checkout -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

fre yamltools

combine-yamls

  • placehold