Usage

Using a set of YAML configuration files, fre make compiles a FMS-based model, and fre pp postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2024; continue to use FRE Bronx frerun.

YAML Framework

In order to utilize FRE 2024.01 tools, a distrubuted YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with [key]: [value] fields.

Yaml Formatting

Helpful information and format recommendations for creating yaml files.

  1. You can define a block of values as well as individual [key]: [value] pairs:

section name:
  key: value
  key: value

  1. [key]: [value] pairs can be made a list by utilizing a -:

section name:
  - key: value
  - key: value

  1. If you want to associate information with a certain listed element, follow this structure:

section name:
  - key: value
    key: value
  - key: value
    key: value

Where each dash indicates a list.

  1. Yamls also allow for the capability of reusable variables. These variables are defined by:

&ReusableVariable Value

  1. Users can apply a reusable variable on a block of values. For example, everything under “section” is associated with the reusable variable:

section: &ReusableVariable
  - key: value
    key: value
  - key: value

  1. In order to use them as a reference else where in either the same or other yamls, use *:

*ReusableVariable

  1. If the reusable variable must be combined with other strings, the `!join` constructor is used. Example:

&version "2024.01"
&stem !join [FRE/, *version]

In this example, the reuasble variable stem will be parsed as FRE/2024.01.

Model Yaml

The model yaml defines reusable variables, shared directories, switches, post-processing settings, and paths to compile and post-processing yamls. Required fields in the model yaml include: fre_properties, build, shared, and experiments.

  • fre_properties: Reusable variables

    • list of variables

    • these values can be extracted from fre_properties in a group’s XML, if available

    • value type: string

    - &variable1  "value1"  (string)
- &variable2  "value2"  (string)

  • build: paths to information needed for compilation

    • subsections: compileYaml, platformYaml

    • value type: string

    build:
  compileYaml: "path the compile yaml in relation to model yaml"   (string)
  platformYaml: "path to platforms.yaml in relation to model yaml" (string)

  • shared: shared settings across experiments

    • subsections: directories, postprocess

    shared:
  directories: &shared_directories
    key: "value"               (string)

  postprocess:
    settings: &shared_settings
      key: "value"             (string)
    switches: &shared_switches
      key: True/False          (boolean)

    • Be sure to define directories, settings, and switches as reusable variables as well

      • they will be “inherited” in the post-processing yamls created

  • experiments: list of post-processing experiments

    • subsections: name, pp, analysis

    experiments:
  - name: name of post-processing experiment                                       (string)
    pp:
      - path/to/post-processing/yaml for that experiment in relation to model yaml (string)
    analysis:
      - path/to/analysis/yaml for that experiment in relation to model yaml        (string)

Compile Yaml

The compile yaml defines compilation information including component names, repos, branches, necessary flags, and necessary overrides. This is discussed more in the “Build FMS Model” section.

Platforms Yaml

The platform yaml contains user defined information for both bare-metal and container platforms. Information includes the platform name, the compiler used, necessary modules to load, an mk template, fc, cc, container build, and container run. This yaml file is not model specific.

platforms:
  - name: the platform name
    compiler: the compiler you are using
    modulesInit: ["array of commands that are needed to load modules." , "each command must end with a newline character"]
    modules: [array of modules to load including compiler]
    fc: the name of the fortran compiler
    cc: the name of the C compiler
    mkTemplate: The location of the mkmf make template
    modelRoot: The root directory of the model (where src, exec, experiments will go)
  - container platform: container platform name
    compiler: compiler you are using
    RUNenv: Commands needed at the beginning of a RUN in dockerfile
    modelRoot: The root directory of the model (where src, exec, experiments will go) INSIDE of the container (/apps)
    fc: name of fortan compiler
    cc: name of C compiler
    container: True if this is a container platform
    containerBuild: "podman" - the container build program
    containerRun: "apptainer" - the container run program

Post-Processing Yaml

The post-processing yamls include information specific to experiments, such as directories to data and other scripts used, switches, and component information. The post-processing yaml can further define more fre_properties that may be experiment specific. If there are any repeated reusable variables, the ones set in this yaml will overwrite those set in the model yaml. This is discussed further in the “Postprocess FMS History Output” section.

Build FMS model

fre make can compile a traditional “bare metal” executable or a containerized executable using a set of YAML configuration files.

Through the fre-cli, fre make can be used to create and run a checkout script, makefile, and compile a model.

Fremake Canopy Supports:

  • multiple target use; -t flag to define each target (for multiple platform-target combinations)

  • bare-metal build

  • container creation

  • parallel checkouts for bare-metal build

  • parallel model builds

  • one yaml format

  • additional library support if needed

Note: Users will not be able to create containers without access to podman. To get access, submit a helpdesk ticket.

Required configuration files:

  • Model Yaml

  • Compile Yaml

  • Platforms yaml

These yamls are combined and further parsed through the fre make tools.

Compile Yaml

To create the compile yaml, reference the compile section on an XML. Certain fields should be included under “compile”. These include experiment, container_addlibs, baremetal_linkerflags, and src.

  • The experiment can be explicitly defined or can be used in conjunction with defined fre_properties from the model yaml, as seen in the code block below

  • container_addlibs: list of strings of packages needed for the model to compile (used to create the link line in the Makefile)

  • baremetal_linkerflags: list of strings of linker flags (used to populate the link line in the Makefile

  • src: contains information about components needed for model compilation

compile:
  experiment: !join [*group_version, "_compile"]
  container_addlibs: "libraries and packages needed for linking in container" (string)
  baremetal_linkerflags: "linker flags of libraries and packages needed"      (string)
  src:

The src section is used to include component information. This will include: component, repo, cpdefs, branch, paths, otherFlags, and makeOverrides.

src:
  - component: "component name"                                            (string)
    requires: ["list of components that this component depends on"]        (list of string)
    repo: "url of code repository"                                         (string)
    branch: "version of code to clone"                                     (string / list of strings)
    paths: ["paths in the component to compile"]                           (list of strings)
    cppdefs: "CPPDEFS ot include in compiling componenet                   (string)
    makeOverrides: "overrides openmp target for MOM6"                      ('OPENMP=""') (string)
    otherFlags: "Include flags needed to retrieve other necessary code"    (string)
    doF90Cpp: True if the preprocessor needs to be run                     (boolean)
    additionalInstructions: additional instructions to run after checkout  (string)

Guide

  1. Bare-metal Build:

# Create checkout script
fre make create-checkout -y [model yaml file] -p [platform] -t [target]

# Create and run checkout script
fre make create-checkout -y [model yaml file] -p [platform] -t [target] --execute

# Create Makefile
fre make create-makefile -y [model yaml file] -p [platform] -t [target]

# Creat the compile script
fre make create-compile -y [model yaml file] -p [platform] -t [target]

# Create and run the compile script
fre make create-compile -y [model yaml file] -p [platform] -t [target] --execute

# Run all of fremake
fre make run-fremake -y [model yaml file] -p [platform] -t [target] [other options...]

  1. Container Build:

For the container build, parallel checkouts are not supported, so the -npc options must be used for the checkout script. In addition the platform must be a container platform.

Users will not be able to create containers unless they have podman access on gaea.

# Create checkout script
fre make create-checkout -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] -npc

# Create and run checkout script
fre make create-checkout -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute

# Create Makefile
fre make create-makefile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target]

# Create a Dockerfile
fre make create-dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target]

# Create and run the Dockerfile
fre make create-dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute

Quickstart

The quickstart instructions can be used with the am5-compile examples located in the fre-examples repository: https://github.com/NOAA-GFDL/fre-examples/tree/main/AM5/am5-compile

  1. Bare-metal Build:

# Create checkout script
fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod

# Create and run checkout script
fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod --execute

# Create Makefile
fre make create-makefile -y am5.yaml -p ncrc5.intel23 -t prod

# Create the compile script
fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod

# Create and run the compile script
fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod --execute

  1. Bare-metal Build Multi-target:

# Create checkout script
fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod -t debug

# Create and run checkout script
fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod -t debug --execute

# Create Makefile
fre make create-makefile -y am5.yaml -p ncrc5.intel23 -t prod -t debug

# Create the compile script
fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod -t debug

# Create and run the compile script
fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod -t debug --execute

  1. Container Build:

In order for the container to build successfully, a -npc, or –no-parallel-checkout is needed.

# Create checkout script
fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod -npc

# Create and run checkout script
fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod -npc --execute

# Create Makefile
fre make create-makefile -y am5.yaml -p hpcme.2023 -t prod

# Create Dockerfile
fre make create-dockerfile -y am5.yaml -p hpcme.2023 -t prod

# Create and run the Dockerfile
fre make create-dockerfile -y am5.yaml -p hpcme.2023 -t prod --execute

  1. Run all of fremake:

Currently, run-fremake kicks off the compilation automatically; no --execute option needed.

# Bare-metal
fre make run-fremake -y am5.yaml -p ncrc5.intel23 -t prod

# Container
fre make run-fremake -y am5.yaml -p hpcme.2023 -t prod -npc

Run FMS model

Check back in the latter half of 2025 or so.

Postprocess FMS history output

fre pp regrids FMS history files and generates timeseries, climatologies, and static postprocessed files, with instructions specified in YAML.

Bronx plug-in refineDiag and analysis scripts can also be used, and a reimagined analysis script ecosystem is being developed and is available now (for adventurous users). The new analysis script framework is independent of and compatible with FRE (https://github.com/NOAA-GFDL/analysis-scripts). The goal is to combine the ease-of-use of legacy FRE analysis scripts with the standardization of model output data catalogs and python virtual environments.

In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as fre cmor.

By default, an intake-esm-compatible data catalog is generated and updated, containing a programmatic metadata-enriched searchable interface to the postprocessed output. The catalog tooling can be independently assessed as fre catalog.

FMS history files

FRE experiments are run in segments of simulated time. The FMS diagnostic manager, as configured in experiment configuration files (diag yamls) saves a set of diagnostic output files, or “history files.” The history files are organized by label and can contain one or more temporal or static diagnostics. FRE (Bronx frerun) renames and combines the raw model output (that is usually on a distributed grid), and saves the history files in one tarfile per segment, date-stamped with the date of the beginning of the segment. The FMS diagnostic manager requires that variables within one history file be the same temporal frequency (e.g. daily, monthly, annual), but statics are allowed in any history file. Usually, variables in a history file share a horizontal and vertical grid.

Each history tarfile, again, is date-stamped with the date of the beginning of the segment, in YYYYMMDD format. For example, for a 5-year experiment with 6-month segments, there will be 10 history files containing the raw model output. Each history tarfile contains a segment’s worth of time (in this case 6 months).:

19790101.nc.tar  19800101.nc.tar  19810101.nc.tar  19820101.nc.tar  19830101.nc.tar
19790701.nc.tar  19800701.nc.tar  19810701.nc.tar  19820701.nc.tar  19830701.nc.tar

Each history file within the history tarfiles are also similarly date-stamped. Atmosphere and land history files are on the native cubed-sphere grid, which have 6 tiles that represent the global domain. Ocean, ice, and global scalar output have just one file that covers the global domain.

For example, if the diagnostic manager were configured to save atmospheric and ocean annual and monthly history files, the 19790101.nc.tar tarfile might contain:

tar -tf 19790101.nc.tar | sort

./19790101.atmos_month.tile1.nc
./19790101.atmos_month.tile2.nc
./19790101.atmos_month.tile3.nc
./19790101.atmos_month.tile4.nc
./19790101.atmos_month.tile5.nc
./19790101.atmos_month.tile6.nc
./19790101.atmos_annual.tile1.nc
./19790101.atmos_annual.tile2.nc
./19790101.atmos_annual.tile3.nc
./19790101.atmos_annual.tile4.nc
./19790101.atmos_annual.tile5.nc
./19790101.atmos_annual.tile6.nc
./19790101.ocean_month.nc
./19790101.ocean_annual.nc

The name of the history file, while often predictably named, are arbitrary labels within the Diagnostic Manager configuration (diag yamls). Each history file is a CF-standard NetCDF file that can be inspected with common NetCDF tools such as the NCO or CDO tools, or even ncdump.

Required configuration

  1. Set the history directory in your postprocessing yaml:

directories:

history: /arch5/am5/am5/am5f7c1r0/c96L65_am5f7c1r0_amip/gfdl.ncrc5-deploy-prod-openmp/history

  1. Set the segment size as an ISO8601 duration (e.g. P1Y is “one year”):

postprocess:
settings:

history_segment: P1Y

  1. Set the date range to postprocess as ISO8601 dates:

postprocess:
settings:

pp_start: 1979-01-01T0000Z

pp_stop: 2020-01-01T0000Z

Postprocess components

The history-file namespace is a single layer as shown above. By longtime tradition, FRE postprocessing namespaces are richer, with a distinction for timeseries, timeaveraged, and static output datasets, and includes frequency and chunk-size in the directory structure.

Postprocessed files within a “component” share a horizontal grid; which can be the native grid or regridded to lat/lon.

Required configuration

  1. Define the atmos and ocean postprocess components:

postprocess:
components:

  • type: atmos

    sources: [atmos_month, atmos_annual]

  • type: ocean

    sources: [ocean_month, ocean_annual]

XY-regridding

Commonly, native grid history files are regridded during postprocessing. To regrid to a lat/lon grid, configure your desired output grid, interpolation method, input grid type, and path to your FMS exchange grid definition.

Optional configuration (i.e. if xy-regridding is desired)

  1. Regrid the atmos and ocean components to a 1x1 degree grid:

directories:

pp_grid_spec: /archive/oar.gfdl.am5/model_gen5/inputs/c96_grid/c96_OM4_025_grid_No_mg_drag_v20160808.tar

postprocess:
components:

  • type: atmos

    sources: [atmos_month, atmos_annual]

    sourceGrid: cubedsphere

    inputRealm: atmos

    xyInterp: [180, 360]

    interpMethod: conserve_order2

  • type: ocean

    sources: [ocean_month, ocean_annual]

    sourceGrid: tripolar

    inputRealm: ocean

    xyInterp: [180, 360]

    interpMethod: conserve_order1

Timeseries

Timeseries output is the most common type of postprocessed output.

Climatologies

annual and monthly climatologies less fine-grained than bronx per-component switch coming now it’s one switch for entire pp

Statics

underbaked, known deficiency currently, takes statics from “source” history files

Analysis scripts

Surface masking for FMS pressure-level history

Legacy refineDiag scripts

CMORize postprocessed output

Brief rundown of commands also provided below:

  • Enter commands and follow --help messages for guidance

  • If the user just runs fre, it will list all the command groups following fre, such as run, make, pp, etc. and once the user specifies a command group, the list of available subcommands for that group will be shown

  • Commands that require arguments to run will alert user about missing arguments, and will also list the rest of the optional parameters if --help is executed

  • Argument flags are not positional, can be specified in any order as long as they are specified

  • Can run directly from any directory, no need to clone repository

  • May need to deactivate environment and reactivate it in order for changes to apply

  • fre/setup.py allows fre/fre.py to be ran as fre on the command line by defining it as an entry point. Without it, the call would be instead, something like python fre/fre.py

  • See also, fre cmor’s README

  • See also, fre cmor’s project board

This set of tools leverages the external cmor python package within the fre ecosystem. cmor is an acronym for “climate model output rewriter”. The process of rewriting model-specific output files for model intercomparisons (MIPs) using the cmor module is, quite cleverly, referred to as “CMORizing”.

Generate data catalogs