Generate mapfile for ESGF publication

The publication process on the ESGF nodes requires mapfiles. Mapfiles are text files where each line describes a file to publish, using the following format:

dataset_ID | absolute_path | size_bytes [ | option=value ]

  1. All values have to be pipe-separated.
  2. The dataset identifier, the absolute path and the size (in bytes) are required.
  3. Adding the version number to the dataset identifier is strongly recommended to publish in a in bulk.
  4. Strongly recommended optional values are:
  • mod_time: last modification date of the file (since Unix EPOCH time, i.e., seconds since January, 1st, 1970),
  • checksum: file checksum,
  • checksum_type: checksum type (MD5 or the default SHA256).
  1. Your directory structure has to strictly follows the tree fixed by the DRS including the version facet.
  2. To store ONE mapfile PER dataset is strongly recommended.
Several esgmapfile actions are available to manage your mapfiles:
  • make generates the mapfiles (the default),
  • show displays the expected mapfiles path to be generated.

Default mapfile generation

The default behavior is to pick up the latest version in the DRS. This required version with a date format (e.g., v20151023). If the version is directly specified in positional argument, the version number from supplied directory is used.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/

Mapfile without files checksums

Because this could be time consuming --no-checksum allows you not include the file checksum into your mapfile(s).

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-checksum

Mapfile with pre-calculated checksums

If your file checksum have been already calculated apart, you can submit a file to esgmapfile with the checksums list. This checksum file must have the same format as the output of the UNIX command-lines “*sum”.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --checksums-from /PATH/TO/CHECKSUMS/FILE


In the case of unfound checksums, it falls back to compute the checksum as normal.

Mapfile without DRS versions

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-version

Mapfile name using tokens

The mapfile name is composed by the dataset ID and the dataset version dot-separated. Another template can be specified for the output mapfile(s) name using several tokens. Substrings {dataset_id}, {version}, {job_id} or {date} (in YYYYDDMM) will be substituted where found. If {dataset_id} is not present in mapfile name, then all datasets will be written to a single mapfile, overriding the default behavior of producing ONE mapfile PER dataset.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile {dataset_id}.{job_id}
$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile {date}.{job_id}
$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile MY_MAPFILE.{version}.{date}

Organize your mapfiles

The mapfile(s) are generated into a mapfile folder created in your working directory (if exists). This can be changed by submitting an output directory for your mapfiles.

In addition, a mapfile_drs attribute can be added into the corresponding project section of the configuration INI file(s) (see Configuration). In the same way as the directory_format it defines a tree depending on the facets. Each mapfile is then written into the corresponding output directory. This mapfile_drs directory structure will be added to the output directory if submitted.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --outdir /PATH/TO/MY_MAPFILES/

The output directory is cleaned up prior to mapfile process to avoid uncompleted mapfiles. In the case of several esgmapfile instances run with the same output directory it is recommended to disable the cleanup:

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-cleanup

Walking through latest directories only

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --latest-symlink

Walking through a particular version only

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --version VERSION

Walking through all versions

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --all-versions


This disables --no-version.

Add technical notes

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --tech-notes-url URL --tech-notes-title TITLE

Overwrite the dataset identifier

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --dataset-name DATASET_NAME


All files will belong to the specified dataset, regardless of the DRS.

Show the expected mapfile name and path

The show works as a “dry-run” of the make and supports different types inputs. You can show the mapfiles full path to be generated from:

  • a directory to scan:
   $> esgmapfile show --project PROJECT_ID --directory /PATH/TO/SCAN

- a text file with one dataset ID per line:
   $> esgmapfile show --project PROJECT_ID --dataset-list /PATH/TO/TXT_FILE

- a unique dataset ID:
$> esgmapfile show --project PROJECT_ID --dataset-id DATASET_ID

In the case of --dataset-list if no file submitted, the standard input is used.

$> esgmapfile show --project PROJECT_ID --dataset-list < /PATH/TO/TXT_FILE


In the case of dataset IDs the version suffix is expected.


All the make arguments can be safely combined with show.


Print only mapfile basename instead of the mapfile full path adding --basename flag.


To only print the result without any other info use --quiet flag.

Exit status

  • Status = 0
    All the files have been successfully scanned and the mapfile(s) properly generated.
  • Status > 0
    Some scan errors occurred or files have been skipped. The error code indicates the number of errors.