XDMF output that simultaneously serves for visualization with software like Paraview and as a simulation checkpoint. More...

#include <xdmf.hpp>

Public Types
enum class	OutputTypeFloat : int { Float32 = 4 , Float64 = 8 }
	Used to specify the output type when writing floating point data. More...

enum class	OutputTypeInt : int { Int32 = 4 , Int64 = 8 }
	Used to specify the output type when writing (signed) integer data. More...

Public Member Functions
	XDMFOutput (const std::string &directory_path, const grid::shell::DistributedDomain &distributed_domain, const grid::Grid3DDataVec< InputGridScalarType, 3 > &coords_shell_device, const grid::Grid2DDataScalar< InputGridScalarType > &coords_radii_device, const OutputTypeFloat output_type_points=OutputTypeFloat::Float32, const OutputTypeInt output_type_connectivity=OutputTypeInt::Int64)
	Constructs an XDMFOutput object.

void	set_write_counter (int write_counter)
	Set the write counter manually.

template<typename InputScalarDataType >
void	add (const grid::Grid4DDataScalar< InputScalarDataType > &data, const OutputTypeFloat output_type=OutputTypeFloat::Float32)
	Adds a new scalar data grid to be written out.

template<typename InputScalarDataType , int VecDim>
void	add (const grid::Grid4DDataVec< InputScalarDataType, VecDim > &data, const OutputTypeFloat output_type=OutputTypeFloat::Float32)
	Adds a new vector-valued data grid to be written out.

void	write ()
	Writes a "time step".

Detailed Description

template<typename InputGridScalarType>
class terra::io::XDMFOutput< InputGridScalarType >

XDMF output that simultaneously serves for visualization with software like Paraview and as a simulation checkpoint.

Overview

Writes simulation data (time-series for a constant mesh) and metadata into a directory. The written data simultaneously serves for visualization with tools that can read XDMF files (e.g., Paraview) and can be used to restore the grid data, e.g., to continue a previous simulation (aka checkpoint). This does not involve data duplication. The data required for XDMF is directly used as a checkpoint.

Currently restricted to the spherical shell mesh data structure. Interprets data as block-structured wedge-element meshes.

File overview (what is written)

The mesh data has to be added upon construction. None, one, or many scalar or vector-valued grids can be added afterward.

Each write() call then writes out

binary data files for each added grid data item containing the raw values,
an .xmf file to be used for visualization of all the binary files (read by an XDMF reader).

The first write() call also writes (once)

the mesh data (grid coordinates/topology) to binary files
a checkpoint metadata file required to read in the checkpoints

For time-dependent runs, call write() in, e.g., every timestep. The mesh written in the first call will be referenced from each .xmf file.

Output file precision

The actually written data type can be selected regardless of the underlying data type of the allocated data for the mesh points, topology, and each data grid. Defaults are selected via default parameters. Concretely, you can write your double precision fields in single precision without converting them manually. Note that this means that your checkpoints obviously have the same precision that you specify here.

Other notes

All added data grids must have different (Kokkos::View-)labels.

Uses MPI IO for fast parallel output.

Checkpoints

To recover a checkpoint, use the function read_xdmf_checkpoint_grid() (or read_xdmf_checkpoint_metadata() to just inspect the structure of the checkpoint).

Note that the checkpoint can only be read using the same domain partitioning (i.e., the 'topology' of the DistributedDomain used when the checkpoint was written must be identical) - BUT the number of MPI processes does not need to match (nor does the distribution of subdomains to ranks need to match). So you can technically (if the amount of main memory permits) read a checkpoint from a large parallel simulation with only one or a few processes (possibly useful for post-processing).

Picking the last step during checkpoint recovery

The .xmf file for each write() call is written last (after the binary data). Thus, if the corresponding .xmf step has been written, the parallel binary data output should be completed.

Checkpoint data binary format

All data is written to a single binary file per grid data item and per time step. Each subdomain is written contiguously. Concretely, per subdomain (also applies to the grid point coordinates)

 for ( each subdomain )
 {
    for ( int r = 0; r < subdomain_size_r; r++ )
    {
        for ( int y = 0; y < subdomain_size_y; y++ )
        {
            for ( int x = 0; x < subdomain_size_x; x++ )
            {
                for ( int d = 0; d < point_dim; d++ )
                {
                    stream << data( subdomain_id, x, y, r, d )
                }
            }
        }
    }
}

The subdomain order depends on the various factors and can be basically random. The concrete ordering (as well as the data type) is reflected in the checkpoint metadata (see below).

Checkpoint metadata format (required only once per time series)

See terra::grid::shell::SubdomainInfo for how the global subdomain ID is encoded.

version:                                      i32
num_subdomains_per_diamond_lateral_direction: i32
num_subdomains_per_diamond_radial_direction:  i32
subdomain_size_x:                             i32
subdomain_size_y:                             i32
subdomain_size_r:                             i32
radii:                                        array: f64, entries: num_subdomains_per_diamond_radial_direction * (subdomain_size_r - 1) + 1
grid_scalar_bytes                             i32 // new in checkpoint version 1, number of float bytes for writing the grid (4 or 8 byte float)
num_grid_data_files:                          i32
list (size = num_grid_data_files)
[
    grid_name_string_size_bytes:              i32
    grid_name_string:                         grid_name_string_size_bytes * 8
    scalar_data_type:                         i32 // 0: signed integer, 1: unsigned integer, 2: float
    scalar_bytes:                             i32
    vec_dim:                                  i32
]
checkpoint_subdomain_ordering:                i32
if (checkpoint_subdomain_ordering == 0) {
    // The following list contains the encoded global_subdomain_ids (as i64) in the order in which the
    // subdomains are written to the data files. To find out where a certain subdomain is located in the
    // file, the starting offset must be set equal to the index of the global_subdomain_id in the list below,
    // multiplied by the size of a single chunk of data per subdomain.
    // That means that in the worst case, the entire list must be read to find the correct subdomain.
    // However, this way it is easy to _write_ the metadata since we do not need to globally sort the subdomain
    // positions in the parallel setting, and we get away with O(1) local memory usage during parsing.
    // Lookup is O(n), though.
    //
    // An alternative would be to sort the list before writing the checkpoint and store the _position of the
    // subdomain data_ instead of the global_subdomain_id. Since the global_subdomain_id is sortable, an
    // implicit mapping from global_subdomain_id to this list's index can be accomplished.
    // Then look up the position of the data in O(1).
    // However, that would require reducing the entire list on one process which is in theory not scalable
    // (plus the sorting approach is a tiny bit more complicated).
    // Use a different value for checkpoint_subdomain_ordering in that case and extend the file format.
    list (size = "num_global_subdomains" = 10 * num_subdomains_per_diamond_lateral_direction * num_subdomains_per_diamond_lateral_direction * num_subdomains_per_diamond_radial_direction)
    [
        global_subdomain_id: i64
    ]
}

Member Enumeration Documentation

◆ OutputTypeFloat

template<typename InputGridScalarType >

enum class terra::io::XDMFOutput::OutputTypeFloat : int

strong

Used to specify the output type when writing floating point data.

Values are the number of bytes.

Enumerator
Float32
Float64

◆ OutputTypeInt

template<typename InputGridScalarType >

enum class terra::io::XDMFOutput::OutputTypeInt : int

strong

Used to specify the output type when writing (signed) integer data.

Values are the number of bytes.

Enumerator
Int32
Int64

Constructor & Destructor Documentation

◆ XDMFOutput()

template<typename InputGridScalarType >

terra::io::XDMFOutput< InputGridScalarType >::XDMFOutput	(	const std::string &	directory_path,
		const grid::shell::DistributedDomain &	distributed_domain,
		const grid::Grid3DDataVec< InputGridScalarType, 3 > &	coords_shell_device,
		const grid::Grid2DDataScalar< InputGridScalarType > &	coords_radii_device,
		const OutputTypeFloat	output_type_points = `OutputTypeFloat::Float32`,
		const OutputTypeInt	output_type_connectivity = `OutputTypeInt::Int64`
	)

inline

Constructs an XDMFOutput object.

All data will be written to the specified directory (it is a good idea to pass an empty directory).

Construction does not write any data, yet.

Parameters

directory_path	Path to a directory that the data shall be written to. If the directory does not exist, it will be created during the first write() call. If it does already exist, data will be overwritten.
distributed_domain	DistributedDomain instance.
coords_shell_device	Lateral spherical shell grid coordinates (see subdomain_unit_sphere_single_shell_coords()).
coords_radii_device	Spherical shell radii (see subdomain_shell_radii()).
output_type_points	Floating point data type to use for mesh coordinate output.
output_type_connectivity	Integer data type to use for mesh connectivity output.

Member Function Documentation

◆ add() [1/2]

template<typename InputGridScalarType >

template<typename InputScalarDataType >

void terra::io::XDMFOutput< InputGridScalarType >::add	(	const grid::Grid4DDataScalar< InputScalarDataType > &	data,
		const OutputTypeFloat	output_type = `OutputTypeFloat::Float32`
	)

inline

Adds a new scalar data grid to be written out.

Does not write any data to file yet - call write() for writing the next time step.

◆ add() [2/2]

template<typename InputGridScalarType >

template<typename InputScalarDataType , int VecDim>

void terra::io::XDMFOutput< InputGridScalarType >::add	(	const grid::Grid4DDataVec< InputScalarDataType, VecDim > &	data,
		const OutputTypeFloat	output_type = `OutputTypeFloat::Float32`
	)

inline

Adds a new vector-valued data grid to be written out.

Does not write any data to file yet - call write() for writing the next time step.

◆ set_write_counter()

template<typename InputGridScalarType >

void terra::io::XDMFOutput< InputGridScalarType >::set_write_counter ( int write_counter )

inline

Set the write counter manually.

This will only affect the step number attached to the file names. The geometry is still written once during the first write() call.

◆ write()

template<typename InputGridScalarType >

void terra::io::XDMFOutput< InputGridScalarType >::write ( )

inline

Writes a "time step".

Will write one .xmf file with the current counter as part of the name such that the files can be opened as a time series.

The first write() call after construction will also write the mesh data (binary files) that will be referenced from later .xmf files, as well as checkpoint metadata.

For each added data grid, one binary file is written. The data is copied to the host if required. The write() calls will allocate temporary storage on the host if host and device memory are not shared. Currently, for data grids, some host-side temporary buffers are kept after this method returns (the sizes depend on the type of data added) to avoid frequent reallocation.

The documentation for this class was generated from the following file:

xdmf.hpp

Public Types

Public Member Functions

Detailed Description

Overview

File overview (what is written)

Output file precision

Other notes

Checkpoints

Picking the last step during checkpoint recovery

Checkpoint data binary format

Checkpoint metadata format (required only once per time series)

Member Enumeration Documentation

◆ OutputTypeFloat

◆ OutputTypeInt

Constructor & Destructor Documentation

◆ XDMFOutput()

Member Function Documentation

◆ add() [1/2]

◆ add() [2/2]

◆ set_write_counter()

◆ write()