ncdlgen in a nutshell: A library that enables creating interfaces for reading and writing structured typed data in multiple formats.

ncdlgen is a C++ library that provides a NetCDF cdl-parser and a code generator for C++ code that reads/writes NetCDF files to reads/writes data with ZeroMQ.

NetCDF is a platform independent data file format for structured data. It is commonly used in Earth observation missions as data storage format. NetCDF cdl-files describe structured data and are used together with the NetCDF library tools. The Netcdf CDL grammar is available here.

ZeroMQ provides sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast.

In the future ncdlgen could be used as a code generation tool for other structured data formats as well.

See concrete example for using ncdlgen as a library to enable code generation and writing to different pipes under examples/. Main current features include generating interface code based on .cdl file and reading the same data in multiple formats.

Interface is described via .cdl file:

netcdf Data {
  variables:
    int foo ;
    float bar ;
}

And passing this data

// Data in generated during compilation with the interface code generator
generated::Data data{{1}, {2.0}};

// Create ZeroMQPipe at local socket and send data
ncdlgen::ZeroMQPipe pipe();
generated::write(pipe, data);

// Receive the data (other process, node etc.)
generated::read(pipe, data);

// Write received data to file
ncdlgen::NetCDFPipe ncpipe("data.nc");
generated::read(ncpipe, data);

Clone the repository

cd ncdl-gen
git clone git@github.com:miikama/ncdl-gen.git

Get all dependencies with conan. Alternatively, download dependencies and configure cmake by hand

conan install .

Install the library for downstream consumers (with conan)

conan build .
conan export .

After installing libraries with conan ncdlgen can be install with

cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/ncdlgen -DCMAKE_PREFIX_PATH=$(pwd)/Release/generators ..
make -j6 && make install

After running conan install . all the required packageConfig.cmake files for ncdlgenare now under build/Release/generators for finding the dependencies. It is also possible to manually download all the primary and transitive dependencies without Conan. Currently primary dependencies are

• fmt
• netCDF
• GTest
• cppzmq

NOTE: add permanent setting with conan profile update settings.compiler.libcxx=libstdc++11 default

To build and run the tests, enable them separately by setting cmake -DBUILD_TESTING=ON .. && make && make test.

Take an example cdl-file

netcdf simple {

    group: foo {

        dimensions:
            dim = 5 ;

        variables:
            int bar ;
            float baz ;
            ushort bee(dim) ;
            int foobar(dim, dim) ;
    }
}

Result of parsing the file contents:

${installation_directory}/parser data/simple.cdl
Group simple
  Group foo
      Dimensions
          dim = 5
      Variables
          int bar
          float baz
          ushort bee (dim)
          int foobar (dim, dim)

Take the same example data/simple.cdl file but use it as an input for the code-generator:

${installation_directory}/generator data/simple.cdl --header --target_pipes NetCDFPipe --interface_class_name generated_simple

results in the following generated code

#pragma once

#include "stdint.h"

#include "netcdf_pipe.h"

#include <vector>

#include "vector_interface.h"

namespace ncdlgen {

struct simple
{
  struct foo
  {
      int bar;
      float baz;
      std::vector<ushort> bee;
      std::vector<std::vector<int>> foobar;
  };

  foo foo_g{};
};

void read(NetCDFPipe& pipe, simple&);

void read(NetCDFPipe& pipe, simple::foo&);

void write(NetCDFPipe& pipe, const simple&);

void write(NetCDFPipe& pipe, const simple::foo&);

};

The corresponding source file can be generated with

${installation_directory}/generator data/simple.cdl --source --target_pipes NetCDFPipe --interface_class_name generated_simple

This can be integrated as part of a CMake build (as done for the test/CMakelFiles.txt)

# Run generator to create test wrappers
add_custom_command(
                   OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h
                   OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp
                   COMMAND generator ${CMAKE_SOURCE_DIR}/data/simple.cdl --header > ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h
                   COMMAND generator ${CMAKE_SOURCE_DIR}/data/simple.cdl --source > ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp
                   DEPENDS generator
                   DEPENDS ${CMAKE_SOURCE_DIR}/data/simple.cdl
                   VERBATIM
                   )
# Add dependency to generated code
add_custom_target(generated-test-code
                  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h
                  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp  )
set_source_files_properties(${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h
                            ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp
                            PROPERTIES GENERATED TRUE)
set(GENERATED_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp)

If you want, the available pipe read/write entries can either be disabled with

./generator data/simple.cdl --header --target_pipes  {} --interface_class_name generated_simple --interface_namespace_name generated

Or you can generate for all the supported pipes

./generator data/simple.cdl --header --target_pipes NetCDFPipe ZeroMQPipe --interface_class_name generated_simple --interface_namespace_name generated

Which generates the following additional interfaces on

#pragma once

#include "stdint.h"

#include "pipes/netcdf_pipe.h"
#include "pipes/zeromq_pipe.h"

#include <vector>

#include "vector_interface.h"

namespace generated {

struct simple
{
  struct foo
  {
      int bar;
      float baz;
      std::vector<uint16_t> bee;
      std::vector<std::vector<int>> foobar;
  };

  foo foo_g{};
};

void read(NetCDFPipe& pipe, simple&);

void read(ZeroMQPipe& pipe, simple&);

void read(NetCDFPipe& pipe, simple::foo&);

void read(ZeroMQPipe& pipe, simple::foo&);

void write(NetCDFPipe& pipe, const simple&);

void write(ZeroMQPipe& pipe, const simple&);

void write(NetCDFPipe& pipe, const simple::foo&);

void write(ZeroMQPipe& pipe, const simple::foo&);

};

The generated code for reading and writing to pipes can be used to read/write the contents of the entire file or its subgroups

#include "generated_simple.h"

ncdlgen::simple root;
ncdlgen::NetCDFPipe pipe{"generated.nc"};
pipe.open();

// Write contents of 'simple' struct to a netcdf file
ncdlgen::write(pipe, root);

// Read the contents of a netcdf file into 'simple' struct
read(pipe, root);

pipe.close();

// Configure ZeroMQPipe
ncdlgen::ZeroMQPipe zeromq_pipe {};

// Push the contents through ZeroMQPipe
ncdlgen::write(zeromq_pipe, root);

// Read the contents through ZeroMQPipe
ncdlgen::read(zeromq_pipe, root);

See example for downstream usage under the example directory.

Clone the ncdlgen repository. In repository root, run

conan install .
conan build .
conan export .

If using conan yourself, you can make a small conanfile with the wanted dependencies

[requires]
 ncdlgen/0.3.0
 netcdf/4.8.1
 cppzmq/4.10.0

And run conan install . --build=missing. Which compiles and installs the ncdlgen library.

Then, configure your project with cmake with

cd build
cmake -DCMAKE_PREFIX_PATH=$(pwd)/Release/generators -DCMAKE_BUILD_TYPE=RELEASE ..

Then compile.

The example uses the generator binary compiled with the ncdlgen build. This has to be in PATH

export PATH="$PATH:<ncdlgen-install-dir-with-generator-binary>"
make

After installing ncdlgen you can use the library in your projects CMakeLists.txt

cmake_minimum_required(VERSION 3.16)
project("ncdlgen-examples")

find_package(ncdlgen REQUIRED)

add_executable(custom_parser custom_parser.cpp)
target_link_libraries(custom_parser PRIVATE ncdlgen::ncdlgen)
`

The example CMakeLists.txt runs the interface generator to build example_data.h interface during compilation. This file is included in the example custom_parser.cpp file.

There is a Dockerfile that setups a build environment for the current user. Build the docker file with

docker build \
    --build-arg USER=$(whoami) \
    --build-arg USER_ID=$(id -u $(whoami)) \
    --build-arg GROUP_ID=$(id -g $(whoami)) \
    --tag=ncdgen \
    .

After building, start the container with

docker run --rm -it -v $(pwd):/home/$(whoami) ncdgen bash

This mounts the repository at the home directory of the container user.

Main features for each release

0.1.0

• Initial relase with NetCDF cdl parser
• Code generator for generating code for interface reading/writing

0.2.0

• Support Conan 2
• Support multidimensional containers in interfaces
• Support multidimensional containers in code generation
• Update gtest version
• Improve code generation configurability
• Support global attributes outside of variables: section
• Resolve untyped attribute types by finding corresponding variable
• Make NetCDF and optional dependency

0.3.0

• Rename NetCDFInterface as NetCDFPipe
• Introduce ZeroMQPipe
• Add cppzmq/4.10.0 optional dependency
• Add cli11/2.4.2 dependency
• Improve code generation configurability
• Make project exportable with conan

Official guides

Getting started with extensions

VSCode language extensions

The language extensions add a language Contribution Point

Text mate grammar guides

Writing grammar

Notes about textmate language grammar

Some inspiration is derived from the first example I found, which is the Jakt language in SerenityOS

Jakt language syntax