This repository is the official implementation of Learning Event-triggered Control from Data through Joint Optimization by N. Funk, D. Baumann, V. Berenz and S. Trimpe, which has been published in the IFAC Journal of Systems and Control. Additional video material depicting the performance of the trained models can accesssed here.

If you use code or ideas from this work for your projects or research, please cite it.

@article{funk_learn_etc,
title = {Learning event-triggered control from data through joint optimization},
journal = {IFAC Journal of Systems and Control},
volume = {16},
pages = {100144},
year = {2021},
issn = {2468-6018},
doi = {https://doi.org/10.1016/j.ifacsc.2021.100144},
url = {https://www.sciencedirect.com/science/article/pii/S2468601821000055},
author = {Niklas Funk and Dominik Baumann and Vincent Berenz and Sebastian Trimpe}
}

1. (recommended but not required) Install Conda, using Python 3 or higher.

2. Clone the repo

3. Install the required packages.

   1. If conda has been installed, inside z_additionals/conda_env there is the conda_env.yml file which can be used to obtain all the required packages. Thus, execute

      conda env create -f conda_env.yml 
      

   2. Otherwise, make sure that your python environment you want to use contains the packages depicted in the yaml file

4. Activate your Python environment. If conda has been used:

   conda activate jl_etc
   

5. Install the components of this repository:

   cd PATH_TO_THIS_REPO/baselines
   

   pip install -e .
   

6. (recommended but not required) General Remark: The results depend on Mujoco as well as the OpenAI Gym. For me it was easiest to install both of these components from source as follows:

   1. Clone the Gym repository, commit used: a6bbc269cf86b12778206d6ddda7097510e1328d

      cd gym
      

      git checkout a6bbc269cf86b12778206d6ddda7097510e1328d
      

      pip install -e .
      

   2. Clone Mujoco-py repository, commit used: 452b3629da92c5f9227430f5e79788db8ef0b71

      cd mujoco-py
      

      git checkout 1452b3629da92c5f9227430f5e79788db8ef0b71
      

      pip install -e .
      

   3. add the MuJoCo license, and the mjpro150 binaries to "/.mujoco"

Perform the steps, exactly as described above. Further, instead of using the conda_env.yml; use the yaml: verification_env.yml. The conda environment is called mujoco-veri instead of jl_etc

In addition to the previously presented steps, we also have to install Marabou:

• Clone Marabou repository, commit used: 9a40623e2cff35c4a2adcad1217ff0741817ceee

  cd Marabou
  

  git checkout 9a40623e2cff35c4a2adcad1217ff0741817ceee
  

  mkdir build
  

  cd build
  

  cmake .. -DBUILD_PYTHON=ON
  

  cmake --build .
  

• Remark: On Ubuntu 14.04 these instructions worked directly as described here. However, on Ubuntu 16.04 I had trouble as the building process failed due to an error of Asan. In order to circumvent the error, I simply set option(RUN_MEMORY_TEST "run cxxtest testing with ASAN ON" OFF) to off in the CMakeLists, then the commands also worked as described.
• Important: for the framework to work, the path to Marabou has to be set correctly in the retrain_proc/checkpol.py file, inside this repository

• The instructions how to train and evaluate the available models are provided inside the README of the functioning_implementations folder

• The instructions how to launch the retraining procedure are contained in the README of the retrain_proc folder

• The folder pretrained_models contains the models, trained using our algorithm that are presented in the publication.

Short overview over the repo:

• baselines folder mainly includes the original OpenAI baselines repository, which has been slightly modified such that all sorts of algorithms can be trained
• eval_code folder contains the script for evaluating the models
• nfunk folder contains the customized gym environments (especially the Pendulum environment), helper functions needed for the baselines package and the implementation of LQR agents
• retrain_proc folder contains the files required for retraining NN policies
• functioning implementations folder contains all the source files for training the different implemented policies. Further, also the commands how to launch the training and evaluation are provided.
• pretrained_models folder contains the models that have been presented in the publication, again with commands how to evaluate them
• z_additionals contains several files:
  • the exported conda environments, required to reproduce the results
  • it also contains a folder called modified_ant_env. Inside this folder are the instructions how to add the modified Ant environment (called Antnff-v8) to the OpenAI gym implementations

While for the most part, the repo is self-contained, a prerequisite for using the retraining is the successfull installation of Marabou. Further, a MuJoCo licence is required for the experiments in higher dimensions.

This repository is based on previous work:

It contains parts of the OpenAI baselines repository inside the folder baselines. Inside this folder you can also find the corresponding license.

The implementation of our proposed hierarchical reinforcement learning algorithm is based on prior work by Martin Klissarov et. al and their PPOC repository.

For the stability verification algorithm we use parts from the NNet repository. The license, as well as the files that we use from this repo are placed in the folder retrain_proc/utils.

If you use this code, please kindly cite the publication.