IaCGen

IaCGen is a LLM improvement framework in Infrastructure-as-Code (IaC) generation.

DPIaC-Eval is the first deployablility-focused IaC benchmark that focuses on CloudFormation and AWS.

Check the paper at FSE 2026.

1. Download the project
2. Install AWS CLI and setup credentials
3. Download required libraries in the requirement.txt
4. *Obtain the following LLM model inference API keys as appropriate. Currently IaCGen support all models from the following three providers:

• OpenAI API: for GPT-4o and o3-mini (Used in the paper)
• Anthropic API: for Claude-3-5-Sonnet and Claude-3-7-Sonnet
• DeepSeek API: for DeepSeek-R1 and DeepSeek-S3

5. Add a .env file under the IaCGen directory with your own API key:

GEMIN_API_KEY=your_actual_gemini_api_key_here
CHATGPT_API_KEY=your_actual_gpt_api_key_here
CLAUDE_API_KEY=your_actual_claude_api_key_here
DEEPSEEK_API_KEY=your_actual_deepseek_api_key_here

6. Follow the instructions in Code/README.md to execute the IaCGen.

• You can check our benchmark (DPIaC-Eval) dataset under the Data folder.
• You can check the code for IaCGen framework under the Code folder.
• Note: Please check the README.md file in each of the folders for a detailed description.
• Note: You can simply download the project and run the main.py in Code folder to test IaCGen. You can edit the variables in the last part of the Python file to control how you want to use IaCGen, such as the type of model and which IaC problem/s you want to test with.

This project is released under Apache License 2.0. For commercial collaborations, enterprise use, or licensing inquiries, please contact (tianyi2332@163.com).

Submit a PR if you want to contribute to the project.

If find bugs, issues, or have suggestions, please share them via GitHub Issues.

IaC-Eval