Floowandereeze & Modding ETL

This project is an Extract, Transform, Load (ETL) pipeline designed to extract metadata from the global version of the Yu-Gi-Oh! Master Duel game via reverse engineering, transform it into a structured format, and store the results in Parquet files, mainly to be used by the Floowandereeze & Modding tool.

This project does not extract the assets themselves (e.g. png files containing art of cards), it only gets the references to the bundles containing the data, which can be used for modding.

• Extracts game metadata by reverse engineering Unity game files. The data currently extracted includes:
  • Card Arts.
  • Card Descriptions and Names (en-us versions).
  • Card Faces.
  • CARD_* Files (en-us versions).
  • Deck Boxes.
  • Fields.
  • Player Icons.
  • Sleeves.
• Transforms raw game data into a structured format, and cleans bad data.
• Loads the processed data into Parquet files for efficient storage and querying.

• Technologies Used
• Installation
• Usage
• Testing
• Important Files Structure
• Configuration
• CI/CD
• License
• Credits

• Python for scripting and data processing
• UnityPy for reverse engineering Unity assets
• pandas for data manipulation
• PIL for image manipulation

Ensure you have the following installed:

• Python 3.8+
• pip package manager
• Yu-Gi-Oh! Master Duel, with the in-game download complete

# Clone the repository
git clone https://github.com/Nauder/floowandereeze-and-modding-etl.git
cd floowandereeze-and-modding-etl

# Install dependencies
pip install -r requirements.txt

Replace the empty string in config.json with your game path.

Run the extraction script to pull data from Unity game files:

python .\etl\main.py

After extracting the data, the script will prompt about the layout of the game fields it found, as determining the type of each field has not been automated yet.

Finally, the data will be available as Parquet files inside the data/ folder, as well as a version.txt file containing the date of the last script run.

The project uses pytest for unit and integrity tests.

Install the development dependencies (separate from the main pipeline requirements):

pip install -r requirements-dev.txt

# Run all tests with verbose output
python -m pytest tests/ -v

# Or simply (pytest.ini configures testpaths = tests)
pytest

• conftest.py adds etl/ to sys.path, mirroring how python etl/main.py is run.
• The integrity tests (test_parquet_integrity.py) validate the data/ Parquet files that are committed to the repository and require no game installation.
• Unit tests mock GameService and other I/O-heavy components, so no game path is needed.

├── data/                 # Final Parquet files
├── etl/
│   ├── decode/           # Decoding logic
│   ├── services/         # Pipeline logic
│   ├── main.py           # Main script
│   └── util.py           # Utility functions
├── tests/                # pytest test suites
├── config.json           # Configurable parameters
├── requirements-dev.txt  # Development dependencies (pytest)
└── README.md             # This

The data/ folder in the repository contains the game metadata as of it's last push, so if there was no game update after the last push it should already be up to date.

Modify the values in config.json, the configurations are:

• game_path path to your Master Duel installation's user data, up to the 0000 folder.
• ocg_game_path path to the OCG Master Duel version's user data, up to the 0000 folder. If omitted or invalid, OCG asset metadata extraction is skipped with a warning.
• num_threads amount of threads to use when extracting data, performance varies by hardware.
• excluded_sleeves sleeve assets to be ignored when building the list of sleeves. The game names sleeve materials the same way as animated sleeve frames, so they are removed manually.

The only mandatory configuration is the game_path, the rest come with default values that should be appropriate for most cases.

This project uses GitHub Actions for continuous integration and code quality checks:

• Pylint: Runs on every push and pull request to the main branch to ensure code quality and catch potential errors. The check requires a minimum score of 9/10.
• Black: Runs on every push and pull request to the main branch to enforce consistent code formatting across the project.
• Tests: Runs the full pytest suite on every push and pull request to the main branch.

The etl/decode folders code is excluded from checks, as it is external code and not maintained in this repository.

This project is licensed under the GNU General Public License.

• Akintos and Timelic for the decoding scripts.