fwaun-tools

Tooling for training the fwaun model family. Two command groups under one binary:

• dataset — a tag and caption editor for local Stable Diffusion LoRA datasets. Combines manual edits, an automatic WD14-family tagger, Qwen3-VL captioning, and Danbooru API fetches under one editing surface so the user doesn't have to think about provenance while curating.
• model — diffusion-checkpoint utilities over safetensors files: task-vector merge-diff, LoRA extraction (extract-lora), and INT8+ConvRot quantization (quant-int8). Pure-Rust/CPU, available in every build.

Built primarily for ANIMA preview and Krea 2 LoRA training, but the data model and export profiles are not ANIMA-specific.

The dataset commands are designed around the input formats consumed by kohya-ss/musubi-tuner and kohya-ss/sd-scripts, and the model commands around ComfyUI. Use with other trainers or inference stacks is not tested.

日本語版 README

• One chip list, three sources. Manual tags, auto-tagger output, and Danbooru tags live in the same list — color-coded, but the user curates without thinking about provenance.
• Negative-tag suppression that survives model swaps. Mark -foo once; re-running the tagger preserves the suppression even with a different model.
• Dataset-wide tags in one line. common_tags in fwaun-tools.toml applies the trigger word and its -hair_colour suppressions to every image without touching a single sidecar — including images added later. Any image can opt out individually.
• Curation-only organizational tags. A positive manual tag starting with an underscore (_foo) is kept in the data and counted for tag-group sorting but never exported — so you can mark "reviewed, none of these" distinctly from "not yet reviewed".
• Tag groups + Kanban view. Declare mutually-exclusive tag sets (e.g. costume variants) in fwaun-tools.toml; the GUI shows one column per tag with drag-and-drop to switch.
• Full-size preview. Double-click a thumbnail to check the image at screen resolution, ← / → to step through the filtered list, Esc to close — or hand it to your usual viewer with the right-click menu's Open in default app / Show in folder.
• Per-folder configuration via fwaun-tools.toml. Pick the tagger model, captioner, export profile and threshold per dataset.
• Two output modes. export writes one <image>.txt per image (sd-scripts DreamBooth/LoRA caption-file mode); metadata writes a single meta.json (sd-scripts fine-tune mode).
• Fast re-opening. The GUI's Reload button re-scans the open folder and only regenerates thumbnails for images that actually changed, so picking up a CLI run or a handful of new files is instant instead of a full rebuild. Across sessions, an on-disk thumbnail cache (bounded and clearable from the config modal's App tab) does the same job.
• Bilingual GUI. English / 日本語 toggle, defaults to host locale.
• CLI for batch operations, GUI for curation. The GUI also has a Model tools tab (Dataset / Model tools mode switch) that front-ends the merge-diff / extract-lora / quant-int8 checkpoint utilities for people who'd rather not use the CLI.

curl -fsSL https://raw.githubusercontent.com/fwaunstp/fwaun-tools/main/install.sh | sh

irm https://raw.githubusercontent.com/fwaunstp/fwaun-tools/main/install.ps1 | iex

Both installers download the latest GitHub release, verify SHA256, and drop both binaries side-by-side:

Pin a specific version with --version v0.2.1 (or -Version v0.2.1 on PowerShell).

By default both binaries are installed, except on a headless Linux host (no $DISPLAY / $WAYLAND_DISPLAY), where only the CLI is installed — the GUI needs a display to run. Override the selection with --both / --cli-only / --gui-only:

# CLI only (e.g. a training box you only SSH into)
curl -fsSL https://raw.githubusercontent.com/fwaunstp/fwaun-tools/main/install.sh | sh -s -- --cli-only
# force both even on a headless host (you'll use a remote / forwarded display)
curl -fsSL https://raw.githubusercontent.com/fwaunstp/fwaun-tools/main/install.sh | sh -s -- --both

The GUI is a single self-contained binary (built with egui) — no .app, no AppImage, no MSI. On Linux it depends on the standard X11 / Wayland system libraries that ship with every desktop distribution, but no extra runtime install is required.

On macOS the binary is not notarized. The installer clears the com.apple.quarantine attribute, but if Gatekeeper still blocks it when launched from Finder, run it from Terminal once (~/.local/bin/fwaun-tools-gui).

The Linux release binaries are full builds (see Build variants), so they link against the glibc shipped on Ubuntu 24.04 (glibc 2.39). They will not run on Ubuntu 22.04, Debian 12, or earlier — the prebuilt ONNX Runtime that the local tagger / captioner depend on references __isoc23_* symbols introduced in glibc 2.38. Upgrade, or install the light CLI from source — it links no ONNX Runtime and has no glibc floor, so it runs on those older distros:

cargo install --git https://github.com/fwaunstp/fwaun-tools fwaun-tools-cli

cargo install builds the default light variant (needs a Rust toolchain; no prebuilt light binaries are published — see Build variants). The install.sh script prints this same hint if it detects an older glibc.

The maintainer develops on macOS and Linux. Windows builds are produced by CI but not regularly exercised — please file an issue if anything breaks.

Requires Rust 1.87+ (edition 2024). On Linux, install standard X11 / Wayland dev headers (libx11-dev, libxcb1-dev, libxkbcommon-dev, libwayland-dev, libgl1-mesa-dev, or your distro's equivalents) for the GUI:

git clone https://github.com/fwaunstp/fwaun-tools
cd fwaun-tools
# light build (default) — no local ONNX inference, runs anywhere
cargo build --release -p fwaun-tools-cli
cargo build --release -p fwaun-tools-gui
# full build — adds the local WD14 tagger + Qwen3-VL captioner (glibc 2.38+)
cargo build --release -p fwaun-tools-cli --features full
cargo build --release -p fwaun-tools-gui --features full

Two build flavors, selected with the full cargo feature:

The published release binaries are full. A light binary drops the two local ONNX models (WD14 tagging, Qwen3-VL captioning) but keeps everything else — including captioning via any OpenAI-compatible endpoint (llama.cpp, Ollama, LM Studio, vLLM, …). Running a local-ONNX-only command in a light build fails fast with a message telling you to install the full build. Use light when you caption over an API and/or need to run on an older-glibc host.

1. Launch fwaun-tools-gui (or run the CLI directly — see below).

2. Open folder… → pick a directory of images.

3. (Optional) Config… → write fwaun-tools.toml for the dataset. Sensible defaults apply if you skip this.

4. Select images, then click Run tagger / Run captioner / Fetch booru. The first run downloads the relevant ONNX models into the HuggingFace cache (~/.cache/huggingface/hub). Set HF_HOME to relocate the cache, or HF_ENDPOINT (e.g. https://hf-mirror.com) if you cannot reach huggingface.co directly.

5. Curate: add manual tags, suppress unwanted auto/booru tags (× strikes them through), edit captions.

6. Export to disk:

   fwaun-tools dataset export <dir>          # one .txt per image
   fwaun-tools dataset metadata <dir>        # single meta.json

fwaun-tools.toml lives in the dataset directory. Everything is optional — without it, defaults kick in. See crates/core/fwaun-tools.toml.example for the annotated full schema. Highlights:

default_profile   = "anima"
default_tagger    = "wd-eva02-large-v3"
default_captioner = "qwen3-vl-4b"

# Applied to every image in the dataset; never written to a sidecar.
common_tags = ["himeko", "-red_hair", "-yellow_eyes"]

[export.anima]
threshold = 0.35
shuffle = false
category_prefixes = { artist = "@" }

[tagger.wd-eva02-large-v3]
repo = "SmilingWolf/wd-eva02-large-tagger-v3"
input_size = 448
storage_threshold = 0.10

[captioner.qwen3-vl-4b]
repo = "onnx-community/Qwen3-4B-VL-ONNX"
subdir = "qwen3-vl-4b-instruct-onnx-vision-fp32-text-int4-cpu"
prompt = "Describe this image in detail."

Dataset curation (fwaun-tools dataset <verb>):

fwaun-tools dataset tag <dir>      [--model NAME] [--threshold X] [--force]
fwaun-tools dataset caption <dir>  [--model NAME] [--force]
fwaun-tools dataset booru <dir>    [--source danbooru] [--force]
fwaun-tools dataset export <dir>   [--profile NAME] [--threshold X]
fwaun-tools dataset metadata <dir> [--profile NAME] [--threshold X] [--output PATH]
fwaun-tools dataset add-tag <dir>    --tags TAG[,...] [--dry-run] [--per-image]
fwaun-tools dataset remove-tag <dir> --tags TAG[,...] [--dry-run] [--per-image]
fwaun-tools dataset replace-tag <dir> --from OLD[,...] --to NEW[,...] [--dry-run] [--per-image]
fwaun-tools dataset mv <dir> <dest>  --tags TAG[,...] [--dry-run]
fwaun-tools dataset status <dir>
fwaun-tools dataset tokens <dir>
fwaun-tools dataset validate-tag-group <dir> --group NAME [--problems-only] [--json]

Batch passes through an existing ComfyUI server. Both write to a separate output directory (default a <dir>_upscaled / <dir>_edited sibling), carry each .ron sidecar across, and skip images that already have an output — so an interrupted run resumes where it stopped:

fwaun-tools dataset upscale <dir> [--profile NAME] [--upscale-model FILE] [--workflow JSON] [--max-edge N] [--force] [--dry-run]
fwaun-tools dataset upscale-models     [--profile NAME] [--base-url URL]
fwaun-tools dataset edit <dir>    [--profile NAME] [--prompt TEXT] [--model NAME] [--resolution 1K|2K|4K] [--workflow JSON] [--limit N] [--force] [--dry-run]
fwaun-tools dataset edit-models        [--profile NAME] [--base-url URL]

edit applies one instruction to every image through the server's Gemini image (Nano Banana) API node — the pass that turns an illustrated set into one with photographic backgrounds. That node is billed per image, so check the count with --dry-run and try the prompt on a few shots with --limit first.

It also needs a comfy.org account API key: ComfyUI authenticates paid API nodes per request, so being signed into the web UI in a browser doesn't cover a run driven over the HTTP API — without a key every image comes back "Please login first to use this node". Generate one at https://platform.comfy.org and export it (the api_key profile field works too, but a dataset-local fwaun-tools.toml tends to travel with the dataset):

export COMFY_API_KEY=comfyui-...        # PowerShell: $env:COMFY_API_KEY='comfyui-...'

tag, caption, booru, upscale, and edit work one image at a time, so they print a progress counter while they run:

caption [  12/340 ]   3.5%  1:04 elapsed  ETA 28:37  …_0013.png

It goes to stderr and is rewritten in place, so the per-image result lines on stdout are unchanged and … > run.log still gets exactly those. When stderr isn't a terminal (redirected to a file, CI) the counter switches to one plain line per 5% of the run instead.

Checkpoint tools (fwaun-tools model <verb>) — operate on safetensors files, not a dataset directory:

fwaun-tools model merge-diff   --base B --tuned T --target G -o OUT [--multiplier M] [--model krea2|anima|auto] [--save-dtype bf16|fp16|fp32]
fwaun-tools model extract-lora --base B --tuned T -o OUT [--rank R] [--alpha A] [--model krea2|anima|auto] [--include RE] [--exclude RE]
fwaun-tools model quant-int8   SRC [DST] [--dry-run] [--include RE] [--exclude RE] [--min-gemm N] [--verify-report PATH]

merge-diff transfers a full fine-tune delta (tuned − base) onto another checkpoint; extract-lora factorizes that delta into a kohya-ss/ComfyUI LoRA by SVD; quant-int8 writes the comfy-kitchen int8_tensorwise + ConvRot layout. All three are CPU/f32 and stream key-by-key, so peak RAM stays small.

These three are also available in the GUI's Model tools tab (switch modes at the top of the window), for the same operations without the command line.

quant-int8 follows the INT8+ConvRot scheme from Comfy-Org's quant_int8_convrot.py.

add-tag / remove-tag bulk-edit the manual tag layer across a directory: add-tag appends each tag verbatim (foo positive, -foo suppression marker), remove-tag deletes matching manual entries case-insensitively (pass --tags=-foo to drop a suppression marker).

When <dir> is the dataset root, both commands edit common_tags instead — see Dataset-wide tags.

replace-tag renames one: each match is rewritten where it sits, so the entry keeps its place in manual_tags, and images that don't carry the old tag are left alone. That's what remove-tag + add-tag can't do — the pair moves the tag to the end of the list and lands the new one on the whole directory.

fwaun-tools dataset replace-tag ./dataset --from long_hair --to very_long_hair
# a whole rename table in one pass — the Nth --from goes with the Nth --to
fwaun-tools dataset replace-tag ./dataset --from cape,hat --to red_cape,straw_hat --dry-run

Matching follows remove-tag: case-insensitive, with the leading - part of the entry (--from=-foo --to=-bar renames a suppression marker; --from Foo --to foo fixes an entry's casing). A new tag the image already has absorbs the renamed entry instead of duplicating it. Unlike the other two this always walks the sidecars, and at the dataset root it also renames the entry in common_tags — --per-image leaves the config alone.

The GUI has the same operation for a selection: right-click a manual chip in the bulk panel → Rename this tag…, or type the pair into the rename row under the chips.

A character LoRA wants the same manual entries on every image: the trigger word, plus -foo suppressions for the traits that should be absorbed into that trigger rather than learned as separate tags. Declare them once:

common_tags = ["himeko", "-red_hair", "-yellow_eyes", "-slit_pupils"]

Same syntax as a sidecar's manual_tags (foo positive, -foo suppression, _foo curation-only). The list is applied as a virtual layer under each image's own entries and is never written into a sidecar, so images added to the directory later are covered without re-running anything, and deleting a line here removes it everywhere. Common positives are emitted first, so the trigger word heads the exported tag string.

An image opts out by naming the same tag itself, in any form — a per-image red_hair cancels the dataset-wide -red_hair for that image alone, and a per-image -himeko drops the trigger from a shot the character isn't in.

The layer counts everywhere an image's tags are read: export, metadata, caption prefixes/suffixes, tag-group classification, the Kanban view, and mv --tags.

A bulk tag edit that covers the whole dataset is a statement about the dataset, so it lands in the config rather than in every sidecar:

# <dir> holds fwaun-tools.toml → writes common_tags, not the .ron files
fwaun-tools dataset add-tag ./dataset --tags=himeko,-red_hair
fwaun-tools dataset remove-tag ./dataset --tags=-red_hair

A subdirectory is only part of the dataset, so it keeps the per-sidecar behaviour; --per-image forces that at the root too. --dry-run reports the config edit without writing it. A dataset-wide remove-tag leaves per-image copies of the tag alone — they may be deliberate overrides — and tells you how many it found.

In the GUI the same rule applies to the tag input and the bulk panel's chips: with every image selected in a folder that owns the config, adding or removing a tag edits fwaun-tools.toml. Deselect one image to go back to editing sidecars. Inherited entries show up as their own chip colour ([C] in the single-image detail), and clicking one overrides it for the selection.

Config edits made this way are surgical — comments and layout in a hand-maintained fwaun-tools.toml survive.

Declare named groups of tags that should be mutually exclusive on each image. The CLI's validate-tag-group reports each image as one of the group's tags, "unset", or "violation" (multiple group tags coexist — informational, not an error). The GUI's View → Kanban mode renders the same buckets as columns; thumbnails are draggable between them, and each drop rewrites manual_tags to record the new state.

Costume separation for a character LoRA — every image lands in exactly one column, so it's easy to spot stragglers:

[tag_group.official_costumes]
tags = ["official_school_uniform", "official_lounge_wear"]

Single-tag groups are valid too — handy as a "is tag X set?" sanity pass on a dataset:

[tag_group.solo_check]
tags = ["solo"]

A positive manual tag starting with an underscore (_foo) is an organizational tag: it's kept in the data and counted for tag-group classification, but never written to the exported .txt. (Suppression markers use -foo; organizational tags use _foo.)

This solves a common ambiguity in character/style tagging: an image with no group tag could mean either "not reviewed yet" or "reviewed, and it's deliberately none of these". Add an organizational tag as a group member to give the latter its own Kanban column, separate from "unset":

[tag_group.character]
tags = ["character_a", "character_b", "_no_character"]

Images you drag into _no_character are marked reviewed without the underscore tag ever leaking into the training caption.

fwaun-tools dataset validate-tag-group ./dataset --group official_costumes

• DEVELOPMENT.md — architecture, crate layout, ONNX session shapes, ort version notes. Read this before contributing.
• crates/core/fwaun-tools.toml.example — annotated configuration example.

This is a personal tool, developed to build the author's own models — fwaun-anima, fwaun-krea2, and fwaun-style on Civitai. It is not extensively tested and likely still has plenty of bugs.

New features aren't guaranteed: the author will add what they find necessary for their own work, but may not take on requests beyond that. Bug fixes to existing functionality, on the other hand, are actively welcome — issues and pull requests are appreciated.

Dual-licensed under either of:

• MIT License
• Apache License, Version 2.0

at your option.