NiceTryGPT ☕🤖

NiceTryGPT is a tiny Agent Skill that takes an existing, authorized CTF challenge, solves it end-to-end, identifies cheap LLM shortcuts, and applies the smallest useful change to reduce them.

Without making the challenge worse for humans.

Increase uncertainty, not complexity.

NiceTryGPT is intentionally small: one skill, five tiny demos, reproducibility tooling, no framework.

Since publishing NiceTryGPT, I have discovered an entirely unplanned side effect of the project: I have never received so many unsolicited offers to buy GitHub stars, increase citations to my DOI, boost the repository's visibility, or achieve guaranteed success — sometimes, apparently, starting from just $1.

I am still not entirely sure which particular property of this repository triggered such enthusiasm from the less scientific corners of the Internet.

At this point, however, the phenomenon appears sufficiently reproducible that I am seriously considering it an undocumented NiceTryGPT benchmark.

Further investigation may be required. 😄

For a project-local Claude Code skill, copy the nice-try-gpt directory into your CTF repository as:

.claude/skills/nice-try-gpt/
└── SKILL.md

Then ask:

Use NiceTryGPT on this CTF. Solve it first, identify the cheapest LLM shortcut,
make the smallest useful change, and verify the result end-to-end.

Claude Code discovers project skills from .claude/skills/<skill-name>/SKILL.md. Custom skills can also be packaged and uploaded where supported. See the official Agent Skills documentation.

NiceTryGPT is also structured as a native Claude Code plugin. The repository root contains .claude-plugin/plugin.json, and Claude Code auto-discovers the mirrored skill under skills/nice-try-gpt/.

For local plugin testing:

claude --plugin-dir /path/to/NiceTryGPT

A standalone marketplace manifest is also prepared so users can add the repository directly and install the plugin with:

/plugin marketplace add aleff-github/NiceTryGPT
/plugin install nice-try-gpt@nicetrygpt

The original standalone skill layout remains available under nice-try-gpt/, so existing ZIP/project-local installation workflows continue to work. See docs/claude-plugin.md for plugin structure, synchronization, and distribution notes.

NiceTryGPT has also been verified with the open skills installer. The installer discovers exactly one nice-try-gpt skill from this repository and can install it for Claude Code and other compatible agents:

npx -y skills add aleff-github/NiceTryGPT --skill nice-try-gpt

The compatibility check was run with telemetry disabled; no install count was generated by the project test.

NiceTryGPT keeps one canonical source repository while allowing directories and installers to point back to it.

Current distribution signals include:

• an open upstream submission to anthropics/skills (#1798);
• an existing indexed copy in AI Skill Store's marketplace;
• compatibility with the open skills CLI used by skills.sh and other Agent Skills tooling;
• crawler-friendly metadata through this README, SKILL.md, llms.txt, CodeMeta, and the project website.

For directory maintainers, canonical links, a short description, categories, keywords, installation commands, and the evidence boundary are collected in docs/distribution.md.

Directory inclusion is not an endorsement, and a pending submission is not described as accepted until its maintainers publish or merge it.

UNDERSTAND
    ↓
SOLVE ORIGINAL
    ↓
FIND ONE CHEAP SHORTCUT
    ↓
MAKE 0–2 SMALL CHANGES
    ↓
SOLVE AGAIN
    ↓
REPORT

If the original challenge cannot be reproduced, NiceTryGPT stops. If the challenge is already fine, NO CHANGE NEEDED is a valid result.

A successful transformation keeps:

• the same vulnerability class;
• the same learning objective;
• the same prerequisite knowledge;
• the same flag/success semantics;
• roughly the same human difficulty band.

The default is one resistance change. A second change is justified only when the first one is insufficient and the human-cost gate still passes.

NiceTryGPT currently ships with five deliberately tiny examples:

All keep the original vulnerability class and learning objective.

The traversal example is intentionally useful as a generalization check: it uses no honeypot. The only change is moving one solve-relevant fact from static behavior into ordinary runtime behavior.

The SQLi example deliberately avoids runtime randomization. The vulnerable query is unchanged; the player simply has to connect two nearby, static application clues before applying the same injection primitive.

The demo uses a deliberately restricted toy shell rather than operating-system command execution. The transformation changes the cue, not the injection primitive.

The template engine is intentionally restricted and does not evaluate arbitrary Python or operating-system code.

No third-party Python packages are required.

python tests/test_demo.py
python scripts/run_demo_adapters.py
python tests/test_release.py

The suite verifies that each original challenge is solvable, the identified cheap shortcut stops working after transformation, normal functionality still works, and the intended vulnerability still reaches the runtime flag.

The v0.4 artifact contract adds a machine-readable companion to each transformation report. The Markdown report explains the design decision; the JSON sidecar records the preservation checks, required human cost, evidence class, and transformation patterns in a form CI can validate.

Run the complete offline research-artifact verification with:

python scripts/verify_research_artifact.py

This validates deterministic demos, evaluation harnesses, the experiment manifest, raw-run classification, generated evidence, plugin/skill synchronization, and the derived Human Cost Gate. It does not run paid model benchmarks.

See docs/methodology.md and schemas/transformation-report.schema.json.

v0.5.0 adds an executable adapter contract for every bundled demo and a generated structural-generalization matrix across all committed transformation reports.

The current matrix contains 7 validated transformations: 5 bundled deterministic demos and 2 independently authored external transformations, spanning 7 recorded vulnerability classes and all 5 resistance patterns. Every bundled demo has an executable adapter.

This is intentionally a coverage statement, not a population-level model claim. Only Interstellar Ingress and DiceMiner currently contribute valid fresh-solver observations; the five bundled demos remain deterministic evidence only. Human difficulty remains a bounded structural criterion rather than a human-subject measurement.

Regenerate the snapshot with:

python scripts/analyze_generalization.py > docs/generalization-status.md

See docs/generalization-status.md and schemas/challenge-adapter.schema.json.

Build a deterministic ZIP containing only the installable skill:

python scripts/package_skill.py

Output:

dist/nice-try-gpt-v0.5.0.zip

The ZIP keeps nice-try-gpt/ as its root directory, so it can be inspected or copied directly into a compatible Agent Skills location.

NiceTryGPT includes a reproducible external evaluation protocol under evals/, with independently authored public CTFs, fresh solver contexts, fixed tool access, raw run accounting, and explicit infrastructure failure handling.

The current evidence milestone combines:

• a completed 5 BEFORE + 5 AFTER GPT pilot on Interstellar Ingress;
• a larger GPT study on DiceMiner, with the full 10-run BEFORE cell and a resource-bounded partial AFTER cell;
• a preregistered Claude cross-model replication attempt on Interstellar Ingress that produced no valid solver run because of an infrastructure / access-limit failure and is therefore not counted as model evidence.

Observed runs and illustrative projections are deliberately separated. Infrastructure/usage-limit failures remain auditable but are excluded from solver-performance denominators. Missing runs are never filled with synthetic results.

Generate the current evidence snapshot directly from the local CSV with:

python3 evals/analyze_evidence.py --write evals/evidence-status.md

See evals/evidence-status.md, evals/protocol.md, and the public CTF candidate registry.

The evaluation supports preliminary / proof-of-concept claims only. It does not establish universal or general LLM resistance.

v0.5.0 — structural generalization adds two new deterministic vulnerability classes, covers all five resistance patterns, standardizes bundled challenge adapters, and generates a cross-challenge structural coverage matrix. It adds no new paid model observations; the v0.3.0 external evidence remains the empirical basis.

See ROADMAP.md.

NiceTryGPT currently uses a deliberately small menu:

• Pattern break — remove a cue that practically names the exploit.
• Runtime discovery — make one fact observable through normal interaction.
• Context split — connect two nearby pieces of application behavior.
• State dependency — let a small amount of ordinary state matter.
• Semantic decoy — add one plausible path that is cheap to rule out.

These are options, not a checklist. Most challenges should need zero or one.

See resistance-patterns.md.

NiceTryGPT is neither a CTF-solving benchmark nor an anti-cheat system. Its narrow focus is minimal transformation of an existing, verified challenge while preserving its learning objective and bounding additional human effort.

See docs/related-work.md for the current positioning against CTF-agent benchmarks and recent LLM-aware challenge-design work.

NiceTryGPT ships with machine-readable CITATION.cff metadata, so GitHub can expose Cite this repository with generated APA and BibTeX formats.

For research or evaluation work, cite the release or commit you actually used and record the model version, tool access, run count, and evaluation protocol. The current release is v0.5.0. Its Zenodo DOI will be added after the release deposit is minted; the previous v0.2.0 archive remains available under DOI 10.5281/zenodo.22858477. See CITING.md for the canonical human-readable citation and codemeta.json for CodeMeta software metadata.

NiceTryGPT separates distribution, preservation, and citation:

• GitHub Releases provide versioned project releases;
• Software Heritage preserves the repository independently of GitHub; the first completed snapshot is swh:1:snp:6c77799e7623abf2653ab9363d3e2f57899174cf;
• Zenodo preserves the v0.2.0 source archive under DOI 10.5281/zenodo.22858477; the v0.4.0 DOI is intentionally not predeclared before Zenodo mints it.

See docs/preservation.md for the preservation and persistent-identifier strategy.

It will not intentionally make a challenge annoying just to slow down an AI.

That means no:

• CAPTCHA or human-verification gimmicks;
• brute force as a design requirement;
• token/context flooding;
• pointless encoding layers;
• obscure trivia;
• fake flags or destructive traps;
• artificial five-stage exploit chains;
• piles of honeypots.

If LLM resistance and human experience conflict, the human player wins.

NiceTryGPT does not claim to prove that a challenge is AI-proof.

Here, “resistance” means reducing an identified cheap shortcut while preserving the intended challenge. A same-context self-review is not model evidence; deterministic validation, solver observations, infrastructure failures, and projections are reported as distinct evidence classes.

NiceTryGPT/
├── README.md
├── CHANGELOG.md
├── VERSION
├── LICENSE
├── CITATION.cff
├── CITING.md
├── codemeta.json
├── SECURITY.md
├── CONTRIBUTING.md
├── .claude-plugin/
│   ├── plugin.json
│   └── marketplace.json
├── skills/
│   └── nice-try-gpt/
│       ├── SKILL.md
│       └── references/
├── nice-try-gpt/
│   ├── SKILL.md
│   └── references/
│       └── resistance-patterns.md
├── adapters/
│   └── *.json
├── examples/
│   ├── mini-idor/
│   ├── mini-traversal/
│   ├── mini-sqli/
│   ├── mini-command-injection/
│   └── mini-ssti/
├── evals/
│   ├── README.md
│   ├── protocol.md
│   ├── solver-prompt.txt
│   ├── results.csv
│   └── summarize.py
├── scripts/
│   ├── analyze_generalization.py
│   ├── run_demo_adapters.py
│   ├── validate_challenge_adapters.py
│   ├── package_skill.py
│   └── sync_plugin_skill.py
└── tests/
    ├── test_demo.py
    └── test_release.py

v0.5.0 — structural generalization.

The repository now has five deterministic demos, executable adapters for every bundled challenge, seven recorded vulnerability classes across all committed transformation reports, and coverage of all five resistance patterns. The empirical solver evidence is unchanged from v0.3.0 and remains preliminary; no universal LLM-resistance, human-subject, or cross-model generalization claim is made.

See CHANGELOG.md.

If you design, organize, or teach CTFs, feedback on the methodology is especially useful.

The most valuable questions are:

• does the Human Cost Gate match real challenge-design constraints?
• which transformations feel fair versus annoying?
• which vulnerability classes are most affected by one-shot LLM solving?
• what evidence would make you trust a before/after transformation?

Use the CTF author feedback issue form. No model-evaluation results are required to give design feedback.

Small, focused contributions are welcome. Read CONTRIBUTING.md first.

NiceTryGPT is intended for CTF challenges, training labs, and systems you own or are explicitly authorized to test. It is not intended to automate testing against third-party systems without authorization.

Maintained by Alessandro Greco (@aleff-github).

GNU General Public License v3.0. See LICENSE.