Add agent feedback behavior evals - #98887
Merged
Merged
Conversation
Contributor
Tests PassedCommit: 870089d |
Base automatically changed from
codex/experimental-agent-feedback
to
canary
September 18, 2026 16:18
aurorascharff
marked this pull request as ready for review
September 18, 2026 16:20
devjiwonchoi
approved these changes
Sep 18, 2026
aurorascharff
enabled auto-merge (squash)
September 18, 2026 16:33
aurorascharff
added a commit
that referenced
this pull request
Sep 23, 2026
## Summary - reduce the managed agent-feedback block from 199 to 160 words while restoring four scannable issue-category bullets - explicitly authorize the feedback pass without coupling it to `agentRules` - explain that `next dev` restores the managed block and that committing `AGENTS.md` keeps the tree clean - preserve the network-sandbox retry and tell agents to read the command output once without piping or truncating it - clarify that prepared reports remain in a browser URL fragment until the user submits them - keep the AI agents guide, API reference, focused integration coverage, and eval fixtures in sync - restore the `CLAUDE.md` compatibility import inside eval sandboxes only, without generating it for users - replace subjective feedback transcript judges with deterministic source and report-payload checks The existing `agentRules` managed block is unchanged. ## Eval results | Eval | Existing instructions | Shortened instructions | | --- | --- | --- | | Routine debugging | 10/10 passed in #98887 | Earlier runs produced no reports; a post-harness-fix rerun is pending | | Anonymization | 2/2 reporting checks passed in #98887 | The post-harness-fix rerun reached AI Gateway timeouts; deterministic schema and redaction checks are in place | | Distinct issues | 1/1 passed in #98887 | The post-harness-fix run produced two distinct valid reports; both deterministic report checks passed | The historical results come from #98887. The shortened-instruction runs use the same private `agent-feedback` treatment and are not published to nextjs.org/evals. The fixtures explicitly enable `experimental.agentFeedback`, matching the production opt-in. The previous transcript judges mixed feature behavior with subjective judgments about browser-opening attempts. The fixtures now check the actual product outcomes directly: routine work produces no report, a qualifying issue produces one valid anonymized report, and distinct issues produce separate valid reports. ## Verification - `pnpm testonly packages/next/src/server/lib/generate-agent-files.test.ts` (10/10 passed) - `NEXT_SKIP_ISOLATE=1 HEADLESS=true pnpm test-dev-turbo test/development/app-dir/agent-rules-auto-generate/agent-rules-auto-generate.test.ts` (12/12 passed) - eval fixture dry-runs for `agent-057-agent-feedback-anonymization` and `agent-058-agent-feedback-distinct-issues` - targeted Prettier and ESLint through the pre-commit hook <!-- NEXT_JS_LLM -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds eval coverage for the experimental agent feedback workflow:
Also adds repeat-run and variant controls so trigger frequency can be measured across multiple runs.
Agent-feedback and selected local-only Skill evals are marked
publish: false, with scopedevals/AGENTS.mdinstructions preventing them from being exported to the public benchmark.Why?
We need to measure trigger precision, anonymization, issue splitting, and duplicate prevention before expanding the experiment.
How?
The
agent-feedbacktreatment uses the managed block and bundled reporting protocol. Only the remote rollout gate is forced on inside the eval sandbox so runs are deterministic.The first attempted run did not reach the agent because the local sandbox was not linked to a Vercel project. It is infrastructure setup and is not included in the results above.
The first distinct-issues run produced the expected two separate payloads. Its scorer rejected them because the parser did not allow the existing
tokenquery parameter and the browser criterion required an open attempt even when the agent environment exposed no browser capability. After correcting those assertions and clarifying the stopping-point wording, the scored rerun passed.Both anonymization treatments produced one valid payload with none of the seeded customer, project, route, local-path, internal-URL, or secret values. The corrected rerun also selected
misleading-errorand passed every reporting assertion. An inherited.next-preservation assertion was removed from this fixture because it measuresnext-dev-loopbehavior, not agent-feedback anonymization; Skill queue coordination belongs in a separate focused eval.