Advanced Evaluation skill
Design LLM judges with evidence-based rubrics, swapped pairwise comparisons and human calibration.
Community · Muratcan Koylan| Author | Muratcan Koylan |
|---|---|
| Licence | MIT |
| Source | muratcankoylan/Agent-Skills-for-Context-Engineering · SKILL.md on GitHub |
| Works in | Claude Code · Codex, Gemini CLI, Cursor, Copilot |
| Read from the source on |
The advanced-evaluation skill designs model-assisted judging for agent outputs.
It covers direct scoring and pairwise comparison, with explicit rubrics, evidence and bias checks. Its Python examples show prompt structures and expected results rather than calling a live judge.
From the skill's SKILL.md: ## Gotchas
What it does
- Choose direct scoring or pairwise comparison for the task.
- Define scoring levels and require evidence for judgments.
- Swap candidate order and calibrate against human labels.
What you get
- Judge prompts, scoring rubrics and a bias-calibration plan.
When to use it
- Comparing outputs whose quality cannot be decided by deterministic checks alone.
When not to use it
- Treating an uncalibrated model score as objective truth or a guaranteed probability.
What it needs
- A file-capable agent, candidate outputs, task-specific criteria and human-labelled examples.
- Live judging requires a model integration and may incur provider charges. The bundled Python example returns prepared structures without a live model call.
Files in the skill
| File or folder | What it is for |
|---|---|
SKILL.md | Judge design, pairwise comparison and calibration guidance. |
references/implementation-patterns.md | Python examples and reference guides for implementation, bias, metrics and pipelines. |
scripts/evaluation_example.py | Python examples and reference guides for implementation, bias, metrics and pipelines. |
Licence as found in the repository on 2026-11-20: MIT.
Example prompts
Type one of these after installing; naming the skill makes sure it is used.
- Draft an evidence-based rubric for these support answers.
- Compare both candidate orders and map winners back to the original identities.
- Plan human calibration and a stress test for verbosity bias.
How to install advanced-evaluation
Keep the reference folder intact
Copy the complete skill directory. The context-engineering marketplace installs the collection, not only this judge-design skill.
No provider key is needed to read or inspect the examples. Supply a real judge client and budget for live evaluation; hosted upload and API execution are not verified here.
A skill is a folder with a SKILL.md file. You install it once; the agent reads its short description at the start of each session and loads the full instructions only when a request matches. Pick the place you use:
Claude Code: from the plugin marketplace
- Start Claude Code in a terminal with
claude. - Add the marketplace (once per computer):
/plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering - Install the plugin. Claude Code opens its details first: check what it will add (skills, hooks, MCP servers), then choose whether it is for you, for everyone on this repository, or for you in this repository only.
/plugin install context-engineering@context-engineering-marketplace - Type / and look for
/context-engineering:advanced-evaluation, or simply ask for the task.
From your shell instead of a session:
claude plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
claude plugin install context-engineering@context-engineering-marketplaceThe plugin installs as a whole, so other skills come with it, including context-fundamentals, context-degradation, context-compression, context-optimization, self-managed-context, multi-agent-patterns, long-horizon-prompting, memory-systems, tool-design, filesystem-context, hosted-agents, evaluation, harness-engineering, project-development, bdi-mental-states, latent-briefing, self-improvement-loops.
Claude Code: copy the folder into your skills folder
Put the skill's folder in ~/.claude/skills/ to use it in every project on this computer, or in .claude/skills/ inside a repository to share it with everyone who works there (commit it).
git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p ~/.claude/skills
cp -R Agent-Skills-for-Context-Engineering/skills/advanced-evaluation ~/.claude/skills/Claude Code notices new or changed skills in these folders during the session. If the skills folder did not exist when the session started, run /reload-skills once. Then type /advanced-evaluation or just describe the task.
On Windows, run the commands in Git Bash or WSL, or copy the folder into the same place with File Explorer.
Codex, Gemini CLI, Cursor and GitHub Copilot
These agents read the same SKILL.md folders. The one folder all four accept inside a project is .agents/skills/, so this copies the skill there:
git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p .agents/skills
cp -R Agent-Skills-for-Context-Engineering/skills/advanced-evaluation .agents/skills/| Agent | Project folder | Your user folder | Notes |
|---|---|---|---|
| OpenAI Codex | .agents/skills/ | ~/.agents/skills/ | Type $ or run /skills to pick a skill; Codex also chooses one when your task matches. |
| Gemini CLI | .agents/skills/.gemini/skills/ | ~/.agents/skills/~/.gemini/skills/ | Or install from Git with gemini skills install <git URL> --path <folder> --consent. Google replaced Gemini CLI with Antigravity CLI for free-tier and Google One users on 18 June 2026; Antigravity uses .agents/skills/ in a project and ~/.gemini/config/skills/ for you. |
| Cursor | .agents/skills/.cursor/skills/ | ~/.agents/skills/~/.cursor/skills/ | Type / in Agent chat to pick a skill. Cursor also reads Claude Code's .claude/skills/ folders. |
| GitHub Copilot | .github/skills/.agents/skills/.claude/skills/ | ~/.copilot/skills/~/.agents/skills/ | Works in Copilot CLI, the cloud agent, and agent mode in VS Code, Visual Studio and JetBrains (IDE support depends on the version). Or run gh skill install OWNER/REPOSITORY SKILL (public preview, GitHub CLI 2.90.0 or later). |
Sources for these steps
Checked against each vendor's documentation on 2026-11-14:
Evidence before the aggregate score
Define what each score level means and require supporting observations. Validate deterministic constraints before asking a model to judge remaining quality questions.
Keep judge inputs and rubrics versioned. Judge-model changes can alter scores even when the candidate output stays the same.
Calibration and disagreement
Evaluate agreement with human labels and use task-appropriate metrics. A reported confidence value is not a calibrated probability unless that calibration has been tested.
Inspect disagreements and bias stress cases. Escalate consequential or inconsistent judgments to human review rather than hiding them in one average score.
Questions about advanced-evaluation
Does the Python example call an LLM?
No. It illustrates prompts, rubrics and prepared output structures. Connect and validate a real model client separately.
Why reverse candidate order?
Position can affect the judgment. Map each result to the original candidate before checking agreement; contradictory results need review or reduced confidence.
Does reversing order eliminate every bias?
No. Also test verbosity, self-preference and authority cues, and compare judgments with human-labelled examples.
When should direct scoring be used?
Use it when explicit criteria and scoring levels are useful. Pairwise comparison can help with relative preferences; calibrate either approach on the actual task.
More AI skills
Related skills
Agent Evaluation skill
Community · Muratcan KoylanPlan agent regression suites with deterministic checks, separate quality dimensions and versioned baselines.Context Degradation skill
Community · Muratcan KoylanDiagnose long-context failures and plan recovery using verified facts and workload-specific comparisons.Dev-Browser skill
Community · Sawyer HoodAutomate Chrome with persistent named pages, Puppeteer scripts and a local command-line service.Tool Design skill
Community · Muratcan KoylanDesign agent-facing tool contracts with clear inputs, outputs, selection guidance and recoverable errors.Brainstorming skill
Community · Jesse Vincent / Prime RadiantTurn an undeveloped idea into an agreed design before writing implementation code.Writing Plans skill
Community · Jesse Vincent / Prime RadiantConvert an approved design into reviewable engineering tasks with interfaces and verification.