Skip to content

Advanced Evaluation skill

Design LLM judges with evidence-based rubrics, swapped pairwise comparisons and human calibration.

Community · Muratcan Koylan
AuthorMuratcan Koylan
LicenceMIT
Sourcemuratcankoylan/Agent-Skills-for-Context-Engineering · SKILL.md on GitHub
Works inClaude Code · Codex, Gemini CLI, Cursor, Copilot
Read from the source on

The advanced-evaluation skill designs model-assisted judging for agent outputs.

It covers direct scoring and pairwise comparison, with explicit rubrics, evidence and bias checks. Its Python examples show prompt structures and expected results rather than calling a live judge.

From the skill's SKILL.md: ## Gotchas

This is a community skill by Muratcan Koylan, not checked by us or by Anthropic. Skills can tell the agent to run code, read files or call the network. Before installing, read its SKILL.md and every script in the folder, and look for anything that does not match what it says it does.

What it does

  1. Choose direct scoring or pairwise comparison for the task.
  2. Define scoring levels and require evidence for judgments.
  3. Swap candidate order and calibrate against human labels.

What you get

  • Judge prompts, scoring rubrics and a bias-calibration plan.

When to use it

  • Comparing outputs whose quality cannot be decided by deterministic checks alone.

When not to use it

  • Treating an uncalibrated model score as objective truth or a guaranteed probability.

What it needs

  • A file-capable agent, candidate outputs, task-specific criteria and human-labelled examples.
  • Live judging requires a model integration and may incur provider charges. The bundled Python example returns prepared structures without a live model call.

Files in the skill

File or folderWhat it is for
SKILL.mdJudge design, pairwise comparison and calibration guidance.
references/implementation-patterns.mdPython examples and reference guides for implementation, bias, metrics and pipelines.
scripts/evaluation_example.pyPython examples and reference guides for implementation, bias, metrics and pipelines.

Licence as found in the repository on 2026-11-20: MIT.

Example prompts

Type one of these after installing; naming the skill makes sure it is used.

  • Draft an evidence-based rubric for these support answers.
  • Compare both candidate orders and map winners back to the original identities.
  • Plan human calibration and a stress test for verbosity bias.

How to install advanced-evaluation

Keep the reference folder intact

Copy the complete skill directory. The context-engineering marketplace installs the collection, not only this judge-design skill.

No provider key is needed to read or inspect the examples. Supply a real judge client and budget for live evaluation; hosted upload and API execution are not verified here.

A skill is a folder with a SKILL.md file. You install it once; the agent reads its short description at the start of each session and loads the full instructions only when a request matches. Pick the place you use:

Claude Code: from the plugin marketplace

  1. Start Claude Code in a terminal with claude.
  2. Add the marketplace (once per computer):
    /plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
  3. Install the plugin. Claude Code opens its details first: check what it will add (skills, hooks, MCP servers), then choose whether it is for you, for everyone on this repository, or for you in this repository only.
    /plugin install context-engineering@context-engineering-marketplace
  4. Type / and look for /context-engineering:advanced-evaluation, or simply ask for the task.

From your shell instead of a session:

claude plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
claude plugin install context-engineering@context-engineering-marketplace

The plugin installs as a whole, so other skills come with it, including context-fundamentals, context-degradation, context-compression, context-optimization, self-managed-context, multi-agent-patterns, long-horizon-prompting, memory-systems, tool-design, filesystem-context, hosted-agents, evaluation, harness-engineering, project-development, bdi-mental-states, latent-briefing, self-improvement-loops.

Claude Code: copy the folder into your skills folder

Put the skill's folder in ~/.claude/skills/ to use it in every project on this computer, or in .claude/skills/ inside a repository to share it with everyone who works there (commit it).

git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p ~/.claude/skills
cp -R Agent-Skills-for-Context-Engineering/skills/advanced-evaluation ~/.claude/skills/

Claude Code notices new or changed skills in these folders during the session. If the skills folder did not exist when the session started, run /reload-skills once. Then type /advanced-evaluation or just describe the task.

On Windows, run the commands in Git Bash or WSL, or copy the folder into the same place with File Explorer.

Codex, Gemini CLI, Cursor and GitHub Copilot

These agents read the same SKILL.md folders. The one folder all four accept inside a project is .agents/skills/, so this copies the skill there:

git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p .agents/skills
cp -R Agent-Skills-for-Context-Engineering/skills/advanced-evaluation .agents/skills/
AgentProject folderYour user folderNotes
OpenAI Codex
.agents/skills/
~/.agents/skills/
Type $ or run /skills to pick a skill; Codex also chooses one when your task matches.
Gemini CLI
.agents/skills/
.gemini/skills/
~/.agents/skills/
~/.gemini/skills/
Or install from Git with gemini skills install <git URL> --path <folder> --consent. Google replaced Gemini CLI with Antigravity CLI for free-tier and Google One users on 18 June 2026; Antigravity uses .agents/skills/ in a project and ~/.gemini/config/skills/ for you.
Cursor
.agents/skills/
.cursor/skills/
~/.agents/skills/
~/.cursor/skills/
Type / in Agent chat to pick a skill. Cursor also reads Claude Code's .claude/skills/ folders.
GitHub Copilot
.github/skills/
.agents/skills/
.claude/skills/
~/.copilot/skills/
~/.agents/skills/
Works in Copilot CLI, the cloud agent, and agent mode in VS Code, Visual Studio and JetBrains (IDE support depends on the version). Or run gh skill install OWNER/REPOSITORY SKILL (public preview, GitHub CLI 2.90.0 or later).
Only install skills you trust. Anthropic's advice is to use skills only from trusted sources, and to read every file of any other skill before using it: a skill can direct the agent to run commands, read files or send data in ways its description does not mention.

Sources for these steps

Checked against each vendor's documentation on 2026-11-14:

Evidence before the aggregate score

Define what each score level means and require supporting observations. Validate deterministic constraints before asking a model to judge remaining quality questions.

Keep judge inputs and rubrics versioned. Judge-model changes can alter scores even when the candidate output stays the same.

Calibration and disagreement

Evaluate agreement with human labels and use task-appropriate metrics. A reported confidence value is not a calibrated probability unless that calibration has been tested.

Inspect disagreements and bias stress cases. Escalate consequential or inconsistent judgments to human review rather than hiding them in one average score.

Questions about advanced-evaluation

Does the Python example call an LLM?

No. It illustrates prompts, rubrics and prepared output structures. Connect and validate a real model client separately.

Why reverse candidate order?

Position can affect the judgment. Map each result to the original candidate before checking agreement; contradictory results need review or reduced confidence.

Does reversing order eliminate every bias?

No. Also test verbosity, self-preference and authority cues, and compare judgments with human-labelled examples.

When should direct scoring be used?

Use it when explicit criteria and scoring levels are useful. Pairwise comparison can help with relative preferences; calibrate either approach on the actual task.

More AI skills

Related skills

Skill categories