Skip to content

Agent Evaluation skill

Plan agent regression suites with deterministic checks, separate quality dimensions and versioned baselines.

Community · Muratcan Koylan
AuthorMuratcan Koylan
LicenceMIT
Sourcemuratcankoylan/Agent-Skills-for-Context-Engineering · SKILL.md on GitHub
Works inClaude Code · Codex, Gemini CLI, Cursor, Copilot
Read from the source on

The evaluation skill designs quality checks for agent behavior across realistic tasks.

It combines deterministic validation with separate quality dimensions and human review. Judge prompts and bias calibration belong to the companion advanced-evaluation skill.

From the skill's SKILL.md: ## Gotchas

This is a community skill by Muratcan Koylan, not checked by us or by Anthropic. Skills can tell the agent to run code, read files or call the network. Before installing, read its SKILL.md and every script in the folder, and look for anything that does not match what it says it does.

What it does

  1. Define expected outcomes and deterministic checks.
  2. Build representative cases across task complexity and budgets.
  3. Compare versioned baselines and investigate regressions.

What you get

  • An evaluation plan with rubrics, regression cases and quality gates.

When to use it

  • Changing an agent, its tools or its context strategy and needing repeatable comparisons.

When not to use it

  • Treating a sample score as factual verification or production certification.

What it needs

  • A file-capable agent, representative cases, expected outcomes and execution traces.
  • The Python example includes placeholder scores and simulated agent output. A real runner, verifiers and any paid model integration must be supplied separately.

Files in the skill

File or folderWhat it is for
SKILL.mdEvaluation dimensions, test design and regression guidance.
references/metrics.mdA Python evaluator example and a reference guide to metrics.
scripts/evaluator.pyA Python evaluator example and a reference guide to metrics.

Licence as found in the repository on 2026-11-20: MIT.

Example prompts

Type one of these after installing; naming the skill makes sure it is used.

  • Design regression tests for this research agent with citation checks.
  • Separate answer quality from tool-use efficiency in the report.
  • Define a quality gate using our human-labelled baseline cases.

How to install evaluation

Use the examples as scaffolding

Copy references and scripts with SKILL.md. The marketplace plugin installs the broader context-engineering collection.

The example uses Python standard-library code, but does not run a real agent. Hosted upload and API execution are not verified here; any model-backed checks use your provider and budget.

A skill is a folder with a SKILL.md file. You install it once; the agent reads its short description at the start of each session and loads the full instructions only when a request matches. Pick the place you use:

Claude Code: from the plugin marketplace

  1. Start Claude Code in a terminal with claude.
  2. Add the marketplace (once per computer):
    /plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
  3. Install the plugin. Claude Code opens its details first: check what it will add (skills, hooks, MCP servers), then choose whether it is for you, for everyone on this repository, or for you in this repository only.
    /plugin install context-engineering@context-engineering-marketplace
  4. Type / and look for /context-engineering:evaluation, or simply ask for the task.

From your shell instead of a session:

claude plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
claude plugin install context-engineering@context-engineering-marketplace

The plugin installs as a whole, so other skills come with it, including context-fundamentals, context-degradation, context-compression, context-optimization, self-managed-context, multi-agent-patterns, long-horizon-prompting, memory-systems, tool-design, filesystem-context, hosted-agents, advanced-evaluation, harness-engineering, project-development, bdi-mental-states, latent-briefing, self-improvement-loops.

Claude Code: copy the folder into your skills folder

Put the skill's folder in ~/.claude/skills/ to use it in every project on this computer, or in .claude/skills/ inside a repository to share it with everyone who works there (commit it).

git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p ~/.claude/skills
cp -R Agent-Skills-for-Context-Engineering/skills/evaluation ~/.claude/skills/

Claude Code notices new or changed skills in these folders during the session. If the skills folder did not exist when the session started, run /reload-skills once. Then type /evaluation or just describe the task.

On Windows, run the commands in Git Bash or WSL, or copy the folder into the same place with File Explorer.

Codex, Gemini CLI, Cursor and GitHub Copilot

These agents read the same SKILL.md folders. The one folder all four accept inside a project is .agents/skills/, so this copies the skill there:

git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p .agents/skills
cp -R Agent-Skills-for-Context-Engineering/skills/evaluation .agents/skills/
AgentProject folderYour user folderNotes
OpenAI Codex
.agents/skills/
~/.agents/skills/
Type $ or run /skills to pick a skill; Codex also chooses one when your task matches.
Gemini CLI
.agents/skills/
.gemini/skills/
~/.agents/skills/
~/.gemini/skills/
Or install from Git with gemini skills install <git URL> --path <folder> --consent. Google replaced Gemini CLI with Antigravity CLI for free-tier and Google One users on 18 June 2026; Antigravity uses .agents/skills/ in a project and ~/.gemini/config/skills/ for you.
Cursor
.agents/skills/
.cursor/skills/
~/.agents/skills/
~/.cursor/skills/
Type / in Agent chat to pick a skill. Cursor also reads Claude Code's .claude/skills/ folders.
GitHub Copilot
.github/skills/
.agents/skills/
.claude/skills/
~/.copilot/skills/
~/.agents/skills/
Works in Copilot CLI, the cloud agent, and agent mode in VS Code, Visual Studio and JetBrains (IDE support depends on the version). Or run gh skill install OWNER/REPOSITORY SKILL (public preview, GitHub CLI 2.90.0 or later).
Only install skills you trust. Anthropic's advice is to use skills only from trusted sources, and to read every file of any other skill before using it: a skill can direct the agent to run commands, read files or send data in ways its description does not mention.

Sources for these steps

Checked against each vendor's documentation on 2026-11-14:

Independent quality dimensions

Report correctness, completeness, source support and tool behavior separately. A weighted total can hide a critical failure in one dimension.

Run deterministic format and constraint checks first. Use human sampling and calibrated model judges for questions those checks cannot resolve.

A repeatable regression process

Version cases, prompts, models and scoring rules. Include realistic budgets and difficult cases so improvements are not measured only on easy examples.

A monitoring plan is not a deployed monitoring service. Implement trace collection, sampling, retention and escalation in the actual production system.

Questions about evaluation

Does the script run my agent automatically?

No. Its sample runner produces simulated output. Connect your actual agent and capture real results before comparing scores.

Does finding a citation prove it is correct?

No. Text markers and citation patterns do not verify source existence, relevance or factual support.

Must successful runs follow one exact tool sequence?

Usually assess the required outcome and constraints. Multiple valid tool paths may solve the same task.

Are the example pass thresholds universal?

No. Set thresholds from task risk, human labels and your measured baseline; the examples are not certification standards.

More AI skills

Related skills

Skill categories