Agent Evaluation skill
Plan agent regression suites with deterministic checks, separate quality dimensions and versioned baselines.
Community · Muratcan Koylan| Author | Muratcan Koylan |
|---|---|
| Licence | MIT |
| Source | muratcankoylan/Agent-Skills-for-Context-Engineering · SKILL.md on GitHub |
| Works in | Claude Code · Codex, Gemini CLI, Cursor, Copilot |
| Read from the source on |
The evaluation skill designs quality checks for agent behavior across realistic tasks.
It combines deterministic validation with separate quality dimensions and human review. Judge prompts and bias calibration belong to the companion advanced-evaluation skill.
From the skill's SKILL.md: ## Gotchas
What it does
- Define expected outcomes and deterministic checks.
- Build representative cases across task complexity and budgets.
- Compare versioned baselines and investigate regressions.
What you get
- An evaluation plan with rubrics, regression cases and quality gates.
When to use it
- Changing an agent, its tools or its context strategy and needing repeatable comparisons.
When not to use it
- Treating a sample score as factual verification or production certification.
What it needs
- A file-capable agent, representative cases, expected outcomes and execution traces.
- The Python example includes placeholder scores and simulated agent output. A real runner, verifiers and any paid model integration must be supplied separately.
Files in the skill
| File or folder | What it is for |
|---|---|
SKILL.md | Evaluation dimensions, test design and regression guidance. |
references/metrics.md | A Python evaluator example and a reference guide to metrics. |
scripts/evaluator.py | A Python evaluator example and a reference guide to metrics. |
Licence as found in the repository on 2026-11-20: MIT.
Example prompts
Type one of these after installing; naming the skill makes sure it is used.
- Design regression tests for this research agent with citation checks.
- Separate answer quality from tool-use efficiency in the report.
- Define a quality gate using our human-labelled baseline cases.
How to install evaluation
Use the examples as scaffolding
Copy references and scripts with SKILL.md. The marketplace plugin installs the broader context-engineering collection.
The example uses Python standard-library code, but does not run a real agent. Hosted upload and API execution are not verified here; any model-backed checks use your provider and budget.
A skill is a folder with a SKILL.md file. You install it once; the agent reads its short description at the start of each session and loads the full instructions only when a request matches. Pick the place you use:
Claude Code: from the plugin marketplace
- Start Claude Code in a terminal with
claude. - Add the marketplace (once per computer):
/plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering - Install the plugin. Claude Code opens its details first: check what it will add (skills, hooks, MCP servers), then choose whether it is for you, for everyone on this repository, or for you in this repository only.
/plugin install context-engineering@context-engineering-marketplace - Type / and look for
/context-engineering:evaluation, or simply ask for the task.
From your shell instead of a session:
claude plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
claude plugin install context-engineering@context-engineering-marketplaceThe plugin installs as a whole, so other skills come with it, including context-fundamentals, context-degradation, context-compression, context-optimization, self-managed-context, multi-agent-patterns, long-horizon-prompting, memory-systems, tool-design, filesystem-context, hosted-agents, advanced-evaluation, harness-engineering, project-development, bdi-mental-states, latent-briefing, self-improvement-loops.
Claude Code: copy the folder into your skills folder
Put the skill's folder in ~/.claude/skills/ to use it in every project on this computer, or in .claude/skills/ inside a repository to share it with everyone who works there (commit it).
git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p ~/.claude/skills
cp -R Agent-Skills-for-Context-Engineering/skills/evaluation ~/.claude/skills/Claude Code notices new or changed skills in these folders during the session. If the skills folder did not exist when the session started, run /reload-skills once. Then type /evaluation or just describe the task.
On Windows, run the commands in Git Bash or WSL, or copy the folder into the same place with File Explorer.
Codex, Gemini CLI, Cursor and GitHub Copilot
These agents read the same SKILL.md folders. The one folder all four accept inside a project is .agents/skills/, so this copies the skill there:
git clone --depth 1 https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering.git
mkdir -p .agents/skills
cp -R Agent-Skills-for-Context-Engineering/skills/evaluation .agents/skills/| Agent | Project folder | Your user folder | Notes |
|---|---|---|---|
| OpenAI Codex | .agents/skills/ | ~/.agents/skills/ | Type $ or run /skills to pick a skill; Codex also chooses one when your task matches. |
| Gemini CLI | .agents/skills/.gemini/skills/ | ~/.agents/skills/~/.gemini/skills/ | Or install from Git with gemini skills install <git URL> --path <folder> --consent. Google replaced Gemini CLI with Antigravity CLI for free-tier and Google One users on 18 June 2026; Antigravity uses .agents/skills/ in a project and ~/.gemini/config/skills/ for you. |
| Cursor | .agents/skills/.cursor/skills/ | ~/.agents/skills/~/.cursor/skills/ | Type / in Agent chat to pick a skill. Cursor also reads Claude Code's .claude/skills/ folders. |
| GitHub Copilot | .github/skills/.agents/skills/.claude/skills/ | ~/.copilot/skills/~/.agents/skills/ | Works in Copilot CLI, the cloud agent, and agent mode in VS Code, Visual Studio and JetBrains (IDE support depends on the version). Or run gh skill install OWNER/REPOSITORY SKILL (public preview, GitHub CLI 2.90.0 or later). |
Sources for these steps
Checked against each vendor's documentation on 2026-11-14:
Independent quality dimensions
Report correctness, completeness, source support and tool behavior separately. A weighted total can hide a critical failure in one dimension.
Run deterministic format and constraint checks first. Use human sampling and calibrated model judges for questions those checks cannot resolve.
A repeatable regression process
Version cases, prompts, models and scoring rules. Include realistic budgets and difficult cases so improvements are not measured only on easy examples.
A monitoring plan is not a deployed monitoring service. Implement trace collection, sampling, retention and escalation in the actual production system.
Questions about evaluation
Does the script run my agent automatically?
No. Its sample runner produces simulated output. Connect your actual agent and capture real results before comparing scores.
Does finding a citation prove it is correct?
No. Text markers and citation patterns do not verify source existence, relevance or factual support.
Must successful runs follow one exact tool sequence?
Usually assess the required outcome and constraints. Multiple valid tool paths may solve the same task.
Are the example pass thresholds universal?
No. Set thresholds from task risk, human labels and your measured baseline; the examples are not certification standards.
More AI skills
Related skills
Dev-Browser skill
Community · Sawyer HoodAutomate Chrome with persistent named pages, Puppeteer scripts and a local command-line service.Context Degradation skill
Community · Muratcan KoylanDiagnose long-context failures and plan recovery using verified facts and workload-specific comparisons.Tool Design skill
Community · Muratcan KoylanDesign agent-facing tool contracts with clear inputs, outputs, selection guidance and recoverable errors.Advanced Evaluation skill
Community · Muratcan KoylanDesign LLM judges with evidence-based rubrics, swapped pairwise comparisons and human calibration.Brainstorming skill
Community · Jesse Vincent / Prime RadiantTurn an undeveloped idea into an agreed design before writing implementation code.Writing Plans skill
Community · Jesse Vincent / Prime RadiantConvert an approved design into reviewable engineering tasks with interfaces and verification.