Author(s): Ethan Mark

Originally published on Towards AI.

Embedded AI Evaluation: The Access Contract Enterprises Need Before They Trust Their Agents

How to give independent evaluators enough access to test an AI agent honestly — without turning an assessment into a data leak, a ceremonial review, or a fight over findings.

Useful evaluation needs a clear window into the system — not unrestricted access to everything around it.

A team can spend months building an AI agent, pass a polished demo, and still be unable to answer one painful question: who outside the build team has seen how it behaves under pressure?

Not who watched the launch. Not who skimmed a dashboard. Who had enough access to inspect the model version, the tools, the prompts, the data boundaries, the failed traces, and the release decision — and could report an uncomfortable result without losing the engagement.

That is the practical promise of embedded AI evaluation. It moves qualified evaluators closer to the work while the system is still being designed, tested, and changed. The idea became newly concrete when Anthropic announced an embedded-evaluation partnership on September 18. Its description is refreshingly direct: today there are no settled standards for what access evaluators need or how they should report findings. Anthropic’s announcement is about frontier-model oversight, but the underlying engineering problem already belongs to ordinary enterprise AI teams.

When a customer-support agent has tools, retrieval, workflow rules, and permission to act, an evaluation cannot be only a model score. The evaluator needs a safe way to inspect the whole consequence chain. The company needs a safe way to provide that access. Both need a shared record of what was actually tested.

An evaluator who sees only curated prompts can confirm a demo. An evaluator who sees controlled evidence can test a system.

This guide turns that principle into a buildable AI evaluation access contract: a small operational agreement that defines scope, credentials, safe data, evidence, reporting, and what happens when the evaluator finds a real problem.

Why a normal security review is not enough

Security review is essential, but it answers a different set of questions. It asks whether systems are protected, identities are controlled, dependencies are acceptable, and known risks are mitigated. A privacy review asks what data is collected and where it goes. A model evaluation asks whether the system meets defined quality or safety targets.

An embedded evaluation connects those questions to live behavior. It asks things such as:

• Does the agent honor the permission boundary when a user asks it to do something plausible but unauthorized?
• Can an evaluator reproduce a reported failure using a pinned version of the agent and its tool policy?
• Does a retrieval update change the answer in a way the existing acceptance tests miss?
• Can a reviewer distinguish a model failure from a tool failure, a policy refusal, or a broken scorer?
• Does the release owner see the same evidence as the person who recommends a release?

These are workflow questions. A secure system can still mislead users. A model with good benchmark results can still call the wrong tool. A strong test suite can still fail to show that the business changed an approval rule between two runs.

The usual response is to invite an external expert near the end, show a few cases, and ask for a sign-off. That is better than nothing. It is also too easy to game by accident. If the evaluator has no clear environment, no access to failures, no way to inspect configuration, and no channel outside the delivery team, the assessment becomes a presentation.

Embedded does not mean unrestricted

The word embedded can make engineering leaders nervous. It should not mean a third party receives production credentials, raw customer records, source-code access to every repository, or standing access after the assessment ends.

It means the evaluator gets access that is sufficient for the claim being tested. The mechanism is not trust alone. It is deliberately narrow, logged, time-bounded access to the artifacts that explain the system’s behavior.

Think in four layers:

1. System view: versioned architecture, tool inventory, policy boundaries, dependency and deployment context.
2. Behavior view: representative tasks, traces, outcomes, errors, refusal reasons, and evaluator-created cases.
3. Evidence view: immutable run IDs, hashes, configuration snapshots, raw measurements, and known limitations.
4. Governance view: who can receive findings, who can challenge them, who can accept risk, and who decides whether the issue is closed.

The four layers give a simple test for every requested permission: what evaluation claim becomes impossible if this access is missing? If nobody can name the claim, do not grant the access. If the claim is legitimate but customer data would be exposed, provide a safer substitute such as a representative fixture, redacted trace, isolated replay environment, or escorted session.

Start with an evaluation charter

Before provisioning an account, write a one-page charter. This saves both security review and evaluation rigor. It tells everyone what success looks like before there is an uncomfortable finding to argue about.

The minimum evaluation charter

• Decision: the release, model change, permission expansion, or incident claim the evaluation informs.
• System boundary: agent version, prompts, tools, retrieval corpus version, deployment environment, and excluded components.
• Claims: precise statements to test, such as “the agent does not submit a refund without an approved case ID.”
• Methods: scenario tests, trace review, red-team attempts, code checks, interviews, or a mix.
• Evidence required: the logs, artifacts, approvals, and measurements that support each conclusion.
• Decision rights: who receives the final report and who can accept an unresolved risk.

Notice the difference between a claim and a goal. “Assess whether the agent is safe” is a goal. “Attempt to induce an unauthorized write through tool selection, prompt injection, and confused-deputy paths; record whether policy enforcement stops it” is an evaluable claim.

Make the charter versioned. If the team changes a model, a retrieval index, a tool, or the definition of success mid-evaluation, the charter should show that change. Otherwise the final report can quietly describe a different system from the one that started the review.

Build a safe evaluator workspace

A sound access path is narrow at the boundary and rich in evidence inside the evaluation workspace.

The best evaluator environment resembles a carefully designed staging system, not a shared production login. The goal is to let someone test meaningful behavior while constraining blast radius and protecting people whose data was never meant to become evaluation material.

Use a separate identity and a short-lived grant

Give the evaluator an individual identity, not a shared “audit” account. Bind it to the charter and expiration date. Use least privilege, multi-factor authentication, and logs that show when the evaluator entered which environment and what privileged artifact they accessed.

Do not hand over a general administrator role because it is convenient. Instead, create task-specific roles: trace reader, fixture runner, red-team operator, configuration viewer, or supervised production observer. The role should mirror the method in the charter.

Prefer replay and simulation before production

Most important claims can be tested using a frozen snapshot: a pinned agent configuration, synthetic or consented data, a tool simulator, and stored traces. A replay environment is especially valuable for agent systems because it lets the evaluator run edge cases repeatedly without charging a customer, modifying a record, or sending an email.

For an agent that performs account changes, create a tool double that produces the same schemas and policy errors as the real service but writes only to disposable records. For a retrieval agent, pin the corpus revision and attach source metadata to each result. For a computer-use workflow, run against a dedicated test tenant with seeded accounts and harmless actions.

Expose the policy and configuration, not only the conversation

A transcript is often insufficient. The same user message can lead to different behavior because of model choice, system instructions, retrieval context, tool schemas, permission mode, feature flags, or a new routing rule. Give evaluators a readable configuration manifest. Redact secrets, but do not hide the control surfaces that shape behavior.

{
 "evaluation_run": "ev-2026-09-19-042",
 "agent_release": "support-agent@4.8.1",
 "model": "approved-model-alias",
 "prompt_bundle": "sha256:8d7...",
 "retrieval_corpus": "support-kb@2026-09-18",
 "tools": ["search_case", "draft_reply", "request_refund"],
 "tool_policy": "refund-approval@3",
 "environment": "isolated-evaluation",
 "fixture_set": "refund-boundaries@12",
 "evidence_retention_days": 90
}

This is not a universal schema. It is a reminder that an evaluation result should identify the system it describes. A score without a configuration basis is hard to reproduce and easy to overstate.

Treat evidence as a product, not a report appendix

Teams often collect screenshots, a few transcripts, and a final slide deck. That makes it hard to revisit the conclusion after a release changes. Better evaluation evidence has a direct path from claim to observation.

For each result, retain:

• the test case or trigger, including its version and safety classification;
• the exact system manifest and environment identifier;
• the trace, tool calls, policy decisions, and relevant outputs;
• the rubric or deterministic assertion used to judge it;
• the evaluator’s observation, confidence, limitations, and recommended action;
• the disposition: accepted, fixed, deferred with an owner, or contested with a documented reason.

Use append-only storage or signed artifacts where practical. The point is not to create a bureaucratic ledger. It is to stop a critical finding from becoming a sentence nobody can trace back to the run that produced it.

There is a useful design principle here: evidence should be accessible at the same level of detail as the decision it supports. A broad release decision needs an understandable summary. A contested high-risk finding needs the raw trace, environment snapshot, and method. Do not make the board read raw logs; do not ask an evaluator to defend a conclusion using only a slide.

Preserve independence in the workflow

Independence is not a feeling. It is a set of operational choices. If the team being evaluated controls the evaluator’s tasks, access, interpretation, and reporting audience, they can shape the result without ever falsifying data.

Build guardrails around four moments:

1. Scope selection

The product team can explain the system, but it should not be able to exclude every uncomfortable scenario. Let evaluators add a bounded number of cases, especially after reviewing real incident patterns, customer complaints, and failed tests.

2. Access disputes

Security and privacy teams should be able to refuse access, but the refusal must be recorded with the claim it affects and an offered alternative. “We cannot provide that” is not enough. The record should say whether the finding is limited because a simulator, trace slice, or interview did not substitute for the missing evidence.

3. Finding review

Give the build team a factual-correction period. They may identify a wrong environment, an outdated policy, or a mistaken assumption. They should not silently rewrite the evaluator’s conclusion. Keep the original finding, the response, and the final disposition together.

4. Escalation

Define one route that does not depend on the delivery manager. High-severity issues need a direct path to a named risk owner, security leader, or oversight group. The evaluator should know the route before starting, not discover it after finding something serious.

The AI Evaluator Forum’s recent minimum conditions are written for frontier AI companies, but they offer a useful north star for enterprise teams: disclose conflicts, describe evaluator access, limit redactions to legitimate needs, and permit direct communication with oversight bodies. Read the full conditions for the wider governance discussion.

Run the evaluation loop alongside delivery

Embedded evaluation is most useful when it is not a single gate at the end. You do not need an evaluator in every stand-up. You need a rhythm that keeps evidence current as the product changes.

1. Plan: write or refresh the charter for the change that matters.
2. Prepare: pin the release candidate, provision the evaluator role, and generate safe fixtures or replays.
3. Test: run agreed scenarios plus a small evaluator-selected set of challenge cases.
4. Interpret: connect findings to raw evidence, uncertainty, and affected users.
5. Decide: fix, constrain, defer with explicit risk acceptance, or release.
6. Learn: turn confirmed failures into regression cases and retire misleading tests.

This loop has a surprising benefit: it prevents teams from treating external evaluation as a grade. A finding is not an embarrassing exception; it is a new test that should make the next release more reliable. Practitioner discussions about production LLM evaluation repeatedly point to this missing loop: teams have traces and scores, but they struggle to convert failures into versioned cases that survive the next model or prompt change.

What should be automated, and what should stay human?

Automation can preserve the trace. Humans still decide whether the evidence supports a release.

Automate the stable, repeatable mechanics. Generate manifests. Create disposable test tenants. rotate short-lived access. Capture traces. run deterministic checks. preserve artifact hashes. Notify owners when a finding is open. Those tasks are tedious precisely because they should be consistent.

Keep judgment human-owned. A human should decide whether a test represents a real customer risk, whether a simulated tool is faithful enough, whether the evaluator’s method is proportionate, whether a limitation invalidates a conclusion, and whether the organization accepts a residual risk. An LLM can help summarize evidence or generate challenge cases; it should not be the only authority that grades its own deployment.

Common failure modes — and the practical fix

The evaluator sees only the happy path

Fix: provide redacted failures, refusal traces, incidents, and a limited right to add cases. A report based only on successful demos is an adoption artifact, not evidence.

The evaluator receives production data by default

Fix: begin with synthetic or consented fixtures, isolated replays, and field-level redaction. Escalate to supervised production observation only when the claim cannot be tested another way.

Findings disappear into a ticket queue

Fix: give every finding an owner, severity rationale, due date, and disposition. Preserve the connection to the evidence even after the ticket closes.

The team claims a result applies to every release

Fix: bind conclusions to a named agent release, model alias, prompt bundle, tool policy, and corpus version. Re-evaluate after material changes.

Confidentiality rules silence the report

Fix: agree on a redaction review that protects customer data, secrets, and active security issues without suppressing the fact that a relevant limitation or finding exists.

A practical first pilot

Do not begin with the most sensitive agent in the company. Choose one workflow that is important enough to have real consequences but bounded enough for a safe replay environment: a support triage agent, internal knowledge assistant, document workflow, or code-review helper with a controlled repository.

Invite an evaluator with relevant technical and domain expertise. Give them a 30-day individual account, a pinned staging release, a tool simulator, a manifest, recent redacted failures, and a direct report path to the named risk owner. Ask them to test three claims and add up to five challenge cases. At the end, keep one short decision record and turn confirmed gaps into regression tests.

That pilot will reveal more than a policy document. It will show whether your traces explain behavior, whether your staging environment is realistic, whether your access controls are usable, and whether your release process can hear bad news early enough to act on it.

The headline promise of AI agents is autonomy. The operational promise should be verifiability. Embedded AI evaluation is how an organization starts earning that second promise before it asks customers to trust the first.

FAQ

What is embedded AI evaluation?

Embedded AI evaluation gives an independent evaluator controlled access to the systems, evidence, and people needed to assess an AI system while it is being built or deployed. It is more than a one-time benchmark or a polished product demo.

Is embedded AI evaluation the same as an AI audit?

They can overlap, but an audit often checks compliance against a defined standard. Embedded evaluation is an operational practice for testing behavior, safeguards, evidence, and release decisions as a system changes.

How can an evaluator test an AI agent without seeing customer data?

Use isolated replays, synthetic or consented fixtures, redacted traces, tool simulators, and supervised observation when necessary. Each substitute should be documented so everyone understands what it can and cannot prove.

What should an AI evaluation access contract include?

Include the decision being informed, system boundary, claims, methods, required evidence, evaluator identity and expiration, data rules, reporting route, redaction process, and finding-disposition workflow.

When should an enterprise re-evaluate an AI agent?

Re-evaluate after material changes such as a new model, prompt bundle, tool, permission, retrieval corpus, routing rule, deployment environment, or incident. The larger the potential impact, the stronger the need for a fresh evidence review.

Can an LLM judge its own agent evaluation?

An LLM can help score narrow criteria or summarize traces, but it should not be the only authority. Pair automated judging with deterministic checks, human review, and a clear record of the model, rubric, and limitations.

Sources and further reading

• Anthropic: Partnering with Accenture on embedded evaluation
• AI Evaluator Forum: Minimum Conditions for Embedding Evaluators
• AWS: Automated agent evaluation with AgentCore and GitHub Actions
• Anthropic: Building effective agents

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI