# How to evaluate AI governance tools

Canonical: https://aiagentgovernance.dev/guides/ai-governance-tools/
By AI Agent Governance. Reviewed 2026-11-09.

AI governance tools support different parts of managing AI: finding deployments, recording owners, evaluating behavior, enforcing permissions or retaining evidence. Choose a tool against a specific missing capability and test the complete integration. A registry, an observability system and an action gateway are not interchangeable.

Write the requirement before opening a vendor comparison. "We need to know which agents can change checkout production" asks for identity, ownership and effective-access information. "Every restart must wait for approval" asks for enforcement before execution. A product can satisfy one requirement without satisfying the other.

Use your [governance framework](https://aiagentgovernance.dev/guides/ai-agent-governance-framework/) to identify the current gap. Keep the accountable decision with a named team even when a tool automates part of the process.

## Compare documented roles

The examples below are selected to illustrate categories, not ranked recommendations. The documentation supports the stated role; it does not establish that a product covers your integration or every governance function.

| Category and example | Documented role | What your trial must still establish |
| --- | --- | --- |
| Identity inventory: Microsoft Defender | Lists and investigates supported non-human identities | Whether every agent deployment maps to an owner and all its downstream identities |
| Evaluation: LangSmith | Offline dataset experiments and online evaluation | Whether assertions observe your actual permissions and side effects |
| Policy decisions: Open Policy Agent | Evaluates policy against structured input | Whether every execution route supplies trusted facts and enforces the returned decision |
| Remote action control: Emisar | Declared infrastructure actions with policy, approval and audit | Whether the selected actions, targets and alternate access routes meet your intended boundary |

[Microsoft's identity documentation](https://learn.microsoft.com/en-us/defender-xdr/investigate-non-human-identities) describes its supported identity types. An identity inventory is useful input to an [agent inventory](https://aiagentgovernance.dev/guides/ai-agent-inventory/), but one identity can serve several deployments, and one deployment can use several identities.

[LangSmith distinguishes offline and online evaluation](https://docs.langchain.com/langsmith/evaluation). [OPA explicitly separates policy decisions from enforcement](https://www.openpolicyagent.org/docs). These distinctions determine which parts your application must integrate and operate.

[Emisar](https://emisar.dev/) exposes declared remote infrastructure actions through CLI or MCP, with policy checks, approvals and audit records. It fits the action-control part of a governance system; enterprise policy ownership, model evaluation and identity lifecycle management remain separate responsibilities. Emisar shares ownership with this publication.

## Ask for evidence at the boundary you need

Do not infer an undocumented feature from a product category. Mark it "not verified" until documentation or a trial establishes it. This includes export formats, retention controls, offline behavior and support for a particular execution route.

For a registry, ask how discovered identities join to deployments and owners. For an approval system, inspect whether the approved target and arguments are checked before execution. For observability, find which component supplies the final outcome and whether sampling can omit it.

| Requirement | Useful demonstration |
| --- | --- |
| Named owner | Find one deployment and follow its record to a person who can suspend it |
| Scoped permission | Allow the assigned target and deny a nearby unassigned target |
| Specific approval | Change the target after approval and verify no dispatch |
| Reliable evidence | Reconstruct a request from independent executor records |
| Export | Export the selected record and inspect identifiers and missing fields |
| Failure handling | Interrupt a required dependency and observe the protected system |

The [tool evaluation worksheet](https://aiagentgovernance.dev/resources/governance-tool-evaluation.csv) keeps requirement, evidence and remaining work together. It contains no product scores.

## Run a trial around one real decision

For a hypothetical trial, choose a test assistant that can read service health and request one restart. Use disposable resources. Define an allowed read, a forbidden cross-service read, an approval-required change and a lost-response case.

Before testing, write the acceptance conditions. A permission check must occur before the protected read. A stale approval must not release a queued change. An unknown execution outcome must stay unknown until target-side evidence resolves it.

Then involve a person who will operate the system. Ask them to find the deployment, understand a held request, deny it and suspend new work. A technically complete integration can still be hard to operate during an incident.

Keep three evidence levels visible: the vendor documents a capability; a trial demonstrates it in the chosen setup; your production deployment verifies it under its actual configuration. Do not use one as proof of another.

## Review integration and operating costs

Count more than the subscription. Someone must keep identity mappings current, maintain policy, review alerts, manage credentials and test upgrades. A flexible engine may require substantial application integration. A managed system may add a dependency whose failure behavior needs review.

Inspect what data leaves your environment. Evaluation and observability tools may receive prompts, retrieved documents or tool results. Decide which fields are necessary, who can see them and how long they remain available. Raw transcript capture should be a deliberate choice.

Test an export and a rollback during the trial. If replacing the tool later would strand important evidence, identify the retained format and access plan before depending on it.

## Record the responsibilities that remain

The purchase decision should name the requirements met, evidence observed, gaps, integration owners and operating obligations. Keep the scope narrow enough that the next engineer can repeat the trial after an update.

Use the [access-control guide](https://aiagentgovernance.dev/guides/ai-agent-access-control/) for the enforcement review and the [audit guide](https://aiagentgovernance.dev/guides/ai-agent-audit-trail/) for evidence reconstruction. A tool is useful when it closes a documented gap and the team can keep that capability working.

## Source documents
- [Microsoft Defender non-human identities](https://learn.microsoft.com/en-us/defender-xdr/investigate-non-human-identities)
- [LangSmith evaluation documentation](https://docs.langchain.com/langsmith/evaluation)
- [Open Policy Agent documentation](https://www.openpolicyagent.org/docs)
- [Emisar declared actions, policies and audit](https://emisar.dev/)