Skip to content

Agent Tests ​

An Agent Test is a semantic test. An AI agent reads the content of your Sources, such as code, documents, and tickets. The agent then compares the content with the Pass Criteria that you write in words. Agent Tests can check Controls that before only a person could check, because the check is a judgment that a Metric Test cannot express. The agent returns a verdict with a report.

What this is ​

In Modulos, Runtime Inspection shows all tests in one table, with two types of test:

  • A Metric Test compares one metric from one Source with a threshold. See Tests & Schedules.
  • An Agent Test gives an AI agent a Test Objective and Pass Criteria in your own words. The agent examines the Sources of the AI System and returns a verdict.

Use an Agent Test when the agent must read and give a verdict, not compare a number. For example:

  • A repository holds a model card with the fields that a Control specifies.
  • Each incident ticket has an owner.
  • The access log shows no access from a region other than the permitted regions.

The agent only reads. Its tools cannot change the Sources it examines.

Where in Modulos ​

  • AI System → Runtime Inspection shows the tests. The Type column shows an Agent or Metric badge, and the Type filter shows one type.
  • AI System → Runtime Inspection → New Test → Agent Test opens the New Agent Test dialog.
  • AI System → Runtime Inspection → select an Agent Test opens the page of the Agent Test with the Test and Results tabs.
  • AI System → Settings → Sources holds the Sources the agent can examine.

Each Agent Test gets a code of the format AT-1. Metric Tests get codes of the format T-1.

If Agent Test is not in the New Test dialog, contact Modulos. Modulos activates Agent Tests for each organization.

New Test dialog with two cards, Metric Test and Agent Test, each with a one-line description.
New Test asks for the type of test first.
  1. 1
    Metric Test
    Compares one metric from one Source with a threshold.
  2. 2
    Agent Test
    An AI agent examines the Sources against your Pass Criteria.

Who can do what ​

Permissions

  • Owners and Editors can add, change, and run Agent Tests, and put them in Jobs.
  • Viewers can see Agent Tests, their runs, and their reports.

The definition ​

An Agent Test has these fields:

FieldWhat it holds
NameA name that tells what the test examines.
Test ObjectiveThe question that the agent investigates, in words. It tells the agent which facts to collect from the Sources, as the metric does in a Metric Test. It does not set the verdict.
Pass CriteriaThe rule for the verdict, in words. It does what the threshold does in a Metric Test. Only the Pass Criteria set the verdict. Write the conditions for each verdict: passed, failed, and error. To make the test stricter, change only the Pass Criteria.
OwnerThe member of the AI System who is accountable for the test. By default, Modulos sets you as the Owner.
SourcesThe Sources the agent reads. If you select none, the agent reads all Sources of the AI System. Select Sources only to make the scope smaller. The agent cannot read Datadog or Prometheus Sources, and it gets no data from the Modulos Client Source (see Supported Source types). For pushed metrics, use a Metric Test.
Mapped ControlsThe Controls the test monitors. A failed run sets each mapped Control that is Executed to Not Executed.

In the Test Objective and the Pass Criteria, type @ to add a mention. A mention points to a Control, a Requirement, or a different concept of the AI System. The agent gets only a link to the concept, not its content. Write the content that the agent needs in the Test Objective or the Pass Criteria.

Make the Test Objective and the Pass Criteria clear to a reader with no context. A general question, such as "Is the AI System compliant with the GDPR?", gives a general verdict. A narrow question that mentions the Control to examine gives a clear verdict. If the Pass Criteria are not clear, the verdict is not clear.

New Agent Test dialog with Name, Test Objective, and Pass Criteria on the left, and Owner, Sources, and Mapped Controls on the right.
The New Agent Test dialog. The definition is on the left, the configuration on the right.
  1. 1
    Test Objective
    What the agent must examine.
  2. 2
    Pass Criteria
    The conditions for passed, failed, and error. Only these set the verdict.
  3. 3
    Owner
    A member of the AI System. Modulos sets you as the Owner.
  4. 4
    Sources
    All Sources of the AI System, unless you select some.
  5. 5
    Mapped Controls
    The Controls that a failed run sets to Not Executed.
  6. 6
    Create and Run
    Saves the test and starts the first run. Create Test only saves it.

How to add an Agent Test ​

  1. Open AI System → Runtime Inspection.
  2. Select New Test.
  3. Select Agent Test.
  4. Give the test a Name.
  5. Write the Test Objective.
  6. Write the Pass Criteria.
  7. Keep the default Owner, which is you, or select a different member of the AI System.
  8. If necessary, select Sources to make the scope smaller.
  9. If necessary, select Mapped Controls.
  10. Select Create Test to save the test, or Create and Run to save it and start the first run.

Create and Run opens the page of the Agent Test. When the run finishes, the Results tab shows its report, if the agent produced one.

A Metric Test needs an enabled Source before you can add it. An Agent Test does not. Connect the Sources the agent must read before you run the test.

How a run works ​

Each time an Agent Test operates, Modulos records a run. A run:

  1. Records the definition at its start: Name, Test Objective, Pass Criteria, Sources, Mapped Controls, and Owner. The verdict applies to that recorded definition, also if you change the test after the run starts.
  2. Goes through the statuses pending and running, and ends as completed or error.
  3. Ends with a verdict, passed, failed, or error. When the agent produced a report, the run also has a Run Report that gives the findings.

The agent sets the verdict from the Pass Criteria only:

  • passed: the facts agree with the Pass Criteria.
  • failed: the facts do not agree with the Pass Criteria. A Source that the agent can read but that does not contain the data that the Pass Criteria refer to also gives failed.
  • error: the agent could not collect the facts, for example because it could not read a Source. See Errors.

A run does not change after it ends. Modulos keeps the runs of a test when you change the test.

Only one run of a test can be active at a time. A second Run Now during an active run gets the message "A run of this agent test is already in progress".

What starts a run ​

The Triggered by column of the runs table tells what started the run:

  • Run manually: a member selected Run Now or Create and Run.
  • Job · name: a Job ran the test. The entry links to the Job.

To run Agent Tests on a schedule, select Create Job on the Runtime Inspection page and select the tests. One Job can hold Metric Tests and Agent Tests.

Errors ​

An error run did not apply the Pass Criteria. The run shows one of these messages:

MessageCause
"A source of this test could not be read." or "This source could not be read: " with the Source type, for example datadogThe agent could not read a selected Source. For example, the connection failed, the Source is not active, its type is unavailable for the organization, or agents cannot read its type (Datadog or Prometheus).
"The run reached its budget ceiling."The run used the maximum work permitted for one run.
"The run did not finish in time."The run did not complete before the time limit.
"The organization has reached its AI usage limit for this month."No AI usage quota of the organization remains for this month. See AI usage.
"The agent could not complete this run."An internal error stopped the agent.

Correct the cause and run the test again. An error run changes no Control.

What a failed run does to Controls ​

Only a completed run with the verdict failed changes Controls. For each mapped Control:

  • If the Control is Executed, Modulos sets it to Not Executed. The log of the Control gives the test name as the cause, and the assignee of the Control gets a notification.
  • Each Requirement of that Control that is Fulfilled becomes Not Fulfilled.
  • Modulos removes the ready for review mark from each Requirement of that Control.
  • A Control that is Not Executed, Out of Scope, or Archived does not change.

The Controls that change are the Mapped Controls recorded at the start of the run.

A passed run does not set a Control back to Executed. Set the Control to Executed again on its page. See Controls. An error run changes no Control.

Mapping Agent Tests and Controls ​

  • In the New Agent Test and Edit Agent Test dialogs, Mapped Controls lists the Controls of the AI System. Controls that are Out of Scope or Archived are not shown.
  • On a Control page, Associated Runtime Inspections lists the Metric Tests and Agent Tests mapped to the Control, each with its latest verdict. Associate to a Test and Remove are available while the Control is Not Executed.

How to read the results ​

Results tab of an Agent Test with the runs table and an open Run Report.
The Results tab lists the runs of an Agent Test. A run opens its Run Report.
  1. 1
    Result
    The verdict of each run.
  2. 2
    Triggered by
    Run manually, or the Job that ran the test.
  3. 3
    Run Report
    The findings behind the verdict.

The page of an Agent Test has two tabs:

  • Test shows the Condition, that is the Test Objective and the Pass Criteria, and the Last Result with a link to the Run Report. Before the first run, it shows "No runs yet. Press Run Now to start one."
  • Results lists the runs with Date, Result, Triggered by, and Duration. Select a run to read its Run Report. During a run, the page tells you that the report appears when the run finishes. A run without a report shows "This run produced no report."

The right side of the page shows the type, the Owner, the Sources, and the Mapped Controls. Owners and Editors can change the Owner there. Edit opens the dialog for the other fields.

Scout and Agent Tests ​

Scout reads the Agent Tests of an AI System by code, for example AT-3, with their latest run. A question to Scout about the tests of an AI System gets an answer for Metric Tests and Agent Tests. The answer keeps them apart. See Scout.

Important considerations ​

  • Write Pass Criteria that name the facts the agent must find, for example "each incident of the last 90 days has an owner and a closed date".
  • Use the Test Objective to limit what the agent reads. A narrow objective gives a faster run and a clearer report.
  • Map a test to a Control only when a failed verdict must remove the Control from Executed.
  • Read the Run Report before you do work because of a verdict. The report gives the findings behind the verdict.
  • Each run uses the AI usage quota of the organization. See AI usage.