Skip to content

Monitor Phase ​

In the monitor phase, governance continues after the implementation. You connect Sources, start tests on a schedule, and keep the Evidence and the risk values current when the AI System changes.

Monitor is the third of the lifecycle phases:

Outcome ​

After this phase, you have:

  • Sources that give Modulos the signals of the AI System in production
  • tests that examine the conditions that are important to you, on a schedule
  • a remediation loop that connects each Failed result to the governance work

Time to first value: 60–120 minutes
Prerequisites: an AI System and the AI System Owner role. To add tests, the Editor role is sufficient.

Path at a glance ​

  1. Connect Sources Add the Sources that give metrics to the AI System.
  2. Add tests Change each governance question into a condition.
  3. Set schedules Start the tests on a schedule and examine the results.
  4. Correct the work Connect failed results to Controls, Evidence, and risk.

Step 1: Connect Sources for live signals ​

Goal: give the tests the signals of the AI System.

Where in Modulos

  • AI System → Settings → Sources

Do this

  • Select Add Source and add a Source that gives metrics, for example Prometheus, Datadog, Azure, or AWS.
  • Make sure that the Connection column shows Healthy for the Source.
  • If a Source type is not available, an Organization Admin can turn it on in Organization → Source & Connectors.

If your organization activates Modulos Client, each AI System shows it with its Sources. Your code sends the metrics to Modulos through it. See Sources.

You are done when

  • the tests can read metrics from the Source

Step 2: Add tests as governance signals ​

Goal: write each governance question as a condition that Modulos can examine.

Where in Modulos

  • AI System → Runtime Inspection

Do this

  • Select New Test and make a Metric Test for each condition that is important to you.
  • Select the Source, the metric, the operator, and the value of the condition.
  • Open the test and select Associate to a Control to link the test to the Controls that it monitors.
  • For a check that an AI agent does from instructions in natural language, make an Agent Test. See Agent Tests.

You are done when

  • each test has a clear purpose and a condition that Modulos can measure

Step 3: Start tests on a schedule and monitor the results ​

Goal: get test results continuously.

Where in Modulos

  • AI System → Runtime Inspection → select a test → Edit → Schedule to set the schedule of a Metric Test
  • AI System → Runtime Inspection → Create Job to start many tests, or Agent Tests, on one schedule
  • AI System → Jobs to see the Jobs that start the scheduled tests
  • AI System → Runtime Inspection → select a test → Results to see the results

Do this

  • In the Schedule section of the test, select Schedule Testing.
  • Set a Frequency (Daily, Weekly, Monthly, or Quarterly) that agrees with the risk.
  • Set the Time, and select Save.
  • Examine the results on the Results tab of each test.

Modulos starts each scheduled test as a Job. For the schedule rules, see Tests & Schedules and Jobs.

You are done when

  • each test gives a result on its schedule: Passed, Failed, or Error

Step 4: Remediate and keep governance current ​

Goal: connect the test results to the governance work.

Where in Modulos

  • AI System → Controls and AI System → Evidence for the remediation work
  • AI System → Risks to quantify the risks again when the AI System changes

Do this

  • When a test fails, find the cause and correct it.
  • Attach the new Evidence to the Control that the test monitors.
  • Set the Control to Executed again.
  • If the AI System changes, quantify the affected threats again with Recalculate.
  • Add a comment on the Control when Comments and Logs does not tell the cause of a change.

A Failed result, and an Error result of a Metric Test, set each mapped Executed Control to Not Executed. For the rules, see Results & Remediation. Modulos logs each status change in Comments and Logs.

You are done when

  • each Failed result goes back into the Controls, the Evidence, and the risk values

Next ​