Appearance
Monitor Phase
In the monitor phase, governance continues after the implementation. You connect Sources, start tests on a schedule, and keep the Evidence and the risk values current when the AI System changes.
Monitor is the third of the lifecycle phases:
Outcome
After this phase, you have:
- Sources that give Modulos the signals of the AI System in production
- tests that examine the conditions that are important to you, on a schedule
- a remediation loop that connects each Failed result to the governance work
Time to first value: 60–120 minutes
Prerequisites: an AI System and the AI System Owner role. To add tests, the Editor role is sufficient.
Path at a glance
- Connect Sources Add the Sources that give metrics to the AI System.
- Add tests Change each governance question into a condition.
- Set schedules Start the tests on a schedule and examine the results.
- Correct the work Connect failed results to Controls, Evidence, and risk.
Step 1: Connect Sources for live signals
Goal: give the tests the signals of the AI System.
Where in Modulos
AI System → Settings → Sources
Do this
- Select Add Source and add a Source that gives metrics, for example Prometheus, Datadog, Azure, or AWS.
- Make sure that the Connection column shows Healthy for the Source.
- If a Source type is not available, an Organization Admin can turn it on in
Organization → Source & Connectors.
If your organization activates Modulos Client, each AI System shows it with its Sources. Your code sends the metrics to Modulos through it. See Sources.
You are done when
- the tests can read metrics from the Source
Step 2: Add tests as governance signals
Goal: write each governance question as a condition that Modulos can examine.
Where in Modulos
AI System → Runtime Inspection
Do this
- Select New Test and make a Metric Test for each condition that is important to you.
- Select the Source, the metric, the operator, and the value of the condition.
- Open the test and select Associate to a Control to link the test to the Controls that it monitors.
- For a check that an AI agent does from instructions in natural language, make an Agent Test. See Agent Tests.
You are done when
- each test has a clear purpose and a condition that Modulos can measure
Step 3: Start tests on a schedule and monitor the results
Goal: get test results continuously.
Where in Modulos
AI System → Runtime Inspection → select a test → Edit → Scheduleto set the schedule of a Metric TestAI System → Runtime Inspection → Create Jobto start many tests, or Agent Tests, on one scheduleAI System → Jobsto see the Jobs that start the scheduled testsAI System → Runtime Inspection → select a test → Resultsto see the results
Do this
- In the Schedule section of the test, select Schedule Testing.
- Set a Frequency (Daily, Weekly, Monthly, or Quarterly) that agrees with the risk.
- Set the Time, and select Save.
- Examine the results on the Results tab of each test.
Modulos starts each scheduled test as a Job. For the schedule rules, see Tests & Schedules and Jobs.
You are done when
- each test gives a result on its schedule: Passed, Failed, or Error
Step 4: Remediate and keep governance current
Goal: connect the test results to the governance work.
Where in Modulos
AI System → ControlsandAI System → Evidencefor the remediation workAI System → Risksto quantify the risks again when the AI System changes
Do this
- When a test fails, find the cause and correct it.
- Attach the new Evidence to the Control that the test monitors.
- Set the Control to Executed again.
- If the AI System changes, quantify the affected threats again with Recalculate.
- Add a comment on the Control when Comments and Logs does not tell the cause of a change.
A Failed result, and an Error result of a Metric Test, set each mapped Executed Control to Not Executed. For the rules, see Results & Remediation. Modulos logs each status change in Comments and Logs.
You are done when
- each Failed result goes back into the Controls, the Evidence, and the risk values