Docs

Appearance

Vijil ​

The Vijil Source connects an AI System to one agent in the Vijil Console. Metric Tests read the scores of the latest Vijil evaluation of that agent. Scout, Agent Tests, and the Risk Agent read the agent and its evaluation results.

Used byMetric Tests, Agent Tests, Scout, Risk Agent
ModePull
CredentialsClient ID, client secret, and agent ID
Metric selectionSelect the score in the menu

Where in Modulos ​

  • AI System → Settings → Sources → Add Source → Vijil adds the Source.

Before you start ​

  • An Organization Admin makes Vijil available. See Source & Connectors.
  • The agent must have one or more evaluations in Vijil.

How to prepare Vijil ​

  1. In the Vijil Console, open Settings → API Keys.
  2. Create a key.
  3. Copy the client ID, which starts with vk_.
  4. Copy the client secret. The Vijil Console shows the secret one time only.
  5. Get the ID of the agent in the Vijil Console.

How to connect Vijil ​

  1. Open AI System → Settings → Sources.
  2. Select Add Source.
  3. Select Vijil.
  4. Type a Source Name.
  5. Paste the client ID into Client ID.
  6. Paste the client secret into Client Secret.
  7. Paste the agent ID into Agent ID.
  8. Select Save.
Add Source dialog with Vijil selected: the connection instructions on the left, the fields on the right.
Add a Vijil Source.
  1. 1
    Client ID
    It starts with vk_.
  2. 2
    Agent ID
    The agent whose evaluation scores Modulos reads.

When you save, Modulos signs in to the Vijil Console and reads the agent. If the credentials are incorrect or Modulos does not find the agent, the Source is not saved.

Metrics for Metric Tests ​

A Metric Test on a Vijil Source reads one score of the latest evaluation of the agent:

  1. In the test dialog, select the Vijil Source.
  2. Select the Metric.

The menu shows the harness, scenario, and probe metrics of the latest evaluation. Below the metric, the dialog shows its category, its ID, and its type. If the agent has no evaluation, the menu shows "No evaluation data found. Run an evaluation in Vijil first."

Vijil gives scores from 0 to 1. Modulos multiplies them by 100. Because of this, a Metric Test compares values from 0 to 100.

Scout, Agent Tests, and the Risk Agent ​

Scout, Agent Tests, and the Risk Agent read a Vijil Source with read-only queries:

  • Agent: the name, the model, and the status of the agent.
  • Evaluations: the latest evaluation, with its name, score, and status.
  • Scores: the harness, scenario, and probe scores of an evaluation by its ID, all scores, or selected scores by their IDs.

Modulos does not start evaluations in Vijil.

Important considerations ​

  • A Metric Test reads the latest evaluation only. If the latest evaluation does not have the metric, the run ends with the error "Metric not found in source". If the metric has no score in that evaluation, the test gets the value 0.
  • Run the Vijil evaluation before the Metric Test runs. The test does not wait for an evaluation.
Sources

All Source types and shared settings

Tests & Schedules

Metric Tests and their schedules

Agent Tests

AI agents that check your Sources