> ## Documentation Index
> Fetch the complete documentation index at: https://docs.botbrains.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitors

> Flag the conversations that meet a condition or fail a scorecard, track the flag rate over time, and alert your team when the count rises

export const feature_0 = "monitors"

export const note_0 = undefined

<Warning>
  [ServiceLens](/concepts/servicelens), which includes {feature_0}, is currently in beta and available per project. Contact [support@botbrains.io](mailto:support@botbrains.io) to turn it on for your project. {note_0}
</Warning>

Monitors flag the conversations you want to review. A monitor watches the conversations that match its filters and flags each one that meets a condition you describe in plain language, fails a [scorecard](/concepts/scorecards), or both. You see how many conversations it flags each day, open them in one click, and can alert your team when the count crosses a threshold. You manage monitors on the [monitors page](https://platform.botbrains.io/~/monitors).

## How Monitors Work

Think of a monitor as a spotlight you shine on specific conversations or behaviors. Filters narrow the light, for example to the conversations of one [topic](/concepts/topics), and a prompt picks out the exact behavior you don't want to see within them, such as the AI agent promising a refund it isn't allowed to give. Every conversation in the light gets a result, so you see how often the behavior shows up and whether it grows or shrinks.

> A monitor decides which conversations get attention, and a scorecard defines how to grade them.

Together with a [scorecard](/concepts/scorecards), a monitor also sets up a review queue: when a monitor watches a scorecard and some of the values that feed into it are still missing because they need a person's input, the monitor appears as a queue on the [Reviews page](https://platform.botbrains.io/~/reviews), and your team answers the missing values in a dedicated review mode. Several people can work through the same queue at once without opening the same conversation, so you can split the work. See [Review Queue](/concepts/review-queue).

Click **New monitor** on the [monitors page](https://platform.botbrains.io/~/monitors) to start from scratch or from a template, then set it up in two steps:

| Step | What you set |
| - | - |
| **1. Which conversations?** | [Filters](#filtering-conversations), the [mode and time range](#continuous-and-one-off), and [sampling](#sampling) |
| **2. When to flag?** | A [prompt, a scorecard with a threshold, or both](#flag-criteria) |

<img src="https://mintlify.s3.us-west-1.amazonaws.com/botbrains/images/monitors/monitor-editor.png" alt="Monitor editor showing step 1 'Which conversations?' with filters, the Continuous and One-off toggle, look back presets and sampling, and step 2 'When to flag?' with a prompt and a scorecard threshold of 70" data-generation-prompt="Navigate to platform.botbrains.io/~/monitors/~/edit?template=unauthorized_advice in a project with ServiceLens turned on. Set viewport to 1920x1080 and collapse the sidebar. Keep the prefilled name, prompt, and 30-day look back. In step 1, add a Channel filter so one filter chip shows. In step 2, select an existing scorecard in the '…or when it fails a scorecard' field and keep the threshold at 70. Capture step 1 'Which conversations?' (filters, Evaluate toggle with Continuous selected, Look back presets, Sampling) and step 2 'When to flag?' (prompt textarea with its character counter, scorecard select with 'below 70 / 100'). Crop out the Alert field, the Test section, and the save button." />

The monitor then evaluates every conversation in scope and gives it one result.

| Result | Meaning |
| - | - |
| **Flagged** | The prompt holds, or the conversation fails the scorecard |
| **Not flagged** | Neither criterion applies |
| **N/A** | Neither criterion flagged the conversation, and at least one couldn't decide, for example because the AI gave no answer or the conversation has no score |
| **Awaiting review** | The scorecard waits for a person's answers, and the prompt didn't flag the conversation. A waiting conversation that the prompt flags shows as **Flagged**. See [Review Queue](/concepts/review-queue). |

Monitors never include [hidden conversations](/concepts/conversations#hiding-a-conversation). Monitors with flag criteria also skip conversations without a customer message. A conversation can appear in several monitors, and each monitor evaluates it on its own.

### What to Monitor

| Goal | How to set it up |
| - | - |
| **Review your AI agent** | Filter **AI involvement** to **Fully autonomous** and describe the mistake in the prompt, such as "the AI agent gave a wrong or made-up answer." |
| **Review your team** | Filter **AI involvement** to **Not involved** or **Public involvement** and add a scorecard with manual items, so your team grades the conversations in a [review queue](/concepts/review-queue). |
| **Catch risks** | Describe the behavior you must never see, such as legal or medical advice, promised refunds, or prompt injection attempts. |
| **Track customer signals** | Filter by a [conversation attribute](/concepts/conversation-attributes) such as frustration. A monitor without flag criteria flags every match. |
| **Measure outcomes** | Describe an outcome, such as "an order was placed." The flag rate then becomes your rate for that outcome. |

<Note>
  Use monitors and scorecards to improve conversation quality, not as the only basis for decisions about individual employees. Have a person check AI-evaluated results first, and see the [EU AI Act limits](/trust/ai-act#prohibited-and-high-risk-use) on HR decisions.
</Note>

### Filtering Conversations

Narrow the scope by channel, AI involvement, topic, status, rating, labels the conversation has or lacks, [conversation attributes](/concepts/conversation-attributes), and scorecard scores. Without filters, the monitor looks at every conversation.

### Continuous and One-off

| Mode | What it evaluates |
| - | - |
| **Continuous** | Conversations within the **Look back**, up to 90 days with presets of 7, 30, and 90 days, then every new conversation |
| **One-off** | Conversations within a **Time range** with a start and an end date up to 90 days back, evaluated one time |

Both ranges apply to the date of each conversation's last message. A continuous monitor evaluates a conversation once it goes quiet. Results usually appear about 10 minutes after the conversation goes quiet, or up to about 40 minutes later for conversations without an AI reply. When a conversation changes and starts or stops matching the filters, the monitor adds or removes it. When a conversation continues after the monitor evaluated it, the monitor evaluates it again once it goes quiet.

### Sampling

Set a sample rate from 1% to 100% to evaluate only a share of the matching conversations, for example so a busy project doesn't send your team more conversations than it can review. The default is 100%. A monitor always picks the same conversations for its sample.

### Flag Criteria

**Prompt.** Describe when to flag a conversation, for example "the customer had to repeat their request." The AI reads each conversation and flags it when the condition holds. A prompt can have up to 2,500 characters.

**What the AI reads.** The AI judges the conversation transcript: the customer's messages, the replies of the AI agent and your team, escalations, and the names of attached files. It doesn't see tool calls, internal notes, your knowledge, guidance, or procedures, so describe behavior that shows in the messages. For example, a prompt about placed orders only catches orders the conversation mentions. botBrains shortens long messages and transcripts.

**Scorecard.** Pick a [scorecard](/concepts/scorecards) in **…or when it fails a scorecard** and a threshold from 0 to 100, which defaults to 70. A conversation fails when it scores below the threshold or fails a critical item. A conversation without a score doesn't fail. The monitor also has the AI fill the scorecard's AI items when they're missing or out of date, and updates its result whenever a score changes.

**Both.** The monitor flags a conversation when either criterion applies.

**Neither.** The monitor flags every matching conversation. Instead of a results list, it shows how many conversations match, up to 10,000+, with a link to open them.

<Note>
  A project can have up to 20 active monitors with a prompt. Monitors that use only a scorecard or only filters don't count toward this limit.
</Note>

## Testing Before You Save

Click **Test on recent conversations** in the editor to run your current settings on the five most recent matching conversations with a customer message. botBrains saves nothing. The test shows how many conversations are in scope, how many of the tested ones the monitor flags, and the result, score, and flag probability of each. For a monitor without flag criteria, the test counts the matching conversations instead.

When you save, the monitor evaluates its scope in the background. If that covers more than 1,000 conversations, the editor asks you to confirm first.

## Reviewing Results

The [monitors page](https://platform.botbrains.io/~/monitors) groups monitors into **Continuous** and **One-off**. For continuous monitors, you choose a period of 7, 30, or 90 days. Click a day in a card's chart to open that day's flagged conversations.

Open a monitor to see its numbers, its definition, and the conversations it evaluated. Continuous monitors compare each number with the previous period of the same length.

<img src="https://mintlify.s3.us-west-1.amazonaws.com/botbrains/images/monitors/monitor-detail.png" alt="Monitor detail page with the Live badge, Flagged, Flag rate, and Evaluated numbers with period changes, a daily chart, and the Flagged tab of the conversation list" data-generation-prompt="Navigate to platform.botbrains.io/~/monitors in a project with ServiceLens turned on and open a continuous monitor with a prompt that flagged conversations on several days in the last 30 days. Set viewport to 1920x1080 and collapse the sidebar. Keep the period at 30 days and the Flagged tab selected, with no day selected in the chart. Capture the header with the Live badge, the row of Flagged, Flag rate, and Evaluated with their change badges, the daily chart, the definition panel on the right, and the first rows of the Conversations list with the Explore with FDE and View conversations buttons." />

| Number | Meaning |
| - | - |
| **Flagged** | Conversations the monitor flagged in the period |
| **Flag rate** | Flagged ÷ (Flagged + Not flagged). N/A results don't count. |
| **Evaluated** | All conversations with a result, including N/A |

The daily chart counts days in Coordinated Universal Time (UTC). Click a bar to narrow the conversation list to that day. For scorecard monitors, the list also shows the score, critical failures, and the weakest scorecard items.

<Note>
  When a monitor's scorecard has [manual items](/concepts/scorecards#unanswered-items), the monitor also works as a [review queue](/concepts/review-queue). On its **Awaiting review** tab, **Start reviewing** opens the oldest pending conversation in review mode.
</Note>

[botBrains Engineer](/concepts/engineer), the Forward Deployed Engineer (FDE), can investigate a monitor for you. Click **Explore with FDE** to start a thread about it.

**View conversations.** Opens the [conversations page](https://platform.botbrains.io/~/conversations) filtered to the flagged conversations. You can also add the **Monitor** filter there yourself.

## Missing Conversations

| Reason | What to check |
| - | - |
| **It's hidden** | Monitors never include [hidden conversations](/concepts/conversations#hiding-a-conversation). |
| **It has no customer message** | Monitors with flag criteria skip it. |
| **It's outside the time range** | The look back and the time range use the date of the conversation's last message. |
| **It doesn't match the filters** | Open the monitor's definition and compare the filters with the conversation. |
| **The sample left it out** | A sample rate below 100% evaluates only a share of the conversations. |
| **It isn't evaluated yet** | Results appear about 10 minutes after the conversation goes quiet. A paused monitor evaluates nothing new. |
| **Its result is N/A** | The AI couldn't decide, or the conversation has no score. N/A results don't appear under **Flagged** or **Not flagged**. |

## Managing Monitors

Changing a monitor only re-evaluates conversations when the change affects its results.

| Change | Effect |
| - | - |
| Filters, mode, look back or time range, sample rate, prompt, or scorecard | The monitor discards its results and evaluates its scope again |
| Scorecard threshold | The monitor applies the new threshold to existing results right away |
| Name, description, or alert | No re-evaluation |

**Re-evaluate now.** Runs an active monitor again from the **More actions** menu and skips conversations that haven't changed since their last result. It isn't available for monitors without flag criteria or while a run is in progress.

**Pause.** A paused continuous monitor keeps its results but evaluates nothing new and sends no alerts. Activate it to resume.

**Delete.** Removes the monitor and all its results. You can't undo this.

**Who can change monitors.** Organization Owners and Admins, Project Owners, and Project Contributors. Project Members and Viewers can view monitors and receive their alerts, but not change them. See [Permissions](/concepts/servicelens#permissions).

## Alerts

Continuous monitors can notify your team when the number of flagged conversations crosses a threshold. In the **Alert** field of the editor, switch on **Notify the team** and choose a count from 1 to 10,000 and a window of one hour or one day. Windows follow UTC clock hours and calendar days, and each flagged conversation counts toward the window of its last message.

| Rule | Detail |
| - | - |
| **Check interval** | Every 10 minutes |
| **Frequency** | At most one alert per monitor and window. At most one email per monitor and UTC day, so later alerts that day arrive in-app only. |
| **Recipients** | Organization Owners and Admins, plus project members whose role can view monitors. All default project roles can view monitors, see [Permissions](/concepts/servicelens#permissions). |
| **Turning alerts off** | Click **unsubscribe** in an alert email to stop these emails for you. People who can change the monitor can switch off **Notify the team** for everyone. |

The alert email links straight to the monitor's flagged conversations for that day.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.