Monitors

Monitors flag the conversations you want to review. A monitor watches the conversations that match its filters and flags each one that meets a condition you describe in plain language, fails a scorecard, or both. You see how many conversations it flags each day, open them in one click, and can alert your team when the count crosses a threshold. You manage monitors on the monitors page.

​

How Monitors Work

Think of a monitor as a spotlight you shine on specific conversations or behaviors. Filters narrow the light, for example to the conversations of one topic, and a prompt picks out the exact behavior you don’t want to see within them, such as the AI agent promising a refund it isn’t allowed to give. Every conversation in the light gets a result, so you see how often the behavior shows up and whether it grows or shrinks.

A monitor decides which conversations get attention, and a scorecard defines how to grade them.

Together with a scorecard, a monitor also sets up a review queue: when a monitor watches a scorecard and some of the values that feed into it are still missing because they need a person’s input, the monitor appears as a queue on the Reviews page, and your team answers the missing values in a dedicated review mode. Several people can work through the same queue at once without opening the same conversation, so you can split the work. See Review Queue. Click New monitor on the monitors page to start from scratch or from a template, then set it up in two steps: The monitor then evaluates every conversation in scope and gives it one result. Monitors never include hidden conversations. Monitors with flag criteria also skip conversations without a customer message. A conversation can appear in several monitors, and each monitor evaluates it on its own.

​

What to Monitor

​

Filtering Conversations

Narrow the scope by channel, AI involvement, topic, status, rating, labels the conversation has or lacks, conversation attributes, and scorecard scores. Without filters, the monitor looks at every conversation.

​

Continuous and One-off

Both ranges apply to the date of each conversation’s last message. A continuous monitor evaluates a conversation once it goes quiet. Results usually appear about 10 minutes after the conversation goes quiet, or up to about 40 minutes later for conversations without an AI reply. When a conversation changes and starts or stops matching the filters, the monitor adds or removes it. When a conversation continues after the monitor evaluated it, the monitor evaluates it again once it goes quiet.

​

Sampling

Set a sample rate from 1% to 100% to evaluate only a share of the matching conversations, for example so a busy project doesn’t send your team more conversations than it can review. The default is 100%. A monitor always picks the same conversations for its sample.

​

Flag Criteria

Prompt. Describe when to flag a conversation, for example “the customer had to repeat their request.” The AI reads each conversation and flags it when the condition holds. A prompt can have up to 2,500 characters. What the AI reads. The AI judges the conversation transcript: the customer’s messages, the replies of the AI agent and your team, escalations, and the names of attached files. It doesn’t see tool calls, internal notes, your knowledge, guidance, or procedures, so describe behavior that shows in the messages. For example, a prompt about placed orders only catches orders the conversation mentions. botBrains shortens long messages and transcripts. Scorecard. Pick a scorecard in …or when it fails a scorecard and a threshold from 0 to 100, which defaults to 70. A conversation fails when it scores below the threshold or fails a critical item. A conversation without a score doesn’t fail. The monitor also has the AI fill the scorecard’s AI items when they’re missing or out of date, and updates its result whenever a score changes. Both. The monitor flags a conversation when either criterion applies. Neither. The monitor flags every matching conversation. Instead of a results list, it shows how many conversations match, up to 10,000+, with a link to open them.

​

Testing Before You Save

Click Test on recent conversations in the editor to run your current settings on the five most recent matching conversations with a customer message. botBrains saves nothing. The test shows how many conversations are in scope, how many of the tested ones the monitor flags, and the result, score, and flag probability of each. For a monitor without flag criteria, the test counts the matching conversations instead. When you save, the monitor evaluates its scope in the background. If that covers more than 1,000 conversations, the editor asks you to confirm first.

​

Reviewing Results

The monitors page groups monitors into Continuous and One-off. For continuous monitors, you choose a period of 7, 30, or 90 days. Click a day in a card’s chart to open that day’s flagged conversations. Open a monitor to see its numbers, its definition, and the conversations it evaluated. Continuous monitors compare each number with the previous period of the same length. The daily chart counts days in Coordinated Universal Time (UTC). Click a bar to narrow the conversation list to that day. For scorecard monitors, the list also shows the score, critical failures, and the weakest scorecard items. botBrains Engineer, the Forward Deployed Engineer (FDE), can investigate a monitor for you. Click Explore with FDE to start a thread about it. View conversations. Opens the conversations page filtered to the flagged conversations. You can also add the Monitor filter there yourself.

​

Missing Conversations

​

Managing Monitors

Changing a monitor only re-evaluates conversations when the change affects its results. Re-evaluate now. Runs an active monitor again from the More actions menu and skips conversations that haven’t changed since their last result. It isn’t available for monitors without flag criteria or while a run is in progress. Pause. A paused continuous monitor keeps its results but evaluates nothing new and sends no alerts. Activate it to resume. Delete. Removes the monitor and all its results. You can’t undo this. Who can change monitors. Organization Owners and Admins, Project Owners, and Project Contributors. Project Members and Viewers can view monitors and receive their alerts, but not change them. See Permissions.

​

Alerts

Continuous monitors can notify your team when the number of flagged conversations crosses a threshold. In the Alert field of the editor, switch on Notify the team and choose a count from 1 to 10,000 and a window of one hour or one day. Windows follow UTC clock hours and calendar days, and each flagged conversation counts toward the window of its last message. The alert email links straight to the monitor’s flagged conversations for that day.