---
title: "How do you measure the start of AI agent usage beyond signup?"
published: 2026-10-15T13:00:00+03:00
updated: 2026-10-23
lang: en
canonical: https://ofershap.github.io/en/posts/agent-onboarding-completed-task-metrics/
original: https://ofershap.github.io/posts/agent-onboarding-completed-task-metrics/
---

# How do you measure the start of AI agent usage beyond signup?

Define separate stages for joining, connecting, task attempt, a checked result, and repeat use. Measure each stage over the same period and the same user group, and mark where progress stopped. A signup is not proof of a completed task.

## In depth

### Start with a definition of useful usage

Before a metrics table, pick one task the user wants to complete. Write down what output will be produced and what will be checked in it. A suggested example: the user asked for a comparison, received a document with sources, and checked that every threshold condition appears in it. The act of sending the message to the agent is an earlier stage.

Task completion should fit the type of work. Downloading a file doesn't prove its content is right. Connecting an account doesn't prove the agent performed a permitted action. If human approval is needed, define upfront whether "completed" includes the approval or stops at the draft.

### Separate events from interpretation

Build a clear stage list. A synthetic illustration for an onboarding flow: joined, completed a permitted connection, chose a task, received an output, checked a result, and returned for another task. Every stage needs an entry condition you can see, not a feeling from whoever is guiding.

In the Google Analytics documentation, an event measures an interaction on a site or app, and you can define a custom event when no existing event fits. That helps plan collection, but the event name is not business proof. An event called task_completed must be sent only when the agreed definition actually holds.

This guide does not recommend installing Analytics in every system. You can also start with minimal consented manual logging, without copying the task content or the conversation.

### Keep a base group and a measurement period

The total number of joiners and the number of this week's finishers are not the denominator and numerator of one measurement. The finishers may have joined earlier, or new joiners haven't had time to perform a task yet.

Choose a cohort of joiners and a follow-up window. A suggested example: everyone who joined in a given week, and whether they reached a checked output within the following seven days. This is an experiment proposal, not a window that fits every product. A task of a few hours and a task dependent on an external vendor need different time frames.

Also note in the report when the data was measured. An old snapshot doesn't become a fresh data point because the post was published again or later. The data date and the publication date should stay distinct.

### Look for the stopping point

A funnel report can show where people move between stages and where they drop. The Google Analytics documentation distinguishes between an open funnel, which allows entry at different stages, and a closed funnel that starts at the first stage. The choice changes who gets counted.

Even without an analytics tool, ask whether a person stopped because of connection, not choosing a task, an unclear output, or waiting for approval. Those are different problems. Sending everyone another reminder message doesn't automatically fix any of them.

The number of events is not the number of people. A repeated attempt can create several events for the same task. Define how you identify an attempt, how you handle duplicates, and what you count when a task is reopened.

### Add a small quality check

Check a sample of outputs against the agreed criteria, with permission to use the data. You can ask the user to mark what helped and what needed fixing without sending a whole private document.

Example: if ten documents were produced and sources were checked in only some of them, don't merge them into one line called "ten successes". You can report separately output that was returned and output that was checked. When there's no evidence of quality, write what's missing instead of completing the number to match expectation.

### End with one change you can test

Pick the prominent blocker and suggest a small change: clearer connection instructions, a shorter first task, or a result check that's easy to understand. Compare the same success definition in the next period, and note what changed.

Measurement should help people reach a result, not collect more information about them. Keep only what the question you defined requires, with proper consent and a clear retention period. A successful onboarding flow is useful usage you can explain, not growth in a number on a card.

## From my experience

My post about the onboarding program separates people who joined through a link from participants in the program. That raises a different question from technical connection: which stage has already happened, and when can you say useful usage has begun? The numbers in the post and the image are historical documentation, not a new measurement in this guide.

## Frequently asked questions

### Does a signup count as agent usage?

It's a stage in the flow. To measure useful usage you need to define a task, an output, and a result check separately.

### How do you compare weeks?

With the same base group, time window, and success definition. Document process changes so you don't attribute every difference to the tool itself.

### Do you need to keep all conversations for measurement?

You can measure stages and limited feedback without keeping the content. The scope of collection and access requires consent and privacy planning.

## About the author

Ofer Shapira builds and leads AI in engineering teams, and writes about agents, decisions, and workflows you can verify.

Last updated: 2026-10-23