kakinang INSIGHTS · 2026-10-04
Which service metrics show an SME whether AI is actually helping?
More AI I want to use reports and human review to assess whether AI helps the team handle enquiries. conversations do not automatically mean better service. Website traffic may have increased, or visitors may be repeating the same unresolved question. Start with the work the company wants to improve: receiving after-hours enquiries, following up on leads, or getting a booking through customer confirmation. Platform numbers offer clues, while a sample of actual conversations and human follow-up is still needed to judge quality. A single “AI resolution rate” cannot carry the whole conclusion.
Define the service outcome before selecting numbers
A three-person service team may want “after-hours enquiries do not disappear”, not “AI sends as many answers as possible”. Observe incoming conversations outside opening hours, those requiring human judgement, those still pending the next day and whether the visitor received a useful next step. Another company may focus on sales enquiries not being missed. It should check contact quality, named follow-up ownership and reply within an actually promised period. A slide filled with every available number will not help if the goal and decision are unclear.
Zendesk's official metrics guidance distinguishes first reply, resolution time, reopened cases and satisfaction, and calls for careful interpretation of the first reply definition. This is a measurement reference, not a claim that kakinang provides all Zendesk fields. NIST's AI risk framework connects measurement to management: monitor outcomes that matter to users, then use the findings to change controls. For a small company, that means a person beside the report who can repair knowledge, staffing or exception handling.
Fix the period and denominator. To compare the proportion of leads followed up across two weeks, define the denominator as new leads in each week after excluding duplicates and tests, and the numerator as those with a first human contact. If the platform does not offer that exact “first contact” metric, record it in the company's own work sheet; do not call a lead count completed follow-up. Mark channel mix, working days and campaigns too. A burst of advertising can otherwise hide the effect of a staffing change.
Connect available reports to the actual workflow
kakinang's tenant analytics can show pending human work, unread conversations, new leads and conversations today, resolved and autonomously resolved cases, average first response and resolution time, and period trends. Separate report sections cover conversations, usage, leads, bookings, commerce and connectors. An authorised user can export CSV from the available report view. Visible sections depend on roles and enabled functions. These values help locate a problem, but first check their definitions, time range and offline work they do not capture.
Read three layers each week. At the entrance, see whether chat opens and new conversations changed. In handling, look for unread or pending human cases and longer resolution. In next steps, inspect whether a person followed up leads and whether customers confirmed bookings. A rise in conversations without more leads could mean public questions were resolved, or that the capture path is broken; it is not automatically good or bad. A booking is formally counted according to customer confirmation and the business record, not merely because AI suggested a time in chat.
Usage reports help explain conversation and token resource consumption, not quality by themselves. If tokens rise, investigate traffic, longer conversations, repeated questions and knowledge lookup before shortening necessary answers. A CSV lets an owner reconcile platform totals with the company's own follow-up log, but operational exports should be handled by authorised roles. Metrics not supplied by the platform, such as actual sales value per opportunity or subjective customer satisfaction, require separate collection. Do not describe them as built-in dashboard numbers.
Use a weekly table to find the next repair
Imagine a small course business receiving forty new website conversations in one week. Eight require human judgement, five leave contact details and three customers confirm trial bookings. These are hypothetical numbers for a method example, not kakinang customer results. It would be misleading to divide three by forty and announce a 7.5% AI conversion rate. The forty can include address questions, repeated tests and unrelated needs. Inspect whether the five leads have usable contact routes and owners, then check whether the three bookings overlap those leads.
Sample the eight conversations requiring a person. If two remain pending on Monday, fix staffing or notification first. If two ask for prices that are already published, check whether knowledge is stale or the answer fails to find its source. If somebody requests work outside the service scope, a clear boundary and referral may be the correct result. This separates the different causes behind “AI did not help”. Time live human takeover and asynchronous leads separately; a weekend flow without staff cannot be judged against a live reply promise.
A useful weekly row needs period, observed value, sampled cases, chosen correction, owner and next review date. After fixing an FAQ, check whether the same question is still repeated. After changing lead notifications, check pending work. After staffing a live period, check actual human replies rather than automated greetings. Reports alone cannot prove causation, and a small sample does not represent every customer, but it can reveal a repairable bottleneck. A metric without a definition, denominator or follow-up action should not become a performance promise. Set a stop condition for each improvement: if wrong prices appear in answers for two consecutive weeks, pause the relevant public knowledge and ask its owner to verify it; if leads remain unattended after notification changes, inspect inbox ownership and holiday staffing. If a metric improves while staff workload rises, review human handling time as well as the automated-resolution share. Each company defines these rules against its own service promise, and people must choose and make the correction. Record the data period and source so the next colleague can recalculate the conclusion.
Questions teams ask
Does a high AI answer rate mean customers are satisfied?
No. An answer can sound complete yet be unsupported; people may repeat questions or receive no follow-up. Check answer sources, unresolved cases and actual replies alongside reported resolution and handoff. If satisfaction was never collected, it cannot be inferred.
Does first response time include an automatic AI message?
Definitions vary by product and report. Zendesk defines its first reply in relation to a public human agent response. Interpret kakinang's average first response time according to its own metric definition. For takeover speed, a team may separately record when human help was needed and when a person actually replied.
Is every lead a successful conversion?
No. Check that contact information works, the need is clear and an owner followed up. Lead count is an entry into a process, while a sale or completed service is another outcome. Without corresponding data, do not use leads as revenue.
Must a small team read many dashboards daily?
Choose a fixed period and a few numbers that can change an action, such as new conversations, pending human cases, unread items, leads and confirmed bookings. Sample success and failure each week and decide whether to update knowledge, notifications or staffing.