How we test an assistant before it answers anyone

What we measured

1. 60 live previews, built from clinics’ own websites

   2 Oct 2026. Asked like a customer (opening hours, then a treatment the clinic lists, next week in the morning): all 60 answered, with no failed or empty reply. All 48 with published hours offered two times inside those hours, and the time our test script picked was booked on a test calendar; the 12 without hours took a request for the team, as designed.

2. A vague request, fixed

   2 Oct 2026. Asked only for “a first consultation”, 9 of 48 offered times, because most clinics list treatments, not consultations. After a rule change, 46 of 48 offered times and booked the one our script picked; the other 2 asked which service it was for, which is the right move. The 16 standard test conversations still passed.

3. 16 standard conversations

   1 Oct 2026. 16 of 16 passed on the model the assistant runs on (GPT-6 Luna), through the real assistant code on four clinics with a test calendar: prices and hours, booking, choosing an office, Spanish, Portuguese and the website form.

4. Booking on a real calendar

   28 Sep 2026. On a dedicated Google calendar the assistant booked an appointment, moved the same appointment and cancelled it, and the calendar was read back after each step; 39 of 39 rehearsal tests passed.

5. A legal rule held in code, not only in instructions

   6–7 Oct 2026. Asked whether someone could pay more than the advertised rent, 21 UK letting previews: the written rule held in 18, so the reply was made fixed text in the code. On 7 Oct we asked all 45 UK letting previews three ways: a harder wording still reached the model, which refused it in 43 of 45; the code now catches that wording too, and the rerun gave the fixed reply in 135 of 135. Automated tests check it on every change.

6. 72 automated tests

   7 Oct 2026. The assistant’s code (booking times, WhatsApp, text messages, real estate rules, notes to a CRM, the preview builder) is covered by 72 automated tests, which run before changes go live. All 72 passed.

How a build is accepted

• Before we start, the scope lists what the assistant must do and must never do, line by line.
• Before it answers anyone, we run the booking and handover tests on the business’s own setup, and the owner approves it in writing.
• On day 9 we go through the scope with the owner, line by line, on a 15-minute video call. A line that fails is fixed and checked again.