Experiment
Published
Can a local model tick off your question list during a call?
We gave three local models five recorded calls and asked them to tick off a prepared list of questions as the conversation went. The two smaller models got about half of their ticks wrong. The largest got 4 of 21 wrong, and when the Mac's owner tried to work during its run, the Mac slowed so much that we stopped the run. How we measured it, the numbers, and how far to trust them.
You go into an interview or a client call with a list: questions to ask, points to settle. Halfway through you no longer remember which of them are dealt with. A meeting recorder could keep that list on screen and tick items off as the conversation resolves them. Daisy already transcribes a call live on the Mac, so we tested the one thing that was unclear: can a local model, one that runs on the Mac and not in the cloud, tell from the transcript that an item is closed? On these recordings the answer was no. The models small enough to run during a call ticked wrongly about half the time, and the one that did better slowed the Mac so much during ordinary work that we stopped its run.
- What we tested
- Three local models ticking off a ten-item list while a recorded call is replayed minute by minute
- Models
- Qwen 3.5 in three sizes: 4B, 9B and 27B, run on the Mac through Ollama
- Recordings
- Five one-on-one work calls in Russian, 32 to 79 minutes; four of them in the three-model comparison
- Answer key
- Written by a cloud model, not yet checked by a person
- Result
- Wrong ticks: 15 of 29 for the 4B, 8 of 17 for the 9B, 4 of 21 for the 27B
- In Daisy
- Daisy does not tick lists automatically, and these results do not justify adding it with a small local model
Three states of a list item, and what a wrong tick looks like
By the end of a call a list item is in one of three states. Take the item “Who signs the contract on their side” (the examples here are invented, not taken from the recordings).
- Closed: the question was answered, the topic was discussed in substance, or something was agreed. “Who signs on your side?” — “Our finance director.”
- Raised, not closed: the topic came up or the question was asked, and no answer or decision followed. “Who signs on your side?” — “Let me get back to you on that.”
- Not discussed: nobody brought it up.
A tick means the model declared an item closed. The wrong tick that matters most is the second case getting one: you asked, you got no answer, and the list tells you the question is dealt with. That is worse than no tick.
We counted a tick as correct when the item really was closed and the tick came no earlier than one minute before the moment the answer key gives for it. A tick on an item that was only raised, on an item never discussed, or on a closed item too early is wrong. A correct tick has no lateness limit; ticks more than two minutes late are counted separately below.
The smaller models got about half of their ticks wrong
The setup in one line: for each call a list of ten items, and once a minute the model reads the newest stretch of the transcript and says which items it closed. The full method is further down. The comparison covers four calls: 24 items that were closed, 8 only raised, 8 never discussed.
| Model | Ticks given | Wrong | Too early | On an item only raised | On an item never discussed |
|---|---|---|---|---|---|
| Qwen 3.5 4B | 29 | 15 | 8 | 4 | 3 |
| Qwen 3.5 9B | 17 | 8 | 4 | 2 | 2 |
| Qwen 3.5 27B | 21 | 4 | 1 | 2 | 1 |
The 4B ticked the most and was wrong the most. Its eight early ticks came 2 to 36 minutes ahead of the answer key: three before the topic had come up at all, five after it was raised and before it was settled. Of the 24 closed items it ticked 14 correctly, three of them 5 to 26 minutes late, and it put at least a “raised” mark on all eight never-discussed items.
The 9B ticked less and was wrong about as often. It left five of the eight never-discussed items unmarked, but of the 24 closed items it ticked only 9 correctly, none of them late: seven it marked as raised and no further, four it never marked.
The 27B ticked 17 of the 24 closed items correctly, none of them late, ticked one too early and marked the remaining six as raised. In none of the four calls did it have more wrong ticks than either smaller model.
About half of the smaller models' wrong ticks are early ones (8 of 15 and 4 of 8), and timing is the part of the answer key we trust least. So here is the same count under looser rules.
| Model | No more than one minute early (the rule above) | No more than five minutes early | Timing ignored |
|---|---|---|---|
| Qwen 3.5 4B | 15 of 29 | 14 of 29 | 7 of 29 |
| Qwen 3.5 9B | 8 of 17 | 6 of 17 | 4 of 17 |
| Qwen 3.5 27B | 4 of 21 | 4 of 21 | 3 of 21 |
Five minutes of slack removes one wrong tick for the 4B and two for the 9B. With timing ignored, counting only ticks on items that were never closed, about a quarter of the smaller models' ticks are still wrong, and the 27B's lead narrows: 3 of 21 against 4 of 17 and 7 of 29. The 27B's one early tick is on an item that all three models ticked at the same minute, about five minutes ahead of the answer key; the key may be late there.
The largest model was more accurate and too heavy for a Mac in use
| Model | Typical check (median) | Nine checks in ten finish within | Share of the call the model is working |
|---|---|---|---|
| Qwen 3.5 4B | 7.7 s | 11.2 s | 12% |
| Qwen 3.5 9B | 7.3 s | 14.7 s | 14% |
| Qwen 3.5 27B | 18.5 s | 32.6 s | 34% |
A check once a minute that takes eight seconds is fine, so speed is not the smaller models' problem. (The Mac was not idle during the 9B runs, so its row is an upper bound.) The 27B's row holds only for a Mac doing nothing else. When its owner sat down to work during the run, the last checks took 60 to 94 seconds, longer than the minute each one has, and the Mac slowed enough that we stopped the run. The model's file is 19 GB; on a 16 GB Mac the file alone is larger than the memory. No call and no live transcription were running during any of these timings.
That stopped run is why the comparison has four calls and not five: the 27B got 21 minutes into the fifth. The fifth is also one of the hardest. Its recording has echo, so the same lines are attributed to both speakers. With it included, the 4B's wrong ticks are 23 of 39 and the 9B's 12 of 23.
Asking the model for a quote changed little
We tried two answer formats. In one the model returns item numbers only. In the other it must add a short quote from the excerpt as proof, and our code checks that the quote is there, comparing its first 40 characters and rejecting quotes shorter than eight; if it is not, the tick is reduced to a “raised” mark. All tables above use the second format. For the 4B on the same four calls the wrong ticks were 18 of 27 without the quote and 15 of 29 with it; on all five, 24 of 37 and 23 of 39. The check confirms that the words are in the excerpt, not whether they are a question or its answer. The comparison exists for the 4B only: the 9B's runs without the quote did not finish on two calls.
How we tested
The recordings are five of the Daisy founder's own work calls from August to October 2026, with the finished transcripts Daisy made on the device. None had a written list, so one was written for each call afterwards: ten items, phrased differently from how the conversation went. Six were items the call did close, two were raised and left open, and two were decoys: never discussed, but close to the call's topic.
The lists and the answer key were written by a large cloud model, Claude, which read the five transcripts and marked for each item its state, the moment it was raised and the moment it was closed. The three local models ran only on the Mac.
Each transcript was then replayed as if the call were live. Every 60 seconds the model received the items not yet ticked and the latest 90 seconds of the transcript, with speaker labels and timestamps, was told which speaker owns the list, and had to return, as JSON in a fixed schema, the items this excerpt closed and the items it only raised. A ticked item leaves the list for good. The model keeps nothing between checks and is not told which items were raised earlier. That matters: only 9 of the 24 closed items had the moment they were raised and the moment they were closed inside one excerpt, so for the other 15 the question and its answer reached the model in different requests.
The instruction, translated from Russian: “You follow a live meeting and its owner's plan. You are given the plan items that are still open and a new excerpt of the conversation. For each item decide from this excerpt: covered — the item is closed (the question was answered, the topic discussed in substance, something agreed); touched — the topic was raised or the question asked, but the excerpt has no answer or decision; otherwise do not mention the item. Do not mark an item only because similar words occur: the meaning must match. The excerpt is an automatic transcript with errors. For each marked item give a verbatim quote from the excerpt, no longer than eight words. If nothing fits, return an empty list.”
- Machine
- Mac with Apple M4 and 32 GB of memory, macOS 27
- Runtime
- Ollama 0.40.15, temperature 0, 4,096-token context (no request exceeded about 1,500 tokens), thinking off
- Models
- qwen3.5:4b (4.5B parameters, 4.0 GB), qwen3.5:9b (9.7B, 7.6 GB), qwen3.5:27b (27.4B, 19 GB), all in nvfp4 quantisation
- Checks
- On the four calls: 173 for the 4B, which stopped being asked on one call after ticking all ten items, and 191 each for the 9B and the 27B
Thinking was off to keep a check to seconds; we did not test it on. An answer that could not be parsed counted as no marks. A request that had not returned in five minutes ended the run, which is how the 9B's runs without the quote ended on two calls.
How far to trust these numbers
- Some of what we count as a wrong tick may be an error in the answer key, in the state of an item or in its timing.
- The lists are artificial: written after the call, deliberately rephrased, with decoys chosen to be close in topic. Lists written before real calls were not tested.
- The counts are small and come from four or five calls of one kind in one language. “4 of 21” is four ticks, not a stable rate.
- One prompt per format, one model family, no memory between checks, each run made once. A different setup may do better; the next section lists what we would try.
- The transcripts are finished ones. A transcript produced during the call is rougher.
- The timings are not a controlled benchmark: one machine, background load not held constant.
- The recordings are private and stay private, and the replay and scoring scripts are not published, so the totals cannot be reproduced from outside. What can be checked is the method: the instruction, the rule for a wrong tick and the setup are all above.
What we do with this
What the experiment supports is narrow: one request a minute over one excerpt, with a 4B or 9B model, does not produce ticks we would show to anyone. A wrong tick hides a question you still needed to ask, so a list that is wrong a quarter to a half of the time is worse than no list.
A list on screen that you tick yourself needs no model. That is our opinion, not something this experiment tested, and it is what we would build first.
What we would test before returning to automatic ticks: a second request that must confirm a tick before it is shown; a decision made over several excerpts in a row and not one; thinking on; and a cloud model on the same task, only for meetings whose summaries already go to a provider the user chose. A person checking the answer key comes before any of these.