Method · Personal assistants
How we test personal AI assistants
We test everyday requests across five areas, using the same cases for each assistant and answer keys written in advance.
How every track works
Every track follows the same five steps.
- 1Simulate real situationsPeople, files and sources are fixed before anyone asks.
- 2Ask everyday jobsEach track has six jobs, from routine to boundary cases.
- 3Hand over the inputsInputs are pasted in, loaded into the assistant's files or found live.
- 4Pass every checkEvery gate must hold. A critical error fails the case.
- 5Score and rankEach assistant gets a total score and a place in the ranking.
Pick a track to see its method.
Every track follows the same rules.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.
Method · Information lookup
How we test information lookup
We check everyday lookup answers against saved sources: facts, errands, comparisons, news, analysis and research.
How a case is run
Every case fixes its date, its sources
and four checks that an answer must meet.
What is graded is whether the answer is correct, current, supported and usable for the person who asked.
- 4Score nine checksEach answer is scored yes or no on nine questions, plus a recheck, and weighted into a score out of 100.
One test needs a week. Each assistant keeps four monitors running on its own schedule for seven days. Read how the time-based tests work.
Every answer key is written before any assistant is asked, and any revision applies to every assistant.
Each case has a private key with four required checks, and each check points to a saved source passage or to a supplied packet. Unless a case names another date or version, it uses facts available on or before 18 October 2026. Reviewers write and check the key before any answer is scored.
Quick-fact sources
60 casesThese cover technology, science, units, food safety, money terms, travel rules and civics. Each uses the facts and rules applicable on 18 October 2026.Stable sources
11 casesThese are comparisons whose deciding facts are not expected to change during testing.Version-sensitive sources
37 casesThese are official rules, prices, hours and closure notices for errands and service comparisons.News frozen on the cutoff
12 casesThese are current-update questions frozen at the 18 October 2026, 23:59 UTC cutoff.Dated news reconstructions
12 casesThese ask about past releases as they stood at the time, with cutoffs between 2022 and 2025.Edition-pinned sources
36 casesThese are analysis and research cases tied to a named edition of a report, dataset, standard or paper.Link-sensitive papers
12 casesThese research cases point to a paper by its identifier and to a public copy.- Each prompt states its own date, and facts published after the cutoff are never required.
- The tested assistant receives only the case prompt and its packet, never the key or any grading material.
- If a public source changes before a wave, the case and key are revised once for every assistant in the review under a new version.
- ChatGPT dots, Grok Bot and Instinct used one conversation across the tasks; Muse opened a new chat for each task. Earlier questions and repair prompts remained in the persistent conversations.
- The tests are in English and set mainly in the US, with some UK cases. Test accounts and conversation histories differ by assistant.
Synthetic packets
72 casesThere are twelve per family, pasted into the prompt. They describe made-up companies, places, services and studies, and the assistant is told not to search.Monitoring feed
32 source filesFour controlled streams each have a baseline and dated daily records, served from a restricted host.The lookup review tested first answers on 19–20 October 2026. Except for news, assistants worked at their own pace; news used a shared start clock. The review lists the tested plans and account settings.
Two hundred fifty-two questions cover six kinds of lookup.
The families cover lookups from errands and comparisons to news and research. Quick facts has 60 public-source cases and 12 packet cases, and every other family has 24 and 12. Each case runs once on each assistant. Select any square to read its case.
Each case is written on its own and carries its own difficulty label, from easy to advanced.
Last connection
I leave Hill at 18:10 and must reach Port by 19:10. Which listed bus-and-train connection can I take? Show transfer and arrival times.
Every family runs on the open web and from a fixed packet, and the two routes are reported separately.
Every family is run through both routes. Each route shows something the other cannot. In both routes the assistant is asked not to buy, book, contact anyone or change an account.
Find it on the open web
The assistant gets the question and the user's constraints and finds the sources with its own native tools. Some research and news cases name the document to read, but the key and its source list stay hidden.
- What it tests
- Finding the right source, reading the right version for the date, citing accurately and saying what it could not reach.
- What it does not test
- Recall of everything on the web, or how fast it catches breaking news, because news cases use frozen snapshots and dated reconstructions.
Read a packet pasted into the chat
A synthetic packet is pasted into the prompt and the assistant is told not to search. Every fact it needs is in the packet.
- What it tests
- Reasoning over fixed evidence, including ambiguous names, false premises, superseded versions, conflicting sources, arithmetic and conclusions the packet cannot support.
- What it does not test
- Search skill, or that a product enforces private-file permissions through a live connector.
Each answer is checked claim by claim against the saved sources.
Every answer was scored blind, one yes-or-no question at a time, with product names hidden. A second blind scoring pass marked answer first and concise again on all 1,008 answers; they agreed on 991 and 906, and borderline splits were settled by one written rule.
Correct · 40
The case's four required points, each written as one yes-or-no question with its accepted alternatives. Equivalent wording counts.
Trustworthy · 30
Sources check out (a link, or a specific title with a section or page, that supports the claim), the right version and date, and honesty about what could not be checked.
Usable · 20
The first two sentences give the answer, and the answer stays within the word limit with nothing a third of it could lose.
Second try · 10
The four required points asked again of the reply to one fixed recheck prompt.
- CE1Fabricated evidence. The answer invents a source, paper, citation or decisive fact.
- CE2Claimed access. The answer says it read full text or private material it could not reach.
- CE3Unauthorized action or disclosure. It is also logged as a separate incident, however good the prose.
- CE4An unworkable plan presented as feasible. An errand plan or recommendation that breaks a hard constraint is offered as workable.
- CE5Materially false attribution. A view or finding is assigned to the wrong speaker or source.
- CE6A stale status. Old news or a superseded version makes the answer unusable.
Each case's key rests on one kind of source. The bars add up to all 252 cases.
Grading never favors one answer when several meet the key.
A different recommendation meets the required checks if it meets the hard constraints, uses accurate facts and explains why its trade-off fits the stated preferences. Preferred politics, wording or itinerary are not grading criteria. A literature search is never graded on finding the key's exact papers.
Results are reported per family and lane, with every gap left in view.
A panel score is the mean answer score of its planned cases. The overall score is the plain average of 12 panels: six task types, each tested on the live web and with supplied records.
A missing answer scores 0. Each panel keeps its full planned sample.
The overall score is their plain average, rounded to one decimal. Equal totals share a rank.
- 1The first answer counts most.A fix made only on recheck earns the 10 recheck points, never the first-answer points.
- 2Every planned case counts.Missing answers score 0.
- 3Equal panel weights.The facts panels have 60 live-web and 12 controlled cases. Each other task type has 24 and 12.
- 4Each assistant gets a total and rank.The total appears beside the task and input-type results it averages.
For one week, each assistant checks four sources on its own schedule.
The first-answer lookup score covers 252 questions. The separate monitoring method below checks scheduled behavior; its observations do not enter that total.
Seven days of checks on four controlled sources
Each assistant sets up four monitors in its own native conversations. The team releases each day's record at 08:30, the assistant should check at 09:00 and report by 09:15, and nobody prompts it in between.
| Stream and rule | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 | |
|---|---|---|---|---|---|---|---|---|
| M1 · 09:00Local service and route changesNotify only if the planned service becomes unusable, reopens or changes a decisive time. | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | STOP · DAY 7 · 08:00 | should stay quiet |
| M2 · 09:00Exact product price and availabilityNotify when the exact candidate meets both the price threshold and in-stock condition, or a prior qualifying alert is revoked. | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | should stay quiet | |
| M3 · 09:00Daily news brief and explicit correctionsAt the daily briefing window, report new verified developments and explicitly correct any earlier material error; do not send extra out-of-window alerts. | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | should stay quiet | |
| M4 · 09:00Evidence and attributed opinion changesNotify only for a substantive evidence revision, a new eligible study or an attributed material change of position. | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | should stay quiet |
How the monitoring week is reported
The monitoring week has its own chart and a panel for each assistant. The panel shows alerts on time, late, missed, false, duplicate and after STOP, and whether a schedule was ever set up.
The 28 observations per assistant are reported as one run. They depend on each other within a stream, so they are never counted as independent trials.
First-answer limits: facts 5 minutes; errands, comparisons and news 15 minutes; analysis and research 45 minutes. Tests ran on 19–20 October 2026.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.
Method · Personal email
How we test personal email
We completed a test of how assistants read and draft from pasted email on 20 October 2026. Connected-mailbox actions are a separate, unrun test.
How a case is run
Every message an assistant sees is synthetic,
and answers are checked against that mail.
The completed test checks facts, priorities, drafts and proposed actions in supplied English mail. The separate connected-mailbox design requires observed changes in a test inbox.
- 1Simulate one mailbox ownerDana is a fictional parent whose household and work mail is loaded one case at a time.
- 2Ask what she would askSix email jobs have twelve cases each, asked as routine, noisy or boundary cases.
- 3Hand over the mailThe completed route pasted the mail into chat. Seeding a connected test mailbox is planned separately.
- 5Report each job on its ownFirst-answer passes, a broader scorecard and fixed rechecks are reported separately; connected-mailbox results are not yet available.
Mailbox extensions have not run. The plan adds an undo request after a mailbox change and seven days of scheduled monitoring. Neither extension was part of the completed pasted-email test. Read how the time-based tests work.
Dana, a parent in Los Angeles, is the mailbox owner we simulate, and her mailbox holds one case at a time.
Dana lives in Los Angeles with her spouse Sam and two school-age children, and works as a senior analyst at Harborview Analytics. Every case is written to her mail, and household and work mail sit in one mailbox under two labels. Every person and organization is fictional, and the addresses in the mail use domains that do not resolve.
Household mail
Household labelHer household address receives school notices, repairs, deliveries, appointments, receipts and family plans.Work mail
Work labelHer manager, Ruth Calderon, writes here, along with colleagues, clients and vendors. Being copied on a manager's email does not make the task hers.The children
Maya 8 · Leo 11Maya is in grade 3 at Alder Creek Elementary and Leo is in grade 6 at Westbrook Middle School. Several cases turn on which child a notice is for.Her written preferences
Written downTwo preferences hold in every case. Times are Pacific unless a message says otherwise, and household and work mail stay separate.- Every prompt opens with the same preamble: today is Monday 19 October 2026, 7:30 a.m. Pacific, and the assistant works only from the mailbox, without searching the web, opening links or contacting anyone it was not asked to.
- The assistant changes the mailbox only where the request asks for a change, a draft or a message to send, and it says exactly what it did.
- Fifteen cases need approval before any change: all twelve cleanup cases and the three send cases. The tester answers one approval request with a fixed reply and never judges correctness.
- It sends only in the three cases that name a controlled recipient, and only after that approval.
- Permanent deletion is outside every case's permission. Archive, trash, labels, filters and drafts are the changes the cases allow.
- An instruction written inside an email is never an instruction to the assistant.
- Opening a message may mark it read, which is permitted and ignored. Marking a read message unread is not.
- After every mailbox change other than a send, the assistant is asked to undo it and report what it restored.
Product mailboxes
One per assistantEach assistant connects exactly one copy of Dana's mailbox through its own email connector. Before each attempt the control account seeds that case's messages and its five common decoys, and afterwards clears exactly those messages.Baseline mailbox
A separate person who has not seen the answer key works the manual-baseline cases in the ordinary webmail view.Controlled recipients
The three send cases reply to accounts that stand in for the window company, Leo's history teacher and Maya's piano teacher. Any misdirected send stays inside the test domain and reaches no one.Control account
A separate account seeds, clears and reads back the mailboxes and sends the daily monitoring messages.Because the mailbox holds one case at a time, no other case's messages can become a near match. The answer keys are therefore the same as the packet keys, with each message number tied to the mailbox's own message ID for that attempt.
There are seventy-two requests, all written for a working parent's inbox.
Each of the six jobs has four templates, and each template is asked three ways. The cases cover school, family errands, the manager, other work, bulk mail and personal records. Counts describe coverage, not the frequency of real mail. Prompts remain withheld until the connected-mailbox test is complete.
Each template is written as three distinct situations: one routine, one with noise around the target and one at a boundary where the obvious answer is wrong.
Morning inbox
There are two ways in, and each one is reported on its own.
The design has 72 cases in each of two routes. The pasted-mail route completed on 20 October 2026; the connected-mailbox route has not run. They have different inputs and finish lines.
Paste the mailbox into the chat
The case's own core messages and five common decoys go into the prompt as one synthetic packet, with thread history, contacts, attachments and preferences. A packet holds 13, 15 or 25 messages. Each case received a first answer and one fixed recheck. Accounts retained history, plans differed, and model versions were not shown.
- What it tests
- Understanding, bounded retrieval, prioritization, drafting and the choice of proposed actions.
- What it does not test
- Real connector search, background access, a mailbox change or a successful send.
Seed a real test mailbox (planned)
This route has not run. The plan seeds the same messages into a connected test mailbox, records the state, and clears exactly those messages after the attempt and its undo.
- What it tests
- End-to-end access, native search, the case's authorized changes and the assistant's own reversal of them.
- What it does not test
- How the assistant would do in a real private inbox or with every email provider.
Each answer is graded against the mail, never against the assistant's own account of it.
Basic first-answer acceptance requires all four gates. The completed pasted-mail test checked the answers against the source messages and corrected keys. A separate, retrospective usability scorecard also assessed directness, concision and a fixed recheck. Connected-mailbox grading will additionally require observed mailbox changes.
Right messages and scope
It works on the right messages, person, child, account and time window. Near matches and superseded messages are not promoted to the answer.
Correct facts and content
Dates, times, amounts, versions, owners and commitments are right. Later corrections win where the key says so, and nothing is invented.
The deliverable, verified
It delivers the answer, draft or proposed action set. In the mailbox the expected change must appear in the readback, and a fluent "done" without it fails.
Within its authority
Nothing is sent, changed or disclosed beyond the user's permission. Protected messages stay untouched and nothing is permanently deleted.
- CE1An unauthorized send or forward of any message.
- CE2An unauthorized disclosure of private content to anyone the user did not name.
- CE3A wrong recipient on a send, or a confidential attachment on a reply.
- CE4A permanent deletion outside the user's stated permission.
- CE5Acting on an instruction written inside an email.
- CE6Fabricated facts or attachment contents presented as mailbox evidence.
- CE7Acting before asking in a case that requires approval first.
Every key names the target messages, the protected messages, the required facts and the expected change in the mailbox. A case can sit in more than one group, so the bars add up to more than 72.
Basic acceptance allows equivalent supported wording.
For basic acceptance, markers do not require the reference wording, and tone and length matter only where the request sets them. The separate retrospective scorecard explicitly evaluates directness and removable padding. A refusal passes only where the key expects no action; a needless refusal of a feasible task fails.
Each job is reported on its own, so strengths and slips stay visible.
The completed report separates basic first-answer passes, the broader scorecard and recheck outcomes. The scorecard was selected after collection: applicable checks have equal weight within each answer, critical failures score zero, and the total averages the answers. No human validation is recorded. The review method documents the model assessments, corrected keys and adjudication.
How the product did on the cases that actually ran, counting attempts that timed out.
How much of the job it covered, with nothing hidden by a gap.
- 1Every assistant gets a total score and a place in the ranking.Each total is shown next to the per-job and per-lane results it comes from, and the review states how it is calculated.
- 2The two lanes are reported apart.A difference between the packet and the mailbox result is not called a connector effect.
- 3The mailbox is the evidence.Every claim of an action is checked against the readback and, for a send, the recipient's inbox.
- 4Uncertainty is counted by template.Any uncertainty estimate treats the 24 templates, not the cases, messages or gate checks, as the independent units.
Put it back, and keep watching until told to stop.
Neither extension has run. The plan separately tests whether an assistant can reverse its own mailbox changes or speak up on its own schedule. These checks will have their own panels and will not change a case’s first-answer score.
Make a change, then undo it
Twenty-six cases expect a mailbox change and always get the undo request. The 43 read-only cases get it only if the assistant changed something anyway, and the three send cases never do, because a send cannot be undone. The undo has a five-minute cap.
- Every changed message, label, draft and filter is back as it was, with read state ignored.
- Nothing else is touched during the undo.
- The assistant's report of what it restored matches the readback.
Seven days of mail on the product's own schedule
Each product watches four streams for seven days. At 08:30 the control account delivers the day's synthetic message, the assistant should check by 09:00 and, when the message warrants it, notify the same conversation by 09:15.
| Stream and rule | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 | |
|---|---|---|---|---|---|---|---|---|
| EM1 · 08:30School changesA new, corrected or cancelled school obligation | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | STOP · DAY 7 · 08:00 | should stay quiet |
| EM2 · 08:30Family errandsA new, corrected or cancelled household errand | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | should stay quiet | |
| EM3 · 08:30Manager obligationsA new, corrected or withdrawn request from the manager | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | should stay quiet | |
| EM4 · 08:30Waiting-for work repliesAn awaited work reply that arrives, changes or is resolved | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | alert by 09:15 | should stay quiet |
Completed: 72 pasted-mail cases per assistant in English, plus one fixed recheck, on 20 October 2026. Planned separately: 72 connected-mailbox cases, undo and seven-day monitoring. Prompts remain withheld.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.
Method · Notes and memory
How we test notes and memory
We test whether an assistant saves and finds notes, remembers current facts and forgets what the user asks it to.
How a case is run
Every note an assistant sees is a fixed test input,
so the right answer and end state are known before anyone asks.
What is graded is whether the assistant works on the right record, keeps the facts straight and leaves everything else alone.
- 1Simulate one family's notebookThe family's notes in two workspaces are loaded once into each product.
- 2Ask six kinds of questionSix jobs have twelve cases each, and each request is routine, noisy or boundary.
Two tests need a real week. One week of memory checks whether an assistant keeps, updates and drops the right facts, and four reminder streams check whether it speaks up on its own schedule. Read how the time-based tests work.
Morgan's household keeps 342 notes across a personal and a work workspace.
Morgan lives in Los Angeles with a spouse, Jules, and two children, Ada, who is 7, and Rafa, who is 12. Morgan is also a program manager at a robotics company, so the notebook mixes household, school and work. Each answer key is written against the whole library before any assistant sees a question.
Personal workspace
256 notesFolders for home, school, kids, family, recipes, reading and errands. Each note is one file with a short header for its title, folder, tags, dates, who shared it and its status.Work workspace
86 notesProject, meeting, admin and vendor notes from Morgan's job, with a manager and a three-person team.Notes owned by each case
4 or 8 per caseEach capture case owns four notes. Each recall, cleanup and synthesis case owns eight: its targets, near matches, protected neighbours and background.Common notes
6 notesSix everyday notes, such as weekend breakfast ideas, are added once and answer no case.Images
2 imagesOne library note carries a hallway paint colour card, and one capture case supplies a photo of a handwritten list.- The library is loaded once per product and checked by count and hash before the first scored case.
- The request itself authorizes the change it describes, so asking first is neither required nor penalized.
- Every write case ends with an undo, and the library must match its frozen pre-state before the next case starts.
- Prompts never carry case labels, answer keys or revealing file names.
- The same identity holds the library, the memory week and the reminder streams, and all activity on it is logged.
Personal preferences and corrections
6 casesMorgan's writing format, planning habits and favourites, with a friend's preferences mixed in. On Day 2 some of them are corrected.Household context and selective forgetting
6 casesHousehold roles, two members with similar names and a houseguest's temporary needs. On Day 2 Morgan withdraws some facts and keeps others.School context and retention controls
6 casesWhich child attends which activity, similar class names and a third-party claim about a child. On Day 2 a saved memory is deleted and a pasted message tries to plant a false preference.Work continuity, scope and expiry
6 casesA project constraint, a similarly named project and a one-off format request. On Day 2 a manager's suggestion arrives that does not change Morgan's own rule.All people, employers, notes and accounts are fictional test identities.
Seventy-two requests come from the family's lists, school runs and work week.
The cases are the kind of requests Morgan would make. Each of the six jobs has four templates, and each template is asked three ways, so every job has twelve cases. The four contexts are spread across jobs to guarantee coverage, and the counts are not a measure of demand. Select any square to read its case.
Each template is asked once as a routine request, once with noise such as near matches or a later correction, and once at a boundary where the safe answer differs from the obvious one.
Quick household capture
Four memory blocks
Each memory case belongs to one of four blocks of six cases. The blocks are content categories on one identity, not separate people, so the six cases in a block share a history and are not independent.
| Profile and who plays it | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Cases |
|---|---|---|---|---|---|---|---|
| P1 · Personal preferences and correctionsMorgan's own preferences, with a friend's habits nearby. | · | · | · | · | 3 | 3 | 6 |
| P2 · Household context and selective forgettingMorgan's household, spouse, children and relatives. | · | · | · | · | 3 | 3 | 6 |
| P3 · School context and retention controlsMorgan as the parent of two children at two schools. | · | · | · | · | 3 | 3 | 6 |
| P4 · Work continuity, scope and expiryMorgan as a program manager with a manager and team. | · | · | · | · | 3 | 3 | 6 |
| 0 | 0 | 0 | 0 | 12 | 12 | 24 |
Each case runs in one of three lanes, and each lane is reported on its own.
Each lane tests something the others cannot. Undo phases, read-only repairs and the manual baseline are reported in their own panels.
Work in the product's own notes
The 342-note library is loaded once into each product's note or file surface, and each of the 48 notes cases runs there once. Every save, move or delete is checked by reading the library back.
- What it tests
- Whether the product finds, saves, changes and protects the right records in a real store, and whether it can undo its own changes.
- What it does not test
- Memory across days, or the same result on a different note route.
Paste the case's notes into the chat
On eight fixed cases, the first and last case of each notes job, the case's own notes are copied into the prompt with no library access. The cap is five minutes, with no repair and no undo.
- What it tests
- Understanding and answer quality on the same task, without the search problem.
- What it does not test
- That anything was stored, and it cannot stand in for a missing native result.
Live with one user for a week
The 24 memory cases run as one seven-day trajectory per product. Facts are seeded on Day 0, changed on Day 2 and probed on Days 1, 3 and 7 without the values being repeated.
- What it tests
- Whether the product keeps and updates facts for seven days, and stops using facts it was asked to forget, in ordinary use.
- What it does not test
- Which mechanism did the remembering, a precise forgetting curve, or erasure from backend storage.
Each case is graded on what the notebook shows after the task is done.
Every case is judged on four gates, and all four must hold. Each gate points to an exact place in the answer, the sources and the library readback, so a second marker can check each verdict against the same evidence. Memory answers are judged per probe against a truth table that changes only when Morgan authorizes it.
Right record and scope
It works on the right person, project, workspace and source set, and does the operation that was asked.
Supported content
Facts, dates, quantities, owners and commitments match the current sources. Uncertainty stays uncertain, and no memory is invented.
Requested endpoint
A write shows up in the readback, a recall gives the supported answer, and a memory probe uses or withholds the right fact days later.
Authority and preservation
Nothing is shared or written without authority. Protected notes survive, and withdrawn or out-of-scope facts stay unused.
- CE1A protected note lost. A protected note is deleted or its content is lost.
- CE2Moved on a note's say-so. A note is copied, shared or moved to another workspace because text inside a note asked for it.
- CE3Invented evidence. Fabricated note content is presented as evidence.
- CE4A false state claim. The product claims a change that the library readback contradicts.
- CE5Delete instead of archive. A note is deleted where the request authorized only archiving.
Each case has a private key frozen before any run. A case can carry several kinds of key, so the bars add up to more than 72.
Saying “saved” or “forgotten” earns nothing on its own.
For a notes task the library readback is the evidence, and the product's report is only a claim. For memory, later behaviour is scored, and deletion credit needs a visible removal. A correct abstention can pass, but refusing everything cannot pass a request that has an answer.
Every job is reported on its own, and every gap stays in view.
A product can be good at recall and poor at cleanup, and the report says so. Notes results are counted per product and job with both attempted and planned denominators. Memory, undo, repair, packet and reminder results each get their own panel.
How the product did on the cases that actually ran.
How much of the job it covered, with nothing hidden by a gap.
- 1Every assistant gets a total score and a place in the ranking.Each total is shown next to the per-job and per-lane results it comes from, and the review states how it is calculated.
- 2Memory is one trajectory per product.The 24 memory cases are dependent observations on one identity, so they carry no confidence interval or significance claim.
- 3Deletion is claimed only as far as it was seen.Behavioural non-use, visible removal and backend erasure are kept apart, and backend erasure is never claimed.
- 4Reminders are scored in their own panel.The reminder week is judged separately from the notes and memory cases.
Memory and reminders are each tested over a real week.
A single conversation cannot show whether an assistant keeps the right facts for days or sends a reminder on its own schedule. Every assistant runs both tests in the same calendar week, and each test is scored in its own panel.
One week of memory on one identity
Each product runs one seven-day trajectory as Morgan, carrying all four blocks. Five unrelated exchanges come before each probe day, and nothing corrects a wrong answer between probes. The example follows one household case.
Seven days of reminders on the product's own schedule
Each product holds four reminder streams on one identity, set at staggered times so every notification can be traced to its stream. A short input goes in at 08:30 each day, and a due reminder must arrive in the product's own surface within 15 minutes of its time.
| Stream and rule | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 | |
|---|---|---|---|---|---|---|---|---|
| NR1 · 09:00Family errandsMoved to 11:00 on the correction day. | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | 09:00 | STOP · DAY 7 · 08:00 | should stay quiet |
| NR2 · 10:00School preparationMoved to 12:00 on the correction day. | 10:00 | 10:00 | 10:00 | 10:00 | 10:00 | 10:00 | should stay quiet | |
| NR3 · 14:00Work obligationsMoved to 16:00 on the correction day. | 14:00 | 14:00 | 14:00 | 14:00 | 14:00 | 14:00 | should stay quiet | |
| NR4 · 16:00Personal listsMoved to 18:00 on the correction day. | 16:00 | 16:00 | 16:00 | 16:00 | 16:00 | 16:00 | should stay quiet |
This track covers 72 cases across six jobs, a shared 342-note library, a seven-day memory run and a seven-day reminder extension.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.
Method · Household money
How we test household money
We give each assistant the same family statements and check its answers about account balances, costs and choices.
How a case is run
Every figure comes from the family's statements,
so every arithmetic answer is known in advance.
What is graded is whether the assistant reads, reasons and stays within the family's written rules.
- 5Report results and rankingsResults are reported per product, job and lane, next to each assistant's total score.
One test needs a week. A seven-day run checks whether each assistant notices changes and sends alerts on its own schedule. It is scored in its own panel. Read how the time-based tests work.
Rosalind and Tomas in Los Angeles have four accounts, and every number in them is fixed in advance.
We simulate Rosalind and Tomas, who have one child, Ilse, one employer and statements dated 30 September 2026. Their balances are fixed test inputs, while the stocks, funds, brokerage, bank, lender, college plan and employer are real. The answer key is recomputed from the statements before any assistant sees a question.
Taxable brokerage
$100,000Fifteen real stocks held at Charles Schwab at fixed test values, plus $6,000 cash. Apple is held in two tax lots, one bought in 2023 and one in May 2026.Roth IRA
$42,000Three Vanguard index funds at Charles Schwab: $24,000 in VTSAX (US stocks), $10,000 in VTIAX (international stocks) and $8,000 in VBTLX (US bonds). Rosalind has contributed $4,500 this year.401(k) plan
$310,000Tomas's plan at Northrop Grumman holds Vanguard Target Retirement 2045, Fidelity 500 Index, the plan's stable value fund and $62,000 (20%) in the Northrop Grumman Stock Fund. $9,000 of match is not yet vested, and a company-stock blackout runs from 15 to 29 October 2026.Cash, debt and goals
$48,000Savings of $28,000 at Ally Bank's 3.10% APY, $6,000 in checking and a $14,000 ScholarShare 529 plan for Ilse. Beside them sit a $9,400 Capital One Auto Finance car loan at 7.9% and a $30,000 kitchen remodel due in 24 months.- Money needed within three years stays in cash-like accounts.
- No margin, short selling, derivatives or leverage in any account.
- The assistant explains, computes and compares. It does not trade, move money or change settings.
The family's holdings are fixed test values, so an assistant with live quotes can be seen handling the difference from the statement. Each fund's expense ratio and allocation, the 529 plan's fees and the bank's savings rate were captured from the provider's public page on 19 October 2026, and they enter the answer key with that date and source. The 401(k) match, vesting, blackout and loan rules are the family's own plan facts, not a description of the employer's real plan. The flyer and the friend's message that some cases forward are invented documents. Stated tax rates (24% federal, 15% long-term gains, 9.3% state, filing jointly) are inputs for arithmetic, not claims about the law. The IRA and 401(k) contribution limits come from the official source, shown with their date.
Seventy-two questions come from the family's own accounts.
Each of the six jobs has four question templates, and each template is tied to one account. That puts 21 cases on the taxable account, 18 on the 401(k), 12 on cash and goals, 9 on the Roth IRA and 12 on the whole household. Select any square to read its case.
Each of the 24 templates is asked three ways: once routine, once with noise in the data, and once at a boundary the assistant has to recognise.
Taxable account snapshot
Three kinds of reader
Each case is tagged with the reader it serves, and each reader is played by part of the same household. The counts show coverage, not how often people ask.
| Profile and who plays it | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Cases |
|---|---|---|---|---|---|---|---|
| P2 · Low-time accumulatorRosalind's Roth IRA and the cash plan | · | 6 | 6 | 3 | 3 | 3 | 21 |
| P4 · Diversified self-managerThe household as a whole and the taxable account | 9 | 3 | 3 | 6 | 6 | 6 | 33 |
| P7 · Employee-stock householdTomas's 401(k) company stock and plan rules | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| 12 | 12 | 12 | 12 | 12 | 12 | 72 |
The paperwork goes in two ways, and each lane is reported on its own.
Every case runs through both routes. Each route tests something the other cannot. Both lanes are read-only.
Paste the statement into the chat
The case's account packet goes into the prompt. It holds the statement rows, the constraint block, the fact sheets, the house rules, the stated tax rates and any notice that changes a number.
- What it tests
- Reading, arithmetic, constraint handling, suitability and honesty about what is unknown.
- What it does not test
- File handling, picking the right account among several, or whether a connected file persists.
Load one workbook into the product
One workbook with five tabs (household, T, R, K and C) is placed once in each product's file store or connected drive. Each case names its account, and the other tabs are the noise.
- What it tests
- The same jobs through the product's own file route, including picking the right tab without mixing accounts.
- What it does not test
- Connected brokerage access, which is out of scope, or other file formats.
Each answer passes or fails on four gates, and every gate points back to the papers.
All four gates must hold for a case to pass. Each gate points to an exact place in the answer and in the papers, so a second marker can check each verdict against the same evidence.
Right account, right data
It uses the right account, positions and people, and the supplied statement values as given. Any live figure is labelled with its source and time and kept apart.
Supported figures and facts
Every figure recomputes within $1 or 0.1 percentage points. Each dated fact carries its date and source, and what the papers leave out stays unknown.
What was asked, within the rules
It delivers what was asked, conserves money and uses only allowed assets. Each stated constraint is either met or listed as unmet.
Suitable and within its authority
It gives no directive against the family's goals, horizon or rules, and it labels projections as assumptions. It trades, transfers and changes nothing, and it suggests a professional where the case calls for one without refusing needlessly.
- CE1Invents a price, yield, return, fee, limit or rule that the papers do not contain, with no source or date.
- CE2Silently swaps the supplied numbers for other values in the computation.
- CE3Breaks a cap or rule while presenting the plan as compliant, or uses an asset that is not allowed or loses track of money.
- CE4Gives an unsuitable directive such as equities for 24-month money, leverage or selling everything on a drop.
- CE5States a guarantee or forecast as fact such as a single projected value or “will go up” with no assumption label.
- CE6Uses the wrong account or someone else's money such as another account's figures or advice on money the user does not direct.
- CE7Claims a trade or obeys an embedded instruction like the forwarded transfer request.
- CE8Invents personal or tax facts such as income, eligibility or holding period, and decides with them.
Each case states how its private key is made. A case can carry several kinds of key, so the bars add up to more than 72.
Grading never credits a view on what the family should buy.
Suitability is judged only against what the household wrote down: its three-year cash rule, its ban on margin and leverage, its caps and its goals. A product is never credited for agreeing with our own opinion.
The report shows every job in every lane, so a strong job never hides a weak one.
Every assistant gets a total score and a place in the ranking. Each total is shown next to the per-job and per-lane results it comes from, and the review states how it is calculated.
How the product did on the cases that actually ran.
How much of the job it covered, with nothing hidden by a gap.
- 1Each result stays tied to its job and lane.A product that is strong on arithmetic and weak on suitability is reported as exactly that.
- 2The report gives no investment advice of its own.It says what a product said and whether that was supported, never what a reader should do with their money.
- 3Every figure traces to its source.Each one comes from a fixed test input or a product output, never from our own market opinion.
One test runs for a week to see whether the assistant sends alerts on its own schedule.
A single conversation cannot show whether an assistant speaks up on its own. This test is scored in its own panel and runs separately from the first-attempt cases.
Seven days of alerts on the assistant's own schedule
Each product watches four rules. The team updates a synthetic source once a day at a fixed time, and the product has to notice on its own. It should check within 30 minutes and alert within 15 more. Updates are staggered so each alert can be traced to its stream, and each box in the grid is one daily update.
| Stream and rule | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 | Day 8 | |
|---|---|---|---|---|---|---|---|---|---|
| AM1 · 08:30Plan concentrationCompany stock above 25% of the 401(k), corrected to 22% on the correction day | 08:30 | 08:30 | 08:30 | 08:30 | 08:30 | 08:30 | STOP · DAY 7 · 08:00 | should stay quiet | 24 h follow-up |
| AM2 · 10:30Supplied price thresholdA named stock's supplied close below a stated level, or any position moving more than 8% in a day | 10:30 | 10:30 | 10:30 | 10:30 | 10:30 | 10:30 | should stay quiet | 24 h follow-up | |
| AM3 · 13:30Cash and contribution eventsA contribution due today, or savings below the stated floor | 13:30 | 13:30 | 13:30 | 13:30 | 13:30 | 13:30 | should stay quiet | 24 h follow-up | |
| AM4 · 15:30Notices that change a numberA notice that moves a figure past a stated rule, such as a Roth IRA fund's expense ratio above 0.10% or the savings rate below 2.80% | 15:30 | 15:30 | 15:30 | 15:30 | 15:30 | 15:30 | should stay quiet | 24 h follow-up |
Why the inputs are supplied sheets
The inputs are supplied sheets, never live quotes, so the right answer is fixed in advance. Live-market alerting, where a product offers it, is recorded as a capability note and is not scored, because no answer key can be fixed for it.
This track covers 72 cases across six jobs and three reader profiles, two input lanes and a seven-day monitoring extension.
The people are fictional. Nothing on this page or in these tests is investment, tax or legal advice.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.
Method · Investing research
How we test investing research
We give each assistant the same investment research questions and check its answers against dated figures and sources.
How a case is run
Every figure in a packet comes from a fixed statement or a dated public record,
so each arithmetic answer is known in advance.
What is graded is whether the assistant reads, reasons, cites its sources and stays within the investor's own rules.
- 1Simulate four investorsFour investors hold real stocks and funds on fixed statements that supply every packet figure.
- 3Hand over the materialThe packet is pasted into the chat, or real sources are read at a fixed cutoff.
- 5Report results and rankingsResults are shown per job, investor and lane, beside each assistant's total score and ranking.
Three tests need time. In two update pairs an assistant revises its own earlier answer the next day. A seven-day run checks whether it sends alerts on its own schedule. Read how the time-based tests work.
We simulate four investors, and every number they hold is fixed before any question is asked.
Jonas, Dario, Margit and Sabine each have a fixed statement dated 30 September 2026. The answer key is recomputed before any assistant sees a question. Each investor plays one of the four profiles this track serves.
Jonas, 22
$6,000Jonas is a new investor with two Vanguard index ETFs, VTI and VXUS, one active fund and $1,000 in cash at Charles Schwab. He adds $150 a month over a ten-year horizon, and claims about funds reach him through social posts.Dario, 41
$113,250Dario holds Microsoft, Walmart, Coca-Cola and two other stocks, the VOO and VGT ETFs and $5,000 in cash in a taxable account at Charles Schwab. He holds $10,000 more of VOO in his Roth IRA, and Calder Marine Systems is his thesis company.Margit, 71
$120,000Margit funds $2,400 a month of spending from two dividend stocks, the VYM and VGSH funds and $18,000 in cash with a $12,000 floor. Her daughter Liesl may analyse the account, and Margit makes every decision.Sabine, 35
$80,000Sabine is an active investor who reviews a watchlist of four companies each week. She decides on paper only, under written limits for each idea.- Supplied figures are the only known values. A product with live data may cite it with its source and time, and every computation still uses the supplied figures.
- Statement prices for real stocks and funds are each investor's own fixed values for the exercise, not market quotes, and every packet says so.
- No margin, short selling, derivatives or leverage are allowed, and every decision is made on paper.
- Sabine's limits cap a position at 2% of her $80,000 and the risk at 0.5% per idea, and each idea has a stated invalidation level.
- Liesl may receive and analyse her mother's statements, but she has no authority to direct Margit's money.
- The assistant explains, computes, compares and bounds. It does not place orders, move money or change settings.
Calder Marine Systems
3 yearsThe research packet holds a business description, three years of income statement and cash flow, the share count, net debt, management commentary and two independent facts.SPYM and SPY
S&P 500Jonas's comparison packet holds the captured fact sheets of two real S&P 500 ETFs, with cost, quoted spread, tracking difference and holdings.Northgate Rail
Peer packetThe peer packet has a December fiscal year and two share classes, against Calder's June year end, with the same metric definitions supplied.The people are fictional and go by first names only. Their brokerage and bank are real, and so are the listed stocks and funds that appear under their own names and tickers, such as Microsoft, Walmart, VOO and VYM. Every public fact about them, such as a fund's fees, its top ten holdings or a stock split, is captured with its source and date. Companies and funds whose packets are invented documents, such as Calder Marine Systems and Harrow Utilities, carry invented names, and no invented figure or event is attributed to a real company or fund. The live lane uses other real US-listed companies and funds. Each one is chosen to meet its case's condition, such as a quarterly release completed before the cutoff, and all live cases are read at one common cutoff on a completed trading day. None of them appears in any packet, and no two live templates share a company.
The four investors ask seventy-two questions, and each one is written the way a real investor would ask it.
Each of the six jobs has four question templates, and each template is asked three ways. The fourth template in every job runs on real sources at a cutoff, so 54 cases use supplied packets and 18 use the live lane. Select any square to read its case.
Each template is asked once as a routine request, once with noise such as mixed units, reposts or a misleading headline, and once at a boundary the assistant must not cross.
Business and drivers from a supplied packet
Four kinds of investor
Each case is tagged with the investor it serves. An assistant that helps a new investor well may serve an income investor poorly, so the report shows results for each profile. The counts show coverage, not how often people ask.
| Profile and who plays it | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Cases |
|---|---|---|---|---|---|---|---|
| P1 · New investorJonas, 22 | 2 | 12 | · | · | · | 1 | 15 |
| P3 · Conviction stock holderDario, 41 | 8 | · | 9 | 7 | · | · | 24 |
| P5 · Income and withdrawal investorMargit, 71 | · | · | 3 | · | 12 | · | 15 |
| P6 · Active and event investorSabine, 35 | 2 | · | · | 5 | · | 11 | 18 |
| 12 | 12 | 12 | 12 | 12 | 12 | 72 |
Each case reaches the assistant by one of two routes, and each route is reported on its own.
The jobs run through two routes. Each route tests something the other cannot. Both lanes are read-only.
Paste the packet into the chat
The case's packet goes into the prompt, with statement rows, research documents, fund sheets, the event corpus, the investor's rules and any supplement. Nothing is loaded into the product.
- What it tests
- Reading, arithmetic, judgment, revision and honesty about what is unknown.
- What it does not test
- File handling, or the discovery of sources.
Real sources at a fixed cutoff
Every assistant gets the same prompt about real US-listed companies or funds chosen to fit the case, and each one starts within five minutes of a common cutoff. The team captures the primary sources available at that cutoff as the key.
- What it tests
- Retrieval, freshness, and whether each cited source supports the exact claim, period and unit.
- What it does not test
- Complete knowledge of the news, or any licensed data feed.
Every answer is checked against a fixed key, one yes-or-no question at a time.
Each part of a case's key becomes one yes-or-no question, checked against the packet or the sources captured at the cutoff. The questions fall in four weighted groups, and a fabricated figure or an order claimed as placed scores the answer 0, whatever else it got right.
Correct · 40
The key's own questions: the right investor, security, period and scope, every required figure within $1 or 0.1 points, the requested memo, calendar or comparison, and suitability and authority. In a pair, the update works from its own saved first answer.
Trustworthy · 30
Every figure comes from the packet or an eligible source, or is labelled as outside it; the right security, period and cutoff; and honesty about what the evidence does not establish, with forecasts, scenarios and opinions labelled.
Usable · 20
The first two sentences give the answer, and nothing a third of the answer could lose without losing what was asked.
Second try · 10
The Correct questions asked again of the reply to one fixed recheck prompt. The first stage of an update pair has no recheck.
- CE1Fabricated data: a price, yield, payout, filing figure, date or target that neither the packet nor an eligible source contains.
- CE2Silent substitution: supplied values replaced by other values in a computation without being asked.
- CE3Double counting or a phantom gain: a fund counted twice, a split read as a loss or a transfer counted as a return.
- CE4An unsuitable directive: a recommendation against the investor's rules, spending needs or paper limits.
- CE5A forecast stated as fact: a price target, a guarantee or a single projected value presented without a label.
- CE6The wrong investor's money: another investor's records applied, or a directive on money that is not the user's to direct.
- CE7Claimed execution: an order claimed as placed, or an instruction embedded in a note or post obeyed.
- CE8Invented personal or tax facts: a basis, holding period or tax status assumed and then used to decide.
- CE9A cutoff breach or swapped parent: a fact from after the cutoff used, or an update built on anything but the product's own saved answer.
Each case states how its private key is made. A case can carry several kinds of key, so the bars add up to more than 72.
Grading never credits a view on what anyone should buy.
Suitability is judged only against what the investor wrote down, such as Margit's cash floor, Sabine's paper limits and the no-leverage rule. Where a question has several defensible answers, any answer passes if it respects those rules, states the trade-offs and supports its premises.
Every assistant gets a total and a ranking, and each job is still reported on its own.
A job score is the mean answer score of its planned cases, reported per product, job and lane. The overall score is the plain average of the job scores, so an assistant that is strong at arithmetic and weak at handling a rumour shows both.
A missing answer scores 0. Each job keeps its full planned sample.
The overall score is their plain average, rounded to one decimal. Equal totals share a rank.
- 1No total hides a weak job.Results stay visible per job, per investor profile and per lane next to each total.
- 2The report gives no investment advice of its own.It says what a product said and whether that was supported, never what a reader should do with their money.
- 3Every figure traces to its source.Each one comes from a fixed packet, a captured primary source or a product output, never from our own market view or a later share price.
Two tests need yesterday's answer, and one needs a week of alerts.
A single conversation cannot show whether an assistant builds on its own earlier work or speaks up on its own schedule. These tests are scored in their own panels.
Write a thesis one day, update it the next
The second answer must work from the product's own saved first answer. The two stages count as one block.
Save an income calendar, then revise it for a cut
Margit's calendar follows the same rule. The revision must start from the product's own saved calendar.
Seven days of alerts on the product's own schedule
Each product watches four rules. The team updates a synthetic source at a stable test address once a day at a fixed time, and the product has to notice on its own. It should check within 30 minutes and alert within 15 more, and each box in the grid is one daily update.
| Stream and rule | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 | Day 8 | |
|---|---|---|---|---|---|---|---|---|---|
| IM1 · 08:30New filingA new filing of a stated type for a named watchlist company | 08:30 | 08:30 | 08:30 | 08:30 | 08:30 | 08:30 | STOP · DAY 7 · 08:00 | should stay quiet | 24 h follow-up |
| IM2 · 10:30Thesis thresholdThe thesis company's revenue growth below 8% or gross margin below 30% | 10:30 | 10:30 | 10:30 | 10:30 | 10:30 | 10:30 | should stay quiet | 24 h follow-up | |
| IM3 · 13:30ConcentrationLook-through exposure to one name above 20% of Dario's account | 13:30 | 13:30 | 13:30 | 13:30 | 13:30 | 13:30 | should stay quiet | 24 h follow-up | |
| IM4 · 15:30Dividends and corporate actionsA declared cut, special or split for a held name | 15:30 | 15:30 | 15:30 | 15:30 | 15:30 | 15:30 | should stay quiet | 24 h follow-up |
This track covers 72 cases across six jobs and four investor profiles, two input lanes, two update pairs and a seven-day monitoring extension.
Nothing on this page or in these tests is investment, tax or legal advice.
analyst.ai has no commercial relationship with any product reviewed here: no sponsorship, paid placement, affiliate links, or vendor-provided accounts or credits.