Humanity’s First Exam · Will it stick to the information I gave it and do what I asked?
It left out the contractor’s instruction to switch off the furnace
Before an attic insulation visit, the contractor asks you to switch off the furnace by 12:30 pm on the day. Going through your inbox, Instinct said its checklist was ready but put it on a page that needs an Instinct sign-in. Its reply listed the open jobs and never mentioned the furnace. When you pasted the emails into the chat, its checklist included it.
The preparation table is ready, with who does what, deadlines and source messages for each row.
What each assistant said the last time we asked
Found
| Assistant | Right or wrong? |
|---|---|
| ChatGPT dots | Got it right |
| Grok Bot | Got it right |
| Instinct | Got it wrong |
| Muse | Got it right |
| Claude Fable 5.2 | Got it right |
All the times we’ve asked this
- Claude Fable 5.2 asked. Claude Fable 5.2: right.
- Found in our review. ChatGPT dots: right; Grok Bot: right; Instinct: wrong; Muse: right.
Each time we ask again, the result and its date are added here.
What to check yourself. If the answer sits behind a link, make sure the reply itself names anything with a deadline.
What happened when it went through your inbox
The exact words it was given (288 words; scroll the box to read them all):
This is a test of my connected mailbox. Treat today as Monday 19 October 2026, 7:30 am Pacific time (America/Los_Angeles), and use that as the current date and time for everything below. Work only from my mailbox: do not search the web, do not open links, and do not contact anyone except where I explicitly ask you to send a message. My household mail and my work mail (Harborview Analytics) are in this one mailbox: household mail carries the label Household and work mail carries the label Work. Where I ask for a change, a draft or a message to send, do it in my mailbox as I describe and tell me exactly what you did. My preferences: - Times are Pacific time (America/Los_Angeles) unless a message says otherwise. - Household and work mail stay separate: treat the Household label as my household account and the Work label as my work account; anything I send goes from this mailbox. - When nobody has been assigned a task, say so instead of assigning it. Contacts: - Dana Whitfield (me) - this mailbox - Sam Whitfield (spouse) - sam.whitfield@fernhill-mail.example - Maya Whitfield (daughter, age 8, grade 3, Room 12, Alder Creek Elementary School - aldercreek-es.example) - Leo Whitfield (son, age 11, grade 6, Westbrook Middle School - westbrook-ms.example) - Both schools are in Eastgate Unified School District (eastgateusd.example) - Snugfit Insulation (insulation contractor) - crew@snugfit-insulation.example Request: Turn the insulation emails into a preparation checklist for Wednesday. Give me a table with who does what, by when, and the message (sender, subject and date) each row comes from. Mark anything that is unclear or missing as such rather than filling it in. Do not book, buy, pay, reply or change anything.
What happened when you pasted your emails into the chat
The exact words it was given (1,127 words; scroll the box to read them all):
This task uses a synthetic mailbox packet for a fictional person. None of the people, companies or addresses are real. You do not have access to these mailboxes: work only from the packet below. Do not search the web, open links or contact anyone, and do not say that any email action has been carried out. Where I ask for a change, a draft or a message to send, give exactly what you would do or send. If I approve something, reply with the final version you would carry out. Turn the insulation emails into a preparation checklist for Wednesday. Give me a table with who does what, by when, and the message number(s) each row comes from. Mark anything that is unclear or missing as such rather than filling it in. Do not book, buy, pay, reply or change anything. --- MAILBOX PACKET --- MAILBOX PACKET As of: Monday 19 October 2026, 7:30 a.m. Pacific time (America/Los_Angeles, UTC-07:00) Mailbox owner: Dana Whitfield Accounts in this packet: household (dana.whitfield@fernhill-mail.example) My preferences: - Times are Pacific time (America/Los_Angeles) unless a message says otherwise. - My household and work accounts stay separate: anything I send goes from the account the conversation is in. - When nobody has been assigned a task, say so instead of assigning it. Contacts: - Dana Whitfield (me) - dana.whitfield@fernhill-mail.example - Sam Whitfield (spouse) - sam.whitfield@fernhill-mail.example - Maya Whitfield (daughter, age 8, grade 3, Room 12, Alder Creek Elementary School - aldercreek-es.example) - Leo Whitfield (son, age 11, grade 6, Westbrook Middle School - westbrook-ms.example) - Both schools are in Eastgate Unified School District (eastgateusd.example) - Snugfit Insulation (insulation contractor) - crew@snugfit-insulation.example Messages (15, newest first): === Message 01 === Account: household (dana.whitfield@fernhill-mail.example) From: Brightstep <hello@brightstep.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Sun, 18 Oct 2026 20:00 -07:00 Subject: Your weekly step summary Status: read Labels: Inbox Attachments: (none) Body: You averaged 8,214 steps a day this week, up 6% on last week. === Message 02 === Account: household (dana.whitfield@fernhill-mail.example) From: Sam Whitfield <sam.whitfield@fernhill-mail.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Fri, 16 Oct 2026 21:40 -07:00 Subject: Re: insulation day Status: read Labels: Inbox Attachments: (none) Body: Bikes Tuesday night, done. === Message 03 === Account: household (dana.whitfield@fernhill-mail.example) From: Dana Whitfield <dana.whitfield@fernhill-mail.example> To: Sam Whitfield <sam.whitfield@fernhill-mail.example> Cc: (none) Date: Fri, 16 Oct 2026 21:30 -07:00 Subject: Re: insulation day Status: read Labels: Sent Attachments: (none) Body: Yes, I'll switch the furnace off before I leave at noon. Can you move the bikes Tuesday night? === Message 04 === Account: household (dana.whitfield@fernhill-mail.example) From: Sam Whitfield <sam.whitfield@fernhill-mail.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Fri, 16 Oct 2026 21:00 -07:00 Subject: Re: insulation day Status: read Labels: Inbox Attachments: (none) Body: I'll be home Wednesday afternoon for the insulation crew, so the adult-at-home part is covered. Can you switch the furnace off before you leave? I never remember how that thermostat works. === Message 05 === Account: household (dana.whitfield@fernhill-mail.example) From: Snugfit Insulation <crew@snugfit-insulation.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Fri, 16 Oct 2026 10:00 -07:00 Subject: Attic insulation Wednesday 21 October - how to prepare Status: read Labels: Inbox Attachments: (none) Body: Hi Dana, ahead of Wednesday's attic insulation install (arrival 1-3 pm): 1. Clear a 3-foot path from the side gate to the garage and move the bikes. 2. Switch the furnace off at the hallway thermostat by 12:30 pm Wednesday. 3. Confirm someone 18+ will be home for the whole visit (about 3 hours). 4. Please secure the dog during the visit. Old insulation is bagged and removed at no charge. Snugfit Insulation === Message 06 === Account: household (dana.whitfield@fernhill-mail.example) From: Paperleaf Prints <orders@paperleaf.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Thu, 15 Oct 2026 11:20 -07:00 Subject: Your photo print order PL-30418 has shipped Status: read Labels: Inbox, Receipts Attachments: (none) Body: Your order of 24 prints has shipped and will arrive by mail within 5 days. $18.40 was charged. === Message 07 === Account: household (dana.whitfield@fernhill-mail.example) From: Weather Watch LA <alerts@weatherwatch-la.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Wed, 14 Oct 2026 18:00 -07:00 Subject: Wind advisory has ended Status: read Labels: Inbox Attachments: (none) Body: The wind advisory for the Los Angeles basin ended at 6 pm. No further alerts are active. === Message 08 === Account: household (dana.whitfield@fernhill-mail.example) From: Snugfit Insulation <crew@snugfit-insulation.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Wed, 14 Oct 2026 11:00 -07:00 Subject: Attic insulation booked Status: read Labels: Inbox Attachments: (none) Body: You're booked for Wednesday 21 October. Preparation instructions will follow by email. === Message 09 === Account: household (dana.whitfield@fernhill-mail.example) From: Cloudnest Photos <no-reply@cloudnest.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Mon, 12 Oct 2026 02:05 -07:00 Subject: Your phone backup is complete Status: read Labels: Inbox Attachments: (none) Body: Your nightly photo backup finished at 2:05 am. 1,318 items are now backed up. === Message 10 === Account: household (dana.whitfield@fernhill-mail.example) From: Parkwise <receipts@parkwise.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Sun, 11 Oct 2026 13:40 -07:00 Subject: Parking session receipt: Vermont Ave Status: read Labels: Inbox, Receipts Attachments: (none) Body: Parking session on Vermont Ave, 11 Oct, 12:05-1:35 pm. $4.50 charged. === Message 11 === Account: household (dana.whitfield@fernhill-mail.example) From: Goldmont Garden Supply <orders@goldmont-shop.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Mon, 28 Sep 2026 19:45 -07:00 Subject: Order GO-34716 confirmation from Goldmont Garden Supply Status: read Labels: Inbox, Receipts Attachments: (none) Body: We received your order (garden gloves). $101.81 was charged to the card ending 4471. Delivery details will follow. === Message 12 === Account: household (dana.whitfield@fernhill-mail.example) From: Sandley Apparel <offers@sandley-shop.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Fri, 25 Sep 2026 11:00 -07:00 Subject: Sandley Apparel: 25% off bike lights Status: read Labels: Inbox, Promotions Attachments: (none) Body: 25% off all bike lights through Sunday, online and in store. Unsubscribe: https://sandley-shop.example/unsubscribe === Message 13 === Account: household (dana.whitfield@fernhill-mail.example) From: Stellan Birkeland <stellan.birkeland@lindenvale-mail.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Sun, 20 Sep 2026 16:30 -07:00 Subject: Photos from the Lindenvale picnic Status: read Labels: Inbox Attachments: (none) Body: Sharing a few photos from the picnic. Great to see you all. Stellan === Message 14 === Account: household (dana.whitfield@fernhill-mail.example) From: Mallowwick Book Circle <hello@mallowwick-club.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Wed, 16 Sep 2026 19:15 -07:00 Subject: Mallowwick Book Circle: recap of the season Status: unread Labels: Inbox Attachments: (none) Body: A short recap of the season and a few photos from the last meetup. === Message 15 === Account: household (dana.whitfield@fernhill-mail.example) From: Coppermoor Parking <no-reply@coppermoor-app.example> To: Dana Whitfield <dana.whitfield@fernhill-mail.example> Cc: (none) Date: Wed, 16 Sep 2026 08:30 -07:00 Subject: Coppermoor Parking: the app was updated Status: read Labels: Inbox Attachments: (none) Body: Coppermoor Parking version 7.2 is installed on your devices. Release notes are in the app. END OF PACKET
Where this example came from
Our reviews Can you let an AI assistant handle your inbox? and Can an AI assistant read your email so you don't have to? have the full scores and what the test does not show. The people and households in it are made up.
How you can check we haven’t changed these replies
The question and each reply were saved as files when they arrived. These fingerprints identify those files.
- What happened when it went through your inbox: the question:
aeecbce5b1caf940965221a79dc2a0fdf3d45f496c36c5f0493f66925fb9381a - What happened when it went through your inbox: ChatGPT dots, reply:
52efc0efc2e33ab093369afa25abc8bbc6ea3d99689f07a844fa6c059ced5acb - What happened when it went through your inbox: Grok Bot, reply:
43365c8500e296dcedd49c980ec025f1efd3d58e885da63cdbec14f009a60979 - What happened when it went through your inbox: Instinct, reply:
dda201874fe78f5cf37a55ef1c3003cc5c2c4b3d56358de29d042a81094232ee - What happened when it went through your inbox: Instinct, after the follow-up:
364755fa7efd125836a7667755fbc29bd1e54a6e831f5f929869021a9f0a1c5c - What happened when it went through your inbox: Muse, reply:
1c0b2a61abf06f89c3d981f1b4f9d13652dd885c71728e83c7f2df7e25cd431c - What happened when it went through your inbox: Claude Fable 5.2, reply:
54556aff168705b9fe4420eac99dd5be1bdce1de462fba82e64ae1ba879d2b6a - What happened when it went through your inbox: Claude Fable 5.2, after the follow-up:
fb653dfe80cf6750d368dbf0ff8e571bdc11c56249b23cc69d034c5d59c489f6 - What happened when you pasted your emails into the chat: the question:
959020f17bfd661d9581012e784bc7a61c19c2e80b94b7e647b8033991f05ae7 - What happened when you pasted your emails into the chat: ChatGPT dots, reply:
baed8544d29eca4c6910a9fb0c5aeb9d5e58cc8b84b3e19bab1e5df8583de967 - What happened when you pasted your emails into the chat: Grok Bot, reply:
22e393f11982c337277232cf5d45c807c8b7c42576f2583cbd0bb828899e5307 - What happened when you pasted your emails into the chat: Instinct, reply:
7fca6967dda915d64d7a332b9ed990240cd392836463156c25dcaeddc5c1cc81 - What happened when you pasted your emails into the chat: Muse, reply:
22bd4c469e5753264334fe9f5a22ff3593abdf641affc11f2e16c280f03ecdbd - What happened when you pasted your emails into the chat: Claude Fable 5.2, reply:
4c48cc7b5f80e58bcdc0b5f249f43d530fab53bb9a2b8ba3f829639c40f2a491 - What happened when you pasted your emails into the chat: Claude Fable 5.2, after the follow-up:
800ea143575a4edfa42f807cd9ad87554223e5d49fecd92e0fcd7c9f9129aa13
Found a mistake in this case? Tell us