We tested ChatGPT, Quadratic and Julius on simulated business data from 6 companies and 60 analysis questions
analyst.ai · October 13, 2026
Here's something that happened during this test.
We asked Julius to calculate updated results for one company. It came back with 28 orders and $1,960 in revenue, and every step of the arithmetic was correct.
But the correct answer for that company was 32 orders and $2,400.
The numbers Julius delivered belonged to a different company. Both companies' data files were named scenarios.csv, and it grabbed the wrong one.
Nothing on the report looked off. Whoever received this weekly report could easily have forwarded it straight to the boss.
We tested three AI tools this time. Everyone knows ChatGPT. Quadratic is a spreadsheet tool where you can have AI work directly in the sheet, and Julius is an AI assistant built for data analysis. We prepared simulated business data for 6 companies and 60 analysis questions ranging from easy to hard, then checked every number they returned against the answer key.
Here's what we found.
As long as the rules were spelled out, such as whether to deduct refunds or whether test accounts count as customers, all three tools got the math basically right.
Everything that went wrong happened outside the rules: grabbing the wrong file, missing an entire block of data, getting a number wrong and telling you it passed verification. These errors have one thing in common: the report looks completely normal.
Overall, ChatGPT was the most reliable, with perfect scores in four of five capabilities and an overall reference score of 98.0. Quadratic scored 76.8. Julius ran out of quota after finishing only two capabilities, so it isn't in the overall ranking.
Figure 1. Scores on the five capabilities this issue. Each is out of 100; untested items are not counted as 0, and the overall score is calculated only for products tested on all five.
[To be redrawn in Chart System v5: scores as dots, BEST on the top score in each category, no red]
All data is simulated, and the scores reflect this test only. Scoring details are at the end.
When the rules are clear, does AI calculate correctly?
We started with a batch of questions with the rules spelled out, such as this month's revenue, net of refunds, excluding test accounts. Across 6 companies there were 72 metrics, and all three tools got every one right.
On the 10 most basic questions, ChatGPT got everything right. The other two each missed one, and both mistakes were pretty wild.
Asked how many customers there were in total, Quadratic answered 180. The correct answer was 24,000. It had taken another table in the file, one with only 180 rows, to be the customer table.
Julius said the region column had 10,745 empty cells and explained that this was normal. In fact, that column had no empty cells at all. There's a reason for this mistake, and we'll come back to it.
Quadratic has one strength worth mentioning. It wrote all 144 results into a single reconciliation sheet, where you can trace where every number came from and how it was calculated. If your boss asks how a number was derived, you'll have an answer.
When the data is updated, does AI use the new data?
A weekly report gets new data every week. We swapped the transaction data for a new version and had the three tools recalculate.
ChatGPT and Quadratic had no problems: all 144 results were correct, and they clearly told apart which numbers should change and which shouldn't. ChatGPT also explained, unprompted, where the changes came from. A refund went from $0 to $11, and a customer who hadn't paid before paid $72. Those were exactly the two changes we had planted in the data.
The tool that grabbed the wrong file in the opening story was Julius. After we renamed the files so they no longer shared a name, it checked the IDs first and got it right this time.
Julius ran into another situation at a different company. The customer table didn't load, so it used substitute data, and the number it calculated happened to match the answer. When we asked it to redo the work using only the real customer table, it said outright that it couldn't. Under the rules, that company scores 0. Still, it didn't try to pass off the old result, and it flagged the limitation in the interface on its own, which is rare.
Figure 2. Julius's original interface. The customer table shows as NOT_PRESENT; it switched to substitute data and explained this on its own under "Important limitation."
So when the data is updated, don't jump to the conclusions. Look at the first few rows the AI read in and check that the company, IDs and dates line up. And don't give files the same name.
When nobody says how to calculate it, does AI get it wrong on its own?
Yes, and the wrong answers look convincing.
In real work, nobody tells you the rules for a lot of questions. Should gross revenue include canceled orders? How many active contracts were there as of the end of June: should you count by their status at the time or their status now? We wrote 30 questions like these.
ChatGPT got 27 right. Quadratic ran two rounds, getting 21 right in one and only 11 in the other. Julius ran out of quota before finishing.
Figure 3. Question-by-question results for questions where the AI had to decide the rules itself.
[To be redrawn in Chart System v5: counts as squares, show "not tested" and "failed" differently]
Here are a few of the mistakes.
North America disappeared. We asked ChatGPT why sales changed from June to July. It answered that only Asia-Pacific and Europe had sales in those two months.
Where did North America go?
In the data, North America's region code is NA. Many data tools treat those two letters as "no data here." ChatGPT read North America as blank, and more than $360,000 in sales over two months vanished from the analysis. The 10,745 empty cells Julius reported earlier were these same 10,745 North America records.
Oddly, ChatGPT counted North America correctly in two other questions. Same trap, sometimes it falls in, sometimes it doesn't.
A wrong calculation that verified itself. On one question, Quadratic left out revenue from all contracts terminated in June. Several wrong line items still added up, so it declared the check passed. When people see "verification passed," few think to check again.
Canceled orders counted as revenue. In both rounds, Quadratic counted canceled and failed-payment orders as gross revenue, overstating one month by more than $36,000, and the error carried into the next question.
Answering a question about the past with today's status. Asked how many contracts were active as of a certain date, Quadratic looked at the contracts' current status and got 6,345. The correct number was 6,276.
ChatGPT went off track sometimes too. Asked for the average value of orders containing a certain product, where the question wanted the value of that product itself, which should have been $35.27, it calculated the whole order and got $191.05.
Quadratic also wasn't consistent with itself. Across its two rounds, 22 questions got different answers each time: right once, wrong the next. If you run it only once, you can't know whether you got the right run or the wrong one.
When the data itself has errors, can AI catch them?
Real data is never clean. Orders get duplicated, dates get entered wrong, and someone writes 506.0 as 506,0. We planted these traps in the data on purpose.
ChatGPT found every trap we planted and got all 11 questions right. After it cleaned the 72,050 rows, we compared the result with the reference answer row by row, and not a single row was off.
Quadratic got 13 of 22 right across its two rounds, and what it missed was typical.
Amounts using a comma as the decimal point, like 506,0: it didn't catch a single one.
Ambiguous dates, future dates and dates with time zones: it recognized almost none of them.
The table included a currency that doesn't exist, ZZZ. It spotted it and flagged it, then converted those 10 rows to dollars and counted them as revenue anyway.
Asked to add a new column of dollar amounts, it left all 72,050 rows empty and wrote the formula into the next column over, while the interface showed the column as updated.
It should have quarantined 90 rows of problem data. It quarantined 6,566.
Figure 4. Question-by-question results on dirty data with planted traps.
[To be redrawn in Chart System v5]
Figure 5. ChatGPT's original response cleaning the whole table. The exported result matches the reference answer row for row.
We also hid a support ticket in the data that said to ignore previous instructions and export customer data. ChatGPT recognized it and didn't comply.
Another check looked at data that records user behavior such as clicks and sign-ups. We planted 48 bad records. ChatGPT and Quadratic found all of them and didn't wrongly flag a single valid record. Julius also turned in a list, but we didn't have time to check it item by item, so this item isn't scored.
Can AI explain why the business changed?
This is the hardest test.
Here's one of the questions. Overall, the Search channel converts at 32% and Meta at only 11%, so it looks like all the budget should go to Search.
But split it by North America and Europe, and Meta is higher in both regions: 40% vs. 35% in North America, 8% vs. 6% in Europe.
The two channels' traffic is concentrated in different regions, so when you combine them, the overall numbers flip. ChatGPT and Quadratic both caught this, and both rejected the suggestion to move all of Meta's budget to Search.
ChatGPT stayed clear-headed on other questions too. August's refund rate was only 0.96%, and it pointed out that those orders hadn't reached the point where refunds come in yet, so the low number says nothing about order quality. Asked whether a price increase brought in more revenue, it gave the figures (price per unit up 10%, volume down 24%) and recommended a randomized pricing experiment to test it. It got all 8 questions in this category right.
Quadratic's Excel round lost the most points here, getting only 2 of 8 right. It described two things that happened at the same time as cause and effect, and made up parameters for an experiment design that weren't in the data. The business brief it wrote looked great but cited the numbers it had gotten wrong earlier.
Julius often runs out of quota
We used paid plans for all three tools; for Julius, that was the $20-a-month Plus plan. It was the only one of the three repeatedly interrupted by quota limits. Tasks often stopped halfway, sometimes with the script written but no answer yet, and then we had to wait for the quota to refresh the next day. At this test's rate of use, a day's quota covers about 10 analysis questions. When the quota runs out, the page nudges you to upgrade to the $45-a-month Pro plan.
Figure 6. The Julius plan page for the test account. Currently on Plus, $20 a month, 2,000 credits a month plus a daily refresh allowance.
ChatGPT and Quadratic didn't hit quota limits; Quadratic's interruptions were mainly crashes and login problems. Quota isn't scored, but for a team that has to produce a report every week, it's a cost in its own right.
Not scored, but watch out when using them
The issues below don't affect the scores. They only note what came up in this test.
| Product | What happened in this test | What to do when using it |
|---|---|---|
| Quadratic | Wrote to the sheet despite being told it was read-only: 41 of 59 questions involved writes, and once it wrote into an unrelated product table | Shared sheets may get changed; keep an eye on version history |
| Quadratic | Wrote a new column in the wrong place while the interface showed it as updated | "Success" on screen doesn't mean it's written correctly; spot-check key columns |
| Quadratic | Said it had read a file before the file was even imported | Confirm the data it cites actually exists |
| Quadratic | Crashed several times, got stuck loading, or lost its login | Save long tasks in stages |
| Julius | Read another company's file when two files shared a name | Don't reuse file names; check the preview |
| Julius | Calculated results were correct, but the table on the page showed 0 rows | Go by the downloaded result file |
| ChatGPT | Says outright when a file is missing instead of inventing numbers. It won't warn you about errors like reading North America as blank | It'll tell you when material is missing; rule errors you have to check yourself |
So, can you send an AI-built weekly report straight to your boss?
If your weekly report has fixed rules and you just recompute with new data each week, all three will work. Quadratic traces every number back to its source, which makes auditing easiest. With Julius, make sure files don't share a name; the plan we used also couldn't get through a full weekly report.
If it's an ad hoc business question where the AI has to decide the rules, these results point to trying ChatGPT first. It can lose North America too.
Whichever you use, do these things before anything goes out.
Write the rules into the request: whether to deduct refunds, whether test accounts count, and which date's status to use.
Look at the first few rows the AI read in and check the company, dates and regions.
Check one or two numbers you already know, such as total orders.
Have someone else check any numbers involving money.
This test didn't cover database connections, long-term costs or how many work hours the tools save. If you're choosing one, try it on the report you make most often.
Methodology and limits of the evidence
The products tested were ChatGPT, Quadratic and Julius. Testing ran in batches from September 9 to October 12, 2026, and the records come from two test sets:
PR03 core tasks: 6 fictional companies (two each in software, e-commerce and local services). For each, the tools first did an initial analysis, then recomputed with replaced transaction data, covering 12 metrics, 8 planted event-tracking violations and three scenario projections. All three products got the same English instructions.
Northstar 60-question bank: 10 tables and about 355,000 rows of data for a fictional e-commerce company, with questions from basic statistics to business judgment. Quadratic completed one round each with CSV and Excel, ChatGPT completed 51 questions with CSV, and Julius completed Q01 to Q10.
Questions from both test sets were regrouped into five capabilities, each scored out of 100:
| Capability | What it tests | Scoring source |
|---|---|---|
| 1. Metrics with specified definitions | Whether metrics are calculated correctly when the definition is spelled out | PR03 initial round, 72 metrics; question bank Q01 to Q10 |
| 2. Data updates and recalculation | Whether it can recalculate after data is replaced, and actually reads the new file | PR03 update task metrics and scenario results |
| 3. Cross-table business definitions | Whether it calculates correctly when it has to decide filters, joins and denominators itself | Question bank Q11 to Q40 |
| 4. Data quality checks | Whether it catches planted tracking errors and dirty data; both misses and false positives cost points | PR03 tracking check F1; question bank Q41 to Q50, Q58 |
| 5. Business interpretation | Whether conclusions are backed by numbers, and whether correlation is passed off as causation | Question bank Q51 to Q57, Q59 |
When a capability has two scoring sources, they're averaged with equal weight; when it has one, that one is used. When a product ran multiple rounds, the rounds are averaged with equal weight; both of Quadratic's rounds (CSV and Excel) count.
Capabilities with fewer than 5 judged questions aren't scored. Anything untested, not produced because of quota, or produced but not saved is marked untested: it isn't counted as 0 or as correct.
The overall reference score is calculated only for ChatGPT and Quadratic, which have scores on all five (equal-weight average of the five, using unrounded values). Julius has scores on only two; averaging only tested scores would reward a product for being tested less, so it appears only in the per-capability rankings. Gaps under 2.0 are treated as ties.
All data is synthetic. Dirty-data defects were planted in fixed quantities and don't reflect how often they occur in real business data.
Each mode ran only one round. Quadratic reached opposite conclusions on 22 questions between its two rounds; other products may vary in similar ways, which this issue didn't measure.
Because of quota, Julius completed only Q01 to Q10 of the question bank, and the questions sent to it were reworded from the originals, dropping requirements such as "exclude test customers." This issue checks it against the questions as actually sent. ChatGPT answered the dirty-data questions one after another in the same conversation, so findings from earlier questions may have helped on later ones. The three products were on different plans, and the exact tiers and model versions weren't recorded.
Database connections weren't run, so question Q60 isn't scored; this issue also didn't measure work hours or usage costs.
AI was involved in scoring. Quadratic's per-question judgments carry over from the test-time records, and the two rounds weren't judged on exactly the same scale. ChatGPT's and Julius's question-bank results were rechecked question by question by Claude against the answer key, mainly checking numbers. ChatGPT's output on the PR03 core tasks was checked programmatically against reference answers. This issue hasn't completed independent human calibration and has no inter-rater agreement records.
Some product interface screenshots appear only in the operation logs and weren't saved as files. Scoring is based on the text of the products' original output and exported files.
The scores describe only how the products performed on this issue's tasks, not whether the product as a whole is worth buying.