# AI hydration audit: method, frozen before collection Written 2026-10-14, before any real answers were collected. Everything that decides a headline is fixed here so the method can't be bent to fit the results. If something has to change after the first real run, add a dated entry under "Changes" saying what and why; never edit the rules silently. ## Question How accurate are the hydration answers that AI assistants give the public, measured against the guidance of the National Academies, CDC, FDA and NIH? ## What is asked The 15 questions in `questions.json`, word for word, in US English, with no system prompt and no follow-ups. Four (`women: true`) form the women's-health subset. Before the run, question wording may be swapped for a phrasing the AI keyword data shows people use more, as long as it asks the same thing and the key still applies. ## Platforms | Platform | How | Why | | --- | --- | --- | | ChatGPT | DataForSEO LLM Scraper (consumer app, US) | what users see, incl. web sources | | Gemini | DataForSEO LLM Scraper (consumer app, US) | as above | | Claude | DataForSEO LLM Responses (API, latest default model, web search on) | no consumer scraper | | Perplexity | DataForSEO LLM Responses (API, web search on) | as above | | Google AI Overview | DataForSEO SERP, Google organic, US desktop | the AI answer most people actually see | Each question is asked **5 times** per platform (fresh session each time). Google AI Overviews are deterministic per query and location, so they're captured once per question and counted as one answer, not five. A question with no AI Overview shown is recorded as "not shown", not as wrong. Collection date and model versions are recorded with every answer. ## Scoring Each answer is scored against its question's `key`, `correct_if` and `partly_if`: - **correct**: meets `correct_if`, and contains no factual error on the question asked - **partly correct**: meets `partly_if`, or is correct but adds a material error elsewhere: a false factual claim about hydration or health, such as a wrong figure or a recommendation attributed to a body that doesn't make it. Extra advice clearly labelled as a rough rule of thumb is not an error. - **incorrect**: gives a figure or claim the key contradicts - **no answer**: declines, or answers a different question Separately, each answer is checked for each **myth** in `questions.json` (`myths`), present or absent. A myth only counts when the answer presents it as fact or official advice; describing it as a myth or rough rule of thumb doesn't count. Two independent judges score every answer: two separate Claude Code sub-agents, each given the same blind file (`score.mjs --export`) in which answers are shuffled and the platform is hidden behind a random id. Neither sees the other's verdicts. Where they disagree on the verdict or any myth flag, a human decides, reading the answer against the key. Agreement rate is published. A random 10% of answers they agree on are also checked by a human; if the human overturns more than 1 in 10 of those, every answer gets human review. Both judges are Claude models, and Claude is one of the platforms scored. Blinding, the human checks and the published raw answers are the safeguards; the write-up says so. (`score.mjs` can instead use two API judges from different companies via OpenRouter, about $5 for the full run, if a reviewer asks for that.) Sources an answer cites are recorded as domains and grouped as: government or academic (.gov, .edu, nih, who.int, nationalacademies, ncbi), major health publisher (Mayo Clinic, Cleveland Clinic, Healthline, WebMD, Medical News Today, Harvard Health, Johns Hopkins), commercial/brand, and other. ## What gets reported - **Accuracy per platform**: % correct, % partly, % incorrect, over all its answers. Headlines use % correct. - **Myth rate per platform**: % of answers repeating any myth, and per myth. - **Women's-health subset**: the same, for the four `women` questions. - **Consistency**: for each platform and question, whether its 5 answers got the same verdict. - **Sources**: the most-cited domains overall and per platform, and the share from government or academic sources. - **Demand**: estimated monthly AI search volume for the questions (DataForSEO AI Keyword Data; labelled as an estimate). Rules for claims: - A difference between platforms is called a difference only if the two-proportion test gives p < 0.05. Otherwise the copy says they performed similarly. - No "best" or "worst" platform unless that platform's correct rate beats or trails every other platform's, by that test. - Every percentage is shown with its count ("31 of 75 answers"). - Every raw answer is published in the CSV, so any claim can be checked. - If no platform falls below 80% correct and the myth rate is under 10% everywhere, the story is "AI mostly gets hydration right" and is published that way. A null result is still a result; the method doesn't change to find a worse one. ## Changes - 2026-10-14, before any real collection, after a pipeline test on two OpenRouter models (judges agreed on 77%): defined "material error" (above) and widened the eight-glasses myth to cover close variants ("8 to 10 cups") and attributing it to a health body. Both judges had read these differently; the wording now says what the rubric meant. Added: an answer repeating a myth cannot be "correct". - 2026-10-14, same day, before collection: judges changed from two API models to two blind Claude Code sub-agents (cost), validated on the same test set. - 2026-10-16, after the 5-call probe and before collection: pinned Claude to `claude-sonnet-5-5` (its current default chat model) and Perplexity to `sonar-pro` (its flagship search model; the API also offers `sonar` and `sonar-reasoning-pro`). Web search is enabled for both, but the model decides whether to search; in the probe Claude did not, and the Gemini scraper returned no sources. Both are recorded as observed. - 2026-10-16, collection done (315 answers, $2.37 DataForSEO): the blind file was too long for one sub-agent, so each judge ran as 4 sub-agents over the same 4 chunks (each answer seen once per judge). Judge A and judge B chunks ran as separate agents with no access to each other. Observed models: ChatGPT app `gpt-6`, Gemini app `3.5 Flash-Lite`. - 2026-10-16, during tie-break review, after scoring: **answer-key error corrected.** Most pregnancy answers gave 8 to 12 cups a day, often citing ACOG. ACOG's own page says exactly that ("During pregnancy you should drink 8 to 12 cups (64 to 96 ounces) of water every day", acog.org Ask ACOG, checked 2026-10-16), but the key only accepted the National Academies' figures, so those answers were wrongly marked partly. The pregnancy key now accepts ACOG's figure. This raises the chatbots' scores; it was applied to all 21 pregnancy answers, which both judges re-scored blind. Pre-correction scores are kept in `scores-before-acog-fix.csv`. - 2026-10-16, tie-break: done by Claude (the study session) rather than a person, at Laurence's request, with Laurence spot-checking 10 decisions. Reviewing the sample surfaced two consistency rules, applied to all answers: a mislabelled figure about a *different* life stage, mentioned in passing, is not a material error (both judges consistently treated it so); and a loose cup range around a stated liter figure ("3 L, roughly 8 to 12 cups") is imprecision, not a false claim. Of the 39 sampled agreements, none was overturned.