AI agent with an eval set
An AI agent that drafts the weekly measles report.
Public health staff have to turn CDC’s weekly measles tables into a readable situation report. The raw data has traps: late reports, tiny baselines, and naming quirks that make a quick draft wrong. This project drafts the report, then checks the draft for accuracy.
Python 3.12+uvAnthropic SDKClaude Opus 5.5pytestD3
01 · At a glance
The question, the data, and the terms.
This one was built for health departments, not hospitals. It is on this site as a worked example of an AI tool that can be checked.
- The question
- Can an AI agent draft the weekly report from CDC’s tables, and can the draft be checked before a person reads it?
- Who it’s for
- Public health epidemiologists and health department staff. Drafts are meant for an epidemiologist to review.
- Data
- Public. CDC NNDSS Weekly Data (data.cdc.gov dataset x9gk-5huc), for “Measles, Indigenous” and “Measles, Imported.” Four frozen weekly snapshots are used for the evals: 2025 week 10, 2025 week 30, 2026 week 20, and 2026 week 44.
- Status
- A working prototype with an eval set and 43 offline tests.
- License
- No license is stated in the repository.
02 · What it does
Code does the math. The model writes.
Code gathers the figures
The agent calls 7 tools to pull the numbers it needs. All arithmetic is done in code, not by the model.
The model writes the report
Every report has the same sections: Summary, Key figures, Geographic distribution, Trend, Data notes, and Recommended actions.
An eval set checks the draft
Ten eval cases run fixed checks against the drafts. An AI judge then scores them on faithfulness, audience fit, and actionability.
A person reviews it
Drafts go to an epidemiologist before release. They are not meant to be published as written.
Why it’s on this siteA pattern for any AI drafting tool
- Keep the numbers out of the model’s hands. Counts and percentages come from code.
- Fix the format. The same sections every week make a draft easy to review.
- Test it every time. An eval set catches a wrong draft before a person has to.
- Publish the misses. The project reports the cases where it fell short.
03 · What the evals showed
Passing checks, and two honest gaps.
These results come from one live run on November 13, 2026, with Claude Opus 5.5.
What passedThe fixed checks
- All 10 eval cases pass the deterministic checks: 132 of 132.
- The AI judge scored most cases 5/5/5 on faithfulness, audience fit, and actionability.
What didn’tThe gaps the project reports
- The unpublished-week case scored 2 on faithfulness, because the draft guessed CDC’s publication lag. The fix is untested in a live run.
- The imported-focus case was not judged, because API credit ran out.
CDC’s “current week” column undercounts badly. In 2025 it summed to 642 against a year-end total of 1,872.
04 · Limits
A draft, not a release.
These limits come from the project’s own documentation.
- NNDSS counts are provisional and change with late reports.
- The weekly tables have no hospitalizations, deaths, vaccination status, ages, or outbreak links.
- “New York” in the data means New York State excluding New York City.
- Drafts need an epidemiologist’s review before release. They are not to be published as-is.
- The eval scores come from a single run, and they have the two gaps described above.
Read the agent, and the tests that check it.
The code, the frozen snapshots, the eval cases, and an example report are on GitHub.