The NHS publishes a huge amount of useful material. The difficult part is finding it consistently.
Board packs, quality accounts, strategies and inspection reports are spread across organisation websites with different structures, naming conventions and archive policies. A question that sounds simple, such as “which mental health trusts mention this priority?”, can begin with hours of opening tabs and downloading PDFs.
I built the NHS Evidence Scraper to make that first stage less manual.
Start with coverage, not cleverness
The platform currently covers 47 mental health trusts, 42 ICBs or commissioners and eight national datasets. It can collect trust board papers, annual reports, quality accounts, strategies and CQC reports alongside sources such as MHSDS, QOF and the National Clinical Audit of Psychosis.
The first useful version did not need artificial intelligence. It needed dependable crawling, sensible file names, source metadata and a summary of what had or had not been found.
That distinction matters. Search and analysis are only useful when the underlying evidence is current, traceable and complete enough to trust.
Public websites are operational systems
The interesting problem was not downloading a PDF. It was coping with the many ways a public website can change.
Some organisations move their publication directory. Others serve documents from a partner domain, load links with JavaScript or rate-limit repeated requests. A crawler which assumes every failure is temporary will repeat the same unproductive work.
The scraper therefore keeps a failure history. Organisations with repeated failures are surfaced in the dashboard. Known-good pages are cached and checked first on future runs. An organisation editor allows a person to correct a starting URL or allowed domain without editing configuration files by hand.
This turned failure from a hidden technical event into something visible and repairable.
Design for the person operating it
Runs can combine trusts, ICBs and national sources while keeping separate progress streams. Users choose document types, lookback periods, organisations, worker counts and request delays. Results can be exported as CSV, and each downloaded document gets a metadata sidecar containing its source, date and classification.
Rate limiting is explicit rather than mysterious. If an NHS site begins returning 429 responses, the operator can increase the delay while the session layer respects Retry-After and retries carefully.
Those controls are not decorative. They reflect the fact that evidence gathering needs supervision, especially across dozens of independently managed websites.
Intelligence comes after provenance
There is an experimental layer for extracting structured insights, searching documents and matching supplier propositions to trust priorities. I treat that part as unfinished.
The more important foundation is provenance: which document said what, when it was published and where it came from. Any useful summarisation layer has to preserve that chain back to the original evidence.
The project reinforced a principle I use in delivery work. Automating the happy path is easy. A useful operational tool also helps someone understand what failed, correct it and continue.