Gates Open Research
Two things the Gates Foundation already publishes, held as one collection: every article on Gates Open Research with its open peer reviews, and the Foundation's committed grants database.
They are published separately and in different shapes — the first as JATS XML behind an article site, the second as a single CSV — and neither knows about the other. Putting them in one collection is what makes the questions in between askable: who reviewed the work this grant paid for, which institutions turn funding into published research, who else works on this problem.
Sources
| Where | How | |
|---|---|---|
| Articles, authors, institutions, reviews | gatesopenresearch.org |
The site lists every published version at /extapi/article/published-xml-urls; each one is fetched as JATS XML and parsed. |
| Grants | gatesfoundation.org/-/media/files/bmgf-grants.csv |
One CSV of every committed grant, updated by the Foundation roughly monthly. |
Neither needs a key. The article site is behind Cloudflare but the XML endpoints answer plainly; the scraper runs one request at a time, which takes about an hour for the full corpus.
Record types
| Type | Notes |
|---|---|
article |
One record per published version. Gates Open Research versions articles in place, so 10.12688/gatesopenres.12785.2 is version 2 of that paper and is a record of its own. |
author |
Keyed by ORCID where the article declares one (about a quarter of them), otherwise by a slug of the name. |
institution |
Affiliations as printed on the papers. Not reconciled against ROR. |
review |
An open peer review report: reviewer, their institution, the recommendation, and which article version it was written against. |
grant |
One row of the committed-grants CSV: grantee, purpose, division, topic, amount, dates, and where the grantee sits. |
What links to what, and what doesn't
article.author_ids, author.institution_ids and review.article_id are internal and resolve.
article.award_ids is different: it is whatever the funding statement of the paper says, and it
resolves to a grant only sometimes. Of roughly 600 distinct award identifiers on the papers,
about 190 match a Gates grant id, covering some 225 of the articles. The rest are real, they are
simply not Gates money — USAID, Wellcome, Grand Challenges Canada, WHO, national research
councils. article.funder_names names them.
Matching needs normalising: papers write INV050591 for the grant recorded as INV-050591.
Upper-case and strip non-alphanumerics before comparing.
grant.institution_id is a slug of the grantee name, so it collides with an institution record
only when a Foundation grantee and a paper affiliation happen to spell themselves identically.
Treat it as a hint, not a join.
Notes
- Amounts are commitments, not disbursements, and the CSV carries no currency column; they are US dollars.
- A grant's
topicanddivisioncan hold several values separated by|. - The grants file is the whole database — around 41,000 grants back to 1997, about $102B committed. An earlier version of this collection carried a truncated download of roughly 7,000 rows, which made the totals and the topic mix wrong; if the numbers look small, the CSV was cut short.
Scraping
pnpm scrape gates/open-research grants # minutes
pnpm scrape gates/open-research articles # about an hour
Both write data/records.json; split that into data/<type>/records.json before pushing, since
the push loads every JSON file under data/ and would otherwise count records twice.
pnpm push gates/open-research --env dev