50 Years of Medical Databases: From 1973 to the AI Era

Thu, 16 Apr 2026 00:00:00 +0000

Every day, thousands of researchers publish papers using public databases. But few know where these databases actually came from.

Today, I’ll walk you through fifty years of history, from 1973 to now. By the end, you’ll know exactly where to start.

1973 — Where it all begins: SEER

The story begins in 1973.

That year, Nixon signed the National Cancer Act. America declared war on cancer, but you can’t fight a war without knowing where the enemy is. So the NCI built SEER, the first national cancer registry. From that point on, every cancer patient’s diagnosis, staging, treatment, and survival was recorded.

Fifty years later, SEER covers half the US population and has directly produced over 17,000 papers. To this day, it remains the number one data source for cancer epidemiology.

1989-1999 — Seeds being planted

Over the next twenty years, more seeds were planted.

NHANES (started 1971, became a continuous biennial national health survey by 1999) — the king of cross-sectional studies, cited over 60,000 times on PubMed.

CHNS (1989) — China Health and Nutrition Survey, jointly launched by UNC Chapel Hill and the Chinese CDC. It tracked health and nutrition changes across China, and was one of the first Chinese health databases open to the world.

PhysioNet (1999) — MIT launched a platform dedicated to sharing clinical data. MIMIC, eICU, PIC — all the databases you’ve heard of, they’re all hosted there. PhysioNet isn’t a database. It’s infrastructure.

2000-2010 — Three forces driving the explosion

After 2000, things accelerated. Three forces pushed at the same time.

First, legislation. ClinicalTrials.gov went online, and by 2007 the law required all clinical trials to be registered. It now has half a million trials, and AACT turned all that data into a queryable database.

Second, journal mandates. Nature said: you want to publish? First deposit your gene expression data in GEO. Data submission became a prerequisite for publication. GEO now has 200,000 studies and 6.5 million samples.

Third, technology breakthroughs. High-throughput sequencing made data explode. In 2006, TCGA launched, profiling 33 cancer types with multi-omics data. Over 20,000 samples, 2.5 petabytes, mentioned in over 29,000 PubMed papers. It completely changed cancer classification from histology to molecular subtypes.

During this period, CTD, the precursor to DepMap (CCLE), and the EPA air quality system all came online.

2015 — The ImageNet moment for clinical AI

In 2015, something game-changing happened.

MIT released MIMIC-III. 60,000 ICU admissions, fully de-identified, freely available worldwide.

This database has been cited nearly 8,000 times. It’s been called the ImageNet of clinical AI — just as ImageNet sparked the boom in computer vision, MIMIC sparked an explosion in clinical prediction models.

ICU mortality prediction, early sepsis warning, mechanical ventilation management. Nearly every critical care AI study started with MIMIC.

Then in 2018, Philips released eICU, 200,000 multi-center ICU records. With eICU, MIMIC models could finally be externally validated.

2019 — China enters

In 2019, China started building its own databases.

Children’s Hospital of Zhejiang University released PIC, the world’s first publicly available pediatric ICU database. Over 13,000 admissions, 12,000 patients, hosted on PhysioNet.

This meant China went from being a user of public databases to a builder.

Around the same time, the Broad Institute launched DepMap. It uses CRISPR for genome-wide screening, telling you which genes each cancer truly depends on. From describing mutations to functional validation. Another paradigm shift.

2021 to now — Designed for AI

By 2021, the direction shifted again.

MIMIC-IV was released — no longer just ICU data. It added 200,000 emergency department records. From an ICU database to a complete acute care database.

CarpeDiem opened in 2023, with 44 clinical parameters per day, simulating what you see on daily rounds.

NWICU opened in 2024, 12 hospitals, 28,000 ICU stays, aligned with MIMIC-IV’s data structure.

Notice the pattern? These new databases aren’t designed for humans to browse. They’re designed for machine learning. Standardized structures, cross-database validation built in.

At the same time, imaging data exploded: CheXpert with 200,000 chest X-rays, PTB-XL with 20,000 ECGs, Kvasir with endoscopy images, CAMELYON with whole-slide pathology. Almost every imaging direction now has a public dataset.

This path, I’ve already walked it

So after all this history, what does it mean for you? Everything.

Fifty years ago, doing research required your own data. No lab, no cohort, you couldn’t do anything.

Today is different. Dozens of public databases are freely available. From critical care to oncology, from epidemiology to genomics, from ECGs to pathology slides.

And databases can be combined. MIMIC for modeling plus eICU for validation. SEER clinical data plus TCGA molecular data. NHANES as primary analysis plus CHNS for cross-population validation. These combination strategies are the real secret to publishing with public databases.

Your direction	Recommended combo
Beginners	NHANES or SEER
Critical care	MIMIC + eICU
Oncology	SEER + TCGA + GEO

All of these databases are free. Most of them just require signing up on PhysioNet and agreeing to the data use terms.

This path, I’ve already walked it.

SEER on Dual Brain Lab