PMID GeneReviews entries (e.g., PMID:31536184) fall back to abstract_only instead of fetching NBK full text

[cmungall]

Summary

When caching PMID:31536184 (GeneReviews chapter for GA1), linkml-reference-validator stores content_type: abstract_only and does not fetch the full NCBI Bookshelf chapter text, even though full text is publicly available.

Version

• linkml-reference-validator==0.1.4rc8

Repro

linkml-reference-validator cache reference PMID:31536184 --force --verbose

Output includes:

• Content type: abstract_only

Cached file frontmatter:

reference_id: PMID:31536184
content_type: abstract_only

Why this is a problem

For GeneReviews PMIDs, users often cite statements that are present in the full NBK chapter but not in the PubMed abstract/summary text. Validation then fails with abstract-only limitations.

Expected behavior

One of:

1. PMID source should detect NCBI Bookshelf/GeneReviews links and fetch chapter full text (e.g., printable view), or
2. Provide a built-in NBK source and/or an automatic PubMed->Bookshelf handoff for PMIDs that resolve to Bookshelf chapters.

Additional evidence

Full chapter is available at:

• https://www.ncbi.nlm.nih.gov/books/NBK546575/?report=printable

(HTML content is large and includes full sections/tables/references, not just the summary.)

Notes

Current PMID implementation appears to do:

• PubMed abstract fetch
• optional PMC full text fetch via PMCID
• fallback to abstract_only when no PMCID

This misses Bookshelf-hosted full text for GeneReviews PMIDs.

[github-actions]

Issue Summary & Enrichment

Context

PMID:31536184 is the GeneReviews chapter for Glutaric Acidemia Type 1 (GA-1 / GCDH Deficiency) — an autosomal recessive metabolic disorder affecting amino acid degradation. The chapter is hosted on NCBI Bookshelf at https://www.ncbi.nlm.nih.gov/books/NBK546575/ (not PubMed Central). The full chapter contains extensive clinical details — diagnosis, management protocols, emergency procedures, genetic counseling — that are absent from the brief PubMed abstract. This makes citation validation against the abstract meaningless for GeneReviews claims.

---

Root Cause (Confirmed in Code)

The PMID fetch pipeline in src/linkml_reference_validator/etl/sources/pmid.py follows this waterfall:

Entrez.efetch(rettype="abstract")         → abstract text
  ↓
Entrez.elink(pubmed → pmc)                → PMCID
  ↓ (if PMCID found)
Entrez.efetch(db="pmc", rettype="xml")    → full text XML
  ↓ (if XML fails)
HTTP GET ncbi.nlm.nih.gov/pmc/articles/   → full text HTML
  ↓ (if no PMCID)
content_type = "abstract_only"            ← GeneReviews lands here

GeneReviews chapters are on NCBI Bookshelf (NBK), not PMC. The elink query uses linkname="pubmed_pmc", which does not traverse to Bookshelf. There is no NBK handling anywhere in the codebase.

---

Implementation Path

There are two NCBI eutils API calls that would solve this:

1. Detect Bookshelf link from PMID:

   Entrez.elink(dbfrom="pubmed", db="books", id=pmid)

   This returns a linked NBK accession (e.g., NBK546575) if the PMID is a Bookshelf chapter.

2. Fetch Bookshelf full text:

   GET https://www.ncbi.nlm.nih.gov/books/{nbk_id}/?report=printable
   

   The ?report=printable URL returns a single-page HTML with the full chapter content including all sections, tables, and references. This is publicly available (no auth required).

Suggested change to _fetch_pmc_fulltext() in pmid.py

After _get_pmcid() returns None, add a new _fetch_bookshelf_text() step that:

1. Calls Entrez.elink(dbfrom="pubmed", db="books", id=pmid)
2. Extracts the first book ID (NBK accession)
3. Fetches https://www.ncbi.nlm.nih.gov/books/{nbk_id}/?report=printable
4. Parses the HTML for the content body
5. Returns ("bookshelf_html", full_text) with a new content type

New content type to add to the model: "bookshelf_html"

Scale of GeneReviews

GeneReviews has ~800+ chapters covering inherited conditions. These are among the most-cited clinical genetics references and are commonly cited in databases like Monarch, HPO annotations, ClinVar, and rare disease knowledge graphs — exactly the kinds of curated sources that linkml-reference-validator is meant to validate. This is a high-value fix.

---

Related Issues

• Cache additional metadata from clinicaltrials.gov entries #39 — ClinicalTrials.gov richer metadata caching (similar pattern: external source with richer data than what's currently fetched)
• Support multiple --text-group values in a single validation pass #40 — Multiple --text-group support (different feature, same session)

---

Recommendation

This is a well-scoped enhancement. The fix is isolated to pmid.py (one new private method + one new elink call in _fetch_pmc_fulltext), with a new content_type value and a new test fixture representing a Bookshelf-backed PMID.

@dragon-ai-agent please implement Bookshelf (NBK) full-text fetching for PMIDs that resolve to NCBI Bookshelf chapters (GeneReviews). In src/linkml_reference_validator/etl/sources/pmid.py, add a _fetch_bookshelf_text(pmid)method that: (1) callsEntrez.elink(dbfrom="pubmed", db="books", id=pmid)to discover any linked NBK accession, (2) fetcheshttps://www.ncbi.nlm.nih.gov/books/{nbk_id}/?report=printable` and parses the HTML body, (3) returns the full text with a new content_type "bookshelf_html". Integrate this into _fetch_pmc_fulltext() as a fallback after _get_pmcid() returns None. Add the bookshelf_html content type to the models. Write tests first (TDD) using a fixture representing PMID:31536184 / NBK546575.`