Claude/research radar v1 setup ks66i3 - #542
Open
ergint wants to merge 12 commits into
Open
Conversation
Scaffolds the local project structure only: config docs for the locked verification/scoring rules, the study JSON schema, a deterministic validator that recalculates every score/tier/priority from raw components (never trusts stored totals), a radar builder that ranks only validated GREEN in-window studies without padding the list, prompt templates for the discovery/verification/scoring phases, a pytest suite locking in the scoring/eligibility rules, and seed records for the three known GREEN studies. No scraping or scheduled automation included. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137g1P4MrXhbcuZ9A1dg3yf
Fills in the three research/verified/ JSON records (measles school-level transmission, stair-climbing cardiovascular mortality, depression/ hippocampal neurogenesis) using only the bibliographic, methodological, and component-score facts already established for this project. Fields with no established value stay null and are listed in provisional_fields rather than invented. No changes to the validator, scoring model, schema, tests, project structure, or ranking rules. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137g1P4MrXhbcuZ9A1dg3yf
Adds scripts/discovery/{pubmed,crossref,normalize,deduplicate}.py and
scripts/run_discovery.py to fetch recent candidate metadata from the
official NCBI E-utilities and Crossref REST API (no HTML scraping) and
write normalized candidates to research/inbox/. Discovery only: every
candidate is written verification_status=YELLOW, ranking_eligible=false,
with null scientific/YouTube scoring fields. A conservative fast filter
rejects only unambiguous non-target publication types (editorial,
correction, erratum, letter, protocol, conference abstract, animal-only by
MeSH); anything uncertain stays YELLOW rather than being rejected.
Deduplicates PubMed/Crossref overlap by DOI, then PMID, then title
similarity, merging without deleting metadata. Supports --dry-run (no
disk writes) and --limit for development. Config lives in
config/discovery.json; NCBI_API_KEY and CROSSREF_MAILTO are read from the
environment only (.env.example added, .env git-ignored).
52 tests pass (28 existing Stage-1 tests untouched + 24 new Stage 2A tests
in tests/test_discovery.py, all against mocked HTTP responses, no live
network). Does not modify scoring rules, verification rules, validator
behavior, evidence tiers, production priorities, seed studies, or existing
tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137g1P4MrXhbcuZ9A1dg3yf
Makes PubMed/Crossref ingestion testable and diagnosable without live
outbound internet:
- tests/fixtures/{pubmed,crossref}/: 7 PubMed + 7 Crossref raw-response
fixtures (esearch, multi-article efetch, DOI/no-DOI, multiple pub types,
structured abstract, editorial, JATS abstract, missing abstract,
multi-author, and a cross-source duplicate pair) shaped like real API
responses.
- scripts/discovery/fixtures.py + `run_discovery.py --fixture-mode DIR`:
replays fixture files through the SAME parse_pubmed_xml/normalize_*/
deduplicate/apply_fast_filter production code a live run uses -- no
separate fake implementation, no network.
- `--save-raw`: dumps raw PubMed/Crossref response BODIES ONLY (never
headers, keys, or env vars) to git-ignored data/raw-discovery/YYYY-MM-DD/
during a live run, for debugging.
- HTTP client hardening (scripts/discovery/__init__.py, pubmed.py,
crossref.py): retries with backoff on transient 429/5xx only, never on
permanent 4xx or on network-level failures (a blocked-by-policy proxy
tunnel is never retried); explicit response-integrity validation before
parsing (empty body, JSON/XML shape, expected root/keys).
- DiscoveryTransportError vs DiscoveryParseError distinguishes "API never
responded" from "response was malformed", surfaced per-source as a
SourceHealth (SUCCESS/API_ERROR/PARSE_ERROR + records_returned +
message) in the discovery summary, so 0 results is never ambiguous with
a failure.
- Deduplication audit: doi_merges/pmid_merges/title_merges counts plus a
deduplication_notes trail on every merged candidate, reported in the
summary alongside pre/post-dedup counts per source.
- tests/test_discovery_fixtures.py: 9 new end-to-end fixture-mode tests,
including a PubMed/Crossref duplicate merging into one YELLOW,
unscored, non-ranking-eligible candidate, and an editorial that
Crossref's coarse `type` field alone can't catch but the merge with
PubMed's PublicationType does.
61 tests pass total (52 previous, unchanged, + 9 new). Does not modify
scoring rules, verification rules, validator behavior, evidence tiers,
production priorities, schema, or seed studies.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137g1P4MrXhbcuZ9A1dg3yf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.