The data-pipeline repository for baseballr.
It builds clean, tidy NCAA college baseball datasets and publishes them as
GitHub release assets on
sportsdataverse/sportsdataverse-data,
where the baseballr load_ncaa_baseball_*() functions read them from. Two
small lookup tables are served directly from this repo's main branch.
flowchart LR
A[stats.ncaa.org] -->|ncaa_schedule_info / ncaa_*_pbp| B(baseballr-data<br/>R creation scripts)
B -->|local rds/csv/parquet| C[(ncaa/ in this repo)]
B -->|sportsdataverse_save| D{{sportsdataverse-data<br/>releases}}
D -->|load_ncaa_baseball_pbp / _schedule| E[baseballr users]
C -->|raw.githubusercontent main| F[load_ncaa_baseball_teams / _season_ids]
F --> E
- The GitHub Actions workflow
daily_ncaa_baseball.ymlruns on a seasonal schedule (and onworkflow_dispatch/repository_dispatch). - It calls
scripts/daily_ncaa_baseball_R_processor.sh, which loops over the requested seasons and runs the creation scripts. - Each creation script writes per-year
rds/csv/parquetartifacts underncaa/(committed back to this repo) and callssportsdataversedata::sportsdataverse_save()to upload those assets to the matching release tag onsportsdataverse-data(overwriting the season's existing assets). baseballr'sload_*()functions download the release assets on demand.
The MLB model tags are built from a committed per-game raw layer, which lives in
its own repo: sportsdataverse/baseballr-mlb-raw
(statsapi feed/live + Baseball Savant per-game Statcast, one gzipped file per
game). It moved there on 2026-09-02 so the capture tree and its ~7 GB backfill
stay out of this repo, which is already ~11 GB.
Read it per game over HTTP rather than cloning it. The per-season manifest is the source of truth for what exists -- one row per scheduled game, with the payload paths filled in only for games actually captured, plus a sha256 per file. Fetch only rows that carry a path; do not derive URLs from the schedule, which lists spring training and exhibition games that are deliberately not captured.
.../baseballr-mlb-raw/main/mlb/raw/manifest/index.csv one row per season (discovery)
.../baseballr-mlb-raw/main/mlb/raw/manifest/{season}.csv one row per game (paths + sha256)
Paths are relative to mlb/raw/, so a full URL is
https://raw.githubusercontent.com/sportsdataverse/baseballr-mlb-raw/main/mlb/raw/<path>.
Capture stages and design notes live in that repo's RUNBOOK.md and
docs/mlb-raw-layer.md.
The daily workflow above maintains the published seasons. Capturing a season
from scratch is a separate, human-run campaign against stats.ncaa.org: it is
slow, rate-sensitive, and every stage is file-exists resumable, so it is driven
from a terminal rather than from CI.
Run the whole thing with the orchestrator — newest season to oldest, committing and pushing per stage:
./scripts/run_backfill_all.sh 2026 2024 # SHARDS=8 by default
tail -f logs/bf_2026_*.logIt stops cleanly once a season's D1 team list comes back empty (the coverage
floor). To drive one stage at a time — re-running a failed stage, or filling a
single division — each has its own launcher. The order below is the one
run_backfill_all.sh itself uses; RUNBOOK.md documents the
manual one-season order and the per-stage detail.
| Stage | Launcher | Network | What it does |
|---|---|---|---|
| 01 | scripts/run_01_schedules_scrape.sh |
online | team lists + team pages → persisted html + ncaa/schedule_master/parquet/{season}.parquet |
| 04 | scripts/run_04_rosters_scrape.sh |
online | teams/{id}/roster → ncaa/rosters_html/{season}/ |
| 02 | scripts/run_02_games_scrape.sh |
online | the 5-tab bundle per uncaptured contest → ncaa/raw/{season}/{contest_id}.json.gz |
| 03 | scripts/run_03_games_parse.sh |
offline | raw bundles + legacy R-era trees → parsed payloads under ncaa/json/ |
| 06 | scripts/run_06_xwalk_build.sh |
mostly offline | NCAA↔ESPN game crosswalk → ncaa/xwalk/espn_game_id/{season}.json |
| 03 | scripts/run_03_games_parse.sh |
offline | re-run, so parsed payloads pick up the ESPN stamps from 06 |
| 05 | scripts/run_05_datasets_build.sh |
offline | persisted html → ncaa/{teams,schedule_master,rosters} reference parquet |
| 07 | scripts/run_07_datasets_publish.sh |
offline + gh |
season frames under ncaa/{dataset}/parquet/, then the release upload |
Stage 03 runs twice on purpose: the second pass is what writes the ESPN game
ids that stage 06 resolves. Stage 02 is the one to chunk (--max) and fan out
across disjoint --shard i/N processes; a ban hard-stops that run with rc=1,
so cool down and re-run — it resumes from what is already on disk.
Each launcher prints its own usage; --season (the calendar year — 2026 is
the spring-2026 season) is required, and --division narrows stages 01 and 04.
flowchart TB;
subgraph A[baseballr-data — scrape + build];
direction TB;
A0[scripts/daily_ncaa_baseball_pbp_scraper.sh]-->A1[python/ncaa_baseball_01_schedules_scrape.py];
A1[python/ncaa_baseball_01_schedules_scrape.py]-->A2[python/ncaa_baseball_02_games_scrape.py];
A2[python/ncaa_baseball_02_games_scrape.py]-->A3[python/ncaa_baseball_03_games_parse.py];
A3[python/ncaa_baseball_03_games_parse.py]-->A4[python/ncaa_baseball_04_rosters_scrape.py];
A4[python/ncaa_baseball_04_rosters_scrape.py]-->A5[python/ncaa_baseball_05_datasets_build.py];
A5[python/ncaa_baseball_05_datasets_build.py]-->A6[python/ncaa_baseball_06_xwalk_build.py];
A6[python/ncaa_baseball_06_xwalk_build.py]-->A7[python/ncaa_baseball_07_datasets_publish.py];
end;
subgraph B[MLB models — mlb_models_cron.yml];
direction TB;
B0[python/mlb_model_publish];
end;
subgraph C[sportsdataverse-data Releases];
direction TB;
C1[ncaa_baseball_schedules];
C2[ncaa_baseball_pbp];
C3[mlb_game_state];
C4[mlb_hitting_models];
C5[mlb_pitching_models];
C6[mlb_fielding_models];
end;
A-->C;
B-->C;
scripts/daily_ncaa_baseball_pbp_scraper.sh is the daily driver (the 00 role);
stage numbers are intended build order, not run order. The R chain
(R/ncaa_01_schedules_creation.R, R/ncaa_02_pbp_creation.R) is the maintained
methodological twin.
The droplet crontab (outside this repo) calls these drivers; each wraps the numbered stages and never re-implements one:
| Script | Schedule | Does |
|---|---|---|
scripts/daily_mlb_capture.sh |
08:30 ET, Mar–Nov | MLB daily capture + statcast (stages 04/06) + publish (stage 05) of the current season |
scripts/daily_ncaa_baseball_capture.sh |
09:30 ET daily Feb–Jun, monthly otherwise | NCAA baseball capture through the Decodo/patchright vendor |
scripts/run_mlb_statcast_backfill.sh |
manual | Statcast backfill over a season range: stages 04 + 06 newest-first, one commit per season (./scripts/run_mlb_statcast_backfill.sh 2026 2015); publish with run_mlb_05_publish.sh --dataset statcast |
Published to sportsdataverse/sportsdataverse-data releases (one tag per
dataset; assets are named *_{year}.rds / .csv / .parquet):
| Release | Loader | Asset pattern |
|---|---|---|
ncaa_baseball_pbp |
baseballr::load_ncaa_baseball_pbp() |
ncaa_baseball_pbp_{year}.rds |
ncaa_baseball_schedules |
baseballr::load_ncaa_baseball_schedule() |
ncaa_baseball_schedule_{year}.rds |
mlb_game_state |
sportsdataverse.mlb.load_mlb_re24_matrix() / _we_table / _wpa (Python) |
mlb_{stem}_{season}.{csv,rds,parquet} |
mlb_hitting_models |
load_mlb_expected_stats() / _expected_hr / _batter_projection |
mlb_{stem}_{season}.{csv,rds,parquet} |
mlb_fielding_models |
load_mlb_oaa() / _catcher_framing |
mlb_{stem}_{season}.{csv,rds,parquet} |
mlb_pitching_models |
load_mlb_xera() / _stuff_plus / _command_plus |
mlb_{stem}_{season}.{csv,rds,parquet} |
Note the schedules release uses the plural tag
ncaa_baseball_scheduleswith singular asset namesncaa_baseball_schedule_{year}.
Served directly from this repo's main branch (not releases):
| File | Loader |
|---|---|
ncaa/teams_info/ncaa_team_lookup.{csv,rds,parquet} |
baseballr::load_ncaa_baseball_teams() |
ncaa/seasons_info/ncaa_season_id_lu.{csv,rds,parquet} |
baseballr::load_ncaa_baseball_season_ids() |
# install.packages("baseballr") # or remotes::install_github("BillPetti/baseballr")
library(baseballr)
# play-by-play / schedule come from sportsdataverse-data releases
pbp <- load_ncaa_baseball_pbp(seasons = 2024)
sched <- load_ncaa_baseball_schedule(seasons = 2023:2024)
# team + season-id lookups come from this repo's main branch
teams <- load_ncaa_baseball_teams()
seasons <- load_ncaa_baseball_season_ids()You can also read release assets directly without baseballr:
# parquet (fastest)
arrow::read_parquet(
"https://github.com/sportsdataverse/sportsdataverse-data/releases/download/ncaa_baseball_pbp/ncaa_baseball_pbp_2024.parquet"
)All times UTC. The workflow refreshes the current season (resolved from
baseballr:::most_recent_ncaa_baseball_season() when no input is given):
| Window | Cadence | Cron |
|---|---|---|
| In-season (Feb–Jun) | Daily | 0 11 * 2-6 * |
| Off-season (Jul–Jan) | Monthly (1st) | 0 11 1 1,7-12 * |
Manual / backfill runs: trigger Update NCAA Baseball Data via
workflow_dispatch with start_year / end_year (and rescrape=TRUE to
re-pull raw JSON from stats.ncaa.org rather than reuse cached files).
baseballr-data/
├── R/ # R pipeline stages and publish toolchain
│ ├── 0000_create_baseballr_releases_init.R
│ ├── 0001_push_existing_release_data.R
│ ├── ncaa_01_schedules_creation.R
│ ├── ncaa_02_pbp_creation.R
│ ├── ncaa_season_ids.R
│ ├── ncaa_teams_info.R
│ ├── ncaa_teams_roster_gapfill.R
│ ├── ncaa_teams_season_team_id_backfill.R
│ ├── ncaa_util_01_schedules_conversion.R
│ ├── ncaa_util_02_pbp_conversion.R
│ └── utils.R
├── docs/ # explainers, model reports and dataset docs
│ └── models/
├── mlb/
│ ├── fielding_models/
│ ├── game_state/
│ ├── hitting_models/
│ └── pitching_models/
├── models/ # model artifacts, cards and the registry
├── ncaa/
│ ├── batter_box/
│ ├── contest_pbp/
│ ├── game_pbp/
│ ├── games/
│ ├── html/
│ ├── json/
│ ├── linescore/
│ ├── pbp/
│ └── … 17 more
├── python/ # Python pipeline stages, numbered in build order
│ ├── mlb_model_publish/
│ ├── ncaa_baseball_data_build/
│ ├── ncaa_pbp/
│ ├── mlb_model_01_game_state.py
│ ├── mlb_model_02_hitting.py
│ ├── mlb_model_03_pitching.py
│ ├── mlb_model_04_fielding.py
│ ├── ncaa_baseball_01_schedules_scrape.py
│ ├── ncaa_baseball_02_games_scrape.py
│ ├── ncaa_baseball_03_games_parse.py
│ ├── ncaa_baseball_04_rosters_scrape.py
│ ├── ncaa_baseball_05_datasets_build.py
│ ├── ncaa_baseball_06_xwalk_build.py
│ └── ncaa_baseball_07_datasets_publish.py
├── scripts/ # bash drivers (the daily/weekly entry points)
│ ├── _env.sh
│ ├── bash_functions.sh
│ ├── daily_ncaa_baseball_R_processor.sh
│ ├── daily_ncaa_baseball_pbp_scraper.sh
│ ├── daily_ncaa_baseball_scraper.sh
│ ├── mlb_models.sh
│ ├── render_model_docs.sh
│ ├── run_01_schedules_scrape.sh
│ ├── run_02_games_scrape.sh
│ ├── run_03_games_parse.sh
│ ├── run_04_rosters_scrape.sh
│ ├── run_05_datasets_build.sh
│ ├── run_06_xwalk_build.sh
│ ├── run_07_datasets_publish.sh
│ └── run_backfill_all.sh
├── statcast/
├── tests/ # test suite
│ ├── fixtures/
│ ├── test_capture.py
│ ├── test_data_build.py
│ ├── test_data_io.py
│ ├── test_data_publish.py
│ ├── test_discover.py
│ ├── test_mlb_model_publish.py
│ ├── test_model_manifest.py
│ ├── test_model_registry.py
│ ├── test_parse_legacy.py
│ ├── test_stage_numbering.py
│ ├── test_stages.py
│ └── test_xwalk.py
└── themes/ # plot themes
| Report | What it is | Last updated |
|---|---|---|
| Model registry | model | artifact |
| Model reports & cards | 4 files, one per item | 2026-09-01 |
| workflow | schedule | last run |
|---|---|---|
|
on repo dispatch / dispatch | 2026-08-01 |
 |
daily 10:30 UTC in Apr-Oct | 2026-08-31 |
 |
on push / PR / dispatch | 2026-08-27 |
 |
on push / PR / dispatch | 2026-08-27 |
| release tag | assets | size | last publish |
|---|---|---|---|
ncaa_baseball_schedules |
59 | 88.0 MB | 2026-08-27 |
ncaa_baseball_pbp |
39 | 2,304.9 MB | 2026-08-27 |
mlb_game_state |
109 | 85.3 MB | 2026-08-31 |
mlb_hitting_models |
106 | 5.2 MB | 2026-08-31 |
mlb_pitching_models |
109 | 7.2 MB | 2026-08-31 |
mlb_fielding_models |
73 | 1.6 MB | 2026-08-31 |
- Secrets: the workflow uploads cross-repo to
sportsdataverse-data, so it usessecrets.SDV_GH_TOKEN(a PAT with write access to that repo), passed to the R scripts asGITHUB_PAT.secrets.GITHUB_TOKENis used for this repo's git operations and dependency installation. - First-time bootstrap (run once, locally or manually):
Sys.setenv(GITHUB_PAT = "<PAT with write access to sportsdataverse-data>") source("R/0000_create_baseballr_releases_init.R") # create empty release tags source("R/0001_push_existing_release_data.R") # backfill historical seasons
- Dependencies:
arrow,data.table,dplyr,glue,purrr,rvest,httr,httr2,optparse, pluspiggyback(release creation) andsportsdataversedata(sportsdataverse_save()); the GitHub CLI (gh) must be available on the runner for asset uploads.
The packages that read what this repo produces:
- R: baseballr — docs at https://baseballr.sportsdataverse.org
- Python:
sportsdataverse.mlb / sportsdataverse.baseball— docs at https://py.sportsdataverse.org
Every numbered pipeline stage in python/ (auto-listed; run subsets with the scripts/*.sh drivers by number or name):
python/mlb_model_01_game_state.pypython/mlb_model_02_hitting.pypython/mlb_model_03_pitching.pypython/mlb_model_04_fielding.pypython/ncaa_baseball_01_schedules_scrape.pypython/ncaa_baseball_02_games_scrape.pypython/ncaa_baseball_03_games_parse.pypython/ncaa_baseball_04_rosters_scrape.pypython/ncaa_baseball_05_datasets_build.pypython/ncaa_baseball_06_xwalk_build.pypython/ncaa_baseball_07_datasets_publish.py
Model release tags published from here: mlb_fielding_models, mlb_game_state, mlb_hitting_models, mlb_pitching_models