Committee/Council award bid capture — built, and bounded to ~1% by the doc-number join key

[alexwolson]

Surfaced by the zero-bid investigation (#163). A distinct gap from the pre-Ariba linkage gap (#124).

The gap

99 awarded solicitations, 2020+, each over $3M — totalling $3.11B — have zero captured bids and nothing else captured either: no Ariba Discovery posting, no Award Summary Form, no staff report on file, no council item. Examples: $197M District 2 Waste Collection, $139M, $137M Containerized Waste Collection, $126M construction, $96M General Contracting (Paramedic), $91M, $90M Pax/RoPax Ferry Vessels, $89M Basement Flooding Protection, $80M sewer, $75M Ashbridges Bay.

Why they're missing

These awards exceed the Bid Award Panel's delegation ceiling, so they were decided by Council or a Standing Committee (Infrastructure, Executive, General Government, etc.), not the Bid Award Panel. The archive's bid capture (#84/#94) only mines Bid Award Panel / Bid Committee agendas (the 891 cached BA/BD pages), so these never enter the corpus. The award itself comes from the OData spine; the bid table is not.

Few items, but disproportionate dollar value ($3.11B) — the exact awards where "was this major spending competitive?" matters most.

Where the bids likely live

Council/committee staff reports on legdocs (plain HTTP, not Akamai-gated — same host we already fetch background_pdf staff reports from). A committee report recommending a large award typically tabulates the bids received, the same way Bid Award Panel reports do.

Suggested direction

1. Identify the target set: awarded solicitations over the panel threshold with no captured bids (the 99 here; refresh the query as new awards land).
2. Locate each award's committee decision + staff report. The committee agendas are on TMMIS (Akamai-gated → headed browser, like the existing council/agency scrapers); the staff-report PDFs are plain HTTP.
3. Parse the report's bid table (a new parser — committee reports differ in layout from BA agendas; validate against real reports per the Agency capture (#135): buyer dimension + TRCA/Zoo board reports, gated bids&tenders portal #136/TRCA _AWARD winner regex over-captures on ~69% of the real corpus #138 discipline).
4. Caveat — some are genuine zero: a handful are emergency / sole-source / "only supplier" (e.g. the $96M Paramedic General Contracting is flagged "emergency"). Refuse those (no bids by nature), don't invent them.

Scope note

This is a new capture source, larger than a linking pass — distinct from #124 (which links already-captured pre-Ariba bids). Bounded and high-value; could be scoped to just the >$3M / >$10M tier first.

[alexwolson]

Closing — not worth building. Investigated + prototyped end-to-end; the feasibility is real but the discovery isn't.

What's confirmed (the good news)

The committee/Council staff reports do tabulate the bids, and a parser handles both shapes cleanly (validated against real reports):

• RFQ (District 2, Doc4553928310, $197M): Table 2: Summary of Bids Received → GFL $36.2M / Halton $37.2M
• RFT (Ashbridges, $85M): opened the following bids → Kenaidan / Torbear / Alberici

A full working source was prototyped on branch feat-164-committee-award-bids (voting-record index → browser item-page discovery → legdocs PDF → parse_committee_bids → attach by document number, like #114). The parser + store are proven — the live run correctly attached GFL + the losing bidder to the solicitation.

Why it's closed (the blocker)

No viable discovery path. The document number → staff-report URL mapping is the problem:

• The plain-HTTP shortcut I probed — the open-data voting-record CSV — holds only 11 award-items-with-doc-numbers total, because it records only items that went to a recorded vote. Committee awards are overwhelmingly adopted on consent, so they never appear. Measured coverage: 5 of 5,062 uncaptured awards (0.1%).
• The only paths to real coverage are browser-scraping every committee agenda (GG/IE/PW/EX/CC across terms — the 891-BA-page pattern, much larger) or per-doc TMMIS search (~365 Akamai-gated searches). Both are substantial builds, and TMMIS is browser-only.

Given the payoff (few items, and the bids for the truly huge ones are already visible under council_items where a report exists) versus a large browser-discovery build, this isn't worth it. The proven parser is on the branch if a discovery path ever justifies resurrecting it.

#167 closes with this (folded in — same blocker, older era).

[alexwolson]

Kept open — complexity updated (not closing). The prior comment's finding stands as the scoping evidence; this reframes it as a complexity/status update:

Feasibility: HIGH — confirmed

Committee reports do tabulate bids and a parser handles both shapes (proven against real reports; the parser + document-number attachment are prototyped and working on branch feat-164-committee-award-bids, live-verified attaching GFL + the losing bidder).

Complexity: HIGH — the discovery is the whole cost

The blocker is document number → staff-report URL discovery, and it is now known to be expensive:

• The cheap plain-HTTP path (open-data voting-record CSV) is a dead end — 0.1% coverage (only 11 award-items-with-doc-numbers exist there; committee awards are adopted on consent, not recorded votes).
• Real coverage requires a large browser build: enumerate every committee agenda (GG/IE/PW/EX/CC across terms — the 891-BA-page pattern, bigger) or ~365 per-doc Akamai-gated TMMIS searches.

So this is a large, browser-heavy effort — not the "bounded" build the original issue estimated. The valuable, low-risk half (the parser + attach-by-doc) is done and parked on the branch; the remaining ~90% of the work is the browser discovery. Revisit when that build is justified.

[alexwolson]

Built and measured. #181 implements the route; this comment records why the original framing
of this issue ("99 awards, $3.11B") is not reachable by it, hence the retitle.

First live run

award items found          : 11
items needing bids         : 8
report urls discovered     : 8      <- 8/8
reports downloaded         : 8      <- 8/8
bids from committee reports: 3      <- from 2 of 8 reports

Every stage works — browser discovery, caching, download, store, supplier rebuild. The yield
is 3 bids.

The ceiling is the join key, not the parser

Discovery can only chase an item whose Agenda Item Title carries a 10-digit document number:

	count
distinct agenda items in the voting records	9,444
award/contract-like items	109
naming a 10-digit doc number	11

Against this issue's target set:

odata awards with no captured bids	4,942
of those, >= $5M	431 ($25.53B)
reachable via voting records	4

Why widening it does not help

1. The 98 award-like items without a doc number are mostly not losses. The sample is
   dominated by Amendment to Blanket Contract and Non-Competitive Contract — no bids by
   nature. Widening the title regex buys far less than 98 suggests.
2. RFTs/RFQs tabulate bids; RFPs narrate them. 6 of 8 reports yielded nothing for this
   reason. The parser refusing them is Exhibition Place procurement (Bonfire) — left the PMMD feed in 2019; everything since is invisible #130 discipline working, not a gap — one RFP report reads
   "The City received one submission ... from: Carla Construction and Maintenance Ltd." in
   prose and never names a losing proponent.
3. Most competitive awards never reach committee, being staff-delegated or panel-handled —
   the same mechanism Download the 3,142 indexed staff-report PDFs #83 measured on titles. Committee is the exception tier, so this route is
   structurally a thin slice.

Recorded in CLAUDE.md so it is not re-litigated, following the #83 precedent.

Left open deliberately

The one cheap extension not built: the prose pattern "received N submission(s) ... from: NAME",
which would pick up at least one of the 8 above. Worth doing; not worth much.

Also still outstanding from the original plan's Task 5: updating #163's accounting and deciding
whether #167 folds in here. Both should be settled against the numbers above rather than the
original headline.