Skip to content

fix(models): don't crash to_markdown() on a malformed heading level - #178

Closed
afonsojanu wants to merge 1 commit into
mldsveda:mainfrom
afonsojanu:fix/heading-level-crash-176
Closed

fix(models): don't crash to_markdown() on a malformed heading level#178
afonsojanu wants to merge 1 commit into
mldsveda:mainfrom
afonsojanu:fix/heading-level-crash-176

Conversation

@afonsojanu

Copy link
Copy Markdown
Contributor

Fixes #176

Summary

ScrapeResult._item_to_markdown() sliced heading["level"][1:] and int()'d it unconditionally (int(heading.get("level", "h2")[1:])), so any level not shaped like h1-h6 raised ValueError ("title", "") or TypeError (None) instead of degrading gracefully. TextExtractor only ever emits h1-h6, but ScrapeResult also accepts user-supplied data (a documented, supported use), where one malformed heading crashed the entire to_markdown() call rather than just that heading.

Fix

Added _heading_level(), which parses the level defensively — falls back to h2 for anything not shaped h<digits>, and clamps in-range-shaped-but-out-of-bounds values (h9, h0) to 1-6 — and routed the heading loop through it, per the fix sketched in the issue.

Testing

  • test_to_markdown_handles_a_malformed_heading_level — parametrized over "title", "", None, "h9", "h0": each renders without raising, with a heading line carrying 1-6 #s.
  • test_to_markdown_clamps_out_of_range_heading_levels — pins the exact clamped output for h1, h6, h9, h0.
  • Confirmed both fail against the old code (git stash on models.py, re-ran): 7 of 9 new cases failed with the exact ValueError/TypeError/assertion mismatches the issue describes.
uv run pytest tests/ -q
# 535 passed, 9 skipped, 19 deselected
uv run ruff check src/pyscrappy/core/models.py tests/test_core/test_models.py
# All checks passed!

mypy src/pyscrappy/core/models.py reports 4 pre-existing errors (missing pandas/yaml stubs, two no-any-return findings on unrelated lines) — none touch _heading_level or the lines I changed; unrelated to this fix.

_item_to_markdown() sliced heading['level'][1:] and int()'d it
unconditionally, so any value not shaped like h1-h6 (e.g. 'title',
'', None) raised ValueError/TypeError instead of degrading
gracefully. TextExtractor only ever emits h1-h6, but ScrapeResult
also accepts user-supplied data, where one malformed heading
crashed the entire to_markdown() call.

Add _heading_level(), which parses defensively and falls back to h2
(clamping in-range values like h9/h0 to 1-6), and route the heading
loop through it.

Fixes mldsveda#176
@vedaant00

Copy link
Copy Markdown
Collaborator

Thanks for this @afonsojanu , and it's a clean fix, the _heading_level helper is nice. Two PRs landed for #176 within a minute of each other though, and I went with #177 since it was green and conflict-free. No reflection on this one. Really appreciate you jumping on it, especially right after #172, if you want another, #166 (bound the on-disk cache) is open and meatier.

@vedaant00 vedaant00 closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_markdown() crashes on a heading whose level isn't h1-h6

2 participants