Skip to content

feat: add parquet and excel exporters - #174

Merged
vedaant00 merged 4 commits into
mldsveda:mainfrom
Siteshcodes:fix/parquet-excel-exporters
Aug 27, 2026
Merged

feat: add parquet and excel exporters#174
vedaant00 merged 4 commits into
mldsveda:mainfrom
Siteshcodes:fix/parquet-excel-exporters

Conversation

@Siteshcodes

Copy link
Copy Markdown
Contributor

Summary

Adds Parquet and Excel exporters to ScrapeResult.

Changes

  • Add ScrapeResult.to_parquet(path) using pandas and PyArrow.
  • Add ScrapeResult.to_excel(path) using pandas and openpyxl.
  • Add .parquet and .xlsx extension dispatch to ScrapeResult.save().
  • Add optional parquet and excel dependencies.
  • Add clear ImportError messages with installation hints.
  • Add tests for exporters, dependency handling, and save() dispatch.

Testing

  • tests/test_core/test_models.py: 42 passed
  • Full test suite: 568 passed, 2 skipped, 19 deselected

Closes #161

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

to_parquet()/to_excel() currently surface a pandas-missing ImportError that suggests installing the wrong extra (pyscrappy[dataframe]), which can mislead users trying to enable Parquet/Excel export.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Parquet and Excel export support to ScrapeResult, including save() dispatch by file extension and optional dependency extras to keep the core install lightweight.

Changes:

  • Add ScrapeResult.to_parquet(path) and ScrapeResult.to_excel(path) implementations using pandas.
  • Extend ScrapeResult.save(path) to dispatch .parquet and .xlsx.
  • Add optional dependency extras and tests covering round-trips and missing optional deps.
File summaries
File Description
src/pyscrappy/core/models.py Adds to_parquet() / to_excel() and wires .parquet / .xlsx into save() extension dispatch.
tests/test_core/test_models.py Adds tests for Parquet/Excel export, missing-dependency errors, and save() dispatch.
pyproject.toml Introduces parquet and excel optional-dependency groups and includes them in the all extra.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/pyscrappy/core/models.py Outdated
Comment thread src/pyscrappy/core/models.py Outdated
Siteshcodes and others added 3 commits August 27, 2026 23:27
@vedaant00

Copy link
Copy Markdown
Collaborator

Thanks @Siteshcodes, nice first contribution. The per-format [parquet]/[excel] split is the right call — it matches the existing extras ([yaml], [stealth], [mcp]) and keeps Excel-only users from pulling pyarrow. Copilot's finding is fixed: the pandas-missing path now points at the right extra, and since both extras include pandas the hint resolves cleanly.

Verified locally: real round-trip tests, missing-dep errors, save() dispatch + parent-dir creation. 573 passed, ruff clean, CI green. Merging. Closes #161.

@vedaant00
vedaant00 merged commit 62630a6 into mldsveda:main Aug 27, 2026
6 checks passed
vedaant00 added a commit to vedaant00/PyScrappy that referenced this pull request Aug 27, 2026
… cache; bump to 1.6.1

Two features merged after the 1.6.0 release cut had no changelog entry:
- mldsveda#174 (mldsveda#161): ScrapeResult.to_parquet()/to_excel() + save() dispatch + the
  pyscrappy[parquet]/[excel] extras.
- mldsveda#180 (mldsveda#166): the on-disk response cache is now pruned to cache_dir_max_size.

Add both to CHANGELOG (1.6.1), document the exporters + extras and the disk-cache
bound in the README, and bump the version across all four sites.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add to_parquet() and to_excel() exporters on ScrapeResult

3 participants