feat: add parquet and excel exporters - #174
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
to_parquet()/to_excel() currently surface a pandas-missing ImportError that suggests installing the wrong extra (pyscrappy[dataframe]), which can mislead users trying to enable Parquet/Excel export.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds Parquet and Excel export support to ScrapeResult, including save() dispatch by file extension and optional dependency extras to keep the core install lightweight.
Changes:
- Add
ScrapeResult.to_parquet(path)andScrapeResult.to_excel(path)implementations using pandas. - Extend
ScrapeResult.save(path)to dispatch.parquetand.xlsx. - Add optional dependency extras and tests covering round-trips and missing optional deps.
File summaries
| File | Description |
|---|---|
src/pyscrappy/core/models.py |
Adds to_parquet() / to_excel() and wires .parquet / .xlsx into save() extension dispatch. |
tests/test_core/test_models.py |
Adds tests for Parquet/Excel export, missing-dependency errors, and save() dispatch. |
pyproject.toml |
Introduces parquet and excel optional-dependency groups and includes them in the all extra. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Thanks @Siteshcodes, nice first contribution. The per-format [parquet]/[excel] split is the right call — it matches the existing extras ([yaml], [stealth], [mcp]) and keeps Excel-only users from pulling pyarrow. Copilot's finding is fixed: the pandas-missing path now points at the right extra, and since both extras include pandas the hint resolves cleanly. Verified locally: real round-trip tests, missing-dep errors, save() dispatch + parent-dir creation. 573 passed, ruff clean, CI green. Merging. Closes #161. |
… cache; bump to 1.6.1 Two features merged after the 1.6.0 release cut had no changelog entry: - mldsveda#174 (mldsveda#161): ScrapeResult.to_parquet()/to_excel() + save() dispatch + the pyscrappy[parquet]/[excel] extras. - mldsveda#180 (mldsveda#166): the on-disk response cache is now pruned to cache_dir_max_size. Add both to CHANGELOG (1.6.1), document the exporters + extras and the disk-cache bound in the README, and bump the version across all four sites.
Summary
Adds Parquet and Excel exporters to
ScrapeResult.Changes
ScrapeResult.to_parquet(path)using pandas and PyArrow.ScrapeResult.to_excel(path)using pandas and openpyxl..parquetand.xlsxextension dispatch toScrapeResult.save().parquetandexceldependencies.ImportErrormessages with installation hints.save()dispatch.Testing
tests/test_core/test_models.py: 42 passedCloses #161