Skip to content

feat(moss-cli): add mapped CSV/JSON bulk import#392

Open
Mahin2076 wants to merge 1 commit into
usemoss:mainfrom
Mahin2076:feat/379-cli-bulk-import
Open

feat(moss-cli): add mapped CSV/JSON bulk import#392
Mahin2076 wants to merge 1 commit into
usemoss:mainfrom
Mahin2076:feat/379-cli-bulk-import

Conversation

@Mahin2076

Copy link
Copy Markdown

Pull Request Checklist

  • I have read the CONTRIBUTING guide.
  • I have updated the documentation.
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have added tests that prove the feature works.
  • New and existing tests pass locally, except for two pre-existing shell-completion failures described below.

Description

Adds a discoverable bulk-import workflow for CSV, JSON, and JSONL files whose source columns do not already use Moss's id and text names.

  • Adds the requested moss documents import command and the shorter moss doc import form.
  • Supports --id-column, --text-column, repeatable --metadata-column, and comma-separated --metadata-columns mappings.
  • Reuses the existing authentication, profile, upsert, wait, timeout, JSON-output, and index-completion behavior.
  • Validates mapped files before creating the client, with row-specific errors for missing or blank required values.
  • Preserves the existing standard document format, including metadata objects, custom embeddings, JSON wrappers, JSONL, and stdin input.
  • Handles quoted multiline CSV fields without retaining an additional copy of every parsed row.
  • Documents the new workflow and adds focused parser and no-network CLI coverage.
moss documents import products -f products.csv \
  --id-column sku \
  --text-column description \
  --metadata-column category \
  --metadata-column price \
  --upsert --wait

Fixes #379

Testing

  • pytest packages/moss-cli/tests/test_documents_import.py -q — 29 passed on Python 3.13 and 3.14
  • Repeated the focused suite with Typer 0.9.0 (declared minimum) and Typer 0.26.8 — 29 passed on each
  • ruff check . --extend-exclude '**/*.ipynb'
  • mypy packages/moss-cli/src/moss_cli/documents.py packages/moss-cli/src/moss_cli/commands/doc.py --ignore-missing-imports
  • black --check and isort --check-only on changed Python files
  • Python 3.10 grammar check on changed Python files
  • Built both sdist and wheel; twine check passed for both
  • Clean-wheel smoke test: package import and moss documents import --help passed

The full Moss CLI suite reports 57 passed and 2 failed under the latest Typer. The same two untouched shell-completion tests (test_completions_bash_outputs_script and test_completions_zsh_outputs_script) also fail on origin/main in the same environment because Typer's completion API changed; this PR does not alter that code.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature which changes existing behavior)
  • Documentation update

@CLAassistant

CLAassistant commented Jul 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a bulk-import workflow to the Moss CLI that supports CSV/JSON/JSONL inputs where the source data doesn’t use the standard id / text field names, addressing the usability request in #379 and making non-code document loading more discoverable.

Changes:

  • Added moss documents import (and moss doc import) to bulk-import documents with --id-column, --text-column, and metadata-column mapping.
  • Enhanced the document loader to support column mapping, better CSV handling (including multiline quoted fields), and more actionable validation errors.
  • Added focused tests plus documentation and changelog entries describing the new workflow.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/moss-cli/tests/test_documents_import.py Adds comprehensive parser + CLI coverage for mapped imports across CSV/JSON/JSONL and stdin.
packages/moss-cli/src/moss_cli/main.py Registers documents as an alias command group for doc to make the feature discoverable.
packages/moss-cli/src/moss_cli/documents.py Implements mapping-aware parsing/validation for CSV/JSON/JSONL while preserving legacy document formats.
packages/moss-cli/src/moss_cli/commands/doc.py Adds the import command and shared submission logic; parses mapping flags and validates input before client creation.
packages/moss-cli/README.md Documents the new import workflow and examples for mapped column usage.
packages/moss-cli/CHANGELOG.md Notes the new bulk import feature in Unreleased.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +176 to +179
columns: List[str] = []
for value in values:
columns.extend(name.strip() for name in value.split(","))
return columns or None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Good first issue: CSV/JSON bulk-import command for the CLI

3 participants