From 8633d182a5bdfac8968808ece7185708ecda2b74 Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Sun, 24 May 2026 21:32:17 -0300 Subject: [PATCH 1/7] Update documentation --- docs/api.rst | 338 +++++++++++++++++++++++++++++++++--------- docs/benchmark.rst | 8 +- docs/cli.rst | 99 +++++++++++++ docs/index.rst | 31 ++-- docs/installation.rst | 64 +++++++- 5 files changed, 454 insertions(+), 86 deletions(-) create mode 100644 docs/cli.rst diff --git a/docs/api.rst b/docs/api.rst index c8adf0b..d08ff93 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -1,75 +1,149 @@ API Reference ============= +Parser Types +------------ + +Lexoid exposes three parser types via the ``lexoid.api.ParserType`` enum: + +* ``LLM_PARSE`` — Sends document pages (rendered to images) to an LLM API. +* ``STATIC_PARSE`` — Non-LLM parsing via ``pdfplumber``, ``pdfminer``, or + ``paddleocr`` for PDFs/images, plus dedicated handlers for HTML, plain + text, CSV, spreadsheets, DOCX, and PPTX. +* ``AUTO`` — Routes to ``LLM_PARSE`` or ``STATIC_PARSE`` based on document + characteristics and the configured routing priority. Falls back to the + alternate parser type on failure. + +Static parsing frameworks +^^^^^^^^^^^^^^^^^^^^^^^^^ + +For PDF inputs, the following ``framework`` values are supported: + +* ``pdfplumber`` (default) — Text extraction with table/heading/hyperlink heuristics. +* ``pdfminer`` — Pure pdfminer-based text extraction. +* ``paddleocr`` — OCR-based extraction (used automatically for image inputs and as a fallback). + Core Function ------------- parse ^^^^^ -.. py:function:: lexoid.api.parse(path: str, parser_type: Union[str, ParserType] = "LLM_PARSE", pages_per_split: int = 4, max_processes: int = 4, **kwargs) -> Dict +.. py:function:: lexoid.api.parse(path: str, parser_type: Union[str, ParserType] = "AUTO", pages_per_split: int = 4, max_processes: int = 4, **kwargs) -> Dict - Parse a document using specified strategy. + Parse a document, image, audio file, or URL. - :param path: File path or URL to parse - :param parser_type: Parser type to use ("LLM_PARSE", "STATIC_PARSE", or "AUTO") - :param pages_per_split: Number of pages per chunk for processing - :param max_processes: Maximum number of parallel processes - :param kwargs: Additional keyword arguments - :return: List of dictionaries containing page metadata and content, or raw text string + :param path: File path or URL to parse. + :param parser_type: Parser type to use (``"LLM_PARSE"``, ``"STATIC_PARSE"``, or ``"AUTO"``). Default: ``"AUTO"``. + :param pages_per_split: Number of pages per chunk for processing. Default: ``4``. + :param max_processes: Maximum number of parallel processes. Default: ``4``. Automatically forced to ``1`` when ``api_provider="ollama"``. + :param kwargs: Additional keyword arguments (see below). + :return: Dictionary containing parsed document data (see "Return value format" below). Additional keyword arguments: - * ``model`` (str): LLM model to use - * ``framework`` (str): Static parsing framework - * ``temperature`` (float): Temperature for LLM generation - * ``depth`` (int): Depth for recursive URL parsing - * ``as_pdf`` (bool): Convert input to PDF before processing - * ``verbose`` (bool): Enable verbose logging - * ``x_tolerance`` (int): X-axis tolerance for text extraction - * ``y_tolerance`` (int): Y-axis tolerance for text extraction - * ``save_dir`` (str): Directory to save intermediate PDFs - * ``page_nums`` (List[int]): List of page numbers to parse - * ``api_cost_mapping`` (Union[dict, str]): Dictionary containing API cost details or the string path to a JSON file containing - the cost details. Sample file available at ``tests/api_cost_mapping.json`` - * ``router_priority`` (str): What the routing strategy should prioritize. Options are ``"speed"`` and ``"accuracy"``. The router directs a file to either ``"STATIC_PARSE"`` or ``"LLM_PARSE"`` based on its type and the selected priority. If priority is "accuracy", it prefers LLM_PARSE unless the PDF has no images but contains embedded/hidden hyperlinks, in which case it uses ``STATIC_PARSE`` (because LLMs currently fail to parse hidden hyperlinks). If priority is "speed", it uses ``STATIC_PARSE`` for documents without images and ``LLM_PARSE`` for documents with images. - * ``api_provider`` (str): The API provider to use for LLM parsing. Options are ``gemini``, ``openai``, ``claude``, ``huggingface``, ``together``, ``openrouter``, ``fireworks``, and ``ollama``. This parameter is only relevant when using LLM parsing. For Ollama, use an explicit provider selection such as ``api_provider="ollama"`` with a local model like ``gemma4:latest``. - * ``return_bboxes`` (bool): Whether to return bounding box information for each text segment. Default is ``False``. + * ``model`` (str): LLM model to use. Defaults to the ``DEFAULT_LLM`` environment variable, or ``"gemini-2.5-flash"``. + * ``api_provider`` (str): API provider for LLM parsing. One of ``"gemini"``, ``"openai"``, ``"anthropic"``, ``"mistral"``, ``"huggingface"``, ``"together"``, ``"openrouter"``, ``"fireworks"``, ``"ollama"``, or ``"local"``. If not set, the provider is inferred from the model name. + * ``framework`` (str): Static parsing framework — ``"pdfplumber"`` (default), ``"pdfminer"``, or ``"paddleocr"``. + * ``temperature`` (float): Temperature for LLM generation. Default: ``0.0``. + * ``max_tokens`` (int): Max output tokens per LLM call. Defaults to ``1024`` (``4096`` for Ollama). + * ``system_prompt`` (str): Override the default parser system prompt. + * ``user_prompt`` (str): Override the default user prompt. + * ``depth`` (int): Depth for recursive URL parsing. Default: ``1``. + * ``as_pdf`` (bool): Convert input (image / webpage / DOCX) to PDF before processing. + * ``verbose`` (bool): Enable verbose logging during LLM parsing. + * ``x_tolerance`` (int): X-axis tolerance for ``pdfplumber`` text extraction. + * ``y_tolerance`` (int): Y-axis tolerance for ``pdfplumber`` text extraction. + * ``save_dir`` (str): Directory to save intermediate PDFs when ``as_pdf=True``. + * ``save_filename`` (str): Filename used when saving the intermediate PDF for a webpage. Defaults to ``webpage_.pdf``. + * ``page_nums`` (List[int]): Specific 1-indexed page numbers to parse (PDFs only). + * ``max_image_dimension`` (int): Maximum width/height (px) to which page images / input images are downscaled before parsing. Defaults to ``DEFAULT_MAX_IMAGE_DIMENSION`` (``1000``). + * ``api_cost_mapping`` (Union[dict, str]): Cost-per-million-tokens dictionary, or path to a JSON file. Sample at ``tests/api_cost_mapping.json``. When provided, the ``token_cost`` key is added to the result. + * ``router_priority`` (str): Routing priority for ``AUTO`` mode. One of: + + - ``"speed"`` (default): Uses ``STATIC_PARSE`` for PDFs without images, else ``LLM_PARSE``. + - ``"accuracy"``: Prefers ``LLM_PARSE``, except for PDFs with no images but with embedded/hidden hyperlinks (uses ``STATIC_PARSE`` since LLMs miss hidden links). + - ``"cost"``: Tries PaddleOCR first; if the extracted character count is below ``character_threshold``, returns it; otherwise falls back to ``LLM_PARSE``. + * ``character_threshold`` (int): Minimum character count for a ``router_priority="cost"`` STATIC_PARSE result to be accepted. Default: ``100``. + * ``autoselect_llm`` (bool): When ``parser_type="AUTO"``, runs the ML-based ``DocumentRankedLLMSelector`` to choose the best LLM for the input document. Default: ``False``. + * ``retry_on_fail`` (bool): When ``True`` (default), automatically retries with the alternate parser type / framework on failure. + * ``return_bboxes`` (bool): If ``True``, attach per-segment bounding boxes (``bboxes`` key on each segment). Default: ``False``. + * ``bbox_framework`` (str): Framework used for bounding box extraction when ``return_bboxes=True``. One of ``"auto"`` (default — chooses ``paddleocr`` or ``pdfplumber`` based on file content), ``"pdfplumber"``, or ``"paddleocr"``. Return value format: A dictionary containing a subset or all of the following keys: - * ``raw``: Full markdown content as string - * ``segments``: List of dictionaries with metadata and content of each segment. For PDFs, a segment denotes a page. For webpages, a segment denotes a section (a heading and its content). - * ``title``: Title of the document - * ``url``: URL if applicable - * ``parent_title``: Title of parent doc if recursively parsed - * ``recursive_docs``: List of dictionaries for recursively parsed documents - * ``token_usage``: Token usage statistics - * ``pdf_path``: Path to the intermediate PDF generated when ``as_pdf`` is enabled and the kwarg ``save_dir`` is specified. + * ``raw``: Full markdown content as a string. + * ``segments``: List of dictionaries with per-segment ``metadata`` (e.g., ``page``) and ``content``. For PDFs, a segment is a page; for webpages, a segment is a section (heading and its content). When ``return_bboxes=True``, each segment also has a ``bboxes`` key. + * ``title``: Title of the document. + * ``url``: Original URL if applicable. + * ``parent_title``: Title of the parent document, if recursively parsed. + * ``recursive_docs``: List of dictionaries for recursively-parsed sub-documents (when ``depth > 1``). + * ``token_usage``: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. + * ``token_cost``: Estimated cost per token category (only when ``api_cost_mapping`` is supplied). + * ``parsers_used``: List of parser names actually used for each chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). + * ``pdf_path``: Path to the intermediate PDF generated when ``as_pdf=True`` and ``save_dir`` is specified. + * ``error``: Present only when an unrecoverable error occurred (parsing returned a fallback empty result). parse_with_schema ^^^^^^^^^^^^^^^^^ -.. py:function:: lexoid.api.parse_with_schema(path: str, schema: Dict, api: str = "openai", model: str = "gpt-4o-mini", **kwargs) -> List[List[Dict]] +.. py:function:: lexoid.api.parse_with_schema(path: str, schema: Union[Dict, Type], api: Optional[str] = None, model: str = "gpt-4o-mini", example_schema: Optional[Dict] = None, alternate_keys: Optional[Dict] = None, fill_single_schema: bool = False, **kwargs) -> List[List[Dict]] - Parses a PDF using an LLM to generate structured output conforming to a given JSON schema. + Parse a PDF (or image) using an LLM to generate structured output + conforming to a given schema. The schema can be a plain ``dict``, a + Python ``dataclass``, or a Pydantic ``BaseModel`` (all are converted to + JSON Schema internally). - :param path: Path to the PDF file. - :param schema: JSON schema to which the parsed output should conform. - :param api: LLM API provider to use (``"gemini"``, ``"openai"``, ``"claude"``, ``"huggingface"``, ``"together"``, ``"openrouter"``, ``"fireworks"``, or ``"ollama"``). - :param model: LLM model name. - :param kwargs: Additional keyword arguments passed to the LLM (e.g., ``temperature``, ``max_tokens``). - :return: A list where each element represents a page, which in turn contains a list of dictionaries conforming to the provided schema. + :param path: Path to the file to parse. + :param schema: ``dict``, ``dataclass``, or Pydantic ``BaseModel`` describing the desired output. + :param api: LLM API provider. One of ``"gemini"``, ``"openai"``, ``"anthropic"``, ``"mistral"``, ``"huggingface"``, ``"together"``, ``"openrouter"``, ``"fireworks"``, or ``"ollama"``. If not specified, inferred from the model name. + :param model: LLM model name. Default: ``"gpt-4o-mini"``. + :param example_schema: Optional example data illustrating the desired filled schema (improves few-shot extraction). + :param alternate_keys: Optional mapping of alternate key names that may appear in the document — helps the model match synonyms. + :param fill_single_schema: When ``True``, the entire document is parsed once and the whole content is used to produce a single instance of the schema (rather than one instance per page). + :param kwargs: Additional keyword arguments (e.g., ``temperature``, ``max_tokens``). + :return: A list of dictionaries — one per page when ``fill_single_schema=False``, or a single-element list when ``fill_single_schema=True``. Each entry conforms to the provided schema. Additional keyword arguments: - * ``temperature`` (float): Sampling temperature for LLM generation. - * ``max_tokens`` (int): Maximum number of tokens to generate. + * ``temperature`` (float): Sampling temperature for LLM generation. Default: ``0.0``. + * ``max_tokens`` (int): Maximum number of tokens to generate. Default: ``1024``. - Return value format: - A list of pages, where each page is represented as a list of dictionaries. Each dictionary conforms to the structure defined by the input ``schema``. + +parse_to_latex +^^^^^^^^^^^^^^ + +.. py:function:: lexoid.api.parse_to_latex(path: str, api: Optional[str] = None, model: str = "gpt-4o-mini", **kwargs) -> str + + Convert a document (PDF or image) into a self-contained LaTeX string by + feeding each rendered page to a vision-capable LLM. The first page emits + the LaTeX preamble and ``\begin{document}``; the last page closes the + document with ``\end{document}``. + + :param path: Path to the file to convert. + :param api: LLM API provider. If not specified, inferred from the model name. + :param model: LLM model name. Default: ``"gpt-4o-mini"``. + :param kwargs: Additional keyword arguments forwarded to the LLM call (e.g., ``temperature``, ``max_tokens``). + :return: The concatenated LaTeX source as a single string. + + +parse_chunk +^^^^^^^^^^^ + +.. py:function:: lexoid.api.parse_chunk(path: str, parser_type: ParserType, **kwargs) -> Dict + + Low-level entry point that parses a single file (or PDF chunk) with the + given parser type. ``parse()`` orchestrates calls to ``parse_chunk`` over + PDF splits; most users should call ``parse()``. ``parse_chunk`` is + wrapped by the ``retry_with_different_parser_type`` decorator, which + implements the ``AUTO`` routing and fallback logic. + + :param path: The file path or URL. + :param parser_type: The :class:`ParserType` to use. + :param kwargs: Same keyword arguments accepted by :py:func:`parse`. + :return: Dictionary containing parsed document data, plus a ``parser_used`` key indicating which parser type actually ran. Examples @@ -82,7 +156,7 @@ Basic Usage from lexoid.api import parse - # Basic parsing + # Basic parsing (AUTO is the default parser_type) result = parse("document.pdf") # Raw text output @@ -91,8 +165,8 @@ Basic Usage # Segmented output with metadata parsed_segments = result["segments"] - # Automatic parser selection - result = parse("document.pdf", parser_type="AUTO") + # Explicitly select the parser type + result = parse("document.pdf", parser_type="LLM_PARSE") LLM-Based Parsing ^^^^^^^^^^^^^^^^^ @@ -102,8 +176,19 @@ LLM-Based Parsing # Parse using GPT-4o result = parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o") - # Parse using Gemini 1.5 Pro - result = parse("document.pdf", parser_type="LLM_PARSE", model="gemini-1.5-pro") + # Parse using Gemini 2.5 Pro + result = parse("document.pdf", parser_type="LLM_PARSE", model="gemini-2.5-pro") + + # Parse using Claude + result = parse("document.pdf", parser_type="LLM_PARSE", model="claude-3-5-sonnet-20241022") + + # Parse using Mistral OCR + result = parse( + "document.pdf", + parser_type="LLM_PARSE", + api_provider="mistral", + model="mistral-ocr-latest", + ) # Parse using a local Ollama model result = parse( @@ -114,17 +199,74 @@ LLM-Based Parsing max_processes=1, ) + # Local SmolDocling / granite-docling + result = parse( + "document.pdf", + parser_type="LLM_PARSE", + api_provider="local", + model="ds4sd/SmolDocling-256M-preview", + ) + + # Local PaddleOCR-VL + result = parse( + "document.pdf", + parser_type="LLM_PARSE", + api_provider="local", + model="PaddlePaddle/PaddleOCR-VL", + ) + Static Parsing ^^^^^^^^^^^^^^ .. code-block:: python - # Parse using PDFPlumber - result = parse("document.pdf", parser_type="STATIC_PARSE", model="pdfplumber") + # Parse using pdfplumber (default static framework) + result = parse("document.pdf", parser_type="STATIC_PARSE", framework="pdfplumber") + + # Parse using pdfminer + result = parse("document.pdf", parser_type="STATIC_PARSE", framework="pdfminer") + + # OCR with PaddleOCR + result = parse("scanned.pdf", parser_type="STATIC_PARSE", framework="paddleocr") + + +AUTO Mode and Routing +^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + # Default AUTO with "speed" priority + result = parse("document.pdf", parser_type="AUTO") + + # Accuracy-first routing (prefers LLM_PARSE) + result = parse("document.pdf", parser_type="AUTO", router_priority="accuracy") + + # Cost-first routing (tries PaddleOCR before falling back to LLM) + result = parse( + "document.pdf", + parser_type="AUTO", + router_priority="cost", + character_threshold=100, + ) + + # Auto-select the best LLM for this document + result = parse("document.pdf", parser_type="AUTO", autoselect_llm=True) + - # Parse using PDFMiner - result = parse("document.pdf", parser_type="STATIC_PARSE", model="pdfminer") +Bounding Box Extraction +^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + result = parse( + "document.pdf", + return_bboxes=True, + bbox_framework="auto", # "auto", "pdfplumber", or "paddleocr" + ) + for segment in result["segments"]: + for text, bbox in segment.get("bboxes", []): + print(text, bbox) # bbox is [x0, top, x1, bottom] normalized to [0, 1] Parse with Schema @@ -134,27 +276,89 @@ Parse with Schema from lexoid.api import parse_with_schema - sample_schema = [ - { - "Disability Category": "string", - "Participants": "int", - "Ballots Completed": "int", - "Ballots Incomplete/Terminated": "int", - "Accuracy": ["string"], - "Time to complete": ["string"] - } - ] + # Plain dict schema (one filled instance per page) + schema = { + "Disability Category": "string", + "Participants": "int", + "Ballots Completed": "int", + "Accuracy": ["string"], + "Time to complete": ["string"], + } + result = parse_with_schema(path="inputs/test_1.pdf", schema=schema, model="gpt-4o") + + # Single instance for the whole document + result = parse_with_schema( + path="inputs/test_1.pdf", + schema=schema, + model="gpt-4o", + fill_single_schema=True, + ) + + # Pydantic schema with example data and alternate keys + from pydantic import BaseModel + + class Invoice(BaseModel): + invoice_number: str + total: float + + result = parse_with_schema( + path="inputs/invoice.pdf", + schema=Invoice, + model="gpt-4o-mini", + example_schema={"invoice_number": "INV-001", "total": 199.95}, + alternate_keys={"invoice_number": ["Invoice #", "Invoice No."]}, + ) + + # Dataclass schema + from dataclasses import dataclass + + @dataclass + class Receipt: + merchant: str + amount: float + + result = parse_with_schema(path="receipt.pdf", schema=Receipt) + + +Parse to LaTeX +^^^^^^^^^^^^^^ + +.. code-block:: python + + from lexoid.api import parse_to_latex + + latex_source = parse_to_latex("paper.pdf", model="gpt-4o") + with open("paper.tex", "w") as f: + f.write(latex_source) - pdf_path = "inputs/test_1.pdf" - result = parse_with_schema(path=pdf_path, schema=sample_schema, model="gpt-4o") Web Content ^^^^^^^^^^^ .. code-block:: python - # Parse webpage + # Parse a webpage result = parse("https://example.com") - # Parse webpage and the pages linked within the page - result = parse("https://example.com", depth=2) \ No newline at end of file + # Parse a webpage and the pages linked within it + result = parse("https://example.com", depth=2) + + # Render the webpage to PDF first, then parse + result = parse( + "https://example.com", + as_pdf=True, + save_dir="output/", + save_filename="example.pdf", + ) + + +Audio Transcription +^^^^^^^^^^^^^^^^^^^ + +Audio inputs are routed to ``LLM_PARSE`` automatically and require a +Gemini model (currently the only provider supporting audio). + +.. code-block:: python + + result = parse("interview.mp3", model="gemini-2.5-flash") + print(result["raw"]) diff --git a/docs/benchmark.rst b/docs/benchmark.rst index dcf1187..51e5e5c 100644 --- a/docs/benchmark.rst +++ b/docs/benchmark.rst @@ -22,7 +22,7 @@ The similarity metric is calculated using the following steps (see `calculate_si 3. Whitespace and Punctuation Normalization Extra whitespace and punctuation are removed from both the parsed and ground truth texts. Therefore, the comparison is purely based on the sequence of characters/words, ignoring any formatting differences. -3. Sequence Matching +4. Sequence Matching Python's ``SequenceMatcher`` compares the extracted text sequences, calculating a similarity ratio between 0 and 1 that reflects content preservation and accuracy. Running the Benchmarks @@ -60,11 +60,11 @@ Customizing Benchmarks You can modify the ``test_attributes`` list in the ``main()`` function to test different configurations: -* ``parser_type``: Switch between LLM and static parsing +* ``parser_type``: Switch between LLM and static parsing (``LLM_PARSE``, ``STATIC_PARSE``, ``AUTO``) * ``model``: Test different LLM models -* ``framework``: Test different static parsing frameworks +* ``framework``: Test different static parsing frameworks (``pdfplumber``, ``pdfminer``, ``paddleocr``) * ``pages_per_split``: Adjust document chunking -* ``max_threads``: Control parallel processing +* ``max_threads``: Number of parallel workers (mapped to ``max_processes`` in the public :py:func:`lexoid.api.parse` API) Benchmark Results ----------------- diff --git a/docs/cli.rst b/docs/cli.rst new file mode 100644 index 0000000..0bef137 --- /dev/null +++ b/docs/cli.rst @@ -0,0 +1,99 @@ +Command-Line Interface +====================== + +Lexoid ships with a ``lexoid`` command (installed as a console script) for +parsing documents without writing Python code. You can also invoke it via +the module form ``python -m lexoid``. + +.. code-block:: bash + + lexoid --help + python -m lexoid --help + +Commands +-------- + +The CLI exposes three sub-commands: + +* ``lexoid parse`` — Convert a document into markdown (or JSON with metadata). +* ``lexoid schema`` — Extract structured data conforming to a JSON schema. +* ``lexoid latex`` — Convert a document into LaTeX. + +Common options +^^^^^^^^^^^^^^ + +Available across all sub-commands: + +* ``--input, -i`` (required): Path to an input file (PDF, image, HTML, DOCX, XLSX, PPTX, CSV, TXT, audio) or a URL (``http://``, ``https://``). +* ``--output, -o``: Path to an output file. If omitted, output goes to stdout (clean — status messages are written to stderr so output can be piped). +* ``--verbose, -v``: Enable detailed logging. + +``lexoid parse`` +^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + lexoid parse --input document.pdf + lexoid parse --input document.pdf --output output.md + lexoid parse --input document.pdf --format json --output result.json + lexoid parse --input document.pdf --parser-type STATIC_PARSE + lexoid parse --input document.pdf --model gpt-4o + +Options: + +* ``--parser-type, -p``: ``AUTO`` (default), ``LLM_PARSE``, or ``STATIC_PARSE``. +* ``--model, -m``: LLM model name. Default: ``gemini-2.5-flash``. +* ``--pages-per-split``: Pages per chunk. Default: ``4``. +* ``--max-processes``: Parallel processes. Default: ``4``. +* ``--framework``: Static parsing framework — ``pdfplumber`` or ``paddleocr``. +* ``--format``: ``markdown`` (default; raw markdown text) or ``json`` (full result with segments, metadata, and token usage). +* ``--api``: API provider override. One of ``openai``, ``gemini``, ``anthropic``, ``mistral``, ``together``, ``huggingface``, ``openrouter``, ``fireworks``, ``ollama``. If omitted, inferred from the model name. + +``lexoid schema`` +^^^^^^^^^^^^^^^^^ + +Extract structured data using a JSON schema. The schema can be passed as a +file path or as an inline JSON string. + +.. code-block:: bash + + # Inline schema + lexoid schema \ + --input document.pdf \ + --schema '{"type": "object", "properties": {"title": {"type": "string"}}}' \ + --output result.json + + # Schema from file + lexoid schema --input document.pdf --schema schema.json --output result.json + + # Specify model and API explicitly + lexoid schema --input document.pdf --schema schema.json --api openai --model gpt-4o + +Options: + +* ``--schema, -s`` (required): JSON schema — file path or inline JSON. +* ``--model, -m``: LLM model. Default: ``gpt-4o-mini``. +* ``--api``: API provider (auto-detected from model name if omitted). +* ``--example-schema``: Example data (JSON string or file path) illustrating a filled schema. +* ``--fill-single-schema``: Produce a single schema instance for the whole document instead of one per page. + +``lexoid latex`` +^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + lexoid latex --input document.pdf + lexoid latex --input document.pdf --output output.tex + lexoid latex --input document.pdf --model gpt-4o + +Options: + +* ``--model, -m``: LLM model. Default: ``gpt-4o-mini``. +* ``--api``: API provider (auto-detected from model name if omitted). + +API keys +-------- + +LLM commands require the relevant environment variable to be set +(see :doc:`installation`). The CLI checks for the required key based on +the resolved provider and raises a clear error if it is missing. diff --git a/docs/index.rst b/docs/index.rst index f4d9e78..99b5c5d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,7 +1,7 @@ Welcome to Lexoid's Documentation ================================= -Lexoid is an efficient document parsing library that supports both LLM-based and non-LLM-based (static) PDF document parsing. +Lexoid is an efficient document parsing library that supports both LLM-based and non-LLM-based (static) parsing of PDFs, images, web pages, office documents, and audio files. .. toctree:: :maxdepth: 2 @@ -9,6 +9,7 @@ Lexoid is an efficient document parsing library that supports both LLM-based and installation api + cli contributing benchmark @@ -16,29 +17,39 @@ Key Features ------------ * Multiple parsing strategies (LLM-based and static parsing) -* Automatic parsing strategy selection -* Support for multiple LLM providers (OpenAI, Google, Meta/Llama, Together AI) +* Automatic parsing strategy selection (``AUTO`` mode) with optional ML-based LLM auto-selection +* Routing priorities: ``speed``, ``accuracy``, and ``cost`` +* Support for many LLM providers (OpenAI, Google Gemini, Anthropic, Mistral, Hugging Face, Together AI, OpenRouter, Fireworks) +* Local LLM inference via Ollama, SmolDocling/granite-docling, and PaddleOCR-VL (no API key required) +* Schema-constrained extraction (``parse_with_schema``) accepting ``dict``, ``dataclass``, or Pydantic ``BaseModel`` +* LaTeX conversion (``parse_to_latex``) +* Audio transcription to markdown (via Gemini) +* Multi-format input: PDF, images (PNG/JPG/TIFF/BMP/GIF), HTML, DOCX, XLSX, PPTX, CSV, TXT, audio, and URLs +* Recursive URL parsing * Table detection and markdown conversion * Hyperlink detection and preservation -* Recursive URL parsing -* Multi-format support -* Parallel processing support -* Permissive license -* Reference highlighting and bounding box extraction +* Reference highlighting and bounding box extraction (``return_bboxes``) +* Parallel processing via multiprocessing +* Command-line interface (``lexoid`` / ``python -m lexoid``) +* Permissive Apache 2.0 license Supported API Providers ----------------------- -* Google +* Google (Gemini) * OpenAI +* Anthropic (Claude) +* Mistral (OCR models) * Hugging Face * Together AI * OpenRouter * Fireworks +* Ollama (local inference) +* Local models (SmolDocling/granite-docling, PaddleOCR-VL) Indices and tables ================== * :ref:`genindex` * :ref:`modindex` -* :ref:`search` \ No newline at end of file +* :ref:`search` diff --git a/docs/installation.rst b/docs/installation.rst index 4894788..a40ff62 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -8,27 +8,81 @@ Installing with pip pip install lexoid +This installs both the Python library and the ``lexoid`` command-line entry +point. See :doc:`cli` for CLI usage. + Environment Setup ----------------- -To use LLM-based parsing, define the following environment variables or create a ``.env`` file with the following definitions: +To use LLM-based parsing, define the environment variables for the providers +you intend to use (in a shell, ``.env`` file, or your container environment): .. code-block:: bash - GOOGLE_API_KEY=your_google_api_key - OPENAI_API_KEY=your_openai_api_key + GOOGLE_API_KEY=your_google_api_key # Gemini + OPENAI_API_KEY=your_openai_api_key # OpenAI / GPT + ANTHROPIC_API_KEY=your_anthropic_api_key # Claude + MISTRAL_API_KEY=your_mistral_api_key # Mistral OCR HUGGINGFACEHUB_API_TOKEN=your_huggingface_token TOGETHER_API_KEY=your_together_api_key + OPENROUTER_API_KEY=your_openrouter_api_key + FIREWORKS_API_KEY=your_fireworks_api_key + +Only the providers you actually use require keys. Local backends (Ollama, +SmolDocling/granite-docling, PaddleOCR-VL) do not require an API key. + +Additional environment variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* ``DEFAULT_LLM`` — overrides the default LLM model. Default: ``gemini-2.5-flash``. +* ``DEFAULT_LOCAL_LM`` — overrides the default local model used by ``parse_with_local_model``. Default: ``ds4sd/SmolDocling-256M-preview``. +* ``DEFAULT_STATIC_FRAMEWORK`` — overrides the default static-parsing framework. Default: ``pdfplumber``. +* ``DEFAULT_MAX_IMAGE_DIMENSION`` — maximum pixel dimension for resizing page/image inputs. Default: ``1000``. +* ``OLLAMA_BASE_URL`` — base URL of the Ollama server. Default: ``http://localhost:11434``. +* ``OLLAMA_TIMEOUT`` — request timeout (seconds) for Ollama. Default: ``120``. Optional Dependencies --------------------- -To use ``Playwright`` for retrieving web content (instead of the ``requests`` library): +Playwright (for web content retrieval) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To use Playwright for retrieving web content (instead of the bare ``requests`` +library), install its browser dependencies after ``pip install lexoid``: .. code-block:: bash playwright install --with-deps --only-shell chromium +LibreOffice (for DOCX to PDF on Linux) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +On Linux, ``.doc``/``.docx`` to PDF conversion uses LibreOffice's +``lowriter`` binary (because ``docx2pdf`` is unsupported on Linux). Install +it from your distribution's package manager, e.g.: + +.. code-block:: bash + + sudo apt-get install libreoffice + +On macOS/Windows, ``docx2pdf`` is used automatically (requires Microsoft Word +or compatible installation). + +Ollama (for local LLM parsing) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Install `Ollama `_, pull a vision-capable model, and +keep the server running: + +.. code-block:: bash + + ollama pull gemma4 + ollama serve + +Then call ``parse(..., api_provider="ollama", model="gemma4:latest", max_processes=1)``. +Lexoid forces ``max_processes=1`` for Ollama-backed parsing to avoid local +multiprocess contention. + Building from Source -------------------- @@ -57,4 +111,4 @@ To activate virtual environment: .. code-block:: bash - source .venv/bin/activate \ No newline at end of file + source .venv/bin/activate From 89b72037c597f753d0b06ec431164951fc698acc Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Sun, 24 May 2026 21:56:27 -0300 Subject: [PATCH 2/7] Add skills files --- skills/lexoid-cli/SKILL.md | 120 ++++++++++++++++ skills/lexoid-python/SKILL.md | 260 ++++++++++++++++++++++++++++++++++ 2 files changed, 380 insertions(+) create mode 100644 skills/lexoid-cli/SKILL.md create mode 100644 skills/lexoid-python/SKILL.md diff --git a/skills/lexoid-cli/SKILL.md b/skills/lexoid-cli/SKILL.md new file mode 100644 index 0000000..2d5f698 --- /dev/null +++ b/skills/lexoid-cli/SKILL.md @@ -0,0 +1,120 @@ +--- +name: lexoid-cli +description: Parse and convert documents (PDFs, images, web pages, DOCX/XLSX/PPTX, audio) from the terminal using the `lexoid` CLI. Use when the user wants to extract markdown / JSON / LaTeX from a file or URL without writing Python, run schema-based structured extraction, or batch-parse from shell scripts. Triggers include "parse this PDF", "convert document to markdown", "extract JSON from PDF", "convert PDF to LaTeX", "use lexoid CLI", or any pipe/shell-style document-processing request. +--- + +# Lexoid CLI + +Lexoid ships a `lexoid` console script (also runnable as `python -m lexoid`) for document parsing without writing Python. There are three sub-commands: `parse`, `schema`, and `latex`. + +## When to use this skill + +- The user has a document (PDF, image, HTML, DOCX, XLSX, PPTX, CSV, TXT, audio) or a URL and wants it converted to markdown/JSON/LaTeX. +- The user wants structured extraction (JSON conforming to a schema) from the shell. +- The task is one-off or scripted — no need to build a Python integration. + +If the user wants to embed parsing into a Python application or library, use the `lexoid-python` skill instead. + +## Setup checks + +Before invoking, confirm: + +1. `lexoid` is installed (`lexoid --help` or `python -m lexoid --help`). If not, run `pip install lexoid`. +2. For LLM-based commands, the relevant API key env var is set: + - `GOOGLE_API_KEY` (Gemini, default), `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `MISTRAL_API_KEY`, `HUGGINGFACEHUB_API_TOKEN`, `TOGETHER_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`. + - Ollama / local backends need no key but need a running server / pulled model. +3. For Linux DOCX → PDF, LibreOffice (`lowriter`) must be installed. + +## Commands + +### `lexoid parse` — Markdown / JSON output + +Convert a document to markdown (default) or JSON (with segments, token usage, parser info). + +```bash +# Default: AUTO routing, markdown to stdout +lexoid parse --input document.pdf + +# Save to file +lexoid parse --input document.pdf --output output.md + +# Full result as JSON (includes per-page segments, token usage, parsers_used) +lexoid parse --input document.pdf --format json --output result.json + +# Force static parsing (no LLM, no API key needed for PDFs) +lexoid parse --input document.pdf --parser-type STATIC_PARSE --framework pdfplumber + +# LLM parsing with a specific model +lexoid parse --input document.pdf --model gpt-4o +lexoid parse --input document.pdf --model claude-3-5-sonnet-20241022 +lexoid parse --input scanned.pdf --model mistral-ocr-latest + +# Parse a URL +lexoid parse --input https://example.com --output page.md + +# Tune chunking / parallelism +lexoid parse --input big.pdf --pages-per-split 8 --max-processes 8 +``` + +Key flags: `--parser-type` (`AUTO`/`LLM_PARSE`/`STATIC_PARSE`), `--model`, `--framework` (`pdfplumber`/`paddleocr`), `--api` (override provider), `--format` (`markdown`/`json`), `--pages-per-split`, `--max-processes`, `--verbose`. + +### `lexoid schema` — Structured extraction + +Extract data conforming to a JSON schema. Schema can be a file path or inline JSON. + +```bash +# Inline schema +lexoid schema \ + --input invoice.pdf \ + --schema '{"type":"object","properties":{"invoice_number":{"type":"string"},"total":{"type":"number"}}}' \ + --output invoice.json + +# Schema from file, explicit provider +lexoid schema --input invoice.pdf --schema schema.json --api openai --model gpt-4o + +# Example-guided extraction (improves accuracy) +lexoid schema --input invoice.pdf --schema schema.json \ + --example-schema example.json + +# Treat the whole doc as one instance (vs. one per page) +lexoid schema --input contract.pdf --schema schema.json --fill-single-schema +``` + +Defaults: model `gpt-4o-mini`. The provider is auto-detected from the model name unless `--api` is given. + +### `lexoid latex` — LaTeX conversion + +Convert a document to a self-contained LaTeX source. + +```bash +lexoid latex --input paper.pdf --output paper.tex +lexoid latex --input paper.pdf --model gpt-4o +``` + +## Output piping + +When no `--output` is given, only the parsed content is written to stdout; status messages, token usage, and parser info go to stderr. This means standard piping works: + +```bash +lexoid parse --input report.pdf | grep -i "revenue" +lexoid parse --input report.pdf --format json | jq '.token_usage' +``` + +## Common patterns + +- **Cheap path first**: try `--parser-type STATIC_PARSE --framework pdfplumber` for native-text PDFs. Fall back to `--parser-type LLM_PARSE` for scans, tables-heavy pages, or hand-written content. +- **Scanned PDFs / images**: use `--framework paddleocr` (no API key) or an LLM model with vision. +- **Batch**: drive the CLI from a shell loop (`for f in inputs/*.pdf; do lexoid parse -i "$f" -o "out/${f%.pdf}.md"; done`). +- **Debug**: add `--verbose` to surface loguru logs to stderr. + +## Failure modes to watch for + +- Missing API key → CLI raises a clean error naming the env var. Set it and retry. +- DOCX on Linux without LibreOffice installed → conversion to PDF fails. Install `libreoffice`. +- `--model` and `--api` mismatch → use `--api` only to override an auto-inferred provider (e.g., to send a model through OpenRouter). +- Ollama: must run `ollama serve` and `ollama pull ` first; the CLI does not start the server. + +## See also + +- Full reference: `docs/cli.rst` and `docs/api.rst` in this repo. +- Python equivalent: `lexoid-python` skill. diff --git a/skills/lexoid-python/SKILL.md b/skills/lexoid-python/SKILL.md new file mode 100644 index 0000000..3c67009 --- /dev/null +++ b/skills/lexoid-python/SKILL.md @@ -0,0 +1,260 @@ +--- +name: lexoid-python +description: Parse and convert documents (PDFs, images, web pages, DOCX/XLSX/PPTX, audio) inside a Python program using the `lexoid` library. Use when the user is writing Python code that needs to extract markdown from documents, run schema-constrained extraction, convert files to LaTeX, get bounding boxes, recursively crawl URLs, or integrate document parsing into a larger pipeline. Triggers include `from lexoid` imports, "use lexoid in Python", "parse PDFs programmatically", "extract structured data with a Pydantic/dataclass schema", or any request to embed parsing into a Python app/notebook. +--- + +# Lexoid Python API + +Lexoid's Python API is the right choice whenever the user is writing Python — notebooks, services, batch pipelines, or anything that needs the parsed result as a Python dict/list. For shell one-offs, use the `lexoid-cli` skill instead. + +## When to use this skill + +- The user is writing Python and wants to parse PDFs, images, URLs, DOCX/XLSX/PPTX, audio, or text-format files. +- The user needs per-page segments, token usage, parser metadata, or bounding boxes in code. +- The user wants schema-based structured extraction (`dict`, `dataclass`, or Pydantic `BaseModel`). +- The user is integrating parsing into a larger app (Streamlit, FastAPI, RAG pipeline, etc.). + +## Setup checks + +Before writing code, confirm: + +1. `lexoid` is installed (`pip install lexoid`). +2. Required API key env vars are set for the chosen provider (see below). +3. For Linux DOCX → PDF conversion, LibreOffice (`lowriter`) is on PATH. +4. For Ollama / local LLMs, the server is running and the target model is pulled. + +API keys by provider: + +| Provider | Env var | +|--------------|-------------------------------| +| `gemini` | `GOOGLE_API_KEY` | +| `openai` | `OPENAI_API_KEY` | +| `anthropic` | `ANTHROPIC_API_KEY` | +| `mistral` | `MISTRAL_API_KEY` | +| `huggingface`| `HUGGINGFACEHUB_API_TOKEN` | +| `together` | `TOGETHER_API_KEY` | +| `openrouter` | `OPENROUTER_API_KEY` | +| `fireworks` | `FIREWORKS_API_KEY` | +| `ollama` | none (uses `OLLAMA_BASE_URL`) | +| `local` | none | + +## Public API + +Four entry points in `lexoid.api`: + +- `parse(path, parser_type="AUTO", pages_per_split=4, max_processes=4, **kwargs)` — main function. Returns a dict. +- `parse_with_schema(path, schema, api=None, model="gpt-4o-mini", **kwargs)` — structured JSON extraction. Returns a list of dicts. +- `parse_to_latex(path, api=None, model="gpt-4o-mini", **kwargs)` — returns a LaTeX string. +- `parse_chunk(path, parser_type, **kwargs)` — low-level single-chunk parser; users rarely need this. + +`ParserType` enum: `LLM_PARSE`, `STATIC_PARSE`, `AUTO`. + +## `parse()` return shape + +```python +{ + "raw": str, # full markdown + "segments": [ # one dict per page / section + {"metadata": {"page": int}, "content": str, "bboxes": [...]}, + ... + ], + "title": str, + "url": str, # if input was a URL + "parent_title": str, # set on recursive sub-docs + "recursive_docs": [...], # populated when depth > 1 + "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, + "token_cost": {...}, # only when api_cost_mapping is supplied + "parsers_used": [str, ...], # which parser ran per chunk + "pdf_path": str, # only when as_pdf=True and save_dir is set +} +``` + +## Common recipes + +### Basic parsing + +```python +from lexoid.api import parse + +result = parse("document.pdf") +markdown = result["raw"] +for seg in result["segments"]: + print(seg["metadata"]["page"], seg["content"][:80]) +``` + +### Choose a parser explicitly + +```python +# Native-text PDFs — fastest, no API key +parse("document.pdf", parser_type="STATIC_PARSE", framework="pdfplumber") + +# Scanned PDFs / images — local OCR, no API key +parse("scanned.pdf", parser_type="STATIC_PARSE", framework="paddleocr") + +# LLM parsing +parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o") +parse("document.pdf", parser_type="LLM_PARSE", model="gemini-2.5-pro") +parse("document.pdf", parser_type="LLM_PARSE", model="claude-3-5-sonnet-20241022") +``` + +### AUTO routing with a priority + +```python +# Speed (default): static if no images, LLM otherwise +parse("doc.pdf", parser_type="AUTO", router_priority="speed") + +# Accuracy: prefers LLM, except PDFs with hidden hyperlinks +parse("doc.pdf", parser_type="AUTO", router_priority="accuracy") + +# Cost: tries PaddleOCR first; LLM fallback if extracted text is too short +parse("doc.pdf", parser_type="AUTO", router_priority="cost", character_threshold=100) + +# ML-based LLM auto-selection (uses lexoid/core/llm_selector.py) +parse("doc.pdf", parser_type="AUTO", autoselect_llm=True) +``` + +### Local inference (no API key) + +```python +# Ollama — Lexoid forces max_processes=1 for Ollama +parse("doc.pdf", parser_type="LLM_PARSE", + api_provider="ollama", model="gemma4:latest", max_processes=1) + +# SmolDocling / granite-docling +parse("doc.pdf", parser_type="LLM_PARSE", + api_provider="local", model="ds4sd/SmolDocling-256M-preview") + +# PaddleOCR-VL +parse("doc.pdf", parser_type="LLM_PARSE", + api_provider="local", model="PaddlePaddle/PaddleOCR-VL") +``` + +### Schema-based structured extraction + +```python +from lexoid.api import parse_with_schema +from pydantic import BaseModel + +class Invoice(BaseModel): + invoice_number: str + total: float + +# Per-page list of filled schemas +pages = parse_with_schema("invoice.pdf", schema=Invoice, model="gpt-4o-mini") + +# Single instance for the whole document +[full] = parse_with_schema("contract.pdf", schema=Invoice, + model="gpt-4o", fill_single_schema=True) + +# Dict schema with example data + alternate keys (improves match) +pages = parse_with_schema( + "invoice.pdf", + schema={"invoice_number": "string", "total": "number"}, + example_schema={"invoice_number": "INV-001", "total": 199.95}, + alternate_keys={"invoice_number": ["Invoice #", "Invoice No."]}, +) + +# Dataclass schemas also work +from dataclasses import dataclass +@dataclass +class Receipt: + merchant: str + amount: float + +parse_with_schema("receipt.pdf", schema=Receipt) +``` + +### LaTeX conversion + +```python +from lexoid.api import parse_to_latex +latex_source = parse_to_latex("paper.pdf", model="gpt-4o") +``` + +### URLs and recursive crawling + +```python +# Single page +parse("https://example.com") + +# Crawl 2 levels deep +parse("https://example.com", depth=2) + +# Render webpage → PDF first, then parse and keep the intermediate PDF +result = parse( + "https://example.com", + as_pdf=True, + save_dir="output/", + save_filename="example.pdf", +) +intermediate = result["pdf_path"] +``` + +### Bounding boxes + +```python +result = parse("doc.pdf", return_bboxes=True, bbox_framework="auto") +for seg in result["segments"]: + for text, bbox in seg.get("bboxes", []): + # bbox = [x0, top, x1, bottom], normalized [0, 1] + ... +``` + +### Audio + +Audio inputs require a Gemini model (the only provider with audio support). + +```python +result = parse("interview.mp3", model="gemini-2.5-flash") +print(result["raw"]) +``` + +### Token cost tracking + +```python +result = parse( + "doc.pdf", + model="gpt-4o", + api_cost_mapping="tests/api_cost_mapping.json", +) +print(result["token_cost"]) # {"input": ..., "output": ..., "input-image": ..., "total": ...} +``` + +## Key kwargs reference + +| kwarg | Purpose | +|-------------------------|-------------------------------------------------------------------------| +| `model` | LLM model name (default from `DEFAULT_LLM`, falls back to `gemini-2.5-flash`). | +| `api_provider` | Override inferred provider. | +| `framework` | `pdfplumber` / `pdfminer` / `paddleocr` for STATIC_PARSE. | +| `temperature` | LLM sampling temperature (default `0.0`). | +| `max_tokens` | LLM output token limit (default `1024`, `4096` for Ollama). | +| `pages_per_split` | Pages per parallel chunk. | +| `max_processes` | Parallel workers (forced to 1 for Ollama). | +| `page_nums` | Specific 1-indexed pages to parse (PDFs only). | +| `depth` | Recursive URL parsing depth. | +| `as_pdf` | Convert input to PDF before parsing. | +| `save_dir` | Where to keep the intermediate PDF if `as_pdf=True`. | +| `return_bboxes` | Attach bounding boxes per segment. | +| `bbox_framework` | `auto` / `pdfplumber` / `paddleocr`. | +| `router_priority` | `speed` / `accuracy` / `cost` for AUTO mode. | +| `character_threshold` | Min char count for STATIC accept under `cost` priority. | +| `autoselect_llm` | ML-based LLM choice in AUTO mode. | +| `retry_on_fail` | Fall back to alternate parser on error (default `True`). | +| `max_image_dimension` | Max px to which images / page renders are downscaled. | +| `api_cost_mapping` | Dict or JSON path with per-model cost — enables `token_cost` in output. | +| `system_prompt` / `user_prompt` | Override the default LLM prompts. | +| `verbose` | Verbose logging during LLM parsing. | + +## Things to verify before reporting success + +- The result dict has non-empty `raw`. Empty `raw` with an `error` key means a recoverable failure occurred and Lexoid returned a stub. +- For LLM_PARSE, `token_usage["total"]` is non-zero — zero suggests the API call silently failed. +- For multi-page PDFs, `len(result["segments"])` matches the expected page count (or `len(page_nums)` if used). +- For `parse_with_schema`, each returned dict actually matches the schema keys — the LLM can drift; consider `example_schema` to anchor it. + +## See also + +- API reference: `docs/api.rst`. +- CLI equivalent: `lexoid-cli` skill. +- Example notebooks: `examples/example_notebook.ipynb`, `examples/example_notebook_colab.ipynb`. From fe21df4d13044129ba1064ace95e23cef46759ef Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Sun, 24 May 2026 22:06:23 -0300 Subject: [PATCH 3/7] Add condition to deploy job for main branch only --- .github/workflows/deploy_docs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml index c84dd4d..5eca369 100644 --- a/.github/workflows/deploy_docs.yml +++ b/.github/workflows/deploy_docs.yml @@ -40,6 +40,7 @@ jobs: deploy: needs: build-docs + if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest permissions: pages: write From f0dc9aa9e34a6463995b248bd696b2043ae1becc Mon Sep 17 00:00:00 2001 From: Dilith Jayakody <54039395+dilithjay@users.noreply.github.com> Date: Mon, 25 May 2026 19:02:33 -0300 Subject: [PATCH 4/7] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api.rst b/docs/api.rst index d08ff93..de68653 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -80,7 +80,7 @@ parse * ``parent_title``: Title of the parent document, if recursively parsed. * ``recursive_docs``: List of dictionaries for recursively-parsed sub-documents (when ``depth > 1``). * ``token_usage``: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. - * ``token_cost``: Estimated cost per token category (only when ``api_cost_mapping`` is supplied). + * ``token_cost``: Estimated cost per token category (only when ``api_cost_mapping`` is supplied and contains an entry for the resolved model). * ``parsers_used``: List of parser names actually used for each chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). * ``pdf_path``: Path to the intermediate PDF generated when ``as_pdf=True`` and ``save_dir`` is specified. * ``error``: Present only when an unrecoverable error occurred (parsing returned a fallback empty result). From b6ac34f052e606577bc8d8a0deb007f69d54cc6c Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Mon, 25 May 2026 19:28:24 -0300 Subject: [PATCH 5/7] Address copilot reviews --- docs/api.rst | 58 +++++++++++++++++++++++------------ docs/benchmark.rst | 6 +++- skills/lexoid-python/SKILL.md | 48 +++++++++++++++++++++-------- 3 files changed, 78 insertions(+), 34 deletions(-) diff --git a/docs/api.rst b/docs/api.rst index de68653..e3b6120 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -36,9 +36,13 @@ parse :param path: File path or URL to parse. :param parser_type: Parser type to use (``"LLM_PARSE"``, ``"STATIC_PARSE"``, or ``"AUTO"``). Default: ``"AUTO"``. :param pages_per_split: Number of pages per chunk for processing. Default: ``4``. - :param max_processes: Maximum number of parallel processes. Default: ``4``. Automatically forced to ``1`` when ``api_provider="ollama"``. + :param max_processes: Maximum number of parallel processes. Default: ``4``. Automatically forced to ``1`` when ``parser_type="LLM_PARSE"`` and ``api_provider="ollama"``. :param kwargs: Additional keyword arguments (see below). - :return: Dictionary containing parsed document data (see "Return value format" below). + + Automatic input handling: + + * ``.doc`` / ``.docx`` inputs with any ``parser_type`` other than ``STATIC_PARSE`` are first converted to PDF (``as_pdf`` is forced to ``True``). + * ``.xlsx`` and ``.pptx`` inputs with ``parser_type="LLM_PARSE"`` are silently switched to ``STATIC_PARSE`` (a warning is logged) because LLM parsing is not supported for these formats. Additional keyword arguments: @@ -63,7 +67,7 @@ parse - ``"speed"`` (default): Uses ``STATIC_PARSE`` for PDFs without images, else ``LLM_PARSE``. - ``"accuracy"``: Prefers ``LLM_PARSE``, except for PDFs with no images but with embedded/hidden hyperlinks (uses ``STATIC_PARSE`` since LLMs miss hidden links). - - ``"cost"``: Tries PaddleOCR first; if the extracted character count is below ``character_threshold``, returns it; otherwise falls back to ``LLM_PARSE``. + - ``"cost"``: For PDFs that contain images, tries PaddleOCR first; if the extracted character count is below ``character_threshold`` the PaddleOCR result is returned, otherwise the document is re-parsed with ``LLM_PARSE``. PDFs without images, and non-PDF inputs, fall back to the same routing as ``"speed"``. * ``character_threshold`` (int): Minimum character count for a ``router_priority="cost"`` STATIC_PARSE result to be accepted. Default: ``100``. * ``autoselect_llm`` (bool): When ``parser_type="AUTO"``, runs the ML-based ``DocumentRankedLLMSelector`` to choose the best LLM for the input document. Default: ``False``. * ``retry_on_fail`` (bool): When ``True`` (default), automatically retries with the alternate parser type / framework on failure. @@ -71,30 +75,36 @@ parse * ``bbox_framework`` (str): Framework used for bounding box extraction when ``return_bboxes=True``. One of ``"auto"`` (default — chooses ``paddleocr`` or ``pdfplumber`` based on file content), ``"pdfplumber"``, or ``"paddleocr"``. Return value format: - A dictionary containing a subset or all of the following keys: + A dictionary with the following keys. Keys marked *(optional)* are only present in specific configurations; all others are always present (and may hold an empty string / list / zeroed dict). * ``raw``: Full markdown content as a string. - * ``segments``: List of dictionaries with per-segment ``metadata`` (e.g., ``page``) and ``content``. For PDFs, a segment is a page; for webpages, a segment is a section (heading and its content). When ``return_bboxes=True``, each segment also has a ``bboxes`` key. - * ``title``: Title of the document. - * ``url``: Original URL if applicable. - * ``parent_title``: Title of the parent document, if recursively parsed. - * ``recursive_docs``: List of dictionaries for recursively-parsed sub-documents (when ``depth > 1``). - * ``token_usage``: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. - * ``token_cost``: Estimated cost per token category (only when ``api_cost_mapping`` is supplied and contains an entry for the resolved model). - * ``parsers_used``: List of parser names actually used for each chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). - * ``pdf_path``: Path to the intermediate PDF generated when ``as_pdf=True`` and ``save_dir`` is specified. - * ``error``: Present only when an unrecoverable error occurred (parsing returned a fallback empty result). + * ``segments``: List of dictionaries with per-segment ``metadata`` (e.g., ``page``) and ``content``. For PDFs, a segment is a page; for webpages, a segment is a section (heading and its content). When ``return_bboxes=True``, each segment additionally carries a ``bboxes`` key (a list of ``(text, [x0, top, x1, bottom])`` tuples normalized to ``[0, 1]``). + * ``title``: Title of the document (defaults to the input file's basename). + * ``url``: Original URL if the input was a URL, otherwise an empty string. + * ``parent_title``: Title of the parent document when this result was produced by recursive crawling; otherwise an empty string. + * ``recursive_docs``: List of recursively-parsed sub-documents. Always present; empty unless ``depth > 1``. + * ``token_usage``: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. Counts are zero when only ``STATIC_PARSE`` ran. + * ``parsers_used``: List of parser names that actually ran, one entry per chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). + * ``token_cost`` *(optional)*: Estimated cost broken down by token category. Only present when ``api_cost_mapping`` is supplied and contains an entry for the resolved model. + * ``pdf_path`` *(optional)*: Path to the intermediate PDF generated when ``as_pdf=True``. To keep the file readable after ``parse()`` returns, also pass ``save_dir`` — otherwise the PDF is written inside a temporary directory that is removed on return. parse_with_schema ^^^^^^^^^^^^^^^^^ -.. py:function:: lexoid.api.parse_with_schema(path: str, schema: Union[Dict, Type], api: Optional[str] = None, model: str = "gpt-4o-mini", example_schema: Optional[Dict] = None, alternate_keys: Optional[Dict] = None, fill_single_schema: bool = False, **kwargs) -> List[List[Dict]] +.. py:function:: lexoid.api.parse_with_schema(path: str, schema: Union[Dict, Type], api: Optional[str] = None, model: str = "gpt-4o-mini", example_schema: Optional[Dict] = None, alternate_keys: Optional[Dict] = None, fill_single_schema: bool = False, **kwargs) -> List - Parse a PDF (or image) using an LLM to generate structured output - conforming to a given schema. The schema can be a plain ``dict``, a - Python ``dataclass``, or a Pydantic ``BaseModel`` (all are converted to - JSON Schema internally). + Parse a document with an LLM to generate structured output conforming + to a given schema. The schema can be a plain ``dict``, a Python + ``dataclass``, or a Pydantic ``BaseModel`` (all are converted to JSON + Schema internally). + + In the default per-page mode, only PDF and image inputs are accepted + (each page is rendered to an image and sent to the LLM together with + the schema prompt). When ``fill_single_schema=True``, the document is + first parsed with :py:func:`parse` and the resulting markdown is sent + to the LLM, so the broader set of file types supported by ``parse()`` + (DOCX, HTML, URLs, etc.) becomes available. :param path: Path to the file to parse. :param schema: ``dict``, ``dataclass``, or Pydantic ``BaseModel`` describing the desired output. @@ -104,13 +114,21 @@ parse_with_schema :param alternate_keys: Optional mapping of alternate key names that may appear in the document — helps the model match synonyms. :param fill_single_schema: When ``True``, the entire document is parsed once and the whole content is used to produce a single instance of the schema (rather than one instance per page). :param kwargs: Additional keyword arguments (e.g., ``temperature``, ``max_tokens``). - :return: A list of dictionaries — one per page when ``fill_single_schema=False``, or a single-element list when ``fill_single_schema=True``. Each entry conforms to the provided schema. Additional keyword arguments: * ``temperature`` (float): Sampling temperature for LLM generation. Default: ``0.0``. * ``max_tokens`` (int): Maximum number of tokens to generate. Default: ``1024``. + Return value format: + + The function returns a Python list of the JSON values parsed from the model. The exact shape depends on the mode and on the schema: + + * **Default (per-page) mode** — one entry per page, in page order. Each entry is the JSON value the model produced for that page. If the schema describes a single record, expect one ``dict`` per page; if the schema describes multiple records per page (e.g., table rows), expect a ``list`` of ``dict``\ s per page. Concretely, indexing follows ``result[page_index][record_index]`` when each page contains multiple records. + * **``fill_single_schema=True``** — a single-element list whose lone element is the JSON value the model produced for the whole document (typically a single ``dict``). + + Because the shape is driven by the model's output, callers that need to support both single-record and multi-record schemas should normalize the per-page entries themselves (e.g., wrap ``dict`` entries in a one-element ``list``). + parse_to_latex ^^^^^^^^^^^^^^ diff --git a/docs/benchmark.rst b/docs/benchmark.rst index 51e5e5c..054a975 100644 --- a/docs/benchmark.rst +++ b/docs/benchmark.rst @@ -64,7 +64,11 @@ You can modify the ``test_attributes`` list in the ``main()`` function to test d * ``model``: Test different LLM models * ``framework``: Test different static parsing frameworks (``pdfplumber``, ``pdfminer``, ``paddleocr``) * ``pages_per_split``: Adjust document chunking -* ``max_threads``: Number of parallel workers (mapped to ``max_processes`` in the public :py:func:`lexoid.api.parse` API) + +.. note:: + + The benchmark harness currently hard-codes ``max_processes=1`` when calling :py:func:`lexoid.api.parse`, so configurations under the ``max_threads`` sweep knob in ``benchmark.py`` do not actually change + ``parse()``'s parallelism. To benchmark parallelism, edit ``tests/benchmark.py`` to forward the sweep value to ``max_processes``. Benchmark Results ----------------- diff --git a/skills/lexoid-python/SKILL.md b/skills/lexoid-python/SKILL.md index 3c67009..81477c7 100644 --- a/skills/lexoid-python/SKILL.md +++ b/skills/lexoid-python/SKILL.md @@ -43,7 +43,7 @@ API keys by provider: Four entry points in `lexoid.api`: - `parse(path, parser_type="AUTO", pages_per_split=4, max_processes=4, **kwargs)` — main function. Returns a dict. -- `parse_with_schema(path, schema, api=None, model="gpt-4o-mini", **kwargs)` — structured JSON extraction. Returns a list of dicts. +- `parse_with_schema(path, schema, api=None, model="gpt-4o-mini", **kwargs)` — structured JSON extraction. Returns a Python list whose shape depends on the mode and the model's output (see "Schema return shape" below). - `parse_to_latex(path, api=None, model="gpt-4o-mini", **kwargs)` — returns a LaTeX string. - `parse_chunk(path, parser_type, **kwargs)` — low-level single-chunk parser; users rarely need this. @@ -54,18 +54,20 @@ Four entry points in `lexoid.api`: ```python { "raw": str, # full markdown - "segments": [ # one dict per page / section - {"metadata": {"page": int}, "content": str, "bboxes": [...]}, + "segments": [ # one dict per page / section; always present (may be empty) + # bboxes is included only when return_bboxes=True + {"metadata": {"page": int}, "content": str, "bboxes": [(text, [x0, top, x1, bottom]), ...]}, ... ], "title": str, - "url": str, # if input was a URL - "parent_title": str, # set on recursive sub-docs - "recursive_docs": [...], # populated when depth > 1 - "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, - "token_cost": {...}, # only when api_cost_mapping is supplied + "url": str, # input URL or "" if input was a local file + "parent_title": str, # parent doc title when recursive; "" otherwise + "recursive_docs": [...], # always present; empty unless depth > 1 + "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, # zeros under STATIC_PARSE only "parsers_used": [str, ...], # which parser ran per chunk - "pdf_path": str, # only when as_pdf=True and save_dir is set + # --- optional keys below --- + "token_cost": {...}, # only when api_cost_mapping is supplied + "pdf_path": str, # only when as_pdf=True; file is removed unless save_dir is also set } ``` @@ -131,6 +133,23 @@ parse("doc.pdf", parser_type="LLM_PARSE", ### Schema-based structured extraction +#### Schema return shape + +`parse_with_schema` returns a Python `list`. The exact shape depends on +the mode and on what JSON the model emits: + +- **Default (per-page) mode** — one entry per page. Each entry is the JSON + the model returned for that page: a single `dict` for single-record + schemas, or a `list` of `dict`s for multi-record schemas (e.g., a page + of table rows). Index as `result[page_index]` for the page's value, and + `result[page_index][record_index]` when each page has multiple records. +- **`fill_single_schema=True`** — a single-element list whose element is + the JSON for the whole document (typically one `dict`). + +If you need to support both single- and multi-record schemas in the same +code path, normalize the per-page entries yourself (e.g., wrap a `dict` +in `[dict]`). + ```python from lexoid.api import parse_with_schema from pydantic import BaseModel @@ -139,10 +158,13 @@ class Invoice(BaseModel): invoice_number: str total: float -# Per-page list of filled schemas +# Per-page extraction. `pages` is a list with one entry per page; +# each entry is whatever JSON the model returned for that page. pages = parse_with_schema("invoice.pdf", schema=Invoice, model="gpt-4o-mini") +# e.g., pages[0] -> {"invoice_number": "...", "total": ...} +# pages[0][0] -> first record if the model returned a list per page -# Single instance for the whole document +# Single instance for the whole document — returns a one-element list. [full] = parse_with_schema("contract.pdf", schema=Invoice, model="gpt-4o", fill_single_schema=True) @@ -230,7 +252,7 @@ print(result["token_cost"]) # {"input": ..., "output": ..., "input-image": ..., | `temperature` | LLM sampling temperature (default `0.0`). | | `max_tokens` | LLM output token limit (default `1024`, `4096` for Ollama). | | `pages_per_split` | Pages per parallel chunk. | -| `max_processes` | Parallel workers (forced to 1 for Ollama). | +| `max_processes` | Parallel workers (forced to 1 when `parser_type="LLM_PARSE"` and `api_provider="ollama"`). | | `page_nums` | Specific 1-indexed pages to parse (PDFs only). | | `depth` | Recursive URL parsing depth. | | `as_pdf` | Convert input to PDF before parsing. | @@ -251,7 +273,7 @@ print(result["token_cost"]) # {"input": ..., "output": ..., "input-image": ..., - The result dict has non-empty `raw`. Empty `raw` with an `error` key means a recoverable failure occurred and Lexoid returned a stub. - For LLM_PARSE, `token_usage["total"]` is non-zero — zero suggests the API call silently failed. - For multi-page PDFs, `len(result["segments"])` matches the expected page count (or `len(page_nums)` if used). -- For `parse_with_schema`, each returned dict actually matches the schema keys — the LLM can drift; consider `example_schema` to anchor it. +- For `parse_with_schema`, each per-page entry may be a `dict` or a `list` of `dict`s depending on the schema and the model. Check the type before indexing, and verify that the keys actually match the schema — the LLM can drift; pass `example_schema` to anchor it. ## See also From 88be1b43183e452d8077c88130fce1dbcb10c9b5 Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Mon, 25 May 2026 19:41:59 -0300 Subject: [PATCH 6/7] Address copilot reviews --- docs/api.rst | 10 +++++----- skills/lexoid-cli/SKILL.md | 3 ++- skills/lexoid-python/SKILL.md | 20 +++++++++++++++----- 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/api.rst b/docs/api.rst index e3b6120..c84992b 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -75,16 +75,16 @@ parse * ``bbox_framework`` (str): Framework used for bounding box extraction when ``return_bboxes=True``. One of ``"auto"`` (default — chooses ``paddleocr`` or ``pdfplumber`` based on file content), ``"pdfplumber"``, or ``"paddleocr"``. Return value format: - A dictionary with the following keys. Keys marked *(optional)* are only present in specific configurations; all others are always present (and may hold an empty string / list / zeroed dict). + A dictionary with the following keys. Keys marked *(optional)* are only present in specific configurations; the others are present on the standard parsing path (and may hold an empty string / list / zeroed dict). * ``raw``: Full markdown content as a string. * ``segments``: List of dictionaries with per-segment ``metadata`` (e.g., ``page``) and ``content``. For PDFs, a segment is a page; for webpages, a segment is a section (heading and its content). When ``return_bboxes=True``, each segment additionally carries a ``bboxes`` key (a list of ``(text, [x0, top, x1, bottom])`` tuples normalized to ``[0, 1]``). * ``title``: Title of the document (defaults to the input file's basename). * ``url``: Original URL if the input was a URL, otherwise an empty string. * ``parent_title``: Title of the parent document when this result was produced by recursive crawling; otherwise an empty string. - * ``recursive_docs``: List of recursively-parsed sub-documents. Always present; empty unless ``depth > 1``. - * ``token_usage``: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. Counts are zero when only ``STATIC_PARSE`` ran. - * ``parsers_used``: List of parser names that actually ran, one entry per chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). + * ``recursive_docs``: List of recursively-parsed sub-documents. Empty unless ``depth > 1``. + * ``token_usage`` *(optional)*: Dictionary with ``input``, ``output``, ``total``, and ``llm_page_count`` token statistics. Counts are zero when only ``STATIC_PARSE`` ran. **Absent on the HTML/recursive-URL path** — when ``path`` is a URL that is not a supported file-typed URL (e.g., ``.pdf``/image) and ``as_pdf`` is not set, ``parse()`` returns the output of ``recursive_read_html`` directly, which does not include this key. + * ``parsers_used`` *(optional)*: List of parser names that actually ran, one entry per chunk (e.g., ``["LLM_PARSE", "STATIC_PARSE"]``). **Absent on the HTML/recursive-URL path** for the same reason as ``token_usage``. * ``token_cost`` *(optional)*: Estimated cost broken down by token category. Only present when ``api_cost_mapping`` is supplied and contains an entry for the resolved model. * ``pdf_path`` *(optional)*: Path to the intermediate PDF generated when ``as_pdf=True``. To keep the file readable after ``parse()`` returns, also pass ``save_dir`` — otherwise the PDF is written inside a temporary directory that is removed on return. @@ -124,7 +124,7 @@ parse_with_schema The function returns a Python list of the JSON values parsed from the model. The exact shape depends on the mode and on the schema: - * **Default (per-page) mode** — one entry per page, in page order. Each entry is the JSON value the model produced for that page. If the schema describes a single record, expect one ``dict`` per page; if the schema describes multiple records per page (e.g., table rows), expect a ``list`` of ``dict``\ s per page. Concretely, indexing follows ``result[page_index][record_index]`` when each page contains multiple records. + * **Default (per-page) mode** — one entry per page, in page order. Each entry is the JSON value the model produced for that page. If the schema describes a single record, expect one ``dict`` per page; if the schema describes multiple records per page (e.g., table rows), expect a ``list`` of ``dict``\s per page. Concretely, indexing follows ``result[page_index][record_index]`` when each page contains multiple records. * **``fill_single_schema=True``** — a single-element list whose lone element is the JSON value the model produced for the whole document (typically a single ``dict``). Because the shape is driven by the model's output, callers that need to support both single-record and multi-record schemas should normalize the per-page entries themselves (e.g., wrap ``dict`` entries in a one-element ``list``). diff --git a/skills/lexoid-cli/SKILL.md b/skills/lexoid-cli/SKILL.md index 2d5f698..d9201d8 100644 --- a/skills/lexoid-cli/SKILL.md +++ b/skills/lexoid-cli/SKILL.md @@ -22,7 +22,8 @@ Before invoking, confirm: 1. `lexoid` is installed (`lexoid --help` or `python -m lexoid --help`). If not, run `pip install lexoid`. 2. For LLM-based commands, the relevant API key env var is set: - `GOOGLE_API_KEY` (Gemini, default), `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `MISTRAL_API_KEY`, `HUGGINGFACEHUB_API_TOKEN`, `TOGETHER_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`. - - Ollama / local backends need no key but need a running server / pulled model. + - Ollama needs no key, but needs `ollama serve` running at `OLLAMA_BASE_URL` (default `http://localhost:11434`) and the target model pulled (`ollama pull `). + - Local backends (SmolDocling/granite-docling, PaddleOCR-VL) need no key and no server — they run in-process; the first call downloads weights from Hugging Face. 3. For Linux DOCX → PDF, LibreOffice (`lowriter`) must be installed. ## Commands diff --git a/skills/lexoid-python/SKILL.md b/skills/lexoid-python/SKILL.md index 81477c7..f2f66a1 100644 --- a/skills/lexoid-python/SKILL.md +++ b/skills/lexoid-python/SKILL.md @@ -21,7 +21,8 @@ Before writing code, confirm: 1. `lexoid` is installed (`pip install lexoid`). 2. Required API key env vars are set for the chosen provider (see below). 3. For Linux DOCX → PDF conversion, LibreOffice (`lowriter`) is on PATH. -4. For Ollama / local LLMs, the server is running and the target model is pulled. +4. For `api_provider="ollama"`: an `ollama serve` process is running at `OLLAMA_BASE_URL` (default `http://localhost:11434`) and the target model has been pulled with `ollama pull `. +5. For `api_provider="local"` (SmolDocling/granite-docling, PaddleOCR-VL): **no server needed** — these models run in-process via `transformers` / PaddleOCR. The first call downloads weights from Hugging Face, so the host needs network access (or pre-cached weights) and enough disk/RAM/GPU for the chosen model. API keys by provider: @@ -54,7 +55,7 @@ Four entry points in `lexoid.api`: ```python { "raw": str, # full markdown - "segments": [ # one dict per page / section; always present (may be empty) + "segments": [ # one dict per page / section; may be empty # bboxes is included only when return_bboxes=True {"metadata": {"page": int}, "content": str, "bboxes": [(text, [x0, top, x1, bottom]), ...]}, ... @@ -62,15 +63,24 @@ Four entry points in `lexoid.api`: "title": str, "url": str, # input URL or "" if input was a local file "parent_title": str, # parent doc title when recursive; "" otherwise - "recursive_docs": [...], # always present; empty unless depth > 1 - "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, # zeros under STATIC_PARSE only - "parsers_used": [str, ...], # which parser ran per chunk + "recursive_docs": [...], # empty unless depth > 1 # --- optional keys below --- + "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, + # zeros under STATIC_PARSE-only; ABSENT on the HTML/recursive-URL path + # (URL input that isn't a file-typed URL and as_pdf is not set) + "parsers_used": [str, ...], # ABSENT on the HTML/recursive-URL path (same condition) "token_cost": {...}, # only when api_cost_mapping is supplied "pdf_path": str, # only when as_pdf=True; file is removed unless save_dir is also set } ``` +For URL inputs that resolve to HTML (no `.pdf`/image extension, and +`as_pdf=False`), `parse()` short-circuits to `recursive_read_html()`, +which returns only `raw`, `segments`, `title`, `url`, `parent_title`, +and `recursive_docs`. Code that always reads `result["token_usage"]` or +`result["parsers_used"]` will `KeyError` on that path — use `.get(...)` +or guard with `if "token_usage" in result`. + ## Common recipes ### Basic parsing From ebc3f8a14fcf0995b457ca56680b681e233e110c Mon Sep 17 00:00:00 2001 From: Dilith Jayakody Date: Sun, 31 May 2026 17:18:34 -0300 Subject: [PATCH 7/7] Update skills.md files --- skills/lexoid-cli/SKILL.md | 37 +++++++++++++++++++++++++++-------- skills/lexoid-python/SKILL.md | 18 +++++++++++++++++ 2 files changed, 47 insertions(+), 8 deletions(-) diff --git a/skills/lexoid-cli/SKILL.md b/skills/lexoid-cli/SKILL.md index d9201d8..d009a74 100644 --- a/skills/lexoid-cli/SKILL.md +++ b/skills/lexoid-cli/SKILL.md @@ -26,6 +26,21 @@ Before invoking, confirm: - Local backends (SmolDocling/granite-docling, PaddleOCR-VL) need no key and no server — they run in-process; the first call downloads weights from Hugging Face. 3. For Linux DOCX → PDF, LibreOffice (`lowriter`) must be installed. +### Loading API keys from `.env` + +API keys are commonly stored in a `.env` file at the project root rather than exported in the shell. The `lexoid` CLI does **not** auto-load `.env`, so for any LLM-based command (`LLM_PARSE`, `schema`, `latex`, or `AUTO` when it routes to an LLM) you must load it yourself. + +**Always load `.env` in a subshell so the keys never leak into the surrounding environment** — the parentheses scope the `export`s to that one command, restoring the environment to its previous state automatically afterward. Guard the source with `[ -f .env ]` so the command still runs (using already-exported keys) when no `.env` is present: + +```bash +# Load .env for this command only (if it exists); env is reset to its prior state on exit +( set -a; [ -f .env ] && . ./.env; set +a; lexoid parse --input document.pdf --parser-type LLM_PARSE --model gpt-4o ) +``` + +Do **not** run a bare `set -a; . ./.env; set +a` in the parent shell — that persists secrets into the session. + +**The examples in the rest of this skill are written as plain `lexoid …` for readability.** Apply the wrapper above to any LLM-based command (`LLM_PARSE`, `schema`, `latex`, or `AUTO` when it routes to an LLM). If your keys are already exported in the environment, run the commands as-is. + ## Commands ### `lexoid parse` — Markdown / JSON output @@ -42,14 +57,14 @@ lexoid parse --input document.pdf --output output.md # Full result as JSON (includes per-page segments, token usage, parsers_used) lexoid parse --input document.pdf --format json --output result.json +# Explicit LLM parsing (forces an LLM regardless of routing). LLM-based: load .env first. +lexoid parse --input document.pdf --parser-type LLM_PARSE --model gpt-4o +lexoid parse --input document.pdf --parser-type LLM_PARSE --model claude-3-5-sonnet-20241022 +lexoid parse --input scanned.pdf --parser-type LLM_PARSE --model mistral-ocr-latest + # Force static parsing (no LLM, no API key needed for PDFs) lexoid parse --input document.pdf --parser-type STATIC_PARSE --framework pdfplumber -# LLM parsing with a specific model -lexoid parse --input document.pdf --model gpt-4o -lexoid parse --input document.pdf --model claude-3-5-sonnet-20241022 -lexoid parse --input scanned.pdf --model mistral-ocr-latest - # Parse a URL lexoid parse --input https://example.com --output page.md @@ -63,6 +78,8 @@ Key flags: `--parser-type` (`AUTO`/`LLM_PARSE`/`STATIC_PARSE`), `--model`, `--fr Extract data conforming to a JSON schema. Schema can be a file path or inline JSON. +All `schema` commands are LLM-based — load `.env` first (see "Loading API keys from .env") unless keys are already exported. + ```bash # Inline schema lexoid schema \ @@ -87,6 +104,8 @@ Defaults: model `gpt-4o-mini`. The provider is auto-detected from the model name Convert a document to a self-contained LaTeX source. +LaTeX conversion is LLM-based — load `.env` first (see "Loading API keys from .env") unless keys are already exported. + ```bash lexoid latex --input paper.pdf --output paper.tex lexoid latex --input paper.pdf --model gpt-4o @@ -103,14 +122,16 @@ lexoid parse --input report.pdf --format json | jq '.token_usage' ## Common patterns -- **Cheap path first**: try `--parser-type STATIC_PARSE --framework pdfplumber` for native-text PDFs. Fall back to `--parser-type LLM_PARSE` for scans, tables-heavy pages, or hand-written content. -- **Scanned PDFs / images**: use `--framework paddleocr` (no API key) or an LLM model with vision. +- **Default to `AUTO`**: with no `--parser-type`, the CLI uses `AUTO`, which inspects the document and routes to the best parser (often an LLM for scans, charts, or complex tables). This is the right choice unless the user asks otherwise. +- **`STATIC_PARSE` is opt-in, not the default**: choose it when the user explicitly wants no API calls / no cost, or you know the input is a clean native-text PDF. It returns empty output on scanned/image-only pages, so it is not a safe first guess for unknown documents. +- **`LLM_PARSE` for quality**: force it with `--parser-type LLM_PARSE --model ` for scans, chart/figure-heavy pages, or messy tables where layout fidelity matters. +- **Scanned PDFs / images**: use `--parser-type STATIC_PARSE --framework paddleocr` (no API key) or an LLM model with vision. - **Batch**: drive the CLI from a shell loop (`for f in inputs/*.pdf; do lexoid parse -i "$f" -o "out/${f%.pdf}.md"; done`). - **Debug**: add `--verbose` to surface loguru logs to stderr. ## Failure modes to watch for -- Missing API key → CLI raises a clean error naming the env var. Set it and retry. +- Missing API key → CLI raises a clean error naming the env var. The CLI does not auto-load `.env`; load it via the subshell wrapper (see "Loading API keys from .env") and retry. - DOCX on Linux without LibreOffice installed → conversion to PDF fails. Install `libreoffice`. - `--model` and `--api` mismatch → use `--api` only to override an auto-inferred provider (e.g., to send a model through OpenRouter). - Ollama: must run `ollama serve` and `ollama pull ` first; the CLI does not start the server. diff --git a/skills/lexoid-python/SKILL.md b/skills/lexoid-python/SKILL.md index f2f66a1..1030ca3 100644 --- a/skills/lexoid-python/SKILL.md +++ b/skills/lexoid-python/SKILL.md @@ -39,6 +39,24 @@ API keys by provider: | `ollama` | none (uses `OLLAMA_BASE_URL`) | | `local` | none | +### Loading API keys from `.env` + +Lexoid reads API keys from the process environment; it does **not** load a `.env` file on its own. When keys live in a project `.env`, load them before calling any LLM-based API (`LLM_PARSE`, `parse_with_schema`, `parse_to_latex`, or `AUTO` when it routes to an LLM). `python-dotenv` ships as a Lexoid dependency, so it is already available: + +```python +from dotenv import load_dotenv + +# Loads .env from the current dir (or a parent) into os.environ if the file +# exists; a no-op that returns False when no .env is found. Existing env vars +# are not overwritten unless override=True is passed. +load_dotenv() + +from lexoid.api import parse +result = parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o") +``` + +Call `load_dotenv()` once at program/notebook startup, before the first `parse(...)` call. If the keys are already exported in the environment, this step is unnecessary (and harmless). + ## Public API Four entry points in `lexoid.api`: