Skip to content

Improvements to CLI usability #66

Description

@brainless

Since are building our crawling to detect text templates or HTML node duplicates across a domain, let us separate the preparation and the crawler stages in the CLI.

Tasks

  • Change CLI argument --link to --domain where a user can pass a URL or domain without protocol or path
  • Extract domain name from --domain argument, trim
  • Create a CLI argument --prep, remove --verbose and --template arguments
  • Add tests for these changes, update tests for features being removed

CLI argument --prep

  • In this mode, we will crawl up to 10 pages of this domain (instead of current 3)
  • Create store, if it does exist, for detected templates and element's path they were found at
  • Element's path should be from HTML root of the page and should contain tag name and class list for each element in the path (no id)
  • Run the template detection logic across all crawled pages
  • Print all detected paths, de-duplicated from all pages at the end
  • The print out should be a serialized format that can be used by Rust

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions