Since are building our crawling to detect text templates or HTML node duplicates across a domain, let us separate the preparation and the crawler stages in the CLI.
Tasks
- Change CLI argument
--link to --domain where a user can pass a URL or domain without protocol or path
- Extract domain name from
--domain argument, trim
- Create a CLI argument
--prep, remove --verbose and --template arguments
- Add tests for these changes, update tests for features being removed
CLI argument --prep
- In this mode, we will crawl up to 10 pages of this domain (instead of current 3)
- Create store, if it does exist, for detected templates and element's path they were found at
- Element's path should be from HTML root of the page and should contain tag name and class list for each element in the path (no id)
- Run the template detection logic across all crawled pages
- Print all detected paths, de-duplicated from all pages at the end
- The print out should be a serialized format that can be used by Rust
Since are building our crawling to detect text templates or HTML node duplicates across a domain, let us separate the preparation and the crawler stages in the CLI.
Tasks
--linkto--domainwhere a user can pass a URL or domain without protocol or path--domainargument, trim--prep, remove--verboseand--templateargumentsCLI argument
--prep