Rewrite hf-cli into a full Hugging Face discovery tool

[tamnd]

Rewrite hf-cli into a full Hugging Face discovery tool

Spec: ~/notes/Spec/3001/ (00 overview, 01 surfaces, 02 html, 03 model, 04 graph, 05 commands, 06 implementation).

The goal is to read everything huggingface.co publishes and turn it into structured, linked data.
Two output planes: records (one shape per entity) and a graph (nodes and edges with stable hf:// URIs).
The API is the primary source and the rendered HTML is the fallback, so no field on a page is lost just because no endpoint returns it.

This issue is the checklist. It gets ticked as PRs land.

M0 foundations

• Drop the scaffold: hf/hf.go and hf/domain.go and their tests
• Point the domain at the real host, huggingface.co, with hf.co and www. as aliases
• Meta envelope on every record: uri, url, kind, sources, fetchedAt, aliasOf
• Extra map[string]json.RawMessage on every record, filled by a shared decoder
• Gated three state decode: false, "auto", "manual"
• StringList decode for card fields that are a scalar or a list
• Time decode for the several timestamp shapes the hub sends
• URI space: URI, SplitURI, composite ids for discussion, commit, ref, file, split
• Classify for every input form: hf://, hub URL, arxiv URL, @handle, bare id
• Locate for every kind
• Client: auth from --token, HF_TOKEN, HUGGING_FACE_HUB_TOKEN, $HF_HOME/token
• Client: shared pacer so N workers make one polite stream
• Client: retry on 429 and 5xx, honour Retry-After, exponential backoff with jitter
• Client: on disk response cache, long TTL for taxonomy, no expiry for sha pinned URLs
• Cursor pagination over the Link header, stop on missing next rather than a short page
• Status to exit code mapping: 2 usage, 3 no results, 4 auth, 5 rate limit, 6 not found, 7 unsupported, 8 network
• Redirect detection so a legacy id fills aliasOf

M1 repositories

• Repo base shared by model, dataset, space, kernel
• Model with config, transformersInfo, safetensors, gguf, modelIndex, inference providers
• EvalResult flattened one row per metric out of model-index
• Dataset with promoted card fields, configs, datasetInfo
• Space with runtime, hardware, replicas, domains
• Kernel
• Card front matter parse, universal plus model plus dataset plus space keys
• Card parse failure is not fatal, record keeps going with cardError
• hf model, hf dataset, hf space, hf kernel, hf get
• Expand parameter sets per repo kind, verified against the server's own 400 error list

M2 lists and search

• hf models, hf datasets, hf spaces, hf kernels with full cursor pagination
• Filter sugar: --task, --library, --language, --license, --sdk all folding into filter=
• --author, --search, --sort, --direction, --gated, --full
• hf search across every entity kind, with per type counts
• hf trending
• hf tags, the controlled vocabulary by type
• hf tasks
• hf sitemap

M3 namespaces, collections, papers

• User and Org records with every count field
• UserRef name normalisation, the user and name keys are the same value
• Avatar URLs absolutised
• user or org resolution with a per process memo
• hf user, hf org, hf ns
• Collection with slug split into namespace, short slug, object id
• CollectionItem including the curator note and the page only numParameters
• hf collection, hf collections, including --item reverse lookup
• Paper with authors, daily feed fields, ai summary and keywords
• hf paper, hf papers, hf citations

M4 social

• Post with the token array content, rendered to a body
• BlogPost from the RSS index
• Discussion with the event thread, comments, reactions
• hf posts, hf blogs, hf discussions, hf discussion
• hf likers, hf likes, hf followers, hf following, hf members
• the >=10 minimum limit on social endpoints handled, not passed through as an error

M4b the HTML plane

• Byte scanner for SVELTE_HYDRATER payloads, no DOM parse
• Secret deny list at extraction time: csrf, jwt, sessionUuid, postLoginRedirectUrl, apiUrlPrefix, anything matching token, secret, password
• JSON-LD blocks, OG and Twitter meta, canonical link
• Unknown components kept, so a page hf has never seen still round trips
• Page record and hf page
• Merge precedence ladder: dedicated endpoint, detail endpoint, page payload, JSON-LD, meta, DOM, list stub
• --deep on repo, user, org, collection, paper reads
• A page fetch failure never fails a read that already succeeded

M5 repository contents

• Ref, TreeEntry, Commit, LFSInfo, SecurityStatus, pickle import scan
• Computed download and raw URLs on every tree entry
• hf refs, hf tree, hf files, hf commits, hf paths
• hf cat and hf readme as byte output escape hatches
• hf card

M6 dataset viewer

• Validity, Split, Size, Row, Feature, ColumnStats, Histogram, ParquetShard
• Pointer numeric stats so an all zero column is not confused with an absent one
• hf valid, hf splits, hf size, hf rows, hf head, hf schema, hf stats
• hf dsearch, hf filter, hf parquet, hf croissant
• Rows stream as pages arrive, so hf rows x | head is cheap

M7 the graph

• Node and Edge types with a source on every edge: api, tag, card, page, derived
• Tag parsing for every namespace: license, arxiv, doi, base_model with its four relations, dataset, region, language
• Taxonomy lookup so a bare word becomes a library or a task rather than a free tag
• The full predicate vocabulary from spec 04, all six groups
• hf:derivesFrom emitted alongside each of the four specific derivation predicates
• Eval results reified as blank nodes, the one place blank nodes are used
• Contradictions between card and tag both emitted, with different sources, never silently resolved
• hf graph, hf edges, hf uri, hf url

M8 crawl

• Breadth first frontier with a visited set on canonical URI
• --follow predicate allow list, structural by default
• --depth, --max-nodes, --max-requests, clean stop at a budget with a report
• Streaming emission, never buffer the whole graph
• Resumable through --db
• Reverse edges only when explicitly followed
• Refuse a tag to repo expansion above the node budget and say so

M9 RDF and bulk

• N-Triples, Turtle, JSON-LD, N-Quads with the source URL as graph name
• Typed literals: xsd:integer, xsd:dateTime, xsd:boolean
• schema.org alignment emitted alongside the hf predicates
• Croissant passthrough, not a reimplementation
• hf rdf, hf dump, hf export
• Confirmation prompt when a bulk estimate exceeds 100k requests

Tests

• Recorded fixtures from the live hub for every endpoint in spec 01
• make fixtures to refresh them, not part of make test
• Every fixture asserts extra is empty, so a new upstream field fails loudly and names itself
• Table driven unit tests for Classify, Locate, tag parsing, StringList, Gated, Time
• Golden output tests per format
• No test touches the network

Docs

• README rewritten around the two planes
• docs site pages for surfaces, records, graph, commands
• Examples that are real pipelines, not toy invocations

[tamnd]

PR #4 is up with the whole rewrite: #4

Everything on the list is ticked except two, which I left open on purpose and describe below.

What landed, roughly in the order of the milestones. The scaffold is gone and the domain points at the real host. Every record carries the Meta envelope and an Extra map, so a field the hub added that no record models yet is kept as raw JSON rather than dropped. Gated, StringList, and Time all decode the several shapes the hub sends for the same thing. The URI space, Classify, and Locate cover every input form a person might paste, including composite ids for discussions, commits, refs, files, and splits.

The client does auth from the four places a token lives, one shared pacer so N workers make one polite stream, retry on 429 and 5xx honouring Retry-After, an on disk cache with no expiry for sha pinned URLs, cursor pagination over the Link header, and the status to exit code mapping.

Repos, lists, search, namespaces, collections, papers, social, repository contents, and the dataset viewer are all in, which is 60 or so commands. The HTML plane reads SVELTE_HYDRATER payloads with a byte scanner, keeps components it has never seen, and drops anything on the secret deny list at extraction time. --deep merges those fields into a normal read through the precedence ladder, and a page fetch that fails never fails a read that already worked.

The graph emits nodes and edges with a source on every edge, parses every tag namespace, reifies eval results as blank nodes, and emits hf:derivesFrom alongside each of the four specific derivation predicates. A card and a tag that contradict each other both get emitted, with different sources, rather than one silently winning. RDF comes out as N-Triples, Turtle, JSON-LD, and N-Quads with the source URL as the graph name, with schema.org alignment alongside the hf predicates, and Croissant is passed through rather than reimplemented.

On tests, the suite runs the real command surface over recorded hub traffic. hftest records each response once into hf/testdata/live and replays it after that, so tests are offline and deterministic while the bytes under them are what huggingface.co actually sent. Each scenario checks a shape golden listing every JSON field path the record carries, and it pins paths rather than values so a like count moving does not fail a build. Every scenario also asserts its Extra map is empty. make fixtures re-records against the live hub and is not part of make test.

The table driven unit tests found a real bug worth naming. Locate and Classify are supposed to round trip, and a hub URL for a canonical dataset did not: huggingface.co/datasets/squad came back as a namespace, because one path segment used to mean a namespace whether or not there was a kind prefix in front of it. So squad, glue, and imdb all resolved to a user page. Fixed in the same commit as the test.

Two more defects turned up while writing the docs against the running binary rather than against the plan. The default table printed the whole record as one wall of JSON in a single cell, because nothing had ever chosen columns, and --fields could not name a field the table hides. Both were upstream in any-cli, fixed there in tamnd/any-cli#18, #19, and #20, and hf-cli now pins v0.4.4. Every record type has table tags, and TestTableColumns keeps the whole column set on a golden so adding a command is a visible decision rather than a silent regression.

The two open items:

Resumable crawl through --db. The flag tees records into a store, which works, but a crawl does not read the store back to skip what it already visited. That is a real feature and not a line of code, so it wants its own PR.

Refusing a tag to repo expansion and saying so. The crawler does refuse to follow a tag back to its repos, since that is a filtered list of millions rather than a link, but it refuses silently. It should report what it skipped and why, the same way the budget stop reports what it had when it stopped.