Repository navigation
scan: match arcade sets by a fingerprint of their member CRCs - #19948
Davey-Hughes wants to merge 3 commits into
Conversation
libretro-database can give the arcade records a set fingerprint: the CRC-32 of a set's ROM CRCs, de-duplicated, sorted and taken as big-endian bytes, and the same for a merged parent. Map both keys in c_converter and the Lua converter so the rdb keeps them. A converter without these rows drops them, as it drops every key it does not map, and readers that do not know them skip them.
The arcade databases key each set on the CRC of the whole TorrentZipped archive, so the same ROMs packed any other way - zstd or 7z, other names or order, a merged parent - never match. An optional set_fp1 key holds the CRC-32 of the set's ROM CRCs, de-duplicated, sorted and taken as big-endian bytes; set_fpm1 holds the same for a merged parent. The archive's one directory walk gives its fingerprint, leaving out folders, empty members and cue or gdi sheets; a member that stores no CRC leaves it without one. Each database whose cores open the archive's type and that carries the keys is looked up through an index built like the crc one. A hit writes the record's entry, as a match on the archive's own CRC does, and no archive is read whole for it. An archive of several members is looked up by its fingerprint before its size is. An archive of one member is looked up by it only after its member's CRC missed, so a first-member hit held for a cartridge database still wins. Where several records of a database share the fingerprint, as a set's split and non-merged packings can, the one whose size is the archive's is taken; where several have that size the archive's own CRC decides, and where none does, the one named like the archive, else the first. The readers read a TorrentZip set whole with no grant only while no database that opens zip carries fingerprints, so the fingerprint indexes of those databases are built before the readers start. One that cannot be built counts as carrying none: that database can match the set only by its own CRC. A database without the keys costs one walk per scan; older versions skip the keys.
rar names no member it cannot unpack (solid, encrypted, an older packer), so a fingerprint taken over the members it does name would claim a set that may not be all there. Such an archive now has none and goes on to its other passes. 7z and rar also name no folder, which would look the same, so the scan's directory walk asks them to name one as zip does, ending in '/'. Nothing else sets the flag.
|
This is a big amount of changes. Would this recognize romsets across split, non-merged, full non-merged, and merged? There are existing .dat files that change regularly that currently work with manual scan always, so I'm worried that the archived sets would get out of date quickly. |
|
The only thing that this should change is that FBNeo roms that are archived with different formats (7z, zstd) are able to be scanned now. Basically the existing database just stored a CRC of the entire torrentzip archive, so if you had exactly that archive, it would match. With fingerprinting, it instead is able to match based on the CRC of each item in the archive. Also when I updated the dat to the latest one in the database repo, it had like 99.99% overlap from the last time it was updated. So yes the dats are updated often but in very small ways, and the script in the PR for the database repo can keep the fingerprint field up to date when those dats are updated. The new fields for fingerprint CRCs are optional and only additive. |
Guidelines
C89_BUILD=1Description
A follow-up to #19944. The arcade databases key each set on the CRC of the whole TorrentZipped archive, so the same ROMs packed any other way (zstd or 7z, other member names or order, a merged parent) never match. On a NAS share of 2592 merged FBNeo sets packed as RVZSTD zips, master writes no FBNeo entry; with this PR and a keyed FBNeo database it writes 2309.
Two optional record keys carry a fingerprint of the set's member CRCs.
set_fp1is the CRC-32 of a set's ROM CRCs, de-duplicated, sorted and taken as big-endian bytes;set_fpm1is the same for a merged parent. The scan's one directory walk per archive already has the CRCs, so a fingerprint match reads no member data. The keys come from libretro/libretro-database#1752; until both land, nothing changes.Commit 1 maps the keys in c_converter and the Lua converter. Commit 2 makes the change, with its tests. Commit 3 gives no fingerprint to a rar that leaves out members it cannot unpack.
What changes
set.zip, with no whole read./, when the scan's walk asks.Numbers
A Linux desktop, master 3195d95 (#19944 as merged) against this PR. "Keyed" means
FBNeo - Arcade Games.rdbbuilt from the keyed dats. All runs were under load (1-minute load 13.9-25 on 32 cores), so times are rough; counts and playlists are the firm result.Merged RVZSTD zips on a NAS, over NFS, 2592 files:
All 2309 are fingerprint matches, each a bare
set.zip. These runs predate the name step above; at the final head the counts are the same (2.42 s, warm), andcountrun.zipandfightfev.zipmove from a clone's record to their own.596 TorrentZipped FBNeo parents on local disk, 3 runs each:
These runs predate the shared-fingerprint rule. At the final head, three keyed runs give the same matches and 2 whole reads in 90.7-91.0 ms, and the playlist equals the stock rdb's, 596/596 including
crc32.50 FBNeo sets as 7z, solid and non-solid: 50 of 50 by fingerprint, 0 whole reads, each a bare
set.7z.Compatibility
FBNeo - Arcade Games.rdbgrows from 2215401 to 2421986 bytes (+9.3 %).Notes for review
Later, each its own change: a BIOS key (
set_fpb1), clone entries for merged zips, subset matching for incomplete sets (205 of the 2586 merged sets on the NAS), one index walk for both keys (FBNeo is walked twice per scan today), and MAME and HBMAME keys.Testing
database_scan_test(392 checks; 329 on master) anddatabase_index_test(45; 38 on master) pass, also under ASan, TSan,HAVE_THREADS=0(369 checks), readers off (370), with two runs in one root and in 5 runs beside 12 busy loops. New cases cover zip, 7z and rar sets, repacks, one-member and cartridge-first archives, a database matched by members, shared fingerprints, read-ahead and a refused index.C89_BUILD=1compiles the touched objects without warnings. Default and--disable-threadsbuilds link, and the libretro-db parser test passes.tools/ci_local.py, the samples-tasks, libretro-db, libretro-common, cdfs-io and settings jobs, the xenial i386 build, linux-nomenu and 12 tree-wide checks. All passed. Five more fail on this host only, as at master (missing mingw compilers, ps4/ps5 units, the menu hot-function budget, pipewire headers under C89).Related Issues
[Any issues this pull request may be addressing]
Related Pull Requests
[Any other PRs from related repositories that might be needed for this pull request to work]
Reviewers
[If possible @mention all the people that should review your pull request]