Error when an ignored CVE is missing

[G-Rath]

bundler-audit doesn't track if it actually ignored an unpatched gem due to an ignore - it would be good if it did, and so errored if it was asked to ignore a CVE that doesn't affect any of the scanned gems.

This is useful because it makes it more visible when a previously-ignored vulnerability is patched, and reduced the risks of a vulnerability being reintroduced unknowingly due to outdated ignore lists.

IMO this should be the default, but if you don't want to be breaking you could put it behind a flag:

• --continue-on-missing-ignore if erroring is made the default behaviour
• --error-on-missing-ignore if not erroring is kept as the default behaviour

I'm happy to help implement this :)

[postmodern]

I don't think we should change the meaning of exiting with an error. Exiting with an error should always indicate something bad. Maybe we could instead print the ignored advisories in a different colors (or no color) with [IGNORED] text next to above them to indicate they are still there, just ignored. Maybe add a --show-ignored option for those who want to see the ignored advisories?

[G-Rath]

But that's one of the reasons why we have error codes, to allow signaling specific reasons for exiting - for example npm audit will set its exit code as non-zero if any vulnerabilities are found; yarn audit takes this a set further, setting the exit code as a bitmask based on the range of severities found (1 for INFO, 2 for LOW, etc; so e.g. if only INFO and MODERATE vulnerabilities were found, then the exit code will be 1 + 4 = 5).

Overall the ability to have this cause a non-zero exit code is critical for it to be usable (even if that requires an extra flag), as otherwise you're relying on developers running the tool locally and inspecting the output or you have to have automatic tooling parse output looking for the warning messages which can be brittle.

For example, at my company we run bundler-audit in our CI systems for all of our projects (along with the daily "global auditor" that I've mentioned in my other issue), since that's the most reliable & easiest way to do the auditing (vs say having devs manually run the auditing which would take a lot of time and be very tedious). So for us we need to be able to have this cause CI to fail.