Skip to content

validation.md

Latest commit

 

History

History
89 lines (62 loc) · 2.63 KB

validation.md

File metadata and controls

89 lines (62 loc) · 2.63 KB
Raw

validation model

nightfall uses a three-tier validation model to honestly report what each test actually proves.

tiers

IMPACT

the test demonstrates the end-to-end security consequence. not just that an API call succeeds, but that the intended effect occurs.

examples:

  • D1: after locking VDM files, attempts to open one for writing and confirms ERROR_SHARING_VIOLATION (proves Defender cannot update sigs)
  • K4: queries a process command line through the dam.sys IOCTL and confirms non-empty data is returned (proves information leak)

STATE

the test verifies that the intended mutation occurred, but does not demonstrate the downstream security consequence.

examples:

  • AMSI: confirms the patched bytes in AmsiScanBuffer match the expected values, but does not test whether a known-malicious string passes AMSI scanning after the patch
  • D2: confirms the oplock was acquired and has not broken, but does not trigger a Defender quarantine attempt to prove it blocks
  • W1: confirms the Report.wer file was written to disk, but does not verify SYSTEM-level WER processing consumed it

STATE is honest about the gap: the test proves the setup worked, not that the attack succeeded.

TRIGGER

the API call completed successfully, but neither the mutation nor the consequence is directly verified.

examples:

  • K1: DeviceIoControl returned NT_SUCCESS, confirming the IOCTL was accepted, but the test does not verify the target process was actually enrolled in the DAM job
  • D4: StartTraceW succeeded, but the test does not verify events are being delivered to the callback
  • E1: the mock directory was created, but the test does not execute the binary or verify elevation

SKIP

prerequisites not met (not admin, missing driver, etc). not a failure.

FAIL

the test ran and something did not work as expected.

running validation

python nightfall.py validate           # run all tests
python nightfall.py validate --target D1   # run one test

or in interactive mode:

nf> validate
nf> validate D1

test isolation

AMSI and ETW tests use try/finally to restore original bytes after verification. the tests patch, read back, verify, then restore -- they do not leave the process in a modified state.

K3 (BSOD) is excluded from the suite for obvious reasons.

coverage

the automated suite covers: D1, D2, D4, D5, E1, K1, K4, W1, N1, AMSI, ETW.

not covered: D3 (needs external network listener), K2 (disruptive), K3 (destructive), W2 (not validated), and all experimental features.

this is the biggest gap in the project. every experimental feature should have at least a TRIGGER test before the project is complete.