diff --git a/docs/TechnicalSpikes/006.oss-fuzz-arvo-dataset.md b/docs/TechnicalSpikes/006.oss-fuzz-arvo-dataset.md new file mode 100644 index 0000000..b8a8be6 --- /dev/null +++ b/docs/TechnicalSpikes/006.oss-fuzz-arvo-dataset.md @@ -0,0 +1,75 @@ +# ARVO Dataset (Reproducible OSS-Fuzz) Technical Spike +- [Goal](#goal) +- [Method](#method) +- [Evidence](#evidence) +- [Conclusions](#conclusions) +- [Next Steps](#next-steps) + +## Spike: OSS-Fuzz ARVO Dataset Technical Spike +Conducted by: Xiaoguang Wang (xgwang9@uic.edu). +Backlog Work Items: [Github Issue #42](https://github.com/sysec-uic/AutoPatch-LLM/issues/42), [Github Issue #128](https://github.com/sysec-uic/AutoPatch-LLM/issues/128). + +## Goal +This technical spike explores how the [ARVO dataset](https://github.com/sysec-uic/ARVO-Meta) can be integrated into the AutoPatch-LLM project to support tasks such as **real-world bug** reproduction, LLM-assisted bug reasoning, and patch generation and evaluation. + + +## Method +### 1. ARVO exploration + +[ARVO](https://github.com/sysec-uic/ARVO-Meta) maintains a SQLite database of reproducible real-world software crashes and their corresponding bug fixes. It provides Docker images to facilitate crash reproduction; however, the crash report can be accessed without executing the container. +We use this `libxml2` bug with `localId == 42528804` as an example. + +To reproduce the bug, we can simply run: +``` +$ docker run --rm -it n132/arvo:42528804-vul arvo +INFO: Running with entropic power schedule (0xFF, 100). +INFO: Seed: 2767409826 +INFO: Loaded 1 modules (39373 inline 8-bit counters): 39373 [0xb22417, 0xb2bde4), +INFO: Loaded 1 PC tables (39373 PCs): 39373 [0xa263c8,0xac0098), +/out/xslt: Running 1 inputs 1 time(s) each. +Running: /tmp/poc +==62275==WARNING: MemorySanitizer: use-of-uninitialized-value + #0 0x72cb4d in xmlParserNsLookupUri /src/libxml2/parser.c:1530:9 + ... ... +``` + +According to the [patch](https://gitlab.gnome.org/GNOME/libxml2/-/commit/fc49679316e067d667b71b8e0b987f0b704abce6), the issue can be fixed by manually adding a single line to the `libxml2` source code: + +``` +$ docker run --rm -it n132/arvo:42528804-vul bash +root@4e9ea0b27c87:/src/libxml2# ls /usr/bin/arvo +/usr/bin/arvo +root@4e9ea0b27c87:/src/libxml2# vi parser.c +8849 +# add l.name = NULL; +``` +Next, we can re-compile `libxml2` inside the container and re-execute the test case (`/tmp/poc`): +``` +root@4e9ea0b27c87:/src/libxml2# arvo compile +root@4e9ea0b27c87:/src/libxml2# arvo run +INFO: Running with entropic power schedule (0xFF, 100). +INFO: Seed: 2040443665 +INFO: Loaded 1 modules (39373 inline 8-bit counters): 39373 [0xb22417, 0xb2bde4), +INFO: Loaded 1 PC tables (39373 PCs): 39373 [0xa263c8,0xac0098), +/out/xslt: Running 1 inputs 1 time(s) each. +Running: /tmp/poc +Executed /tmp/poc in 1 ms +*** +*** NOTE: fuzzing was not performed, you have only +*** executed the target code on a fixed set of inputs. +*** +``` +This indicates the bug is gone under the same test case. + +### 2. Better understand the ARVO dataset +Xiaoguang also wrote a few [scripts](https://github.com/sysec-uic/ARVO-Meta/tree/analysis/scripts) to help better understanding the ARVO bugs and their corresponding fixes. Follow the README to analyze how many bugs in `libxml2` are fixed with 1 line of code changes, < 5 lines of code changes, < 10 lines of code changes, and so on. + +## Evidence +N/A + +## Conclusions +The ARVO dataset seems to be a good candidate for real-world evaluation and is suited for integration into the AutoPatch framework. + +## Next Steps +**Step 1**: Select approximately 4 projects from the ARVO (or OSS-Fuzz) datasets and conduct a systematic study to assess the capabilities and limitations of LLMs in reasoning about and fixing real-world memory safety bugs. + +**Step 2**: Integrate these findings and selected case studies into the AutoPatch framework. \ No newline at end of file