A case study of gray-box fuzzing with byte- and tree-level mutation strategies in XML-based applications for exposing security vulnerabilities

Fuzzing is an automated process used to detect crashes and vulnerabilities in software systems, and it is classified as either grammar-based or mutation-based in terms of input generation. While grammar-based fuzzing generates inputs from a specification and handles highly structured data, mutation-based fuzzing creates inputs by randomly modifying input files or abstract syntax trees. There are not many case studies comparing the crash detection capabilities in the scope of mutation-based fuzzing. To add to the body of empirical evidence in this area, this case study compares fuzzing with different mutation strategies to evaluate their effectiveness in three aspects: fault detection effectiveness, fault detection performance, and types of faults detected. Additionally, we evaluate the effects of seed generation techniques on fuzzing effectiveness. We perform the fuzzing on three XML parsers: libxml2, Apache Xerces, and Expat. To perform fuzzing, we use well-known mutation-based fuzzers that implement varying levels of mutation strategies. We conduct investigations to examine the effects of seed generation on fuzzing by utilizing publicly available seeds and probabilistic context-sensitive grammar (PCSG)-based seeds. In terms of fault detection effectiveness and performance, we evaluate the number of crashes and the number of test cases generated. With respect to mutation strategies, our results demonstrate that the bit/byte-level mutation strategy detects more crashes than the tree-level mutation strategy. According to the fuzzing results, PCSG-based seeds can help detect higher number of crashes than publicly available ones. In terms of generated test cases, fewer test cases are produced for PCSG-based seeds compared to publicly selected ones, while bit/byte-level mutations generate more test cases than tree-level mutations. Empirical results show that crash detection capabilities of fuzzing differ importantly based on the mutation strategy used.