Benchmarking Reverse Engineering Tools and Using Tool Output for Further Analysis

With the growing amount of complex legacy code in use by organisations the ability to comprehend codebases through the use of Reverse Engineering tools is increasingly important. Although a wide number of tools exist to perform such reverse engineering there is no objective standard to determine their relative strengths, weaknesses and overall performance or to validate new techniques.

Furthermore for a discipline to mature objective comparison between tools through a form such as a benchmark is required. In order to address this gap we created the Reverse Engineering to Design Benchmark (RED-BM). The benchmark offers the facilitation of tool performance comparison from a standardised set of source code artefacts against a number of initial metrics geared towards the use of reverse engineering tools to aid structural comprehension of software. The benchmark provides a mechanism for configuring and combining performance metrics which be used to form a weighted compound measures allowing for
extensibility into different domains.

As part of RED-BM we assessed the performance of a number of tools for metrics and to provide their output within the benchmark as a yardstick measurement of new techniques. A wide degree of variety was seen in tool performance and also in the supposed standard for output and exchange of reverse engineering information (the Extensible Metadata Interchange format - XMI). Generally speaking it was found that tools are not capable of producing complete accurate models so it is foreseen that a future requirement could be the ability to amalgamate models generated from multiple tools to form a more complete structural picture. To aid the creation of the benchmark tools were developed to automate the comparison between source code and reverse engineering output. The basis of these tools, specifically the ability to parse XMI generated from different reverse engineering tools, is being developed for future applications such as structural analysis and custom UML projection within the Eclipse platform.