Analyzing Java Microbenchmark Harness (JMH) Performance in Open-Source Systems

Abstract

Microbenchmarking is widely used in Java software testing, especially for open-source projects. This study examines performance measurement practices in open-source Java projects, analyzing 586 microbenchmarks across 30 projects. We investigate performance test durations, stability, and variability, revealing insights through exploratory data analysis and statistical modeling. Our findings offer guidance for achieving steady-state performance and understanding performance measurement patterns. Additionally, we explore performance metric variability, emphasizing the impact of outliers. Our study contributes actionable insights for enhancing software quality and performance in opensource projects. We also employ density plots to visually represent performance metric variability. While acknowledging limitations, our research bridges theory and practice, empowering developers and researchers to optimize Java software performance in real-world scenarios.