In the fast-paced world of software development, tools are the lifeblood of developers. They empower programmers to write code more efficiently, leading to functional, reliable, and resilient software. But there’s a crucial aspect that often takes the backseat — security. Ensuring our software is secure from malicious attacks is vital in today’s digital landscape.

Meet SonarQube and SpotBugs, two remarkable open-source tools designed to analyze computer programs and uncover code issues. They are not just any tools; one of their primary goals is to identify potential security vulnerabilities in the code, making our software robust against potential threats.

But how effective are these tools, really? I embarked on a research journey to find out their true value for developers. I wanted to know if they could do more than just point out a line number where a security bug might be hiding. After all, developers need actionable insights to fix these vulnerabilities.

The experiment: shedding light on the usefulness of developer tools

To conduct the experiment, I divided a group of developers into three groups. The first group, the control group, received only a line number where the security bug was detected. The other two groups, however, were given additional information about the security bugs, generated by SonarQube and SpotBugs. I wanted to see if these additional details would make a significant difference.

Each developer was presented with four different code samples, each containing a security bug. Their task was to identify the correct fix from four possible options. The bugs covered a range of issues, including SQL injection, hard-coded credentials, encryption, and sensitive data logging. I intentionally kept the code samples simple, avoiding extra graphics, to maintain consistency across the groups.

A mixed bag of results

The results were intriguing, to say the least. A large majority of developers (83%) couldn’t identify the fix for all four bugs, even with access to additional information. It seemed that merely providing more details about the bugs didn’t significantly impact the overall performance of the developers.

Digging deeper, I analyzed the reasons behind this surprising finding. Through regressions and analyzing open-ended and Likert questions, I found several factors influencing the accuracy of the answers.

Experience in software development emerged as a significant contributor, with seasoned developers outperforming their less experienced counterparts. Interestingly, the choice of the tool used became less critical with more experience.

Furthermore, the developers’ perception of vulnerability severity and their confidence in their answers also played crucial roles in their overall performance. When a developer considered a security bug to be a significant threat and had confidence in their solution, they were more likely to find the correct answer.

How to increase the usability of developer tools?

Understanding developers’ individual differences is essential: Developers come from diverse backgrounds, with varying levels of experience and expertise. To make developer tools more effective, we must tailor them to suit the developer’s background. This could mean asking a few questions during setup to customize the interface based on their responses.

Raise awareness: Moreover, raising developers’ awareness and knowledge about security bugs can lead to more diligent efforts in fixing them. If developers truly understand the potential consequences of leaving a security vulnerability unaddressed, they will be more motivated to take action.

Give code samples: Among the various elements included in the notifications generated by the tested tools, developers especially appreciated one thing — code samples. These tangible examples of both valid and buggy code helped them better comprehend the issues at hand. So, let’s incorporate more code samples into our tools, as they seem to be a hit among developers!

Unleashing the power of developer tools

Security is not an afterthought; it’s a fundamental aspect of software development. SonarQube and SpotBugs have the potential to be game-changers in the quest for secure software. However, their effectiveness lies not just in their functionality, but also in how well they accommodate developers’ diverse needs.

By understanding individual differences, fostering security awareness, and providing helpful code samples, we can unleash the true power of developer tools.

---

Endnote: The full paper with all the details published at the prestigious ACM Conference on Human Factors in Computing Systems (CHI ‘21) and a video recording of the presentation are available here.