I have been writing a series of articles on Bug life cycle and some thoughts are still pending. In this article, I will describe my views on effective bug reports and how the same might help to improve the productivity of software development teams. There are many stories around the web on this topic, but still I would like to add my two cents here.
Objective
The bug reports must help to reproduce the problem with minimum efforts and help the developer to debug and fix the problem. The good bug reports contain most of the information required to stake holders to reproduce, analyze, debug and fix the problem in a given context.
Concerns
There are many instances in my experience where in which the bugs were returned to the testers or the bug reporters seeking more information / clarifications / steps etc.
How often the developers says that the issue is non reproducible at their end and seek clear steps to reproduce the bug?
How often the developers require more information from the bug reporters which is not part of Bug report?
How often do we spend more time on investigating the bug even after the bug has been reported?
How often, we were asked to provide test data for the issues
How often, we were asked for test environment details around the issue
And the list goes on …
We are often a hurry or excitement to report the issue and may provide just basic info and ignore. Some of us are also under the impression that we don’t need to spend so much time of the bug reports and might be waste of time too. Most of the information around the issue will be available only when the issue has been found. It would be tough to provide the same data at a later time.
But in reality, we spend more time even after reporting the problem and try to provide the information asked by stake holders.
The problem lies with the quality of the bug report. We often tend to report the bug with minimum details around the problem and tend to ignore context driven information required for other stake holder to save time.
The Bug Summary
Make it simple and point to the actual problem. Keep in mind that its developer’s initial interaction & the same must grab the attention and points to the actual problem.
Steps to Reproduce
Describe the state of application and the pre-requisites. The flow of user actions and data provided is important. All the steps must be logical and we can remove redundant steps. It’s a good idea to have a relook on all the steps and try it on a fresh system.
Expected Behavior & Actual Behavior
The perceptions and views tend to differ always. It’s good the document the expected behavior and actual behavior of the system under test.
Test data
We provide lot of data to the systems under test at specific user actions. It’s important to list down this data and also capture data around the issue. Provide the following as part of bug report
The input data to the application
Screenshots around the Issue
The Log files
Test Environment
Test Environment plays key role in reproducing the issues and it’s the root cause for many of the non reproducible issues. We must take that extra mile to capture all the environment details like OS, databases, build numbers, browsers. There are no best practices here and we must capture information based on the context.
Bug Tracking Systems
We log bugs in bug tracking tool and assumes that all the things have been taken care. But if try to look at history and pattern of these bugs based releases or environments etc via bug databases, all the issues may not be present. We do miss out many attributes of bug databases that might help us to derive patterns, categorize over specific need in the long term.
Final words
The purpose of a bug report is to showcase that there is a problem that exists and let the developers see the failure. It must defend and advocate the reasons around the problem and finally helps to resolve the bug in the system
Almost an year ago, I have evaluated some low cost and open source Test Management Tools. Managing the Test Efforts & Test documentation is always an issue for small and medium companies.
The following were my requirements for Test Management Tool
Capture the Requirements
Design & Prepare Test Cases
Map Test Cases with Requirements
Link Bug reports with Test Case ID after the test execution
Test Execution Reports
Version Management for the Test Cases
Search Feature
Testuff comes with all the above features and some additional capabilities to record the test execution and link them with the issue tracker. It comes free for single user. Check out the testuff blog for more updates
The Testability Explorer is an open source project that lets you measure the testability of Java code. This is an interesting idea: a metric not of direct quality, or of testing, or of test coverage, but of ease of testing. Presumably code that is easy to test will get tested, and will therefore be of higher quality, other things being equal.
The following information over Testability Explorer is useful
The term quality might mean different things to different people based on their context of operation and it’s tough to have universal definition for the same.My experiments with software development started over a decade ago & exploring the ways the helps to build & deliver good quality code.The quest towards Code Quality Metrics started with the above objectives.
In the process, we do use Static Analysis and Dynamic Analysis to derive the above mentioned metrics. Some more good articles around Code Quality are here
I have been talking about Code Quality through Static Analysis for a while here, here & here. The quality for any application development must begin from it’s gross roots and the Application Code is one of it’s starting point to begin with.Static Analysis techniques help us to identify some metrics over the application code base
Cyclomatic Complexity
Application Design & Dependency Metrics
Exception Handling
Infinite Loops
Dead Code
Performance Issues
Programming Language guidelines & best practices in their context
The above information will be useful and it’s easy to get the same via Test Framework with a single click.
Code Coverage along with a bunch of unit tests is one widely used technique to help regression testing for the dev & test teams.
Now it’s good to see that Agitar combines Code Coverage & Cyclomatic Complexity to derive risk metrics for Java Code Base. They call it as a code Change Risk Analyzer and Predictor (i.e. CRAP) for Java.
Though it’s a prototype and see how the industry receives the same, i see it as a good initiative on the code quality front and might be a metric going forward for white box testing.
Now we have a dedicated site for CRAP4J. This contains latest news, forum discussions and many more. I would say that this a good resource for Code Quality lovers & the good news is that they are designed to be open source tools.
Crap4j is a Java implementation of the CRAP (Change Risk Analysis and Predictions) software metric – a mildly offensive metric name to help protect you from truly offensive code.
The CRAP metric combines cyclomatic complexity and code coverage from automated tests (e.g. JUnit tests) to help you identify code that might be particularly difficult to understand, test, or maintain – the kind of code that makes developers say: “This is crap!” or, if they are stuck maintaining it, “Oh, crap!”.
The best way to learn more about CRAP and Crap4j is to check the various articles, newsgroups and blogs about them.
As a Tester & Developer, it’s pretty common to see NoClassDefFoundError. It’s not that easy always to figureout the required jars for the application.
Whatjar comes with a lot of search capabilities to identify and download the required jars.
Whatjar is a high-performance, search engine written using Java and MySQL to index and search Java JAR contents. Its primary goal is to provide a tool to help the developer find the JAR file required when faced with a class not found exception. In addtion to act as searchable resource of open source java jars.
Whatjar is free, there is no cost to upload a jar or to search our existing database of JARS.
In this post, i will explore different stages of the Bug from it’s inception to closer. The Bug has been found and logged into the Bug Tracking System. It’s my fourth post in the Bug Life Cycle series.
The Bug has been found and logged into the Bug Tracking System. It will be treated as New Bug in the System.
The Bug will be assigned to the concerned Developer for a Resolution.
The developer looks in to the possibilities of the resolution & takes a call on Resolution by fixing it or differing over the information provided.
Tester validates the resolved issue in the build & checks for the regression scenarios over the fix.
If the issue found fixed, then he choose to Close the issue else he / she will Re-open the same.
The Cycle follows again for the re-opened issue till it gets closed.
It worth doing the following activities
Capturing the required and re-usable info to the Bug Report at it’s each stage.
Check for all the closed bugs of Severity 1 & 2 against final build for the release.
In the next post, I will share my thoughts on the useful metrics over the Bug Tracking Repository.
In this post, i will share my views & findings with Open Source Test Management Tools. Usage of tools in the Test Management is becoming the basic need & It will be tough to manage the activities associated there with out any support from the tools.
The commercial tools in this segment are costly and may not fit into upcoming organizations budget. So i have been looking at Open Source Test Management tools. In the last week, I have evaluated some open source test management tools.
The following are my requirements for Test Management Tool
Capture the Requirements
Design Test Cases
Map Test Cases with Requirements
Link Bug reports with Test Case ID after the test execution
Test Execution Reports
Version Management for the Test Cases
Search Feature
The following tools are evaluated after the initial screening.
The above tools work with Apache, MySQL & PHP. Both the tools looks promising and the Test Link has some additional benefits in-terms of better reports and linking with popular issue trackers.
I have been working on Java Projects for quite some time and interested in exploring the source code metrics like size and complexity across project not specific to perticular package or class and looking after LOC (Line of Code) counters to capture some metrics of the source code.
Explored some tools and found that JavaNCSS is a good match for the context of sizing metrics on Java. JavaNCSS is a command line utility which measures some standard source code metrics for the Java programming language. The metrics are collected globally, for each class and/or for each function.
The following are some of the advantages that i have seen
Support for Ant Tasks, so easy to integrate with build process
Reports can be in Text, XML, HTML etc
Support for Stylsheets and easy to get nice HTML reports
Metrics at each level Package / Class / Method
Cyclomatic Complexity Number
List the number of packages / classes / functions / LOC counter at each level
