I have been writing a series of articles on Bug life cycle and some thoughts are still pending. In this article, I will describe my views on effective bug reports and how the same might help to improve the productivity of software development teams. There are many stories around the web on this topic, but still I would like to add my two cents here.
Objective
The bug reports must help to reproduce the problem with minimum efforts and help the developer to debug and fix the problem. The good bug reports contain most of the information required to stake holders to reproduce, analyze, debug and fix the problem in a given context.
Concerns
There are many instances in my experience where in which the bugs were returned to the testers or the bug reporters seeking more information / clarifications / steps etc.
How often the developers says that the issue is non reproducible at their end and seek clear steps to reproduce the bug?
How often the developers require more information from the bug reporters which is not part of Bug report?
How often do we spend more time on investigating the bug even after the bug has been reported?
How often, we were asked to provide test data for the issues
How often, we were asked for test environment details around the issue
And the list goes on …
We are often a hurry or excitement to report the issue and may provide just basic info and ignore. Some of us are also under the impression that we don’t need to spend so much time of the bug reports and might be waste of time too. Most of the information around the issue will be available only when the issue has been found. It would be tough to provide the same data at a later time.
But in reality, we spend more time even after reporting the problem and try to provide the information asked by stake holders.
The problem lies with the quality of the bug report. We often tend to report the bug with minimum details around the problem and tend to ignore context driven information required for other stake holder to save time.
The Bug Summary
Make it simple and point to the actual problem. Keep in mind that its developer’s initial interaction & the same must grab the attention and points to the actual problem.
Steps to Reproduce
Describe the state of application and the pre-requisites. The flow of user actions and data provided is important. All the steps must be logical and we can remove redundant steps. It’s a good idea to have a relook on all the steps and try it on a fresh system.
Expected Behavior & Actual Behavior
The perceptions and views tend to differ always. It’s good the document the expected behavior and actual behavior of the system under test.
Test data
We provide lot of data to the systems under test at specific user actions. It’s important to list down this data and also capture data around the issue. Provide the following as part of bug report
The input data to the application
Screenshots around the Issue
The Log files
Test Environment
Test Environment plays key role in reproducing the issues and it’s the root cause for many of the non reproducible issues. We must take that extra mile to capture all the environment details like OS, databases, build numbers, browsers. There are no best practices here and we must capture information based on the context.
Bug Tracking Systems
We log bugs in bug tracking tool and assumes that all the things have been taken care. But if try to look at history and pattern of these bugs based releases or environments etc via bug databases, all the issues may not be present. We do miss out many attributes of bug databases that might help us to derive patterns, categorize over specific need in the long term.
Final words
The purpose of a bug report is to showcase that there is a problem that exists and let the developers see the failure. It must defend and advocate the reasons around the problem and finally helps to resolve the bug in the system
Bug Database for the products might have thousands of issues over a period of time against various builds and releases. Though these issues fixed over a period of time, it might be hard to derive meaningful metrics over the release.
We need to support these releases over the production systems & it might be helpful to capture the hot spots / risk elements with the release. Most of the issues here to deal with the respective features, compatibility with other features / technologies & performance related issues.
The usual metrics of number of issues against a module and their severity levels may not be of help always.
How easy is it to derive the following from the Bug Database for a given release
Identify the issues have originated from Requirements, Design & Implementation
Identify the issues over their category (Functional, Performance, Security, Compatibility, Usability etc)
Identify the issues along with their origin & category over the features rather than the modules / components
In this post, i will explore different stages of the Bug from it’s inception to closer. The Bug has been found and logged into the Bug Tracking System. It’s my fourth post in the Bug Life Cycle series.
The Bug has been found and logged into the Bug Tracking System. It will be treated as New Bug in the System.
The Bug will be assigned to the concerned Developer for a Resolution.
The developer looks in to the possibilities of the resolution & takes a call on Resolution by fixing it or differing over the information provided.
Tester validates the resolved issue in the build & checks for the regression scenarios over the fix.
If the issue found fixed, then he choose to Close the issue else he / she will Re-open the same.
The Cycle follows again for the re-opened issue till it gets closed.
It worth doing the following activities
Capturing the required and re-usable info to the Bug Report at it’s each stage.
Check for all the closed bugs of Severity 1 & 2 against final build for the release.
In the next post, I will share my thoughts on the useful metrics over the Bug Tracking Repository.
Today, i have come across an interesting issue with BSNL Portal. I have been using this site for online billing and payments for about an year.
Followed the usual steps and my objective is too see the bill and pay it online. To my surprise, the system says
This error message ” Invalid object name ‘RECEIPT_MASTER’ “ is strange and there is no information for me on how do i proceed further. The application might be unable to get the specified object from the Database.
In case if this happens on the last day of the payment, on line users are liable for late payment charges due to this issue.
Users are familiar with the messages like, Unable to process the information as of now & please check back after some time.
The error messages must be meaningful to the business users and we need to make sure that the technology related terms / phrases are not part of the same.
Context driven information is the need of the hour and there is a huge value associated with the same. It’s good to capture the context driven information in the bug reports. My initial experiences with bug reports way back in early 2000 have taught many lessons to improve upon.
Bug reports used to capture what is the problem with the system familiar to the user (tester) who reported the same. People spend very less time to capture all the details required and there are many reasons for the same. I hope some of the upcoming testers will also be in the similar situations.
Some of the Reasons people quote here
We need to test more and less time to capture & write more information in the Bug Reports
It’s tough to capture all the information required
System is complex & It’s tough for the novice users to understand the bug reports
You know capturing all the info is process driven & it may not be worth of efforts
It some times boring activity to collate the info & push it
I can reproduce it on my machine if developer needs it.
This list can go on…
I hope you have come across this situation at least once in your career.
This is my third post in the Bug Life Cycle Series and it’s good to know the different users of your Bugs and their context with them. The mission of your bug report is to provide details and context of the problem and convey the importance of it with a user driven stories.
Your bug report must be the voice of customer and it need play the role of an advocate against the problem. Bug Advocacy from Cem Kaner is an excellent source to begin with. If the bug report unable to specify the need of the context, then it’s better not to write any report
It’s good to explore & capture some of the following problems
Productivity
Performance
Usability
Migration
Stability etc
Try to link your issues with most suited functions listed above. It may not be obvious to other users in the system to explore & analyze the issues in that fashion.
There is another context associated with Bug Reports. That’s with the stake holders of the project. The Bug Tracking system must give the right trends and identify the hot spots. Testers must capture the right kind of data to derive better valuable metrics over the bug repository.
Care must be taken to capture
Capture all the Test Environment details
Detailed classification on the feature. Classify to the maximum possible sub feature/component of the system
The above info helps a lot to identify the trends in bugs and focus on the unstable components / environments.
Final Thoughts
Push the entire context driven information to the bug repository at least for a release cycle and observe the results. Check back with your repository to identify the trends and risk associated with the release and I am sure that it will be in the similar lines of end user feedback.
Bugs are there every where in the software applications. Almost every one who uses software applications for their day to day activities; do come across different kind of problems while working with them. Each of these problems has different meanings to different people on it. There are many instances where in which, we (as end users of the system) do feel that how come they missed this bug, it’s very important in this context (The context where in which the user operates).
Tester deals with Bugs every day and its good know about other people get affected with them. The list includes Customers, Stake Holders, Sales, Professional Services, Technical Support, Architecture & the Development team.
This is my second post in the Bug Life Cycle Series. Before going more into the Bugs and their life cycle, it’s good to know, what it means to different people across the software application. Based on the contexts, the same bug might mean different things to different people. Do go through Role of Software Testing to understand my mission about Software Testing.
A software bug (or “bug”) is an error, flaw, mistake, failure, or fault in a computer program that prevents it from behaving as intended (e.g., producing an incorrect result). Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, and a few are caused by compilers producing incorrect code. A program that contains a large number of bugs, and/or bugs that seriously interfere with its functionality, is said to be buggy. Reports detailing bugs in a program are commonly known as bug reports, fault reports, problem reports, trouble reports, change requests, and so forth.
Tester Perspective
Any deviation from the expected results of the test case will be treated as a bug in the software application.
Customer Perspective
Customer uses the software application to solve his business needs. Any problem while modeling a solution to the business need will be considered as a bug in the software applications. These problems can be classified into two. They are the list of problems that they can live with and the other is the list of problems that need to be addressed to solve the business need.
The second list of problems will be sent to the vendors of the application for the fix. Some times, customer sends only the show stopper problems for his business. It’s true in the fact that as an end user, I will contact the vendor only if the problem is critical for me.
Technical Support Perspective
The Support Team classifies (though it’s critical) the customer requests into New Features, Enhancement, Bug, How to & Enquiries etc with respective severity levels. The decision will be taken by validating the problem with the features in the product.
Developer Perspective
The feature is designed this way and all the cases have passed. The end user might be using some other scenario. Yes, Some times there are bugs in my code. But it functions well if we use the way the application has built.
Management Perspective
Any problem with the application will be treated as Bug if it has impact over the revenue and customer satisfaction.
Final Thoughts
It’s tough to have same perspective across all the people (might happen in an ideal world). The bug for some one may not be the bug for the other person. However there are some set of show stopper bugs for which, the perspective will be the same.
This is my first post in the Bug Life Cycle Series. I need to talk about this because when it comes to the role of testing, it’s not clear. The Role of Software Testing is often mis-understood across the different stake holders of the application development & this list includes testers too.
Testing is considered to be part of Quality Assurance activity since the initial days of Software Development and the same trend is happening as of now too. Even most of the titles like QA Engineer / QA Lead are associated with Testers even though they are not performing the role of QA.
It’s good to capture the mission of Testing & align your test teams in that direction. Every one in the team must be clear on his / her role and see how the same is helping to achieve the mission of the team.
Testing is not about targeting for Bug Free Product. It’s just impossible since you can’t build human brain into systems (Of course humans do commit mistakes).
Testing is not about fighting with the Development Teams. Don’t act like the enemy of developers.
Testing is not about just looking at the documents (so called BRS, SRS, FRS) and writing the test cases.
Don’t fight with Developers on the issues need to be fixed for the release instead write good report that reduce the time to reproduce and debug.
Testing is a process followed to make things better. It helps to take informed decision by providing the relevant information based on the context.
My teachers made me better by giving the relevant feedback at every stage on my performance. This includes nurturing the concepts, performance in the tests & week areas in the subject. This information helped me to identify the areas missed / uncovered and to improve upon.
Testers need to identify the critical issues with the system as soon as possible and make sure that the information supplied is sufficient to reproduce the issue. We need to supply the information on the presence of bugs in the system to the stake holders. The information should help the stake holders to take informed decisions.
The following list helps
Identify the different end users of the system and their interaction with the same.
Capture the important scenarios for each end user. It’s good to note that we need to capture the story around the scenario and not just steps.
Talk to different stake holders including the customers (incase if you have access) on how the feature might be used & capture the scenarios.
Plan towards uncovering the critical issues of the system as early as possible
Final Thoughts
Focus towards what is import for the decision making. Try to uncover important issues first and provide the required information to reproduce and debug the same problem.
I have been talking about Bugs in the Software Applications in my recent posts. They play an important role in Software Development & Testing. The biggest value addition is to uncover the critical issues of the system at the early stage.
I have already written some posts on the Bug Tracking front and find the list below. They talk about how to handle the Bugs, avoid conflicts & the lessons to be learned for the testers.
Last week, i had reviewed the bug reports submitted by my team & found that some of them are not reproducible. I am curious about where things went wrong & checked with the team.
Some of the Responses:
Look Venkat, this bug is not-producible always. So we have captured the log.
It does occur if we re-start the application
It takes good amount of time for the investigation
As a Tester, i have come across of situations where some bugs won’t reproducible again and i infer that many of the testers might have gone through the same episode atleast once.
But then, Non-Reproducible Bugs are errors of the testers even though we may not agree. More over this is true for Technical Support & Testing functions.
It’s tough for the Developers to work with such bugs and testers used to get lot of objections over such bugs. If the count is more, then it might spoil the relationships between testers and developers.
An Insight
I infer from my experiences that systems doesn’t have the powers of human brain. It’s tough to repeat the same mistake again from people since they are conscious. Systems doesn’t have that much capability interms of thinking and they do perform the sequence in which they have coded for.
How do deal with Non Producible Issues
Testers need to improve their skills inorder to reproduce such issues. The following might be a help.
Try to capture the story around the issue and not just steps.
Capture the complete Test Enviroment
Capture all the available Test Execution Results. This includes your test data, screen shots, applications logs, system logs, server logs
Need more patience (they won’t occur easily)
Need more observation skills on the application behavior under test.
Narrate the story around the issue to the stake holders and to corner the issue.
Resources & Time Contraints might be an issue (More true for Tech Support). Bargain for the same
Conclusion
They are bound to take lot of time & resources in the process. It’s suggested to reproduce each & every issue. We must explore more for the critical issues & re-produce them asap. In the process, we learn more too.
In this post, I will be looking at Bugs with an insight that helps testers to learn some new lessons on why the same might have occured. It’s quite common that the testers are blammed for all the missed out bugs in the system as if they are super natural powers to over see all the issues that are there in the system.
Bugs are there every where. Not just software applications, look at any system for that matter & there are issues wrt End User needs and the same might need to evovle over a period of time with the changing needs / requirements from time to time.
What is a Bug ?
It’s tough to arrive at a generic defintion for the bug & people might have heated discussions on what is a bug & what is not. Mostly it’s based on the context in which people are working. We do come across many bugs in our daily life while working with the tools that are needed for the day to day operations.
So in my context, Bug is something that’s unable (doesn’t allow me) to satisfy my needs with the system.What do to do ?
Capture the story around the Bug and not just steps to reproduce. Talk to the reporter and discuss on the scenarios used with the system that lead to the bug.Let’s explore the possible options even though Tester validated the system early
New Test Scenarios for the Testers
We mayn’t even think of that scenario while validating the system. Add the similar test scenarios to the Test Suites
The scenario may not be logical (though it seemed to be for testers) for the user though functionally correct. Add to the similar scenarios to Test Suites
Might have been a compatability issue with a new environment that doesn’t exist in Test Labs
Reduce mistakes and improve test teams capability
Might have been missed by a Tester. Check with the tester who performed the Test earlier and find out on why issue missed and the same marked as a pass even though it fails. Might happen through overlook also. Do more analysis here and try to reduce the same.
Might be over looked through regression tests. Might have failed and passed in the internal builds and finally breaked in the production system. Analyse the reasons here and improve upon on regression testing.
Make it a point to log the bug as soon as they discover the same. People might loose the same from their memory as time goes
