Context driven information is the need of the hour and there is a huge value associated with the same. It’s good to capture the context driven information in the bug reports. My initial experiences with bug reports way back in early 2000 have taught many lessons to improve upon.
Bug reports used to capture what is the problem with the system familiar to the user (tester) who reported the same. People spend very less time to capture all the details required and there are many reasons for the same. I hope some of the upcoming testers will also be in the similar situations.
Some of the Reasons people quote here
- We need to test more and less time to capture & write more information in the Bug Reports
- It’s tough to capture all the information required
- System is complex & It’s tough for the novice users to understand the bug reports
- You know capturing all the info is process driven & it may not be worth of efforts
- It some times boring activity to collate the info & push it
- I can reproduce it on my machine if developer needs it.
This list can go on…
I hope you have come across this situation at least once in your career.
This is my third post in the Bug Life Cycle Series and it’s good to know the different users of your Bugs and their context with them. The mission of your bug report is to provide details and context of the problem and convey the importance of it with a user driven stories.
Your bug report must be the voice of customer and it need play the role of an advocate against the problem. Bug Advocacy from Cem Kaner is an excellent source to begin with. If the bug report unable to specify the need of the context, then it’s better not to write any report
It’s good to explore & capture some of the following problems
- Productivity
- Performance
- Usability
- Migration
- Stability etc
Try to link your issues with most suited functions listed above. It may not be obvious to other users in the system to explore & analyze the issues in that fashion.
There is another context associated with Bug Reports. That’s with the stake holders of the project. The Bug Tracking system must give the right trends and identify the hot spots. Testers must capture the right kind of data to derive better valuable metrics over the bug repository.
Care must be taken to capture
- Capture all the Test Environment details
- Detailed classification on the feature. Classify to the maximum possible sub feature/component of the system
- Clarity on Severity & Priority
- Versions and Build Numbers (Affected & Fixed)
- Bug Classification (Requirements / Design / Implementation etc)
- Bug Types (Functional, Performance, Usability, Security etc)
- It can go on…
The above info helps a lot to identify the trends in bugs and focus on the unstable components / environments.
Final Thoughts
Push the entire context driven information to the bug repository at least for a release cycle and observe the results. Check back with your repository to identify the trends and risk associated with the release and I am sure that it will be in the similar lines of end user feedback.
Happy Testing…
Cyclomatic Complexity (CC) is a software metric (mostly code based metric) used to the number of independent path executions in the application. Introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.
It helps the developers to determine the independent path executions and base line unit tests that they need to validate. Using this, the developers can assure that all the paths have been tested atleast once. It’s a great comfort for the developer and their respective managers.
It’s better to write JUnit Tests for all these linearly-independent paths and integrate it with any code coverage tool. These reports help to focus more on the un covered paths and improve the code coverage.
It also helps to evaluate the risk associated with the application. The following are the results published by SEI and they are being followed widely to determine the health of the code base.
Cyclomatic Complexity Risk Evaluation
1-10 A simple program, without much risk
11-20 More complex, moderate risk
21-50 Complex, high risk program
Greater than 50 Un testable program (very high risk)
Explore more at Cyclomatic Complexity in Software Technology Roadmap from SEI.
Further Reading on the topic
Use metrics to evaluate the risk early in the cycle & improve your test coverage.
Happy Testting …
(more…)
Abstract:
Bugs are there every where in the software applications. Almost every one who uses software applications for their day to day activities; do come across different kind of problems while working with them. Each of these problems has different meanings to different people on it. There are many instances where in which, we (as end users of the system) do feel that how come they missed this bug, it’s very important in this context (The context where in which the user operates).
Tester deals with Bugs every day and its good know about other people get affected with them. The list includes Customers, Stake Holders, Sales, Professional Services, Technical Support, Architecture & the Development team.
This is my second post in the Bug Life Cycle Series. Before going more into the Bugs and their life cycle, it’s good to know, what it means to different people across the software application. Based on the contexts, the same bug might mean different things to different people. Do go through Role of Software Testing to understand my mission about Software Testing.
Wikipedia defines Software Bug as the following
A software bug (or “bug”) is an error, flaw, mistake, failure, or fault in a computer program that prevents it from behaving as intended (e.g., producing an incorrect result). Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, and a few are caused by compilers producing incorrect code. A program that contains a large number of bugs, and/or bugs that seriously interfere with its functionality, is said to be buggy. Reports detailing bugs in a program are commonly known as bug reports, fault reports, problem reports, trouble reports, change requests, and so forth.
Tester Perspective
Any deviation from the expected results of the test case will be treated as a bug in the software application.
Customer Perspective
Customer uses the software application to solve his business needs. Any problem while modeling a solution to the business need will be considered as a bug in the software applications. These problems can be classified into two. They are the list of problems that they can live with and the other is the list of problems that need to be addressed to solve the business need.
The second list of problems will be sent to the vendors of the application for the fix. Some times, customer sends only the show stopper problems for his business. It’s true in the fact that as an end user, I will contact the vendor only if the problem is critical for me.
Technical Support Perspective
The Support Team classifies (though it’s critical) the customer requests into New Features, Enhancement, Bug, How to & Enquiries etc with respective severity levels. The decision will be taken by validating the problem with the features in the product.
Developer Perspective
The feature is designed this way and all the cases have passed. The end user might be using some other scenario. Yes, Some times there are bugs in my code. But it functions well if we use the way the application has built.
Management Perspective
Any problem with the application will be treated as Bug if it has impact over the revenue and customer satisfaction.
Final Thoughts
It’s tough to have same perspective across all the people (might happen in an ideal world). The bug for some one may not be the bug for the other person. However there are some set of show stopper bugs for which, the perspective will be the same.
This is my first post in the Bug Life Cycle Series. I need to talk about this because when it comes to the role of testing, it’s not clear. The Role of Software Testing is often mis-understood across the different stake holders of the application development & this list includes testers too.
Testing is considered to be part of Quality Assurance activity since the initial days of Software Development and the same trend is happening as of now too. Even most of the titles like QA Engineer / QA Lead are associated with Testers even though they are not performing the role of QA.
It’s good to capture the mission of Testing & align your test teams in that direction. Every one in the team must be clear on his / her role and see how the same is helping to achieve the mission of the team.
It’s good to note that
- Testing is not about assuring the quality into the systems because The Tester is not a Quality Police.
- Testing is not about targeting for Bug Free Product. It’s just impossible since you can’t build human brain into systems (Of course humans do commit mistakes).
- Testing is not about fighting with the Development Teams. Don’t act like the enemy of developers.
- Testing is not about just looking at the documents (so called BRS, SRS, FRS) and writing the test cases.
- Don’t fight with Developers on the issues need to be fixed for the release instead write good report that reduce the time to reproduce and debug.
Testing is a process followed to make things better. It helps to take informed decision by providing the relevant information based on the context.
My teachers made me better by giving the relevant feedback at every stage on my performance. This includes nurturing the concepts, performance in the tests & week areas in the subject. This information helped me to identify the areas missed / uncovered and to improve upon.
Testers need to identify the critical issues with the system as soon as possible and make sure that the information supplied is sufficient to reproduce the issue. We need to supply the information on the presence of bugs in the system to the stake holders. The information should help the stake holders to take informed decisions.
The following list helps
- Identify the different end users of the system and their interaction with the same.
- Capture the important scenarios for each end user. It’s good to note that we need to capture the story around the scenario and not just steps.
- Talk to different stake holders including the customers (incase if you have access) on how the feature might be used & capture the scenarios.
- Plan towards uncovering the critical issues of the system as early as possible
Final Thoughts
Focus towards what is import for the decision making. Try to uncover important issues first and provide the required information to reproduce and debug the same problem.
In this post, I will be looking at Bugs with an insight that helps testers to learn some new lessons on why the same might have occured. It’s quite common that the testers are blammed for all the missed out bugs in the system as if they are super natural powers to over see all the issues that are there in the system.
Bugs are there every where. Not just software applications, look at any system for that matter & there are issues wrt End User needs and the same might need to evovle over a period of time with the changing needs / requirements from time to time.
What is a Bug ?
It’s tough to arrive at a generic defintion for the bug & people might have heated discussions on what is a bug & what is not. Mostly it’s based on the context in which people are working. We do come across many bugs in our daily life while working with the tools that are needed for the day to day operations.
So in my context, Bug is something that’s unable (doesn’t allow me) to satisfy my needs with the system. What do to do ?
Capture the story around the Bug and not just steps to reproduce. Talk to the reporter and discuss on the scenarios used with the system that lead to the bug.Let’s explore the possible options even though Tester validated the system early
New Test Scenarios for the Testers
-
We mayn’t even think of that scenario while validating the system. Add the similar test scenarios to the Test Suites
-
The scenario may not be logical (though it seemed to be for testers) for the user though functionally correct. Add to the similar scenarios to Test Suites
-
Might have been a compatability issue with a new environment that doesn’t exist in Test Labs
Reduce mistakes and improve test teams capability
-
Might have been missed by a Tester. Check with the tester who performed the Test earlier and find out on why issue missed and the same marked as a pass even though it fails. Might happen through overlook also. Do more analysis here and try to reduce the same.
-
Might be over looked through regression tests. Might have failed and passed in the internal builds and finally breaked in the production system. Analyse the reasons here and improve upon on regression testing.
-
Make it a point to log the bug as soon as they discover the same. People might loose the same from their memory as time goes
In this post, i will share my views & findings with Open Source Test Management Tools. Usage of tools in the Test Management is becoming the basic need & It will be tough to manage the activities associated there with out any support from the tools.
The commercial tools in this segment are costly and may not fit into upcoming organizations budget. So i have been looking at Open Source Test Management tools. In the last week, I have evaluated some open source test management tools.
The following are my requirements for Test Management Tool
-
Capture the Requirements
-
Design Test Cases
-
Map Test Cases with Requirements
-
Link Bug reports with Test Case ID after the test execution
-
Test Execution Reports
-
Version Management for the Test Cases
-
Search Feature
The following tools are evaluated after the initial screening.
The above tools work with Apache, MySQL & PHP. Both the tools looks promising and the Test Link has some additional benefits in-terms of better reports and linking with popular issue trackers.
Solving puzzles, sudokus & the analytical problems is one of my hobby. This set of puzzles always helps me with some brain food which mandatory to stay tuned in the current trends.
I was going through this Testing Triangles post from Elisabeth Hendrickson from Quality Tree Software. The triangle program is very much popular to the world of testers and many of the books and blogs have already covered the same.
Elisabeth posted this as a testing puzzle and asks the people to uncover the bugs in her program. The beauty of this Triangle program is that whole code is available for the tester since it’s written using JavaScript.
I couldn’t stop my self exploring the program and discovering some issues with Testing Triangles and logging them into comments sections while in hurry too.
I thought for a while on this program in the evening and felt that there are many similarities for this even though it’s very very small when compared to the real world applications we test today. So let me take a re-look at this program and apply my test approach for the problem.
Step1: What is the Problem ?
We need to test a program that takes the three sides of a triangle and tell the user about what kind of triangle is it. Here is that Triangle program which has been written in Java Script.
Find the description from the post here
This version of the program takes as input three numbers representing the size of the sides of a triangle. When the user clicks “Draw” the program draws a picture of the triangle with the size of the sides shown in proportion and also displays the type of triangle.
Step2: Question the problem & get the required information for the Test.
The Testers first need to understand the application and the scenarios in which the same might be used.
So with this perception, let’s re-look at the information available for the application. This is true that the above information provided for the application, acts as Requirements, User Manual and any other info we can call it as.
Like many applications that we test now, even this doesn’t have clarity on the information provided. This is also one of the reason for me to call it as a classic example for testing. If we dig more into it the clarity on the information is missing and there are lot of implied requirements (at least from the developer perspective) /assumptions made by the developer.
It’s not clear from the above description that
- What is a Triangle and how to define the same ?
- How many types of triangles are there and how to define each one of them ?
- Are there any boundary limits for the size of the sides that makes triangle ?
- Can any given three numbers make a triangle ?
- etc..
Since the info is not available and the Triangles are very generic. Let’s try to Google around the same and get some info to explore the application.
So get the info here, here, here, and here
Step 3: Sanity / Smoke Test for Triangle Program
By now, I know what a triangle is to begin with and how the same can be drawn. Let me do some sanity checks here.
- For the Test Data of 1, 1, 1 as the values for three edges of triangle the output of the application is “Triangle Type: Equilateral”
- For the Test Data of 2, 2, 3 as the values for three edges of triangle the output of the application is “Triangle Type: Isosceles”
- For the Test Data of 5, 6, 7 as the values for three edges of triangle the output of the application is “Triangle Type: Scalene”
I have started with some very basic positive tests to make sure that the application is inline as per requirements. It’s not good to test the application against negative tests to begin with.
Let’s explore some more tests.
- Try with 0,1,2 – Invalid – wow // might have checked for 0
- Try with -2,3,4 – Invalid – wow // might have checked for -ve values
- Try with 1,2,3 – Invalid – // what ? why is this invalid
For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Invalid”
The Application says it’s invalid. But as per the above spec it should be a Scalene Triangle since all the edges are different from each other.
The following queries came to my mind
- Is this a bug with Application ?
- Am I missing any more info that’s required for the test ?
- Can any given three edges make a triangle ?
- Is there any logic behind the same ?
- But it could have been good,if the application says why the data is invalid. I have no clue on this and it’s a usability issue.
The general tendency that i see in many of the testers is to report it as a bug the moment they see some different behavior with the application and run into many discussion over the same. I have already blogged on it with a post as That’s Expected Behavior and not a Bug. A little more depth into information gathering will be more effective.
Let’s see why this test data is invalid for the application.
I got some more information again over the web on the edges of the triangle that will help me to understand the issue. And this is the logic behind the same. Get more info on this here.
In order to construct a triangle, the sum of any two edges must be greater than the third one.
So with the above data of 1,2,3 we can’t construct a triangle and the data is invalid. But a error message here on why this is invalid from the application could have saved us lot of time in exploring the same.
One more test with 3,4,5
For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Right”
Oh the behavior is different again. i am expecting of Scalene Triangle. Some more info via message could have been helpful for users on why the same has been labeled as that particular type. Might be the developer expecting all the users to have the domain knowledge.
Step4: Thumb Rules of the game
By now, we have capture a bit more info on the application & let’s list the same here
- Rule 1 : Sum of all the angles inside a triangle must be 180 degrees.
- Rule 2 : In order to construct a triangle, the sum of any two edges must be greater than the third one.
- In case of sides, they are called as equilateral triangle, isosceles triangle, right angled and scalene triangle
- In case of angles, they are called as right-angled triangle, obtuse triangle and cute triangle
It’s a good practice to update this section as and when we get the more info about the app behavior & fine tune the same.
Step5 : Black Box Testing on the Application
We have the following info as of now through above test observations.
- Application doesn’t have any validations for the input value to make sure weather it’s a number or not .
- The is at least one test case that has passed for all the types of triangles
Many testers with so much information available now for testing still trying to attack on the system with negative tests (people used to call it as negative testing) to uncover some issues on the first phase itself. Even though we uncover many issues there it won’t be worth trying out the same without exploring the basic functionality of the application.
Let’s explore the application with some more Tests
The side of the triangle must be a numeric value. So the application should allow only that kind of data. Application must draw the triangle with in the area given for the same.
- Try with 6,8,11 – draws out side the box
- Try with 3,4,5 – Right Angled
- Try with 6,8,10 – Scalene- // Why so. why can’t it be right angled ?
- Try with 3,5,10 – Scalene // what ? Basic violation of thumb rules. it must be invalid. more over it just draws a single line and not a triangle
- Try with 6,7,15 – same as above
There are couple of critical bugs here. For an input data which supposed to be invalid, treated as a triangle and other one is that it draws the triangle out of the box.
So there must be some goof up happened while handling the conditions on input data to satisfy them as valid or invalid. This has been proved now with the way that the above tests revealed the application behavior.
Let’s do some code based testing now since it’s a java script and the whole code is available.
Step6: White Box testing on the Application
Let’s look at some important piece of the code.
if (this.s1 >= this.s2 + this.s3) {
return "Invalid";
}if ((this.s3 == this.s2) && (this.s2 == this.s1)) {
return "Equilateral";
}
if (Math.pow(this.s1, 2) == (Math.pow(this.s2, 2) + (Math.pow(this.s3, 2)))) {
return "Right";
}
if ((this.s3 != this.s2) && (this.s2 != this.s1)) {
return "Scalene";
} else {
return "Isosceles";
}
The above code helps us to get the type of triangle based on the input data. Some observations from the above code
- If there are no validations for the input values, then it’s going to be a major problem. Think of the scenarios like (0,0,0) or (-2,3,4)
- For this (this.s1 >= this.s2 + this.s3) condition we need to assume that the input data must be sorted from max to min being the s1 as max value and s3 as min value. Else this is going to be big blowup
Look at this bunch:
Triangle.prototype.setSides = function(s1, s2, s3) {
sides = [s1, s2, s3];
sides = sides.sort(); this.s1 = Number(sides[2]);
this.s2 = Number(sides[1]);
this.s3 = Number(sides[0]);
this.scale = 310/this.s1;
this.scaledSide1 = this.scale * this.s1;
this.scaledSide2 = this.scale * this.s2;
this.scaledSide3 = this.scale * this.s3;
};
It’s reading the input data, sort it and assign the same for the values for the edges. I am concerned about the sort() method here. It’s a block box for us and need to explore on what it does. Other piece of does simple operations.
Let’s ask the google again for some help here. Find some important reading here on the sort method.
Oh my god. This is the problem. It’s doing a generic sort and not tuned towards the numeric sort. So this is the root cause of all the issues. It’s such an issue that even a programmer might over look the same by it’s nature
Find some info on how JavaScript sort works below from the link
Example
In this example we will create an array containing numbers and sort it:
<script type="text/javascript">
var arr = new Array(6)arr[0] = "10"arr[1] = "5"
arr[2] = "40"
arr[3] = "25"
arr[4] = "1000"
arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort())
</script>
|
The output of the code above will be:
10,5,40,25,1000,11,10,1000,25,40,5
|
Note that the numbers above are NOT sorted correctly (by numeric value). To solve this problem, we must add a function that handles this problem:
<script type="text/javascript">
function sortNumber(a,b){return a - b
}
var arr = new Array(6)arr[0] = "10"arr[1] = "5"
arr[2] = "40"
arr[3] = "25"
arr[4] = "1000"
arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort(sortNumber))
</script>
|
The output of the code above will be:
10,5,40,25,1000,11,5,10,25,40,1000
Step7: Summary:
I have taken a this approach for this application since it’s simple to apply & explore in this context. It’s good to always start exploring the applications by gathering information from all walks.
Keep getting the info, fine tune your tests and observe the application behavior. Set the base ground right and dig more and keep digging. You are bound to uncover all the important issues with the application.
