I have been talking about Code Quality through Static Analysis for a while here, here & here. The quality for any application development must begin from it’s gross roots and the Application Code is one of it’s starting point to begin with.Static Analysis techniques help us to identify some metrics over the application code base

• Cyclomatic Complexity
• Application Design & Dependency Metrics
• Exception Handling
• Infinite Loops
• Dead Code
• Performance Issues
• Programming Language guidelines & best practices in their context

The above information will be useful and it’s easy to get the same via Test Framework with a single click.

Code Coverage along with a bunch of unit tests is one widely used technique to help regression testing for the dev & test teams.

Now it’s good to see that Agitar combines Code Coverage & Cyclomatic Complexity to derive risk metrics for Java Code Base. They call it as a code Change Risk Analyzer and Predictor (i.e. CRAP) for Java.

Though it’s a prototype and see how the industry receives the same, i see it as a good initiative on the code quality front and might be a metric going forward for white box testing.

Some useful links for CRAP4J

• Introducing CRAP4J
  
• The Code C.R.A.P. Metric Hits the Fan – Introducing the crap4j Plug-in
• Junit Factory’s Code C.R.A.P metric

Update on Oct 23rd

Now we have a dedicated site for CRAP4J. This contains latest news, forum discussions and many more. I would say that this a good resource for Code Quality lovers & the good news is that they are designed to be open source tools.

Crap4j is a Java implementation of the CRAP (Change Risk Analysis and Predictions) software metric – a mildly offensive metric name to help protect you from truly offensive code.

The CRAP metric combines cyclomatic complexity and code coverage from automated tests (e.g. JUnit tests) to help you identify code that might be particularly difficult to understand, test, or maintain – the kind of code that makes developers say: “This is crap!” or, if they are stuck maintaining it, “Oh, crap!”.

The best way to learn more about CRAP and Crap4j is to check the various articles, newsgroups and blogs about them.

(more…)

Software Testing is considered as a Cost for some Stakeholders & an Investment for others in their context of operation. It’s good to know the status in your context. I had this query in LinkedIn Answers & thought of sharing the good discussion to my blog readers.

Do you consider Software Testing as an Investment or a Cost in your Context ?

Bug Database for the products might have thousands of issues over a period of time against various builds and releases. Though these issues fixed over a period of time, it might be hard to derive meaningful metrics over the release.

We need to support these releases over the production systems & it might be helpful to capture the hot spots / risk elements with the release. Most of the issues here to deal with the respective features, compatibility with other features / technologies & performance related issues.

The usual metrics of number of issues against a module and their severity levels may not be of help always.

How easy is it to derive the following from the Bug Database for a given release

1. Identify the issues have originated from Requirements, Design & Implementation
2. Identify the issues over their category (Functional, Performance, Security, Compatibility, Usability etc)
3. Identify the issues along with their origin & category over the features rather than the modules / components

(more…)

Context driven information is the need of the hour and there is a huge value associated with the same. It’s good to capture the context driven information in the bug reports. My initial experiences with bug reports way back in early 2000 have taught many lessons to improve upon.

Bug reports used to capture what is the problem with the system familiar to the user (tester) who reported the same. People spend very less time to capture all the details required and there are many reasons for the same. I hope some of the upcoming testers will also be in the similar situations.

Some of the Reasons people quote here

• We need to test more and less time to capture & write more information in the Bug Reports
• It’s tough to capture all the information required
• System is complex & It’s tough for the novice users to understand the bug reports
• You know capturing all the info is process driven & it may not be worth of efforts
• It some times boring activity to collate the info & push it
• I can reproduce it on my machine if developer needs it.

This list can go on…

I hope you have come across this situation at least once in your career.

This is my third post in the Bug Life Cycle Series and it’s good to know the different users of your Bugs and their context with them. The mission of your bug report is to provide details and context of the problem and convey the importance of it with a user driven stories.

Your bug report must be the voice of customer and it need play the role of an advocate against the problem. Bug Advocacy from Cem Kaner is an excellent source to begin with. If the bug report unable to specify the need of the context, then it’s better not to write any report

It’s good to explore & capture some of the following problems

• Productivity
• Performance
• Usability
• Migration
• Stability etc

Try to link your issues with most suited functions listed above. It may not be obvious to other users in the system to explore & analyze the issues in that fashion.

There is another context associated with Bug Reports. That’s with the stake holders of the project. The Bug Tracking system must give the right trends and identify the hot spots. Testers must capture the right kind of data to derive better valuable metrics over the bug repository.

Care must be taken to capture

• Capture all the Test Environment details
• Detailed classification on the feature. Classify to the maximum possible sub feature/component of the system
• Clarity on Severity & Priority
• Versions and Build Numbers (Affected & Fixed)
• Bug Classification (Requirements / Design / Implementation etc)
• Bug Types (Functional, Performance, Usability, Security etc)
• It can go on…

The above info helps a lot to identify the trends in bugs and focus on the unstable components / environments.

Final Thoughts
Push the entire context driven information to the bug repository at least for a release cycle and observe the results. Check back with your repository to identify the trends and risk associated with the release and I am sure that it will be in the similar lines of end user feedback.

Happy Testing…

Cyclomatic Complexity (CC) is a software metric (mostly code based metric) used to the number of independent path executions in the application. Introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.

It helps the developers to determine the independent path executions and base line unit tests that they need to validate. Using this, the developers can assure that all the paths have been tested atleast once. It’s a great comfort for the developer and their respective managers.

It’s better to write JUnit Tests for all these linearly-independent paths and integrate it with any code coverage tool. These reports help to focus more on the un covered paths and improve the code coverage.

It also helps to evaluate the risk associated with the application. The following are the results published by SEI and they are being followed widely to determine the health of the code base.

Cyclomatic Complexity Risk Evaluation
1-10 A simple program, without much risk
11-20 More complex, moderate risk
21-50 Complex, high risk program
Greater than 50 Un testable program (very high risk)

Explore more at Cyclomatic Complexity in Software Technology Roadmap from SEI.

Further Reading on the topic

• Cyclomatic Complexity from Wikipedia
• McCabe Metrics
• Code Improvement Through Cyclomatic Complexity
• Cyclomatic Complexity from High Integrity Software Systems Assurance
• Project Metrics Help – Cyclomatic Metrics

Use metrics to evaluate the risk early in the cycle & improve your test coverage.

Happy Testting …

(more…)

It’s almost a month since my last post on this blog and busy with my upcoming release of QuickRules BRMS. I have been talking to the people around on the Software Testing and felt that it’s not communicated well. Though there is enough information on this subject, i would like to describe my own version of the same here.

Testing is about making things better by providing constructive criticism based on the context (we can also say qualitative information and not being nice) at the right time and in the right direction too.

I like the phrase, “Testers, you are the headlights of the project” from the book Lessons Learned in Software Testing.

I have been thinking about this concept helped for the individuals. This revealed lot of crucial information and i hope this helps my fellow testers to motivate them & their teams.

There are many real life testers (incase if we need to list all of them) who contributed & still contributing a lot for us in every phase of life to grow and improve upon (fix the imp bugs among ourselves).Let’s explore some of them below.

My Parents are the first testers in my life. They contributed invaluable information at each stage (Milestone release) of my life. Instead of saying “Sshhhhhh you can’t do that, they used to tell me further implications that might arise”. An insight into this tells us that it’s not an order, but there is context based information for informed decisions.This helped me to stop for a while, analyze the information and work on the required steps to improve upon the current state.

My teachers helped me a lot by providing the constant feedback (just not being nice) through assignments, tests and covey the areas which are good and bad for me in the respective subjects. They are the best testers because they are the ones who taught the concepts and observed my execution towards the same.

There is a tremendous scope for the improvement, incase if we have acted upon the feedback at the right times.

My Boss at work used to evaluate (Test the Tester) me & provide the feedback on the tasks performed by me. This information helps to analyze to identify the next set of steps to be taken for the improvement.

If we look back, there are many testers around us providing the qualitative information to make things better and improve upon.

The Value of this information is tremendous since it came from people who are more experienced and passed through the current stage where we stand. The value lies in the fact that most of the successful people around, learnt a lot from others (learn from others mistakes too rather than your own) and they have become experts in their own fields.

How does this helps Software Testing

Software testing too comes under the similar lines and its role is to provide context driven information for the stake holders to make informed decisions over the application under test (AUT).

So as being testers, we need to provide the constructive criticism at each stage of the Development. If we look back at the above scenarios, the value addition is more because the people involved there have better skills over the context.

That being said, the current industry lacks skilled testers. The true value addition in Software Testing will be more, if and only if the people involved there have better skills over the context they working with.

Do share your views here or send them to me at venkatreddyc@gmail.com

Happy Testing…

This is my first post in the Bug Life Cycle Series. I need to talk about this because when it comes to the role of testing, it’s not clear. The Role of Software Testing is often mis-understood across the different stake holders of the application development & this list includes testers too.

Testing is considered to be part of Quality Assurance activity since the initial days of Software Development and the same trend is happening as of now too. Even most of the titles like QA Engineer / QA Lead are associated with Testers even though they are not performing the role of QA.

It’s good to capture the mission of Testing & align your test teams in that direction. Every one in the team must be clear on his / her role and see how the same is helping to achieve the mission of the team.

It’s good to note that

• Testing is not about assuring the quality into the systems because The Tester is not a Quality Police.
• Testing is not about targeting for Bug Free Product. It’s just impossible since you can’t build human brain into systems (Of course humans do commit mistakes).
• Testing is not about fighting with the Development Teams. Don’t act like the enemy of developers.
• Testing is not about just looking at the documents (so called BRS, SRS, FRS) and writing the test cases.
• Don’t fight with Developers on the issues need to be fixed for the release instead write good report that reduce the time to reproduce and debug.

Testing is a process followed to make things better. It helps to take informed decision by providing the relevant information based on the context.

My teachers made me better by giving the relevant feedback at every stage on my performance. This includes nurturing the concepts, performance in the tests & week areas in the subject. This information helped me to identify the areas missed / uncovered and to improve upon.

Testers need to identify the critical issues with the system as soon as possible and make sure that the information supplied is sufficient to reproduce the issue. We need to supply the information on the presence of bugs in the system to the stake holders. The information should help the stake holders to take informed decisions.

The following list helps

• Identify the different end users of the system and their interaction with the same.
• Capture the important scenarios for each end user. It’s good to note that we need to capture the story around the scenario and not just steps.
• Talk to different stake holders including the customers (incase if you have access) on how the feature might be used & capture the scenarios.
• Plan towards uncovering the critical issues of the system as early as possible

Final Thoughts

Focus towards what is import for the decision making. Try to uncover important issues first and provide the required information to reproduce and debug the same problem.

In this post, i will share my views & findings with Open Source Test Management Tools. Usage of tools in the Test Management is becoming the basic need & It will be tough to manage the activities associated there with out any support from the tools.

The commercial tools in this segment are costly and may not fit into upcoming organizations budget. So i have been looking at Open Source Test Management tools. In the last week, I have evaluated some open source test management tools.

The following are my requirements for Test Management Tool

1. Capture the Requirements

2. Design Test Cases

3. Map Test Cases with Requirements

4. Link Bug reports with Test Case ID after the test execution

5. Test Execution Reports

6. Version Management for the Test Cases

7. Search Feature

The following tools are evaluated after the initial screening.

• TestLink

• RTH

The above tools work with Apache, MySQL & PHP. Both the tools looks promising and the Test Link has some additional benefits in-terms of better reports and linking with popular issue trackers.

The perception from most of the management is that the Testers are the quality police for the products / projects that they develop. This is true for most of the companies where there is no Quality Assurance group.

What do i mean by a Quality Police ?

1. Is the requirements freeze done for the product
2. Did we write all the specs like SRS, Design Docs, Uses Cases
3. Are the developers writing Unit Tests ?
4. Are the developers testing the builds before pass on the same to Test
5. This list goes on…

There are testers who are trying to do the above and i call them as Quality Police .

Where does this attitude come from ?

The attitude of acting like a Quality Police for testers is injected by the management from the beginning of their careers. So they expects the same as they go along with their career and this attitude spoils the relationships between developers and testers.

Acting like Quality Police and trying to get things done is more of process which need to be followed by all the teams and let’s not take that up. Let the management take care of the process management too.

What is Quality ?

Wikipedia describes quality assurance as follows

Quality Assurance is the activity of providing evidence needed to establish confidence among all concerned, that the quality-related activities are being performed effectively. All those planned or systematic actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality. Quality Assurance is a part and consistent pair of quality management proving fact-based external confidence to customers and other stakeholders that product meets needs, expectations, and other requirements. QA (quality assurance) assures the existence and effectiveness of procedures that attempt to make sure – in advance – that the expected levels of quality will be reached.

Read more on Quality Assurance from wikipedia

It’s worth noting here that Good Quality doesn’t come from less bugs in the product. I have seen people whose assumption is that if a product contains bugs it’s of less quality and in case if it doesn’t it’s of more quality. I would say that we can’t measure the quality of the products just with Bugs. I have never seen any product which doesn’t have bugs in it.

Are the Testers responsible for the Product Quality ?

The Testers test the product and report bugs. Some of these bugs will get fixed for the release in case if they are import. You can’t build quality into systems just by reporting bugs and helping developers to fix them.

Assuring the quality into Products under development is the responsibility of all the teams associated with Product Development

So What should be the Role of Tester then

The role of the Tester to test software, find bugs, report them so that they can be fixed. The Bug Reports should be clear, easy to reproduce, reduce the time to debug the same for developers and the report should motivate the developers to fix the issue asap.

Tester should focuses on the software product itself and gathers important information regarding what it does and doesn’t do. The process of gathering should include all the teams associated with the product. Talks to management, sales, architects, development, support and gather their expectations over the release.

Don’t you think that this is a big job all by itself and need lot of time. As i keep saying the Role of the Tester is to provide qualitative information on the product to the management for the better decisions. So the big challenge here is to provide accurate, comprehensive, and timely information about the product under development.

Introduction to KLOC

Lines of Code (LOC) is one of the software metric that is used by most of the people for Software Measurement. Thousand lines of code is treated as KLOC. This metric helps us in knowing the size and complexity of the Software Application. Click here for more details on KLOC. This info is from wikipedia

What does it mean to Software Testing

We do test applications with the intention to see if the promised functionality works fine or not. Any deviation here will be considered as a Bug. So each of these bugs must be originated from some line of code in the product.

So it’s understood that when the size of the code is more there is a chance for more number of bugs in the prodcut. Even most of the process to talk about some % of issues is fine or acceptable quality per KLOC(even though there is lot of subjectivity).

The Defect Density is arrived at Number of Bugs / KLOC per the product under test. The defect density is one of the metric used to measure the quality of the product. Most of the Quality Process does talk about this metric.

Concerns

The concern in this approach is that how these values are measured. The general bias with KLOC is that people try to see that only the excutable lines of code in the product.

Each every line in the product may not be code at all & we may not execute each and every one of them. So it’s not taken care, then the issues related to documentation, images, installation etc might be misleading.

Since we are looking at KLOC as the size of the product it’s better to include each and every entity that effects the same. Then it’s helpful for both development and test teams.