I have been talking about Code Quality through Static Analysis for a while here, here & here. The quality for any application development must begin from it’s gross roots and the Application Code is one of it’s starting point to begin with.Static Analysis techniques help us to identify some metrics over the application code base

• Cyclomatic Complexity
• Application Design & Dependency Metrics
• Exception Handling
• Infinite Loops
• Dead Code
• Performance Issues
• Programming Language guidelines & best practices in their context

The above information will be useful and it’s easy to get the same via Test Framework with a single click.

Code Coverage along with a bunch of unit tests is one widely used technique to help regression testing for the dev & test teams.

Now it’s good to see that Agitar combines Code Coverage & Cyclomatic Complexity to derive risk metrics for Java Code Base. They call it as a code Change Risk Analyzer and Predictor (i.e. CRAP) for Java.

Though it’s a prototype and see how the industry receives the same, i see it as a good initiative on the code quality front and might be a metric going forward for white box testing.

Some useful links for CRAP4J

• Introducing CRAP4J
  
• The Code C.R.A.P. Metric Hits the Fan – Introducing the crap4j Plug-in
• Junit Factory’s Code C.R.A.P metric

Update on Oct 23rd

Now we have a dedicated site for CRAP4J. This contains latest news, forum discussions and many more. I would say that this a good resource for Code Quality lovers & the good news is that they are designed to be open source tools.

Crap4j is a Java implementation of the CRAP (Change Risk Analysis and Predictions) software metric – a mildly offensive metric name to help protect you from truly offensive code.

The CRAP metric combines cyclomatic complexity and code coverage from automated tests (e.g. JUnit tests) to help you identify code that might be particularly difficult to understand, test, or maintain – the kind of code that makes developers say: “This is crap!” or, if they are stuck maintaining it, “Oh, crap!”.

The best way to learn more about CRAP and Crap4j is to check the various articles, newsgroups and blogs about them.

(more…)

Bug Database for the products might have thousands of issues over a period of time against various builds and releases. Though these issues fixed over a period of time, it might be hard to derive meaningful metrics over the release.

We need to support these releases over the production systems & it might be helpful to capture the hot spots / risk elements with the release. Most of the issues here to deal with the respective features, compatibility with other features / technologies & performance related issues.

The usual metrics of number of issues against a module and their severity levels may not be of help always.

How easy is it to derive the following from the Bug Database for a given release

1. Identify the issues have originated from Requirements, Design & Implementation
2. Identify the issues over their category (Functional, Performance, Security, Compatibility, Usability etc)
3. Identify the issues along with their origin & category over the features rather than the modules / components

(more…)

Context driven information is the need of the hour and there is a huge value associated with the same. It’s good to capture the context driven information in the bug reports. My initial experiences with bug reports way back in early 2000 have taught many lessons to improve upon.

Bug reports used to capture what is the problem with the system familiar to the user (tester) who reported the same. People spend very less time to capture all the details required and there are many reasons for the same. I hope some of the upcoming testers will also be in the similar situations.

Some of the Reasons people quote here

• We need to test more and less time to capture & write more information in the Bug Reports
• It’s tough to capture all the information required
• System is complex & It’s tough for the novice users to understand the bug reports
• You know capturing all the info is process driven & it may not be worth of efforts
• It some times boring activity to collate the info & push it
• I can reproduce it on my machine if developer needs it.

This list can go on…

I hope you have come across this situation at least once in your career.

This is my third post in the Bug Life Cycle Series and it’s good to know the different users of your Bugs and their context with them. The mission of your bug report is to provide details and context of the problem and convey the importance of it with a user driven stories.

Your bug report must be the voice of customer and it need play the role of an advocate against the problem. Bug Advocacy from Cem Kaner is an excellent source to begin with. If the bug report unable to specify the need of the context, then it’s better not to write any report

It’s good to explore & capture some of the following problems

• Productivity
• Performance
• Usability
• Migration
• Stability etc

Try to link your issues with most suited functions listed above. It may not be obvious to other users in the system to explore & analyze the issues in that fashion.

There is another context associated with Bug Reports. That’s with the stake holders of the project. The Bug Tracking system must give the right trends and identify the hot spots. Testers must capture the right kind of data to derive better valuable metrics over the bug repository.

Care must be taken to capture

• Capture all the Test Environment details
• Detailed classification on the feature. Classify to the maximum possible sub feature/component of the system
• Clarity on Severity & Priority
• Versions and Build Numbers (Affected & Fixed)
• Bug Classification (Requirements / Design / Implementation etc)
• Bug Types (Functional, Performance, Usability, Security etc)
• It can go on…

The above info helps a lot to identify the trends in bugs and focus on the unstable components / environments.

Final Thoughts
Push the entire context driven information to the bug repository at least for a release cycle and observe the results. Check back with your repository to identify the trends and risk associated with the release and I am sure that it will be in the similar lines of end user feedback.

Happy Testing…

Cyclomatic Complexity (CC) is a software metric (mostly code based metric) used to the number of independent path executions in the application. Introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.

It helps the developers to determine the independent path executions and base line unit tests that they need to validate. Using this, the developers can assure that all the paths have been tested atleast once. It’s a great comfort for the developer and their respective managers.

It’s better to write JUnit Tests for all these linearly-independent paths and integrate it with any code coverage tool. These reports help to focus more on the un covered paths and improve the code coverage.

It also helps to evaluate the risk associated with the application. The following are the results published by SEI and they are being followed widely to determine the health of the code base.

Cyclomatic Complexity Risk Evaluation
1-10 A simple program, without much risk
11-20 More complex, moderate risk
21-50 Complex, high risk program
Greater than 50 Un testable program (very high risk)

Explore more at Cyclomatic Complexity in Software Technology Roadmap from SEI.

Further Reading on the topic

• Cyclomatic Complexity from Wikipedia
• McCabe Metrics
• Code Improvement Through Cyclomatic Complexity
• Cyclomatic Complexity from High Integrity Software Systems Assurance
• Project Metrics Help – Cyclomatic Metrics

Use metrics to evaluate the risk early in the cycle & improve your test coverage.

Happy Testting …

(more…)

I have been working on Java Projects for quite some time and interested in exploring the source code metrics like size and complexity across project not specific to perticular package or class and looking after LOC (Line of Code) counters to capture some metrics of the source code.

Explored some tools and found that JavaNCSS is a good match for the context of sizing metrics on Java. JavaNCSS is a command line utility which measures some standard source code metrics for the Java programming language. The metrics are collected globally, for each class and/or for each function.

The following are some of the advantages that i have seen

1. Support for Ant Tasks, so easy to integrate with build process
2. Reports can be in Text, XML, HTML etc
3. Support for Stylsheets and easy to get nice HTML reports
4. Metrics at each level Package / Class / Method
5. Cyclomatic Complexity Number
6. List the number of packages / classes / functions / LOC counter at each level

Further Reading:

1. JavaNCSS Home
2. LOC Counters for C++ / Java on Joel Software Discussion group
3. SLOC on Wikipedia

Introduction to KLOC

Lines of Code (LOC) is one of the software metric that is used by most of the people for Software Measurement. Thousand lines of code is treated as KLOC. This metric helps us in knowing the size and complexity of the Software Application. Click here for more details on KLOC. This info is from wikipedia

What does it mean to Software Testing

We do test applications with the intention to see if the promised functionality works fine or not. Any deviation here will be considered as a Bug. So each of these bugs must be originated from some line of code in the product.

So it’s understood that when the size of the code is more there is a chance for more number of bugs in the prodcut. Even most of the process to talk about some % of issues is fine or acceptable quality per KLOC(even though there is lot of subjectivity).

The Defect Density is arrived at Number of Bugs / KLOC per the product under test. The defect density is one of the metric used to measure the quality of the product. Most of the Quality Process does talk about this metric.

Concerns

The concern in this approach is that how these values are measured. The general bias with KLOC is that people try to see that only the excutable lines of code in the product.

Each every line in the product may not be code at all & we may not execute each and every one of them. So it’s not taken care, then the issues related to documentation, images, installation etc might be misleading.

Since we are looking at KLOC as the size of the product it’s better to include each and every entity that effects the same. Then it’s helpful for both development and test teams.

I was going through the post Why counting is a bad idea on Shrini’s Blog on Software Testing. Shirni has listed a classic example on why counting is a bad idea for Testing.

Even though i like this post, I won’t agree fully on the views expressed there. Probably we might need to improve our selves with the process of measuring.

Certain artifacts like test cases, requirements and bugs are not countable things and any attempt to count them can only lead to manipulations and ill-informed decisions.

Wait — Are there any things at all in testing that we can count without loss of effectiveness and usefulness of the information that a counted number might reveal?

Let me describe my stand clearly wrt to this Why counting is a bad idea.

I do agree that it’s not posssible to count all the activities that we do in testing. But then that doesn’t mean like all the Test related metrics are bad. May be, We can’t generalize them since it leads to lot of subjective debate.

When it comes push the usefulness and effectiveness of the information by the metric, this is more of a improvement process. All these metrics are subjective to the respective projects / environments / organizations. Any attempt in making this as a genaral metric will lead to manipulations and ill-informed decisions.

1. Bad Idea for whom ?
   Is this for Customers / Management / Developers / Testers / Others
2. If Counting is a bad Idea, how do we measure the effort spent for the Project ? Do we need to go by the gut feelings of te people ? Will there be any data to justify our decisions ?

Most of the metrics used for Software Development for that matter are subjective. It varies based on the application, technology, domain, global support etc. It’s tough to make them general like we do in other industries. So the same will be applicable for Testing too.

Some thing like below are very much subjective and won’t be accepted in genaral.

Productivity

1. No of Test cases prepared Per Person Per hour = 5
2. No of Test cases executed per person per hour = 15

Look at the subjectivity in the above scenario.

1. What is the structure of the Test Case ?
2. How many user actions will be there in genaral per TestCase ?
3. What is the Volume of Test Data associated with each TestCase ?
4. What information available for these Test Cases (SRS, Design, UseCases etc) ?
5. What kind of test cases are they writing ? Are they writing for Unit, Integration, System, Accepatance, Performance etc..
6. What kind of Test Design Tech are they following (BVA, EP, DT, OA, CE etc)

There will be so many queries that will be come up here and it’s tough to measure some one just based on the number in genaral. But then since it’s subjective to the project (team), the respective stake holder should be able to take some decision based on that number.

But then, we can’t make this number generic. Probably this might be useful to the similar projects to have some rough estimate.

At the end, i shouldbe able to give qualitaive (again this is subjective) to the stake holders. If some one expects 100% qualitaive info here, that’s an ideal world to which we can’t go as of now.

What does that mean ?

I should be able to give a number. My metrics should tell me the number of Test Cases, Issues etc wrt the requirement.

Otherwise we may not have clarity on what we are doing & what are we trying to deliver. Things will be going on the pure gutfeelings

Pradeep has come up with nice replies to my comments on the post.

@Pradeep,

Thanks for sharing your views.

Let me look at your requirement

” This field should take only alphabets and nothing else other than that “

In the first place, the requirement is not clear for me

1. What kind of Application are we taking about ?
2. What is the size of this field (Some thing like password. It should have a Min and Max length)
3. When you say Alphabets, Do you mean to support all the alphabets in all the languages or just english or some thing else?
4. ????

See Pradeep, my queries will go on like this and all these will help me in getting the clarity on the requirement. This is how i try to to understand the system.

This process helps me in knowing on what is the requirement, it’s advantages, it’s limitations and most possible scenarios where this can be used.

All the above will help me in capturing the required tests for the above requirement.

But this may not be generic on the whole but then it can be generic to the Organization or at least to the project people are working with.

See there should be some way to measure on what we are doing. This measure should help the stake holders to take a decision on the quality of the application under test.

I do agree on the fact that we need to improve ourselves in coming up with measures that have qualitative information for the stake holders.

@ Shrini, Thanks for coming up a post that has good debate.