metrics

The following are some typical software testing questions that require measurement to answer:
y

y y y y y y

How big is it? o How long will it take to test it? o How much will it cost to test it? What about the bugs? o How bad were they? What type were they? o What kind of bugs were found? o How many of the bugs that were found were fixed? o How many new bugs did the users find? How much of it has to be tested? Will it be ready on time? How good were the tests? How much did it cost to test it? Was the test effort adequate? Was it worth it? How did it perform?

Good answers to these questions require measurement. If testers don't have good answers to these questions, it is not because there are no applicable metrics; it's because they are not measuring. In this chapter, we discuss the metrics available to answer each of these questions, both fundamental and derived. The techniques used to give you good answers to questions like "How big is it?" are presented throughout the rest of this book; all of these techniques require measurement. In this chapter, I introduce the units and metrics used by these techniques.

A metric is a measure. A metric system is a set of measures that can be combined to form derived measures-for example, the old English system of feet, pounds, and hours. These metrics can be combined to form derived measures as in miles per hour. Measure has been defined as "the act or process of determining extent, dimensions, etc.; especially as determined by a standard" (Webster's New World Dictionary). If the standard is objective and concrete, the measurements will be reproducible and meaningful. If the standard is subjective and intangible, the measurements then will be unreproducible and meaningless. The measurement is not likely to be any more accurate than the standard.

Operational Definitions: Fundamental Metrics

The definition of a physical quantity is the description of the operational procedure for measuring the quantity. For example, "the person is one and a half meters tall." We know from this definition of a person's height by what metric system the person was measured and how to reproduce the measurement. The magnitude of a physical quantity is specified by a number, "one and a half," and a unit, "meters." This is the simplest and most fundamental type of measurement. Derived units are obtained by combining metrics. For example, miles per hour, feet per second, and dollars per pound are all derived units. These derived units are still operational definitions because the name tells how to measure the thing.

What to Measure in Software Testing

Measure the things that help you answer the questions you have to answer. The challenge with testing metrics is that the test objects that we want to measure have multiple properties; they can be described in many ways. For example, a software bug has properties much like a real insect: height, length, weight, type or class (family, genus, spider, beetle, ant, etc.), color, and so on. It also has attributes, like poisonous or nonpoisonous, flying or nonflying, vegetarian or carnivorous.

Fundamental Testing Metrics: How Big Is It?

Fundamental testing metrics are the ones that can be used to answer the following questions.
y y y y

How big is it? How long will it take to test it? How much will it cost to test it? How much will it cost to fix it?

The question "How big is it?" is usually answered in terms of how long it will take and how much it will cost. These are the two most common attributes of it. We would normally estimate answers to these questions during the planning stages of the project. These estimates are critical in sizing the test effort and negotiating for resources and budget.

We quantify "How big it is" with these metrics. These are probably the most fundamental metrics specific to software testing. They are listed here in order of decreasing certainty.

Only time and cost are clearly defined using standard units. Tests and bugs are complex and varied, having many properties. They can be measured using many different units. For example, product failures are a special class of bug-one that has migrated into production and caused a serious problem, hence the word "failure." A product failure can be measured in terms of cost, cost to the user, cost to fix, or cost in lost revenues. Bugs detected and removed in test are much harder to quantify in this way. Time Units of time are used in several test metrics, for example, the time required to run a test and the time available for the best test effort. Let's look at each of these more closely. The Time Required to Run a Test This measurement is absolutely required to estimate how long a test effort will need in order to perform the tests planned. It is one of the fundamental metrics used in the test inventory and the sizing estimate for the test effort. The time required to conduct test setup and cleanup activities must also be considered. Setup and cleanup activities can be estimated as part of the time required to run a test or as separate items. Theoretically, the sum of the time required to run all the planned tests is important in estimating the overall length of the test effort, but it must be tempered by the number of times a test will have to be attempted before it runs successfully and reliably.
y

Sample Units: Generally estimated in minutes or hours per test. Also important are the number of hours required to complete a suite of tests.

The Time Available for the Test Effort This is usually the most firmly established and most published metric in the test effort. It is also usually the only measurement that is consistently decreasing.
y

Sample Units: Generally estimated in weeks and measured in minutes.

The Cost of Testing The cost of testing usually includes the cost of the testers' salaries, the equipment, systems, software, and other tools. It may be quantified in terms of the cost to run a test or a test suite. Calculating the cost of testing is straightforward if you keep good project metrics. However, it does not offer much cost justification unless you can contrast it to a converse-for example, the cost of not testing. Establishing the cost of not testing can be difficult or impossible. More on this later in the chapter.

Sample Units: Currency, such as dollars; can also be measured in units of time.

Tests We do not have an invariant, precise, internationally accepted standard unit that measures the size of a test, but that should not stop us from benefiting from identifying and counting tests. There are many types of tests, and they all need to be counted if the test effort is going to be measured. Techniques for defining, estimating, and tracking the various types of test units are presented in the next several chapters. Tests have attributes such as quantity, size, importance or priority, and type Sample Units (listed simplest to most complex):
y y y y y

A keystroke or mouse action An SQL query A single transaction A complete function path traversal through the system A function-dependent data set

Bugs
Many people claim that finding bugs is the main purpose of testing. Even though they are fairly discrete events, bugs are often debated because there is no absolute standard in place for measuring them.
y

Sample Units: Severity, quantity, type, duration, distribution, and cost to find and fix. Note: Bug distribution and the cost to find and fix are derived metrics.