Introduction

This paper presents a graphic framework for understanding the components necessary for the successful use of data in decision making. We have found that the ideas and concepts presented here can be used by anyone, whether management or practitioner, to improve the effectiveness of a measurement program. Our position is that measurement must be justified by the decisions which require it and that these decisions must be clearly visible to all involved in the measurement program.

Data is sometimes requested simply because someone believes that by merely gathering the data, things will improve. This is naive. What we really want is improvement. We want change with the key goal of real, measurable improvement. Knowing when and what to change will separate teams which will succeed and prosper from those which will be out looking for jobs and wondering what went wrong.

The models presented in this paper have been used over the past several years in an environment of engineers, engineering management, software developers and business management. We have seen the models used many different ways to interpret different project management programs. It is often used as a shorthand to explain a sometimes foreign and complex process. It is also used as a graphic checklist of components and attributes required for management with data programs. Once the model has been presented and used once, the graphics allow people who are swamped with information to quickly recall the fundamental concepts. In this way, they can get down to the important issues without having to internalize the basic concepts each time metrics issues are presented.

This paper explains the model and concept in much the same way as we do. We call the basic model the "Single Engine Model" and always present it first. The model is extensible and we have been amazed at the different ways people use it as a fundamental building block. We present an extension oriented toward process improvement; this extension we call the "Two Engine Model".

Measurement Customers

There have always been those who want to direct change and each has had to express what they wanted to happen. There has typically been little concern with measuring results. The situation is best described by the phrase "Royal Dictates". Figure 1 depicts this behavior. First there is the box labeled "CONTROL". This box represents the agency with the authority to direct a change. This role is sometimes called the process owner or process manager. The "ACTION" represents the change ordered. The bottom box represents the target of the action. In our example it contains those who are to effect the change, the "PROCESS". The "PROCESS" is where the needs of a customer are transformed into deliverables for there end-user.

Figure 1, Royal Dictates

Today, managers know that successful organizations have measurement programs. Some conclude that if they are to be successful, they too need one. Figure 2 depicts behavior best described as "measurement for the curious". This arises when CONTROL asks for data about the PROCESS, but has no visible plan for using the data; it appears that CONTROL is merely curious - or worse, has a hidden agenda. The phrase "what gets measured, gets done" is sometimes wrongly equated with this situation. Measurement for the Curious annoys if the data is not used, for its collection involves additional work. It creates suspicion (and possibly program sabotage) if later used for purposes initially unstated.

Figure 2, Measurement for the Curious.

In Measurement for the Curious, it often makes sense for PROCESS to spend as little effort as possible and still meet the measurement request. There is little reason to do more.

Most initial measurement programs are really "guessing" programs and the quality of data gathered is typically poor. The use of the data for decision making has not been planned or communicated.

Single-Engine Model

PROCESS

The fundamental process transforming Needs into Deliverables and results. It is monitored by MEASUREMENT and its behavior is influenced by ACTION directed by CONTROL.

MEASUREMENT

The quantitative evidence regarding the state of the target PROCESS and its products. There are Process and Product measurements and satisfaction of downstream customers who communicate product Value.

CONTROL

The decision making mechanism using MEASUREMENT, Goals, and Constraints to direct ACTION.

ACTION

The response of CONTROL to influence the target PROCESS in desirable directions.

CYCLE

Dynamic characteristics of the model such as cycle time.

Figure 3, Management with data framework, the Single Engine Model

PROCESS behavior depends on ACTION

• A process must exist at the organizational level indicated by CONTROL. In the absence of such a process, one can only control people directly, a poor basis for quality improvement.
• The process must exhibit measurable behavioral effects in response to the available actions. Otherwise effective control of the process is impossible by the available actions and it is impossible to associate cause and effect .
• It must be known - or at least determinable from experience - what process performance trends are likely to be associated with available actions. Otherwise, the available actions lack predictability. This would suggest the available actions are not strongly correlated with process performance.

MEASUREMENT depends on Process Internals, Entry and Exit Criteria, and Value

• The measurements selected must, at minimum, reflect the degree to which the goals of the process have been achieved. Otherwise "improvement" can't reliably be demonstrated by measurements.
• The measurements selected should, if at all possible, include some which quickly respond to actions taken. Otherwise the lag times may make it difficult to associate cause and effect. Rapid confirmation of improvement is important to motivation.
• The measurements selected must be sufficiently reliable that action can reasonably be based on them. Otherwise "managing with data" is an illusion. Guessing and vagueness in definitions are often the cause of unreliable metric data showing excessive variation.

CONTROL depends on timely and reliable data.

In considering appropriate CONTROL, several factors are of importance:

• Explicit goals and constraints must be known to the control agency. Otherwise improvement is undefined and ACTION is without direction.
• A forum must be specified in which measurement is presented to the CONTROL agency. Otherwise the communication of the state of the process to the CONTROL agency is not reliable and the model degenerates to Royal Dictates.
• Conditions must be stated under which specified measurements are presented to the control agency. Otherwise the data may not be timely or may not be relevant.

ACTION depends on GOALS and MEASUREMENT

• A range of actions must be available. If there is only one possible control action, there is no decision to be made. It is our position that measurement programs can only be justified by the decisions they support.
• A forum must be presented in which ACTION is presented to PROCESS. Otherwise the communication of the ACTION to the PROCESS is not reliable and the model degenerates to Measurement for the Curious. The actions taken must truly be based on the stated goals and measurements. If not, we are not really managing with data, but are driven by hidden agendas and opinion.

CYCLE Attributes

• The time required to initiate an action in response to changes in goals or measurements must be small compared to the response time of the target process. Otherwise the available actions aren't timely in controlling the process.
• The time required to complete an engine cycle should be small in comparison to other changes in the environment. Otherwise changes in needs, constraints, business objectives and even process technology will render measurement ineffective.

Shared Memory

One such repository is in the experiences of the practitioners. We refer to this as "individual professionalism". Unfortunately, this is not a reliably shareable memory. An obviously superior repository of lessons learned is a documented statement of Professional Practice for the group. This documented description can be shared between workers and rationally analyzed. The use of Professional Practices can be an instance of "organizational professionalism".

We next turn to a model of process improvement. We assume the processes are documented in a group or organizational statement of Professional Practice. These practices are to be improved and improvement is to be based on the evidence of our measurements.

Figure 4. The Two Engine Model

Two Engine Model

All processes (Professional Practices) should have parameters which enable tailoring to satisfy local needs. These parameters are controlled in the bottom engine.

We comment on and illustrate some especially important points of the Two Engine Model in the following sections.

Customer

• The bottom engine produces a computer program. The customer is the end-user for that program.
• The bottom engine produces a requirements document. The customer is a software designer.

Process Improvement

In general, a given bottom engine can utilize professional practices from several top engines, each specialized to processes specific to a certain domain. For example, a software development process could utilize coding standards, a project planning process standard and a configuration management process standard.

Similarly, one top engine may produce professional practices which are used by many bottom engines. For example, there could be many software development processes using the same project tracking and oversight standard. The same top engine could be collecting data on the results of the professional practice use and be continuously improving the standard.

Even where both the bottom engine (product development) and the top engine (process development and improvement) are being done by the same individuals and controlled by the same manager, the two engine model still is of value. By understanding the difference between using a process and improving it, the team will be able to differentiate between process changes for improvement and change for the sake of change.

Internal and External Process Measurement

Entry Criteria:

A definition of needed input to the PROCESS and their suitability to the task. For instance, a requirements document is needed by PROCESS and it must be in writing.

Exit Criteria:

A definition of external deliverables produced by PROCESS and measurable attributes of cost, schedule and quality. For instance, source code is produced by PROCESS and the functionality is traceable to the requirements.

Value:

The results of the use of the products of PROCESS as reported by customers or end-users. For instance, customer satisfaction surveys are used to assess the way users view the relationships of cost, schedule and quality. Do they think that they are overpaying for the quality they are receiving?

On the other hand, internal measurements are taken within the development process and relate to intermediate products and process characteristics. In Figure 4, the measurement line marked "Internal" represents such measures. If present, these measures imply that we know something of the internals of PROCESS. For example, if PROCESS is using Professional Practices, we can measure the penetration and compliance of the Professional Practice. If the work flow internal to PROCESS is known, we can measure the difference between actuals and estimates for cost and schedule.

Cycle Time

Limits of a Model

The model does not detail other important characteristics of a successful software organization, for example:

• The incentive process used to motivate the various players
• The processes used for deployment and training.
• The process of the customer in reaction to the products and deliverables of the supplier. Model extensions to this area have been created and are interesting, but are too lengthy to address here.

Conclusion