Measuring improvement is essential, not only because it can confirm that a correct decision was made, but because the process of defining a measurement method forces one to thoroughly analyze exactly what the proposed change is attempting to improve.
On the other hand, one must be aware that defining a measurement can actually affect the problem and solution. Once people become aware of the measurement they will often tend to work directly towards improving the value itself rather than following the policy intended to improve what this value is supposed to be measuring.
For instance, IQ scores are no longer considered a direct measure of
intelligence, but a measure of a person's ability to write IQ tests.
In sporting events, the replacement of human referees by electric
indicators has changed the style of the sport; a fencer's goal has
now become simply to depress the button at the end of the sword by
whatever means are legally allowed.
The justice system measures success by the percentage of convictions,
not by how just those convictions are.
Police in Japan traditionally discourage autopsies in order
to keep their district's murder rate low.
And drug companies love to sell products that alter measurable
attributes of blood tests, even though changing those measurements
may have no beneficial effect on the disorders that produce the symptoms.
As an old Alka Seltzer® commercial
used to admit,
… we wouldn't have invented a disease unless we had something to take for it..
Similarly, if the goal is to improve on-time production, and the measurement is defined to be the percentage of jobs that are finished on time, workers may ignore the policy changes and still greatly improve this measure simply by allowing a few large jobs to be very late, thereby ensuring that all other jobs are completed on time. Such actions may be financially damaging, yet the measurement might lead management to think that the new policy, which is actually being ignored, is working well.
In general, any measurement must be tightly connected to the goal. The more ways there are to change that measurement without affecting the goal, the less reliable and useful that measurement will be. Yet despite this, many people and organizations love to promote the changes in their measured values, ignoring the fact that the original purpose has long since been lost.