You are on page 1of 5

Useful Benchmarking Techniques in Computer

Over the years, there has been a tremendous development in benchmarking
techniques, in terms of both benchmarking practices and method, which are widely
used in different organization to achieve different goals in the developing

Benchmarking is simply the comparison of one organization's practices and
performance against those of others. It seeks to identify standards, or "best
practices," to apply in measuring and improving performance. „Benchmarking‟ is a
term that is now widely used within the quality arena. Benchmarking involves
comparing a set of products or services against the best that can be found within
the relevant industry sector.
Benchmarking at its best is used as a tool to help our business evaluate
opportunities for improvement. There are many benefits of benchmarking, the top
reasons include:
 Identify and priorities specific areas of opportunity
 Understand your customers needs better
 Identifying your strengths and weaknesses
 Set goals and performance expectations
 Monitor your performance and effectively manage change
 Understand your competitors to become more competitive.

Quality Assurance:
Quality Assurance (see also Inglis and McConachie, Danaher, Luck and Jones, this
issue) is a process oriented to guaranteeing that the quality of a product or a service
meets some predetermined standard. Quality assurance makes no assumptions
about the quality of competing products or services. In practice, however, quality
assurance standards would be expected to reflect norms for the relevant industry.
The process of quality assurance therefore compares the quality of a product or
service with a minimum standard set either by the producer or provider or by some
external government or industry standards authority. By rights, this standard
should bear some relationship to best practice, but this is not a necessary condition.
The aim in quality assurance is to ensure that a product or service is fit for the

Quality Improvement:
Quality Improvement (see also Cummings, Phillips, Tilbrook and Lowe, this issue)
is concerned with raising the quality of a product or service. The type of
comparison that is made when engaged in quality improvement is between the
current standard of a product or service and the standard being aimed for. Quality
improvement is concerned with comparing the quality of what is about to be
produced with the quality of what has been produced in the past. Quality
improvement is therefore primarily concerned with self rather than with others.
Processes focused on quality improvement are also focused more on specific
aspects of an organizational unit‟s performance than on overall performance. It is
usually the case that constraints dictate that efforts at improvement need to be
targeted at areas of greatest need.

As computer architecture advanced, it became more difficult to compare the
performance of various computer systems simply by looking at their specifications.
Therefore, tests were developed that allowed comparison of different architectures.
For example, Pentium 4 processors generally operate at a higher clock frequency
than XP processors, which does not necessarily translate to more computational
power. A slower processor, with regard to clock frequency, can perform as well as
a processor operating at a higher frequency. Benchmarks are designed to mimic a
particular type of workload on a component or system. Synthetic benchmarks do
this by specially created programs that impose the workload on the component.
Application benchmarks run real-world programs on the system. While application
benchmarks usually give a much better measure of real-world performance on a
given system, synthetic benchmarks are useful for testing individual components,
like a hard disk or networking device. CPUs that have many execution units —
such as a superscalar CPU, a VLIW CPU, or a reconfigurable computing CPU —
typically have slower clock rates than a sequential CPU with one or two execution
units when built from transistors that are just as fast. Nevertheless, CPUs with
many execution units often complete real-world and benchmark tasks in less time
than the supposedly faster high-clock-rate CPU.
Given the large number of benchmarks available, a manufacturer can usually find
at least one benchmark that shows its system will outperform another system; the
other systems can be shown to excel with a different benchmark.
Manufacturers commonly report only those benchmarks (or aspects of
benchmarks) that show their products in the best light. They also have been known
to mis-represent the significance of benchmarks, again to show their products in
the best possible light. Taken together, these practices are called bench-marketing.
Ideally benchmarks should only substitute for real applications if the application is
unavailable, or too difficult or costly to port to a specific processor or computer
system. If performance is critical, the only benchmark that matters is the target
environment's application suite.

Benchmarking is not easy and often involves several iterative rounds in order to
arrive at predictable, useful conclusions. Interpretation of benchmarking data is
also extraordinarily difficult. Here is a partial list of common challenges:
 Vendors tend to tune their products specifically for industry-standard
benchmarks. Norton SysInfo (SI) is particularly easy to tune for, since it
mainly biased toward the speed of multiple operations. Use extreme caution
in interpreting such results.
 Some vendors have been accused of "cheating" at benchmarks — doing
things that give much higher benchmark numbers, but make things worse on
the actual likely workload.

 Many benchmarks focus entirely on the speed of computational
performance, neglecting other important features of a computer system

Types of benchmarks:
1. Real program
o Word processing software
o Tool software of CAD
o User's application software

2. Component Benchmark
o Core routine consists of a relatively small and specific piece of code.
o Measure performance of a computer's basic components
o May be used for automatic detection of computer's hardware
parameters like number of registers, cache size, memory latency, etc.

Six steps to successful benchmarking:
1. Identify what you're going to benchmark
2. Identify your competitors
3. Look at trends
4. Outline objectives
5. Develop an action plan for your objectives
6. Monitor your results and implement an action plan

 For computer industry “Identify your Competitors” is more useful. Most
businesses benchmark within the same industry. Identify effective tactics
used by your competitors, and areas in which their business is performing
better and note these down.

According to the research of benchmarking activities, there are some reasons
leading to the failure of benchmarking. Davies et al, (1999) listed some of the
reasons as follow: "preoccupation with metrics, industrial tourism, benchmarking
being mistaken for competitive analysis, lack of implementation of findings, lack
of planning - resulting in poor findings, lack of structure in benchmarking project,
failure to involve all levels and areas of organization, perception of the need to
benchmark, belief that a company is unique or studies too large and superficial".