The fundamentals of AGV acceptance testing

Acceptance testing may be the final step in the design and implementation of an
AGV system, but it’s also one of the most important. Here’s a look at some of the
fundamentals that guide the industry today.

Automatic Guided Vehicle Systems Product Section of Material Handling Industry

of America – Copyright 2009

“Accepting testing should never be in doubt and should never be a surprise,” says one
veteran project manager in the automatic guided vehicle systems (AGVS) industry.

What he means is this: If an AGV supplier and customer have worked together to design,
simulate, install and verify the right system from the start, acceptance testing should be a
forgone conclusion.

At the same time, before an AGV system can be handed off, suppliers and customers
alike need a defined process to verify that the system is operating as designed. That’s
where acceptance testing comes into play. After the implementation is complete and a
system goes live, the customer and the supplier measure the performance of the system
against the criteria defined at the outset of the project.

While every AGV system is unique, there are basic practices being used by the members
of the AGVS Product Section of the Material Handling Industry of America (MHIA).

Design for success

For those unfamiliar with the technology, an AGV system consists of driverless vehicles
that are controlled by software and automatically move materials throughout a facility.
While AGV systems are most often found in manufacturing operations, vehicles are also
now being deployed to lower costs and improve productivity in large warehouses and
distribution centers. Automated truck loading vehicles are replacing lift trucks at the
loading dock. Systems may range in size from one or two vehicles that move materials
between work stations to 100 vehicle systems that interface with other materials handling
systems to route materials throughout a facility or between facilities.

Regardless of the size, the first step towards the successful acceptance of an AGV system
is taken long before a solution ever goes live.

During the definition stage of a project, the system supplier and the customer will create a
specification document, also known as a bounded solution, that defines in specific detail
the following:
 1) The problem the system will solve, or the operating environment where the
system will be installed, including the pickups and deliveries the vehicles will
execute, the size of the loads and other systems the vehicles will interface with.
2) The performance characteristics for the system. While throughput, or the number
of loads the system is designed to handle during a day, is the ultimate goal,
performance characteristics might also include variables like response and travel
times, maintenance and materials costs, and system availability down time.

Computer simulation is one tool often used for designing large systems and verifying and
predicting their performance; for smaller systems with only a few vehicles, mathematical
calculations plugged into a spreadsheet are sufficient for verifying performance.

In addition to defining the performance requirements, the specification document should

also specify the acceptance criteria that will be used for acceptance testing. Some
university studies have suggested that contracts and specification documents can be
improved by:

* Clearly defining system failure

* Using precise and consistent terminology, especially for reliability and availability

* Clearly defining the conditions, or load, under which systems will be tested

* Clearly defining the repair process to be used during tests

* Employing statistically derived tests with stated risks, and

* Clearly specifying the implications of test failure.

What’s more, the specification document should also include the resources that the
supplier will be required to provide for the acceptance test, like a technician, controls
engineer and test coordinator, and the costs associated with testing.

Finally, in addition to specification and acceptance testing, the definition stage should
include specifics about how change orders will be handled after the specification
document has been accepted by the customer and the supplier. The reason is simple: One
of the leading causes of a performance failure is scope creep, that is a project that
expands after the initial system has been defined.

The factory acceptance test

Once a system has been designed to everyone’s satisfaction, an implementation plan is

created to install the system and go live. Before that happens, some system suppliers
include an additional step: a factory acceptance test.
The factory acceptance test provides an opportunity for the buyer to preview equipment
and software prior to its arrival on site. The test should be conducted to demonstrate to
the buyer the required or specified functionality for the item being reviewed.

For the control system, the test should demonstrate the functionality of the system in
standalone mode utilizing emulators for equipment interfaces. The buyer can then
critique the demonstration as to the expected functionality, user interface and ease of use.

For the equipment, the test should demonstrate the equipment operating with rated loads
and at rated speeds outlined in the specification document. The test should provide for a
minimum operation time without errors or problems.

Buyer acceptance at the factory indicates that the equipment is ready to be installed at the
site. If the buyer doesn’t have the time or resources to participate in a factory acceptance
test, the supplier should execute the test and certify the results.

Evaluation, delivery and acceptance

Once an AGV system has been installed, integrated with other material handling
equipment, and debugged, it’s ready for acceptance testing. This is where the
performance characteristics of the system meet the acceptance criteria.

The point is to determine whether the system is performing according to the standards
outlined and agreed to in the specification document.

For instance, is the system meeting its throughput numbers, including the time required
for battery charging as well as scheduled and unscheduled down time? In addition to
throughput, a system may be judged on:

* Time-based criteria, including characteristics like the time it takes to load or unload a
vehicle or the time a load spends waiting in a queue for transportation;

* Cost-based characteristics such as the fixed initial cost of the system or the overhead
costs associated with managing the overall system, including programming, controls and
upgrades;

* Quality-based characteristics, including the ability of the system to pickup/deliver the

correct quantity of material or the ability of the system to pickup/deliver to the correct
location.

Since AGV systems are designed for specific operating environments, the customer and
the supplier can determine the performance and acceptance criteria most important for
that application, based on the system design as well as the performance needs and
expectations.
For example, one company with a 30-vehicle AGVS system in a manufacturing
application measured the system by the number of late moves per day. In order to
measure this, the customer established specific time windows for delivery and pick up for
each piece of material handling equipment.
For another company, the acceptance test was based on the ability to pick up outgoing
loads from a station within a specified number of minutes for that given station.

Yet another company required that all loads had to be transported with 15% spare
capacity in the hourly cycle time so that the system would be flexible enough to react to
any subsequent variation in demand.

Often, two types of acceptance tests will be used: a system functionality test and a system
throughput/availability test.

The System Functionality Tests allows the AGV provider to demonstrate that the
installed system meets all the functional requirements purchased by the buyer. The test
may range from several hours for a small system to several days, depending on the
complexity of the system. In many situations, a repeat of the Factory Acceptance Test for
each subsystem is all that will be required. If the system meets the functional
requirements, testing can proceed to the System Throughput/Availability Test.

The System Throughput/Availability Test demonstrates that the system can meet the
expected performance requirements. During this test, the system will demonstrate that
vehicles are moving a specified number of pallets per hour, moving a certain number of
loads in and out of an AS/RS system every hour, or moving a certain number of pallets to
the shipping area. At the same time, the system should also be demonstrating its ability to
achieve the required availability, or the readiness of the system to perform its intended
purpose.

How long should a System Throughput/Availability Test last? That depends on the size
and complexity of the system. For a simple system, an eight-hour test may be sufficient;
for a more complex system up to a week may be required.

As part of that process, it’s important to document any errors in operation during the test.
Once the test is complete, those errors should be analyzed by the customer and the
supplier to separate system errors that can be corrected by the system provider, from non-
system errors that were caused by other factors. For example, a delay caused by a
navigational error, is a system error that should be corrected by the vendor. A delay
caused because the AGV was blocked by a lift truck is a non-system error that is out of
the AGVS supplier’s control.

In either case, it’s possible for a system to meet the throughput requirement but not the
availability requirement or vice versa. In those instances, the supplier would be expected
to make any necessary corrections to the system and re-run the test.
Two notes of caution. A 2006 system reliability study conducted by Brigham Young
University found that when automated equipment is first operated by the buyer’s
personnel, there is a learning curve. It’s not uncommon for an automated system,
including an AGV system, to operate below expectations during the first year only to
meet or exceed performance numbers in year two as operators and maintenance personnel
become familiar with the system.

What’s more, it’s not unusual for an AGV system to be up and running while the
manufacturing line it will interface with is not complete, or is only operating at partial
capacity. In that instance, the supplier and customer may come up with a creative solution
to create a valid acceptance test. Some customers may do that by running the system with
fewer vehicles; still others may create artificial loads to fill up the interface points and
simulate full production conditions.

Remember, if you’ve done development, planning and implementation right, acceptance

should be a foregone conclusion: Never in doubt and never a surprise.

Key words and definitions

Performance characteristics: A performance characteristic is something that is

specifically determined through the direct observation of system operations, estimated
through the use of computer simulation, or computed through the use of mathematical
analysis. For example, the 3 minutes required for a vehicle to pickup, transport and
deliver a load from point A to Point B is a performance characteristic. Performance
characteristics may be compared later to the acceptance criteria value during acceptance
testing.

Acceptance criteria: The acceptance criteria represents the specified quantitative

standards that will determine acceptability. Acceptance criteria includes:

Maximum value: This is a value which is not to be exceeded, such as the waiting time
for all loads shall be no more than 15 minutes per day.

Minimum value: This is a value which must be met or exceeded, such as the minimum
number of loads per day that the system must deliver one point to another.

Target value: A specific numerical value which is to be met, such as the number of loads
the system will deliver from one point to another per day.
Tolerance: This is a value, or the variation, within which a performance characteristic
must fall, such as the arrival time of a vehicle must be within +/- 1 minute of the
scheduled arrival time.

Three types of performance characteristics

Three types of characteristics may be used to define the performance characteristics of a
system: time-based, cost-based and quality-based. While every system is unique,
researchers suggest that it is unlikely that all the characteristics will be relevant for any
given system. Instead, the AGVS supplier and customer should determine the
characteristics that are most important to the processes being supported by that system.

Time-based characteristics

Total Transport Time: the time required to transport a load from point A to point B.
This would include the load time, loaded travel time and unload time.

Vehicle Travel Time: this may be expressed as loaded travel time or unloaded travel
time.

Response Time: the time from when a pick-up request is made until the vehicle arrives at
the pick-up station.

Load/Unload Time: the time required to load or unload a vehicle,

Output: the number of completed transportation events in some stated period of time.

Throughput Rate: the number of completed transportation events per unit time.

Load Waiting Time: the time a load spends waiting in a queue for transportation.

Vehicle Waiting Time: the time a vehicle spends waiting in the system.

Vehicle Blocking Time: the time a vehicle spends waiting because it is blocked by
another vehicle.

Vehicle Charging/Fueling Time: the spent by a vehicle during recharging or refueling

operations.

Resource Waiting Time: the time spent by a resource waiting for a transportation event.
Lateness: the difference between the actual time of an event and the scheduled delivery
time for that event.

Utilization: the time, or percentage of time, the system or component is actively used.

Availability: the amount of time the system, or a component within the system, spends in
a functional state.

Preventative Maintenance Time: the amount of time on preventative maintainenace.

Repair Time: the amount of time spent correcting system/component failures.

Time To Failure: the actual times to failure of the system or its components.

System Warm-Up Time: the amount of time between the initial power-on and the actual
start of active system operation.

Cost-based characteristics

Investment Cost: the fixed initial cost of the system.

Installation Cost: the fixed initial cost of the system.

Energy Cost: the variable cost of power to operate the system, over a prescribed time.

Labor Cost: the variable cost of labor required to operate the system, over a prescribed
period of time.

Maintenance Cost: the variable cost of maintenance over a prescribed period of time.

Material Cost: the variable cost of materials or other replenishable supplies, materials or
spare parts required to operate the system.

System Overhead: the overhead costs associated with managing the overall system,
programming, controls and upgrades.

Quality-based characteristics

System/Component Failures: the number of times during a state period of time that the
system or a component within the system fails. Also known as reliability.
Load Accuracy: A broadly defined category that refers to the ability of the system to
pickup/deliver the correct material.

Load Quantity Accuracy: a broadly defined category that refers to the ability of the
system to pickup and deliver the correct quantity of material.

Load Location Accuracy: a broadly defined category that refers to the ability of the
system to pickup and deliver to the correct location.

Load Damage: the number or percentage improvement in transported loads without

damage as a result of the AGVS.

System Safety Incidents: the number of reported safety incidents.

Distance Traveled: the total distance traveled over some stated period of time by
vehicles in the system.