Protecting Your Investment: Creating

an Effective FAT/SAT/BAT Procedure

October 21st, 2009

Presented in conjunction with the

Laboratory Robotics Interest Group

New England

Open Source Software Initiative

 Agenda

Introduction
Types of Laboratory Automation Systems
Creating a Project Plan
Factory Acceptance Testing
Site Acceptance Testing
Biological / Chemical Acceptance Testing
Lessons Learned
Summary
 Some Definitions

FAT - Factory Acceptance Testing

SAT - Site Acceptance Testing

BAT - Biological Acceptance Testing

CAT - Chemical Acceptance Testing

 Introduction

Purchasing Laboratory Automation is an investment

What are you buying (workstation or integrated system?)
Adding a new capability or enabling technology?
Not always about efficiency

Properly plan for success

Define your project goals and criteria for success before
engaging any vendor
Design a Test Plan which uses the actual workflow(s)
Test the new capability successfully adds value
 Getting Started
Map your process
Add Reagent Incubate Add Reagent Read

Build a Project Charter

Why automate?
Added value?, Criteria for success?
What can kill the project?

Setup a Project Team and Determine Stakeholders

Include an Automation Lead/ Project Manager, Scientific Lead, &
Purchasing Representative
A Facility Lead and Software Lead may be necessary depending
on the complexity of the system
 Getting Started (II)

Capture User Requirements Specification (URS)

Use requirements to define an objective Request for

Proposal (RFP) process

Select a vendor which..

Is willing to work with you to properly test the system
Is willing to tie milestone payments to project phases

URS RFP Vendor Contract

Selected
 Workstations

Workstations are standalone pieces of equipment

integrated with a small # of peripheral devices
Typically stacker based and designed for a specific
workflow
 Integrated Systems
Integrated Systems vary in complexity and usually
include 1 or more robotic arms and a number of
peripheral devices
Typically support a number of different workflows and
are controlled with a scheduler package
 Integrated Systems Continued

Integrated systems can be designed to serve a

number of processes

Historically integrated systems have been built

to support:
High Throughput Screening (HTS)
Ultra High Throughput Screening (uHTS)
Compound Storage and Replication
Combinatorial Chemistry
Cell Culture
 Project Plan

In general, the project plan is similar in all cases

and should include:
project goals
the process map(s)
Schedule and milestones
Resource plan
Payment terms
Reference to test plan documents (ATPs)
 Project Plan Continued
Consider breaking project into separate milestones with payment
percentages tied to the completion of these milestones

Ensure that there is a written and clear understanding with

vendor for what acceptance testing will be required

Recommend leaving a percentage of total project payment as a

final milestone payment

Identify a clear procedure for change management

Consider applying payment penalties for

Missing key deadlines
Not satisfying system / design requirements
 Example Project Plan Timeline

Define
PO
System Build and Test Ship and Install
Accepted
Specifications

Kickoff Factory
Meeting Acceptance
Define Functional Test
Design Spec
Site Acceptance
Define Test
Assay/Consumables/
Bio/Chem
Acceptance
Agreement on Test
Acceptance
Testing
 Acceptance Test Plan (ATP)

Draft a formal document (ATP) for testing procedures for

each test phase (FAT / SAT / BAT)

ATP should include language which

Outlines requirements for testing procedures
Outlines Testing Scripts
Defines criteria for success / failure
Defines when / what / if interventions are acceptable
Roles & responsibilities and more
 Example Assay

Labware
Source Plate/Assay Plate

Stated Protocol
Dispense 10l BufferA
Dispense 10l BufferB
Incubate 60 min at Room Temperature
Wash
Read
 Additional Assay Information

Unstated Protocol
Read Barcodes from Source and Assay Plate
Shake for 1min after 2nd dispense
Seal plate with clear heat seal before reading
Use Barcode for name of resulting reader file
 Acceptance Test Plan (ATP) (II)
Any contract language regarding payment structure,
penalties etc. should be in the purchase agreement and
not in the ATP

The ATP can reference milestone payments

Between the Project Plan and/or Purchase Agreement,

all parties should have a clear understanding of financial
responsibilities

Request that the FAT and SAT be performed on a wet

system
 Acceptance Test Plan (ATP) (III)

The ATP document should be finalized and agreed upon

well in advance of the actual FAT
allows the vendor to run through the FAT testing (pre-FAT)

Vendor is responsible for unit testing of Custom and

Standard devices prior to FAT

The Actual FAT should not occur until the vendor has
confirmed that they have gone through and tested the
FAT process (pre-FAT )
 FAT: Workstations and Integrated Systems

FAT will test the purchased equipment using your

materials and protocols at the vendors site

FAT should use all of the equipment on the system

Should not be limited to 1 giant all encompassing test
should entail the primary task(s) for which the system was
purchased

FAT should not execute an experiment or protocol which

will not be used in actual lab
 Workstation FAT
Not all vendors will agree to complete a FAT for a workstation
purchase
Insist upon executing a FAT for any system with customization
If the vendor is unwilling to include a successful FAT as criteria for
payment, offer to pay for application labor to execute the FAT

FAT should include:

module level testing
error recovery testing
testing of new interfaces and any data handling that the system will
generate
safety requirement testing
QC testing

Not all vendors will agree to QC testing, however most will provide
an exception in the event it does not work in the SAT or BAT
 Integrated System FAT
All modules should be tested
If any of the modules will be run in standalone mode confirm that
they will operate in that mode
Module level and system level error recovery should be tested

QC and Safety testing are required

1 sample should be run from start to end so the entire

process can be monitored, before running multiple
samples

If appropriate, test throughput matches requirements

 Integrated System FAT (II)
Run the most complex process using the maximum
expected run size (regardless of run duration)

Information System testing should be done as part of

FAT

Always check the error recovery option

Review System documentation

Confirm the most current layout is provide
Manuals should be provide for standard and custom equipment
Cable layout a bonus, in particular when labeled.
 Navigating through FAT defects
System requirements will define when interventions are
acceptable. For example:
Day time run semi-automated system Interventions Acceptable
Fully automated system which runs overnight Interventions Not
Acceptable

When FAT runs require interventions

refer to the ATP
acceptable to proceed without starting over?

The # of successful continuous days of running should

be defined in advance and required for the system to
pass the FAT
 Navigating through FAT defects (II)

Any exceptions / defects should be documented and

witnessed by both the vendor and customer

Only accept defects to be fixed later when you are

confident they will be resolved easily

It is typically riskier, more expensive and time consuming

to fix problems on the customers site compared to the
vendors site
 Examples of Exception

A non-recoverable error

Communication Errors between hardware and

controlling software
 SAT
SAT should replicate the FAT testing procedures

SAT confirms that the system was not impacted by transport and
installed successfully in the customer lab

Both customer and vendor representatives should be present for all

SAT testing

Repeat testing for any exceptions that occurred during the FAT
confirms defects have been corrected to the customers satisfaction

Document and witness any preexisting or new exceptions that occur

during the SAT

Only accept SAT when all critical system exceptions have been
rectified and re-tested
 To BAT or not to BAT?
A Biological Acceptance Test or Chemical Acceptance Test is a final
testing phase which
Uses actual experiment materials
Confirms the actual experiment / process can be executed successfully
using the system

Vendors rarely agree to execute BAT tied to a payment milestone

Numerous variables outside of their control
External factors can impact experiment performance
Include as requirement within RFP

BAT should be applied when

Purchasing a large / complex integrated system
Experiment or process can be repeated manually with good precision
 BAT

Set expectations for BAT before signing the purchase

agreement / contract

A successful BAT will show that the automation does not

negatively impact experiment / process results
Precision and accuracy of data should be equal to or better than
running the experiment manually

Apply financial recourse for un-successful BAT

Final milestone
 Lessons Learned

Avoid scheduling major milestone at the end of a

quarter or end of the year.
Time constraints may influence the acceptance of the
FAT/SAT/BAT/CAT testing
Financial constraints from users companies

Negotiate service contract quote and associated

services (parts and labor) before contract signoff
 Summary

Understand the process you wish to automate and the

instrumentation you decided to purchase to complete the
task

A FAT, SAT and BAT should be performed on any

integrated system. For workstation a SAT and BAT
should be performed

Work with your team to protect your investment. It may

not be your money but your name is associated with the
purchase
 Acknowledgments

Ted Manley
William Goode
John Davies
Brian Duffy
Matt OBrien
Josh Martin
Agilent/Velocity 11
Backup Slides
Resources
Example Project Plan