57

th
Annual ISA Power Industry Division Symposium
2-4 June 2014, Scottsdale, Arizona
Hilton Scottsdale Resort 1 1
David Herrell and Kyle Dittman
MPR Associates

Bob Cardwell
Southern Nuclear
Replacing an Obsolete Software-
Based Module with an FPGA-
Based Module
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
2
Author Short Biography Slide
• David Herrell is an Executive Engineer at MPR Associates,
Alexandria, VA with 35+ years of nuclear digital I&C experience
• Part of MPR’s senior I&C technical staff, he works with
suppliers and nuclear power plants around the world
• Worked for a system supplier, as a seconded contractor, and
as a utility employee at Salem and Hope Creek prior to MPR
• Bachelors and Master degrees in Electrical Engineering
• Member of IEEE Nuclear Power Engineering Committee
(NPEC), member and current chair of Subcommittee 6 on
Safety Systems, and member of Working Group 6.4
responsible for IEEE Std. 7-4.3.2
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
3
Background
• In the 1980s, Edwin I. Hatch
Nuclear Power Plant
replaced the electromagnetic
timers on the Unit 2 EDGs
with commercial software-
based equipment
• Three cabinets of equipment
were installed, each
containing 1 dc-to-dc
converter, terminal blocks, 2
control modules, 2 alarm
relay outputs, and 2 counters
with relay interfaces to the
counters
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
4
Background (cont’d)
• Multiple failures and obsolescence concerns of Rochester
Instrument Systems (RiS) modules initiated a project to
generate form, fit, and function replacements for the control
modules
• SNC awarded a contract to MPR to re-engineer and provide
replacement modules as basic components
• MPR and SNC decided to base the replacement module
architecture on a field programmable gate array (FPGA)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
5
Background (cont’d)
• The equipment was reverse-engineered and implemented on
an FPGA-based module
• The Product Design (PD) Group has been developing FPGA-
based designs for medical devices under FDA regulations
• The Nuclear Group used the PD Group’s capabilities, by
adapting the PD Group plans, procedures, and instructions
into a safety-related Programmable Logic lifecycle process
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
6
Background (cont’d)
• Early in the project, MPR considered the possibility of relying
solely on 100% testable and tested logic, but decided to
continue with IV&V as good engineering practice
• MPR had Gavial Engineering and Manufacturing procure
components, assemble, solder, and preliminary test the
modules under their 10CFR50 Appendix B compliant Nuclear
QA program
• The use of commercial components required the performance
of commercial grade dedication activities on the fabricated
module
• Commercial grade dedication was performed per our 10CFR50
Appendix B compliant Quality Assurance Program


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
7
Existing Design

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
8
Existing Chassis


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
9
Analysis of Existing Design
• Only limited design documents for the original modules were
available, including cut sheets, limited functional design
descriptions, and schematics from Rochester Instrument
Systems
• Having schematics for modules and for cabinet wiring
eliminated the need for extensive wire tracing and generation
of replacement prints
• Verification (as-built) of the cabinet schematics was performed
• The functional design descriptions were adequate to
determine how the module worked in sufficient detail to avoid
the need to disassemble the Fairchild F8 microprocessor
software

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
10
Analysis of Existing Design (cont’d)

• SNC provided a spare working cabinet for use in the design
activities
• Much of the software documentation (e.g., flowcharts) had
little value other than showing how the module worked
– FPGA implementation does not include problematic software
features such as sequential instructions, jumps, multitasking, and
hardware interrupts
• It was determined that building a generic replacement was not
appropriate
– Replacement module functions were customized
– Features of the OEM module were not needed
– DG safety function could be simplified

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
11
Re-Engineered Requirements
• Unused inputs and outputs (12 inputs to 3, 12 outputs to 10),
were eliminated
– Reduced complexity,
– Increased reliability (fewer parts),
– Reduced power consumption, and
– Allowed for enhanced diagnostics
• Diagnostics enhanced to verify that the output relay coils have
continuity, rather than just checking that the output switch
turns on or off
– Diagnostics use the inductive characteristics of the relay coil to
check continuity
• Diagnostics considered active (inject current) or passive
(monitor voltage)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
12
Re-Engineered Requirements (cont’d)
• System/Plant external wiring limits the extent of diagnostics
• Existing OEM module was designed for minimal EMC
• Design constraints for “new” EMC requirements; meets United
States Nuclear Regulatory Commission Regulation Guide
1.180 requirements including:
– Electrically fast transients requirements,
– Electrostatic discharge requirements, and
– Surge withstand requirements
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
13
System, Hardware, and Software
Requirements
• A single document was written for both module hardware and
Programmable Logic requirements
• Many of the detailed design decisions made during the
original module’s design are now constraints on the
replacement module (e.g., module size, electrical connection,
and pinout; front panel size; chassis arrangement and wiring)
• Requirements were created to address issues with the original
design (e.g., weak ground connection from module to chassis,
minimal distance from module to chassis necessitated paper
insulator to protect OEM module contact with chassis)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
14
System, Hardware, and Software
Requirements (cont’d)
• There was no attempt to recreate the original generic module
requirements, as the module is being used for a single
purpose in a single plant
• Programmable Logic architecture created, showing how the
parallel action embedded in the logic actually functions
• Requirements and detailed design iterated to point where
VHDL code implementation and module schematic could be
started
• Design iteration continued through completion of
implementation, with final passes to resolve any remaining
IV&V clarity issues
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
15
System, Hardware, and Software
Requirements (cont’d)
• MPR performed hazards analysis throughout life cycle, to
inform the design, implementation, and V&V processes –
hazards external to the replacement module could not be
resolved
• Hazards that could not be resolved involved constraints in the
existing design, which were present from the initial installation;
no new hazards were added
• Hazards analysis activities augmented the testing program by
verifying that all hazards for which testing could be performed
were included in the testing program, and that those which
could not be tested were reviewed independently
• Routine surveillance testing covers the external hazards to the
replacement module
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
16
Software Tools
• Software tools are at least equally important for FPGAs as for
software-based devices
• Programmable logic requires evaluation of internal FPGA
signal timing, which cannot be externally measured
• Only way to evaluate internal timing is by use of simulation
and timing evaluation software tools
• Internal timing verification cannot depend on testing, since
hardware may work while violating vendor internal timing
constraints (e.g., setup, hold)
• Most vendors provide frequent updates to their software tools,
which should be considered for use, as tool errors are
corrected (important) in addition to new FPGA support

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
17
Software Tools (cont’d)
• MPR uses National Instruments LabVIEW for IV&V testing
and for equipment qualification testing
• Custom LabVIEW application designed, verified, and validated
as a means of stimulating the module and measuring,
recording, and analyzing the module’s response
• MPR also uses a tool to generate requirements traceability
matrices
• Automatic generation of RTMs based on metadata tags
embedded in documents eliminates the pain associated with
manual generation and correction of generation errors
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
18
Prototype Redesign


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
19
Design Enhancements


• As a basic component “form, fit, and function replacement,”
enhancements have to fit within the module and cannot
require external change; sensible enhancements are not
precluded
• Increase in computed MTBF based on the design changes
• Original module required replacement of EPROM for each
unique program and timing sequence
• Replacement module has switch selectable sequence, one
module with three selectable sequences

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
20
Design Enhancements (cont’d)
• Original module had 3 LEDs – Power, Running,
and Failed
• Replacement keeps Power and Failed, eliminates
Running as there is no equivalent in FPGA space
• Adds LEDs for 3 external field contact states
• Adds LEDs for 10 demanded relay output state –
choice made to show demanded rather that actual
state
• Adds 2 numeric LEDs to provide:
– Replaces external obsolete external counters
(abandoned in place)
– Added display of FPGA failure status code
– Added display of selected sequence

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
21
Design Enhancements (cont’d)
• Enhanced diagnostics for Hatch’s specific application:
– Can now diagnose limited amount of relay and internal wiring
failures
– Did include separate watchdog timer, such that FPGA does
not annunciate its own failure
– Diagnostics driven by hazards analysis

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
22
Process and Implementation
Considerations
• All activities were performed under our 10CFR50 Appendix B
compliant Nuclear Quality Assurance Program, including
Programmable Logic life cycle
• Modifications were required to fit software life cycle to VHDL
• Many design and review topics for software were not
applicable to VHDL (e.g., interrupts, multi-tasking, constraints
of sequential execution, loops, jumps, memory
allocation/deallocation, paging, etc.)
• Few new topics were added, including evaluation and checks
of logic signal timing internal to the FPGA
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
23
Process and Implementation
Considerations (cont’d)
• Since no mathematical functions, there are no typed variables;
everything is either a bit or a collection of bits
• There are no widespread industry consensus guidelines for
good coding practices, as there are for software
• MPR did not apply the equivalent of a static analyzer to the
VHDL
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
24
Process and Implementation
Considerations (cont’d)
• VHDL does allow for sharing signals between modules, with
only explicit definition of the sharing
• Program instrumentation (e.g., “printf()” in “C”) is simple in
software; more complex in FPGAs and requires interesting
logic to support simple scanning
• Can still implement “stubs” and “drivers” for VHDL code for
testing stimuli, just like for procedural languages
• Software tools exist to simulate the internal logic, including
delay times based on the placed and routed VHDL, which may
be necessary to resolve timing issues
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
25
Process and Implementation
Considerations (cont’d)
• With self-implemented math, precision and accuracy are still
concerns
• VHDL code must still initialize memory prior to use
• Exception handling still must be designed in to the VHDL, with
inputs and outputs checked for reasonability
• Designers still make the same mistakes (e.g., bad
assumptions, missing punctuation, and erroneous but
compliable syntax, incomplete switch statements, etc.)
• Generating good, complete unit tests is still as complex
• For both software and VHDL, the quality of the product is a
function of the designer’s experience and capabilities
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
26
QA Plans, Procedures, Processes
• MPR has extensive experience with FPGAs in medical devices,
under FDA rules
• MPR’s Nuclear QA program requires generation of a task-
specific QA plan for safety related projects, explaining how the
project will work under 10CFR50 Appendix B constraints.
• With the FDA-compliant processes tailored to 10CFR50
Appendix B vocabulary, work performed in accordance with our
Nuclear QA program
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
27
Verification and Validation
vs. 100% Test
• Intended to use the “100% testable and tested” approach
• With all diagnostics and multiple state machines considered,
the complexity required to apply all possible input combinations
to all possible states becomes unreasonable
– State Machines include: main state machine (8 states),
sequencing step counter (62 states), field contact input debounce
state machines (3 sets of ~22 states), active diagnostics
– External and internal inputs include: 47 diagnostic failures, 3 field
contact inputs, and front panel reset switch, or 2
51
combinations)
• MPR used a traditional IV&V process, and notes that IV&V
found design errors that testing would not have uncovered


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
28
Conclusions
• Redesigning obsolete digital equipment is possible with no
design documentation, but even a little documentation
simplifies the process
• Replacing an analog device is simpler than replacing a digital
device
• Replacing software-based devices successfully with FPGA-
based devices requires thought, understanding of the original
equipment, and familiarity with both software and FPGAs
• Consider and implement modular reuse of VHDL code
• Licensing an FPGA-based replacement is not significantly
different than licensing a software-based device

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
29





Questions?

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.