Tips for Troubleshooting Pilot Plant and

Laboratory Equipment
 Published on June 1, 2020

Status is reachable

Richard Palluzi
Pilot Plant and Laboratory Engineering, Safety, and Design Consultant at Richard P Palluzi LLC
85 articles Following

Troubleshooting is a tedious and time consuming operation. Often one hears that X is ‘a good
troubleshooter” or has “good troubleshooting skills” almost as if these are innate qualities one
must be born with. Nothing could be less true. Troubleshooting is a skill anyone can learn and
improve at as long as they understand some simple basic principles. While troubleshooting skills
are similar across many fields and activities, I am going to focus on those most useful in a
research environment (laboratory to pilot plant to demonstration unit scales) since that is where
my interests and experience reside.

·        Haste always makes a troubleshooter less effective. Taking some time to assess the
situation before doing anything while everyone is agitated over the issue is far from easy.
Operators want you to start doing something right now! A more effective approach is to take a
moment before you touch anything to look carefully at the system. Note the position of all the
key valves. Look at the pressures and temperatures. Note what is, and is not, operating. Often
taking the time to do this will reveal the problem very quickly and obviously. The author once
solved a problem defying the combined efforts of a much more senior group because he could
not get close enough to the area to participate and, while waiting, noted that the feed valve had
been inadvertently closed.

·        It may often be necessary to ask the operator to give you a moment to think. Alternately
you may need to patiently wait for the operator to run through their explanation while only
partially listening so you can have some time to think. One way I have found to address their
concern and still function effectively is to think out load. This shows that you are working the
issue while keeping them involved. This does not always work, as some operators insist on
arguing or commenting on each point or trying to get you to stop and move in the direction they
believe best. A polite, but pointed comment, that you are not as familiar with the problem as the
operator and need a few moments to come up to speed often helps. (An occasional “Could you
please stop and let me think for a minute” may, hopefully less often, be necessary!)

·        Write down what the exiting conditions and equipment status before you do anything
including the positions of all the key valves. Update these as you make changes being sure they
are clear enough and legible enough to understand what you did and when you did it. This may
sound like wasteful added work, but after doing 5 or 6 things it is very easy to be uncertain if the
last test was with the valve on manual or automatic or if the nitrogen was or was not flowing.
Are we sure the pressure on the tank started at 10 psig? Was the system temperature constant
during both tests? It is so easy to get confused and not know what exactly you have done without
some attention to documenting each step. It is even easier to read 10 psig as 100 psig or fail to
note the temperature is in F while the operator is giving you C. Jot down each step as you make
changes. Make sure that you can (1) read what you wrote, (2) understand what that your
abbreviation meant, and know that the top number is the pressure and the bottom the
temperature. (In other words, take a moment to be neat, semi organized and careful to label
things.) I can’t tell you how many times I have had to use this information to unravel a confused
operator (or myself).

·        Treat everything the operator or person discovering the problem says with a very large
grain of salt. People will leave out key items because they think they are unrelated, fail to tell
you something they did out of embarrassment, be reluctant to admit they have no idea what the
pressure gauge was reading or if the feed valve was on, or simply get confused and swear they
put it on manual when, in fact, they didn’t. And occasionally, even in the best organizations,
sometimes an operator senses they may have created the problem and do not want to give you all
the background for fear of making their culpability apparent. After wasting an hour finding out
the operator simply forgot to set something properly or did something in the wron sequence it is
very hard not to let some annoyance show. But being open enough to ask without criticism often
helps. “It’s like the cooling pump wasn’t on when you started. Is there any chance that that might
have happened?” A reply of something vague like “I guess that is possible” often allows saving
face and solving the problem.

·        Be suspicious of the person who has “solved” the problem and just wants you to fix
something. The author has wasted much too much time getting a pump changed out, a controller
pulled for repair, and a regulator replaced only to find the pump was not working because of an
unsatisfied interlock, the controller because of an incorrectly set pressure balance, and the
regulator because the gas cylinder was empty. I often try to address these “solutions” by a
discussion about how the solver is positive that this is the problem. While often not received
well, good operators rarely mind explaining their thought processes; weaker operators can often
be helped to understand that perhaps they are not 100% certain and allows a more grounded
discussion to commence. In either case, the time spent is usually well worth the time saved on an
incorrect solution. Even experienced operators can be badly misled by the similarity with a past
problem. The current erratic flow (±50%) that last time was due to a pump seal leak is often not
recognized as being a lot different than the previous ±10%.

·        Train yourself to start from the beginning of a system or the end of a system and work your
way to the other end. This is often felt to be counter intuitive as most people want to start in the
middle and feel they can easily identify if the issue is behind or in front of their starting point.
Sadly, they are wrong. Starting at one end or the other of the system avoids false moves and
saves time in the end. Is the breaker on or does the equipment have power? Either is an
unambiguous starting point for further checking. That JB3 is not energized may be much less so.
And complex systems often have numerous paths to hold out a system making the ability to
figure out which direction to go suspect at best.

·        Do not assume that the most complex or troublesome component is the culprit. We all tend
to think that the flow has stopped because that (expletive deleted) valve has hung up again or that
difficult to access filter clogged yet again. That may be the case but it is just as likely to be
something entirely different like the feed valve being closed. Troublesome components exert a
magnetic pull on troubleshooting efforts that often hinder a more logical, and ultimately faster
approach. Learn to avoid the pull. (And yes, the author freely admits he often wasted 1 minute on
a fast check of the pressure drop across the filter or a fast confirmation the valve is moving
before doing much else. He also must admit it has often led him astray.)

·        Recognize that everyone has a “hunch” where the problem may reside. While these are
sometimes correct based on past experience or simply a good intuitive sense, I have seen hours
wasted checking an issue that a few moments analysis would show could not explain what you
are seeing. Sometimes this becomes even more of a problem as if you don’t trust a component
you tend to keep circling back to it, often needlessly. This is where the next step helps.

·        Develop a plan before you do anything. The author has seen too many things done to
troubleshoot a problem which, when asked what this test will prove, is met with dead silence as
people realize the proposed test will not prove anything. Taking the time to draw a simple flow
chart and show where the results of each step will lead to is invaluable and well worth the time to
develop. Often it will show that a certain path will show very little or that you need to do an
additional step first to make sure that you can reach a valid conclusion later. And don’t be
surprised when you get to a point and suddenly have to say, something like “and if the pressure
rises …. then I have no idea what that means!” Sometimes you get lost in the process and forget
what the step can or cannot show. Talking through the steps with a knowledgeable individual
often helps you avoid faulty logic or skip over another, more viable, path.

·        For particularly troublesome problems, review what you have done and seen with
someone. If no one is available, writing down a summary can sometimes work. Many times, the
effort to organize your thoughts and explain the problem will help you solve the issue or at least
identify the next steps. The person can often ask you a question or challenge an unfounded
assumption.

·        When you have done everything you can think of and nothing has fixed the problem, resist
the urge to go do it all over again. Instead take a step back and try and identify what you have not
checked. While it is possible you checked something wrong, it is more likely you just are not
looking in the right area. I often have found the problem by asking simply “what have ee not
looked at so far?” The key you lost is always in the last place you look.
·        Doubt everything you are told no matter how much you trust and respect the operator. All
too often the tank is empty, the cylinder not connected, the power not on, or some other
incredibly obvious issue that someone failed to note. I have stood near too many valves I swear I
opened and too many switches I swear I closed to not recognize how easy it is to get confused or
fail to look closely enough at something.

·        Look carefully at the system and pay attention to any nagging points. Why do you keep
looking at that pump? Why is your attention always drawn to that feed system? Often your
subconscious is noting something is not right and if you look at it a bit longer you will realize the
issue.

·        Never ever, do more than one thing at a time. Keep a close eye on everyone involved so
that you are not starting something while some else has stopped it. One step and one step only at
a time. Having someone double check that all 5 valves are all in the right position or double
checking that you traced that feed line to its source correctly often saves incredible amounts of
time and effort.

·        Make sure you understand the status of everything involved before the problem started to
develop. The author once pushed an operator to remember if anything was different right before
a heater stopped working. It took some effort for him to remember to mention the electrician
who borrowed a ladder a few minutes before the problem; a fast trip confirmed that the
electrician had locked out the wrong circuit by mistake and cut all the power to our panel.

·        Conversely, be willing to let go of something that you just cannot find any logical linkage
to the problem. The lights may well have flickered in the control room just before the problem
but if the DCS system is working properly it may well have nothing to do with the problem.

·        Always remember that the proposed issue cannot violate basic physics and chemistry. I do
not care what logic path is involved, closing the valve does not increase the flow. Of course, you
have to be very diligent in making sure you have all the right information. If the valve was
reverse acting then perhaps you are actually opening and not closing the valve.

·        Never trust any drawings without verifying they are correct. Yes, I might not go down this
path as the first thing I do but I always try to confirm how the equipment is piped and wired. It is
too easy for a drawing to be wrong. And good MOC systems have reduced, but far from
eliminated this problem. Operators who resist this step loudly arguing they know their units often
have to be cajoled into letting you check anyway. My favorite tactic was to agree but note that
unless I verified it I would always have a lingering doubt.

·        Always gather all available documentation on intermittent problems. Make sure all the
observable facts are written down when the problem was discovered not afterwards. Doing it
later makes it is too easy to make a mistake. Make sure the operator knows that if they forgot to
note the pressure it is better to admit that then put in a “normal” value. Make sure the operator
knows to stick to verifiable facts and not opinions. “The seal started leaking sometime just before
the problem” is valid if the drip pan is always dry and the operator saw it a few hours before. If is
not so valid if the drip pan is soaked and it just reflects the operator’s opinion that there is more
than the usual liquid in it. The author once spent a week trying to find why a 30 psig rupture disk
kept bursting when the system never had more than 5 psig on it. Only after reviewing the
documentation that we finally forced ourselves to keep was it discovered that the disk burst
whenever someone turned on the GC. However, ridiculous that sounded, it allowed us to find the
incorrect line that that fed 60 psig of helium into our system.

·        Always ask the people involved about anything odd they notice, particularly the recurrence
of anything when the problem appears. I finally was able to find why a gas monitor
intermittently shut down a unit despite there being no releases when an operator casually noted
that it always seemed to happen when another operator was cleaning his reactor. (The monitor
was much more sensitive to the cleaning solution.)

·        When you appear to have fixed the problem, try and find a way to prove you are right. If
you can go back and prove that when the regulator is set too low the problem appears and
disappears when it is set properly you can be fairly sure you found the true culprit. But often
ancillary things you do during troubleshooting fix the problem and you don’t realize it. The
author once changed out a metering pump twice before he realized the issue was vapor build up
and not the pump. Each time we changed out the pump we carefully bled the lines fixing the
problem until it built up again. Obviously, this is not always practical but if the replacement
component, for example, looks fine, a fast off line test may raise real suspicions that you have
not found the problem and so may not have fixed it.

·        More is not always better. Involving numerous people in the troubleshooting process often
simply slows it down (everyone needs to have their say), creates the potential for confusion (too
many people doing different things at once), and rarely solves the problem faster. A better
approach is usually to step aside and let someone else have a try when you appear to have hit a
dead end. A fresh set of eyes or a new approach, without you breathing down their neck, often
can solve the problem.

·        Stepping away from the problem for a time can sometimes also help. I am always amazed
at how many problems are solved faster the next morning than the previous evening.

·        Problems sometime fix themselves. (They don’t but sometimes you fix them without
realizing it.) Sometimes you don’t understand why doing X solves the issue. Both cases are rare
and worthy of further investigation. Yes, I have walked away from some and still, to this day,
share the stories in the hope someone smarter than me will tell me how I fixed it but these are
rare. So be careful in assuming the problem has truly been fixed of gone away in these cases.
Often, they are hiding a very subtle or intermittent problem.

And, even if you follow all these guidelines religiously, troubleshooting will take time and effort
and often be incredibly challenging. However, I think you will find it will, on average, go a bit
faster and easier.