Best of TestRail Quality Hub

Hiring Testers
One of the most important responsibilities of a QA leader is to build a testing team. But
it isn’t easy. In this era of Shift-Left testing, you may have to first justify why your team
needs testers at all. In addition, an effective tester must understand testing principles
and have technical abilities. But there is a relative lack of higher education courses in
testing. So it can be hard to find someone with the knowledge and skills you need.

At TestRail, we believe in the essential contributions that testers make to the develop-
ment of quality software. So, we’ve gathered some of the best articles from the TestRail
Quality Hub to help you hire great testers.

Contents

Do We Still Need Dedicated Software Testers? 3

The Ideal Tester to Developer Ratio 9

Find, Interview and Hire Great Software Testers 14

25 Challenging Tester Interview Questions 25

21 Questions to Ask When You Are Being Interviewed 31

Retaining Your Software Testers 38

Making Do When You’re Understaffed 45

How to Hire the Agile Team You Need 52

Hiring Testers

Do We Still Need
Dedicated Software
Testers?
Simon Knight, TestRail Product Manager

3
Most of you will have heard at some point or another about startups or much bigger or-
ganisations that have decided they have no need for dedicated testers. Facebook is an
often cited example of a large company who has decided that by using various mech-
anisms including dogfooding and production monitoring, they can eliminate testers.
They are choosing instead to release lower quality products directly into the public
domain.

Such stories beg the question that, if it’s okay for Facebook and other companies not
to have dedicated testers, can my organisation save time and money by eliminating
the testing function also? Instead of having a tester do it, can’t I just get the developer
responsible for building the software in the first place to check his or her own code?

Life Without Testers

Even without dedicated testers and even if the developers get it wrong, you can have
other safeguards in place, right? Just like Facebook does. You can do things like:

• Have the developers write lots of automated tests so that you don’t need to spend
any time manually testing our software.

• Have your own people dogfood for problems with a pre-release version of the
software.

• Build some more logging and traceability into the software so that you can easily
pinpoint where problems are occurring if you see any.

• Monitor current activities with previous versions of the software to see whether
there’s some indication of different user behaviours and whether those behaviours
are desired or not.

Well, maybe. And maybe not. Let’s take a look at some of those ideas in some more
detail and see whether they hold up to scrutiny.

4
Automated Testing
Apparently, automated testing gets all the focus now. It seems like more or
less every team in the world is varying degrees of Agile and is now writing
code using some kind of model driven development – typically test, behaviour
or acceptance test driven (TDD, BDD and ATDD respectively).

Normally in a model-driven-development kind of a scenario, the developer is

attempting to write their code using their model of choice, mocking out any
interfaces or integration points so that the code they’re working on can be
tested in isolation and when any resulting automated testing is run as part of
the build process, the test is run isolated from any other code dependencies.
There are a few problems here though.

For a kick-off, the very notion of automated testing is problematic. You can’t
really automate testing. Testing is a cognitive, analytical, human activity – not
something that can be performed by a machine. A machine can only do what
you tell it to. If you tell it to assert against a boundary or edge-case, that’s
what it will do. If you don’t, it won’t. So that means that for an automated test
to run, you have to know in advance exactly what it is you are going to test
for. It doesn’t take into account the things that you might find out about your
code along the way.

When humans test, they generate information about your program. When
a machine tests, they generate bits of data. 1’s and 0’s might give you some
information about your software, but it’s a black and white kind of a situation.
If you want the full spectrum of greys, you need to pull in some human
intelligence.

Dogfooding
Having your own people test pre-release code is actually a pretty decent
strategy. But it largely depends on what the nature of your software is and
how much people can use it without disrupting their normal, business as
usual activities.

5
Microsoft has had some success with this approach over the years, but its
bread and butter is productivity software. Microsoft can have its people
dogfood an email or word-processing application very easily since a good
proportion of their people are going to be using that stuff all the time anyway!
For more niche applications, this approach might prove more difficult.

Dogfooding works well for web applications. It can help to reduce the cost of
up front testing where the complexities of the production environment are
prohibitively difficult to simulate. The nature of web applications also makes
it possible to carry out controlled experiments like A/B-split testing or con-
trolled test flights where potentially dangerous code is floated out to a small
percentage of the user base. Combined with monitoring, dogfooding can be a
very powerful tool indeed.

Logging and Production Monitoring

Adding logging to the software seems like a no-brainer. Why wouldn’t you
want logging? Anytime you get an unhandled exception in the code you can
trace the activity through the stack and see what the user was doing when
things went wrong, and on what kind of an environment or browser they were
working in.

Likewise with monitoring. If you have historical data that we can compare
the current situation with, then changes in user behaviour can be monitored
over time and you can see whether or not new features are having the desired
impact. If you find that users are changing their behaviour in ways either you
or they don’t like, you can take action accordingly.

Monitoring also helps you to see whether or not there are any non-function-
al concerns. You can see for example how long it takes a user to complete a
specific task like searching for and landing on a specific item, or making a
purchase. You can identify how real users are carrying out their various activ-
ities and look for improvements in both flow and response times.

6
 You can also look for potential security issues, some of which may relate to
business logic, as well as the more specific vulnerabilities that inevitably rise
to the surface over time.

The Opportunity

None of the activities listed above require a dedicated tester to carry them out. In
many organisations, they can and are being carried out by developers, operations and
support staff. With that in mind, it may seem like testers shouldn’t be much in demand.
But that’s actually not the case. Though it’s fair to say that the market for testers has
changed considerably over the last 5 years or so.

There’s very little call for Manual Functional Testers as they used to be called, for
example. The market’s all about Agile Testers and Developers in Test these days. There
are obviously some specific competencies associated with those job titles. Whether
you have those vogue skills will have some bearing on your career trajectory, without a
doubt. But there’s some skills you can work on that never go out of fashion.

If you can prove your worth on an agile development team by employing critical thinking
skills throughout the backlog design and grooming phases of the project, exploring
requirements with domain knowledge and specific examples, employing the art of
humble enquiry along the way, you’ll make many developer friends and help them deliver
a top quality product as a result.

Then, using your “mad experiment design skillz” while they’re cutting code, you can help
them to design effective automated tests/checks that help constrain development to
the functionality that needs to be delivered and prove it works the way the business

7
intended. Having that layer of automation will ultimately make life easier as you carry
out some manual testing, since you’ll be able to use it to ensure you get a reliable build
and potentially as a taxi-service to get the build exactly where it needs to be for you to
test it.

The Bright Future

Do you think your services as a tester are no longer in demand? Do you think the job
market has no place for testing professionals any longer? You couldn’t be more wrong.
But you need to make sure you have the right skills. By working on your test design and
critical thinking skills:

You can be responsible for the dogfooding – working directly with the business users
to help them test the pre-release product, capturing any issues as they arrive and
reporting them back to the development team.

You can help define the areas that, based on your exploratory testing, you think need
additional logging in case they do go wrong and help the team with analysing faults if
they do occur.

You can work with the production monitoring team or system to carry out various kinds
of testing in production, providing valuable results and feedback to your team to help
them in the next iteration.

If you can add value to your team by helping provide information about what users really
experience in the production or pre-production environments, by designing dogfood
and production experiments – you’ll always be in demand.

8
Hiring Testers

The Ideal Tester to

Developer Ratio
Justin Rohrman, Excelon Development

9
I was working on a pretty tough project early in my career. This was back in the bad
old days, when waterfall was the go-to project management tool. We were at the end
of a release, in the testing phase, and all the testers were working evenings and some
weekend days to get through the queue of work. We were all left feeling burned out at
the end of that release.

The Monday after the release, I asked if we could hire a new tester for our team. The
answer I got from my manager was that I needed to show how that would make sense
financially. I’m not an accountant, so I never did the analysis.

What is the right test-to-dev ratio, and how can you figure out what’s right for your team
without working in the accounting department?

Analyze The Problem

Software development is often a process of handoffs — even in agile, and even when we
do two-week sprints.

Developers might start a sprint with a handful of tickets to work on apiece. They work
for a day or two, then I check in and ask how things are going. The developer tells me
that everything seems to be going well, but they need a little more time. They don’t
really have anything testable just yet.

After a day more I get a small change from one developer and start working on that.
I find a problem where entering a bad date in a date field caused an exception to be
thrown when I submit the page, and another where a button is overlapping a text field
in older versions of Internet Explorer. I report those issues to the developer, and he lets
me know he’ll get them in his queue of work.
10
New changes ready to test start rolling in a couple of days before the end of the sprint
and our inevitable deploy to production. We have to test a new report, a new manage-
ment workflow, and a new page for managing those workflows — oh, and the two bugs I
found a few days before. All of that is a lot for a team of two testers with two days to go
before a release.

Presto: We now have too much work and not enough testers to do a good job. If we did
have enough testers, we would have the new problem of fixing and retesting all the
problems they find in time to release.

We had several problems. First, we started the sprint with large swaths of work, and
a single card in our bug-tracking system might take around five days to complete, on
average. Next we have varying amounts of developer testing. Some developers created
unit tests, and some did nothing aside from quickly open up a browser after they were
done writing code. Last, we had physical and mental separation between roles on the
development team.

Our tester-to-developer ratio was completely inappropriate for how we were develop-
ing software, especially right before we wanted to put out new software. We had three
options: recruit and hire an additional tester, work to make our development process
more effective and efficient, or both of those things. Here is where I would start.

Break Work Into Smaller Chunks

The first thing I would do in this scenario is learn to break down work into smaller
pieces. In my experience, a day or two is a good upper limit for completing a change.
Smaller changes are easier to understand, isolate and, more importantly, test.

11
When I look at longer running changes I have worked on, ranging from a week to several
months, there were inevitably surprises. We discovered new scope, forgot to test
things or had to rework parts of the code that were implemented poorly. This philos-
ophy can easily stink of big, upfront design if we aren’t careful. The goal here isn’t 100
percent reduction of uncertainty; it is understanding a change just enough that you can
get it done in a day or two.

Developers tend to start performing some testing after you get in the habit of making
very small changes. Because very small changes are easier to understand, I see devel-
opers begin to consider how and what to test.

I like pairing with developers. We start by thinking about what we would test, and then
we write that test. After that, we write the product code that will make the test pass. On
a normal change, we do this back-and-forth process of writing tests and then produc-
tion code until we get a gut feeling. That feeling lets us know that we are probably far
enough along to create a new test environment and do some exploring.

This isn’t about pairing, though; this is about managing the tester-to-developer ratio.
So why does this matter?

Software that is written in very small chunks and is built by people who care about
quality tends to be pretty good. I don’t generally see dead-on-arrival builds, back-and-
forth bug reports and bug fixes, or time spent in bug triage meetings trying to figure out
what matters, what is a bug and what is a “feature request.” There are fewer bugs, and
the bugs that are there tend to be more interesting and harder to find.

This workflow requires fewer testers — they just also need to be more highly skilled.

12
Figure Out Your Ratio

It is easy to look at a bottleneck — getting a large amount of testing done right before a
release, for example — and automatically think you need to hire another person. But the
tester-to-developer ratio tends to solve itself if you examine the actual problems you
are having.

How long does it take to get one code change done? How much developer testing and
quality emphasis do you see in the development process? How closely do the different
roles on your development team work?

If you are working on small changes that last somewhere around a day, test-driving
where you can and pairing sometimes, and you still have a bottleneck, then maybe you
do need another tester. By that point, though, it should be apparent that the next step
is hiring a new person; in this situation, there’s no accountant needed.

13
Hiring Testers

Find, Interview and

Hire Great Software
Testers
Simon Knight, TestRail Product Manager

14
Are you a test manager, QA lead or recruiter tasked with building and growing your QA
team? Or were you asked to find and recruit your company’s best next tester, quickly? If
so, you can probably empathize with my fictitious test manager’s diary entry.

Another day, another inbox full of recruiter emails. All of them have got CV’s or
resumés attached that I somehow need to decipher and correlate to the skills
I’m looking for. Most of the emails I’m sent are a complete waste of time.

If the candidates actually made it to an interview, I’d quickly discover that they
don’t have the experience their CV says they do, they can’t challenge assump-
tions the way I think they need to, they’re not committed to learning and they
don’t have the confidence and the communication skills I need from them.

Recruitment is a time-consuming, energy-draining task that takes me away

from the things I should be focusing my attention on. I could live with it, if ulti-
mately we managed to find the right people and hire them. But we can’t even
seem to do that.

Every so often I’ll find someone who actually is a good fit, and has the skills I’m
looking for. But then we don’t manage to get them through the door. Somehow,
they fall out of the process.

Maybe someone makes them a better offer, or they decide to stay put, or
something else happens to make them change their mind. I feel like I’m wasting
my time here.

Why can’t we find the right candidates? Why aren’t the wrong candidates being
filtered out of the process? Why doesn’t the offer and joining process work? Am
I the only one who feels like this?

Why is hiring testers such a challenge? Where are all the awesome testers hiding? How
do you know whether a tester is awesome or not? And once you’ve found one, how do
you attract him or her to your company?

In this article I’m going to tackle the various stages of tester recruitment, namely:
Finding and attracting them. Interviewing them. Hiring them. Feel free to skip ahead if
a specific section provides the information you’re looking for. But before we get into any
of that, let’s get the most important part out of the way first.

15
Figuring Out What You’re Looking For

Do you really have a clear understanding of what attributes are important for the role
you’re trying to fill? If you don’t, then inevitably at some point further down the line you
are going to encounter some difficulties.

In the best case scenario you will discover that the CV’s you’re receiving don’t match the
kind of person you think you’re looking for. Or, during the interview stage you’ll realise
that the candidate can’t solve the kind of problems you need them for.

Wouldn’t it be better to figure out what your ideal candidate looks like in advance, so
that you can target them more effectively from the outset? This way, when you get their
CV or start to interview them, you already know that they have the skills you’re looking
for, and how to test for them.

Starting the process with a checklist or set of heuristics could work. For example, you
might have a list of questions like the ones below:

• Does the candidate understand what testing is? Can they explain it, clearly
and simply?

• Do they have experience of working on software development projects? Do

they understand the nuances and pitfalls?

• Can they distinguish between different approaches and how they may be
more or less appropriate depending on the project context?

• Can they work well within a team?

• Are they smart?

• Do they get things done?

• Can they design a test? Can they perform testing work? Can they operate,
design and build tools to help them perform testing more skilfully?

• Can they communicate effectively?

• Can they think critically and creatively?

• Do they have some domain knowledge for the industry in which you operate?

16
Or you could use someone else’s list: Huib Schoots has done a great job of identifying
a set of criteria in his article, Heuristics for recognising professional testers. And Ilari
Henrik has some superb ideas about what makes a World Class Tester that may also
be useful in your search. Johanna Rothman has some fantastic resources in her Tips
to Streamline Your Hiring. And Cem Kaner has a detailed guide to Recruiting Software
Testers that contains a number of lists that can help you tighten-up your requirements.

Ultimately, if you want to hire the best testers for your team, it would be wise to spend
some time figuring out what skills and attributes are most important to your organisa-
tion. Once you’ve gone through the process of trying to answer all of those questions,
you can document and communicate the answers, so that all of the people involved in
the hiring process can see exactly what success looks like, making it easy for them to
objectively decide whether or not a candidate meets your requirements.

How to Find Great Candidates

The thing you need to do next is to take those qualities and try to find some testers
that have them. But here’s the bad news. The days of just publishing a job posting in
a newspaper, or on the web, and then waiting for the applications to roll in, are over.
All the great candidates, i.e. the testers you want to hire, already have a great job,
and plenty of options, without needing to scan the job listings or the internet for new
opportunities.

Those testers aren’t looking for you. And they’re not paying attention to the job market.
So how can you find them? You could rely on a third party. Maybe you don’t feel like you

17
even have a choice, because the HR department normally takes care of all that. Them
or the preferred recruitment company that insists on spamming your inbox. Maybe
though. Just maybe, you could get a bit more hands on, by:

• Going to a community event: You may not know it, but testers are everywhere!
There’s almost certainly a meetup near you. And if there isn’t, why not think about
starting one? You don’t have to go far. You could even hold it in-house. Plenty of
other people have!

• Going to a testing conference: Just go along and put the word out. Alternatively,
ask the organiser if you can be a sponsor, or if you can make an announcement of
some kind.

• Connecting with testers on the net: You can start following the testing community
online via Twitter or LinkedIn, and the latest happenings in blogs and testing
forums. The testing community is very vocal. You won’t have to look too hard!

• Publishing material testers want to read: If you’re starting to engage with testers
on social media, you should definitely think about how you can contribute to the
conversation. Creating blog posts and articles that intrigue and excite testers is a
great way of drawing them into your sphere of influence.

• Setting testers a challenge they can’t ignore: If you feel like you still need to
advertise, make it interesting. Attention spans aren’t what they used to be so your
copy really needs to catch the eye. And what stands out more to the kind of tester
you want to hire than a testing conundrum?

If you’ve followed some of the advice above, then it’s quite likely that you will find
someone you want to recruit. Perhaps you bumped into them at a meetup or a confer-
ence, or read a blogpost or Twitter discussion that piqued your interest.

But now you’ve found them, how do you attract them into your organisation? Well – you
could just ask the person in question if they want to come and work for you. But, maybe
they’re not ready for that yet. Perhaps they’re already in an [employment] relationship.

18
 Play the long game
You can frequent some of the places where your tester hangs out. Get to
know the forums or groups they’re a part of on LinkedIn for example. Or find a
great blog post they wrote and comment on it. Let them know you’re interest-
ed in their work.

If you manage to get their attention, then you can take it a step further and
tell them about the opening you’ve got and how you think they might be a
perfect fit. Let them know how impressed you are with their work, and their
background; reference an article they wrote or a time you saw them speak at
a conference – you’ll be on safe territory.

Sell the role. Tell them all about how awesome your organisation is and why
they wouldn’t want to let the opportunity pass them by. Focus on the benefits
to them. Can they work from home? Do they get the best technology?
Fantastic training? Once you’ve got them hooked, it’s time to take the next
step.

How to Interview Testers

By this stage you should have a tester that you think maybe you want to hire. Perhaps
you’re in contact with them because you’ve engaged with them directly, or maybe their
CV has arrived on your desk via some other channel. Irrespective – you need to figure
out whether or not they’re right for your team, and whether they have the skills you’re
looking for.

19
Traditionally, the way that folk have gone about this is by means of an interview. But, if
you pay any attention to advances in psychology then you’ll probably already know that
most interviews are basically an exercise in confirmation bias. In the classic interview
scenario, whoever does the interviewing will make a decision based on their gut
reaction to the candidate within the first 30 seconds. Then, they’ll spend the rest of the
time looking for reasons to justify the decision they’ve already made. If you want a more
scientific outcome, you need to move away from this model. I have a couple of sugges-
tions for ways you might proceed instead.

Give them a call

If you’re going to follow my recommendations, then you should a) warn the
tester in advance what they can expect and b) be certain you are investing
your time and energy into the right people.

Use the telephone call script below to gauge whether or not it’s worth con-
tinuing the process, tailoring the easy and hard questions so that they’re
specific to your own requirements:

1. Introduction

2. Question about recent project candidate worked on

3. Easy Testing Question

4. Hard Testing Question

5. Are you satisfied?

6. Do you have any questions?

If they make it through the screening call, then you can take them to the
next stage.

20
Give them a work test
Setting your candidate a task that is representative of the kind of work they’ll
be doing for you is a great way to get a better idea of how they think and
behave in a work situation. And there’s lots of different approaches you could
take to keep things fun, or more serious, both in and out of the office.

You should keep in mind the objectives of this approach though. What is it
exactly that you want to know about your candidate and how will you measure
those things by giving them a work test?

If you’ve followed my earlier advice and have gotten really clear on what
specific skills, competencies, personality or attributes it is that you’re
looking for from your tester, you should have a good idea of what information
you’re trying to get out of the work test. You just need to tailor it so that it
answers your specific questions, and any additional considerations like:

• Can the candidate communicate their thinking well?

• Can the candidate complete the exercise within the time allocated?

• Does the candidate have specific domain knowledge?

• Does the candidate have specific skills that can be applied – tools,
automation, critical thinking etc.

If you’re stuck for more ideas, Richard Robinson has a couple of great
examples in his Needle in a Haystack article, available in the July 2014 edition
of Testing Circus magazine. What you really need to know next though is what
“good” looks like, once you get the results.

Stick to a script
Understanding what good looks like gets much easier when you follow
exactly the same process every time you speak to a candidate.

If every candidate does the same work test, then over the course of time you
will know what good, bad and mediocre looks like for that test. If you already
have an example of what good, or even excellent looks like – even better.

21
The same can be said of interviews. You need to stick to a script. Try to make
sure whoever is doing the interviewing asks exactly the same questions
every time. And record the results so that you have data from which you can
start to draw meaningful conclusions, even if you don’t actually hire anyone.

The results might surprise you and provide some actionable insights about
your hiring process. The kind of people you’re targeting for interview in the
first place, for example.

Give them a work day

If you want to go the extra mile, or still have some reservations – get the
candidate in to work with your team. You can start to build a picture of how
good of a cultural fit they are and how they relate to and communicate with
their potential future colleagues.

Inform the candidate that they’ll be in for 3-4 hours, then start off with some
introductions and the formal interview. After the formal interview, introduce
the candidate to your team and leave them to get on with some work for a
couple of hours.

Ideally, try to have a specific piece of work lined up. It should be something
that will require them to collaborate with at least one other member of the
team in order to deliver successful results.

After the work, give the candidate a break to gather their thoughts while you
catchup with the team and get their feedback.

Finish with a debrief based on how things have gone so far. Do they have any
questions or insights based on the experience. Do they still want the job? Do
you still want to hire them?

If the work day is a success, then you’re ready for the final stage.

22
How to Hire the Ideal Candidate

Okay, so you’ve attracted an ideal tester and you now know: a) they can do the work and
b) they’re a fit for your organisation. It’s time to seal the deal. How are you going to make
sure that your tester actually makes it through your door, and not one of the other op-
portunities that they may have available to them? What are you going to offer them that
makes your opportunity stand out among all of the other roles that are available to the
brightest, best, most high-flying testers out there?

It’s probably not about the money, right? Maybe you have some flexibility with how much
you’ll remunerate the right person. But there’s a cap. Once you’ve reached it, there’s
no more wiggle room. And let’s get this straight. Beyond a certain amount of money,
folk aren’t motivated to work harder or perform better anyway. So we need to think a
bit more creatively. What else can your organisation offer that may be attractive to a
certain kind of hire?

• Flexible working hours?

• Remote working?

• Unlimited holiday?

• Training and personal development?

• Coaching?

• The latest most powerful hardware?

• A free lunch?

• Time off work to attend conferences?

23
At this point, it’s not even necessarily about what you can or can’t offer. You just need to
be willing to have a conversation about it. Find out what it is that drives and motivates
your new hire. Once you know what it is that will get them through the door, if you can,
give it to them.

Work with them. Figure out how to bridge the gap between what they want and what
you can offer. Attracting the right people is hard enough. Don’t let them get away once
you’ve found someone that you definitely want.

Curing the recruitment headache

At the beginning of this article, my fictional test manager had quite the recruitment
headache. She could have just outsourced the entire process, but I’ve recommended
the following course of alternative therapies:

Identifying the specific characteristics, attributes and skills a tester needs to have in
her organisation.

• Engaging with the testing community in order to find awesome testers that
have the abilities and personality she’s looking for.

• Striking up a relationship with the tester directly.

• Screening potential candidates with an initial telephone call.

• Testing candidates with a work simulation.

• Sticking to a script during the entire process – making it more rigorous and
scientific.

• Inviting the candidate to a work day – soliciting feedback from both them
and the team.

• Negotiating on working practices rather than money during the offer stage.

Recruiting testers may be challenging, but it’s not impossible.

24
Hiring Testers

25 Challenging Tester
Interview Questions
Simon Knight, TestRail Product Manager

25
In the previous article, I wrote about an approach to finding, interviewing and hiring the
best testers. In it, I suggested that when carrying out an interview you should stick to a
script so that, if you need to compare results across a number of interviews, you can at
least be assured that the same questions were asked each time.

To help you out even further, we thought we’d go the extra mile and provide you with
a list of questions too! And to make sure we had the very best questions, we even
approached some of the best thinkers in the testing community and included their
responses below.

Questions For Warming Up the Candidate

When you start your interview, you want to give your candidate every possible opportu-
nity to shine and show you their best qualities. Use the questions below to help break
the ice and get them warmed up ready for some tougher follow-up questions.

1. What is testing to you? Do you have a testing paradigm? Can you explain it?
(Suggested by Kim Knup @punkmik)

2. Why did you choose software testing as a career, and what motivates you to stick
with it? (Suggested by Dan Billing @TheTestDoctor)

3. Tell me about the work you’ve been doing recently. What’s the most interesting bug
that you’ve found, and why? (Suggested by Katrina Clokie @katrina_tester)

4. What kind of challenges does testing present? Can you tell me about some specific
software testing challenges you’ve faced, and how you overcame them?

26
Questions to Test for Behavioral and Character Traits

Hiring an awesome tester isn’t just about making sure they have the right skills for the
work you need them to do. It’s of paramount importance that they are a fit for your
organisation and the team they’ll be working with. Use the questions below to help you
figure out where their strengths and weaknesses lie, and whether their values align with
yours.

5. Have you encountered any challenges working with your colleagues? Tell me about
a specific instance when you were in a difficult situation, and how you dealt with it.
(Suggested by Katrina Clokie @katrina_tester)

6. How about situations where you have to make decisions… Have you ever made a
bad decision? What contributed to you making that decision? How did you deal with
the consequences? (Suggested by Kim Knup @punkmik)

7. How have you added value to the organisations you’ve worked with? Can you give me
a specific example from your last or current position? (Suggested by Dan Billing
@TheTestDoctor)

8. What process are you using for testing currently? Can you describe how you might
improve it? (Suggested by Katrina Clokie @katrina_tester)

27
Questions to Determine A Candidate’s Testing Skills

Now we’re getting to the good stuff. What kind of a tester are you dealing with here?
Do they know their edges from their corner cases? Can they distinguish between a test
and a check? Do they understand their biases and models, and how a worldview can
influence their approach and decision making process? Use the questions below to dig
deep into their testing toolkit and to make sure they don’t view every testing problem as
a nail to be pounded with a blunt instrument.

9. What kind of tests have you been doing? What do you enjoy about them? How do you
develop those tests?

10. When you perform a test, what steps do you take? What’s your process?

11. Have you ever written a test plan? What would you put in one?

12. How do you prioritise your testing? What factors might influence your decisions?

13. How do you know when it’s time to stop testing?

14. What’s the role of risk in your testing? How do you analyse and measure it?

15. Do you measure how effective (or not) your testing is? What metrics do you use?

16. If I left you testing for two hours, what would you have to show me when I returned?
(Suggested by Richard Bradshaw @FriendlyTester)

28
Questions to Determine A Candidate’s Testing Skills

No doubt many readers will be hiring for automation skills; but being able to automate
testing activities isn’t just about understanding the right tool, language or framework.
It’s about identifying where the risks to the product are and then being able to pinpoint
those risks by developing intelligent, reliable and repeatable automated tests. You want
your tester to help you speed up the development and delivery of your product, not slow
it down. Use the questions below to help you figure out whether they have the depth of
knowledge to do so.

17. Have you automated any of your tests? How so?

18. What’s your favourite testing tool? Why? If some technical constraint meant you
were unable to use it, what would you do instead? (Suggested by Richard Bradshaw
@FriendlyTester)

19. How do you know when you (or your automation) has found a bug? What makes
it a bug? Are some bugs more important than others? How do you report them?
(Suggested by Richard Bradshaw @FriendlyTester)

20. What do you do if the developers decide the bug is not a bug?

21. How do you decide which tests to automate? Which tests don’t you automate, and
why? (suggested by Katrina Clokie @katrina_tester)

29
Questions to Determine A Candidate’s Testing Skills

Finally, if you’ve made it to the self-development questions and are still interested in the
answers, you may have found a keeper! But are they in it for the long term? Is software
testing a passion, or just filler until they can find a proper job? How do they learn, and
what keeps them motivated to do so? Use the questions below to probe your testers
long-term commitment to the role.

22. Testing can be challenging. What keeps you motivated? (suggested by Kim Knup
@punkmik)

23. How do you stay at the top of your game? What self-learning do you do? (suggested by
Dan Billing @TheTestDoctor)

24. What have you been criticized for in the past? How did you respond to that
criticism? What did you do about it?

25. Describe the characteristics of your ideal boss, and why.

This completes the list of 25 questions that you might pose to a candidate in an
interview. In the next article, we’ll take a look at 21 questions that a candidate should
ask in return.

30
Hiring Testers

21 Questions to Ask
When You Are Being
Interviewed
Justin Rohrman, Excelon Development

31
In the previous article, Simon Knight described questions that a hiring manager can
ask candidates during an interview. In response to that article, I wrote one on the bias
and problems built into different styles of interviews, and some ways we can counter
that to get a better understanding of the candidate. Ideally, we ask a person the right
questions, and in the right conditions so that we can learn about their skill level and
how they might contribute to our teams.

I also assess candidates based on the questions they ask during an interview. These
questions show that they are curious and paying attention. They show that the person
is able to discover when they don’t know something, and figure out how to seek that
information. And, they of course can learn more about whether or not they want the job
based on the answers.

So, what questions should you ask when you are the one being interviewed?

32
Clarifying the Role

Job advertisements tend to be pretty generic. Most will say something about discov-
ering and reporting bugs, creating and executing test scripts, and generally working to
help the development group produce the best software they can. But, in the same way
that a test case isn’t testing, the job specification isn’t the job. There are whole catego-
ries of things that people forget to describe when they make a job advertisement, and
others that are hard to turn into bullet point lists. These questions are designed to help
you learn what a tester really does at this company.

1. What does a normal day look like for a tester there? What meetings do they go to?
Who do they talk with? How much time do they spend testing software?

2. What skill set do you expect the person you are hiring to have? Can you give some
examples of what tasks you would expect them to be able to perform?

3. What technology and tool stack do your testers need to know? What is the learning
path for someone that is not familiar with a piece of technology or process?

4. How do testers work with developers, product people, and decision makers?

5. How is the relationship between testers, developers, and product people?

33
Test Approach

Companies, and even teams within a company, approach testing from vastly different
perspectives. Some groups deliver software hundreds of times every day and do
most of their testing by layering automation and using radical collaboration tools like
Extreme Programming. Others companies completely separate the testing, develop-
ment, and product groups. Testers are expected to do the testing equivalent of big
design up front, and then perform those plans once there is software to look at. Most
companies today are somewhere between those two extremes. You will probably want
to understand how the company thinks about and approaches software testing.

6. How much time do your testers spend doing things that aren’t testing, such as doc-
umentation, meetings, emails, and status reports?

7. How free are your testers to make decisions on how their testing work is done?

8. How close to development do your testers work? Do you ever have developers and
testers pairing on a code change?

9. What tools do your testers use? What happens if they want to use something new or
different?

10. How does a team know when they are ready to stop testing and push a code change
to production?

34
Culture

Culture is shorthand for the ways people in a company interact, the goals they strive
for, their value system and ethics, and how employees treat each other. Some cultures
foster an environment where lone individuals can control broad swaths of the product,
and are responsible for its problems as well. Others are collaborative, congenial, and
encourage open discussion about where improvements can be made and who the
appropriate people to do that work are. There are also the very important questions of
diversity and inclusion practices. Getting a feel for culture can help the interviewer, or
you, to figure out if you are a fit for this organization and how long you might want to
work there.

11. Do you give your technical team members the opportunity to work from home?

12. How often do you find yourself in a ‘crunch time’ situation where you require people
to come in early, stay late, or work on weekends?

13. What sort of diversity and inclusion practices do you use to make sure you can build
and maintain diverse, high-functioning teams?

14. Does your company financially help testers that want to attend conferences or
attend training. If not, do they offer time off without having to take PTO?

15. How do you prevent burnout of your technical team members?

35
Clarifying the Team

Team compositions vary quite a bit. The last company I worked with had a completely flat
structure. There were 6 or 7 developers, 2 testers including me, and we all sat in the same
room. We had some people that were acting as leads in practice, but they didn’t have the
title or the power that comes along with that title. We all reported up to one development
manager when we needed something, or when he needed something from us.

Other teams I have worked with were steep pyramids. One company had a handful of
testers per group, each group had a lead tester, and a couple of groups combined would
work with someone called a test architect. There were also managers and directors in the
mix. Everyone reported up to someone else and the ladder was tall. My personal preference
is to work in a more flat structure. I feel like it is easier to get things done, and decisions
come through much faster. These questions can help understand how flat or deep the
organizational hierarchy is at a company, and how that will affect the testing work.

16. What team would I be working on assuming you decide to hire me?

17. How exactly do the different roles work together? Do they sit in the same room?
Do they sit at the same table? Do they have to schedule meetings to talk?

18. Who would I report directly to? What is that person’s relationship to the team?

19. Why are you wanting to add a tester to the team?

20. What is the biggest complaint your testers have now?

21. What do your testers like the most about the team and company?
36
Emergent Questions

This is a tricky category because it is dependent entirely on the flow of conversation. I

usually have a laundry list of questions I want to ask people I’ll be working with and what
you see above is an example of that. But, I also come up with a lot of questions based on
what we are talking about at any point in time. Technology work is full of lingo, some of
it is overloaded to mean different things at different points in time. In testing, we don’t
have a lexicon, or a standard dictionary. A lot of my questions focus on that. An inter-
viewer might start talking about their regression testing process and I’ll say “When I say
that term I mean the testing we do to discover new problems introduced by a change. Is
that what you mean?” I also like to flip questions back on the interviewer to learn about
them. If they ask me how I would test something, after I give my answer I like to ask
them the same question. This can be a learning experience, and can also be a litmus
test for how the conversation is going.

This is just a few of the questions I like to ask. They help me learn much more about
what sort of job I would be getting into. Sometimes, they help the people on the inter-
viewing side to better understand what they are looking for, and what they really need in
a new tester.

37
Hiring Testers

Retaining Your
Software Testers
Simon Knight, TestRail Product Manager

38
I’ve had a bit of experience with wanting to leave software testing gigs. During my
career as a tester I’ve worked for a number of organizations in various different capac-
ities. From my days as a rookie through various layers of senior, technical and automa-
tion roles right up to the dizzying heights of test management.

Over that time, as you might well imagine, I’ve had a range of experiences in the
groups, teams, departments and corporations within which it’s been my pleasure (and
sometimes my displeasure) to work.

In total I have worked for 14 different companies during my testing career so far.
Granted, 11 of them were on a contract basis – so I was never in any real danger of
having to stick around for too long, but nevertheless; a few patterns have started to
emerge. I have been able to make a number of observations about which companies
I would have chosen to stick around with and work for in the longer term, and which
companies I definitely wouldn’t.

In my experience, those factors break down into several commonalities which I believe
can probably be applied to most testers. They are as follows:

• Opportunities for learning and development, or lack thereof.

• Recognition for contributions, or lack thereof.

• Pay, relative to the organization and the wider marketplace.

And that’s basically it. In any situation where you’ve got someone wanting to move on
from a role, there’s probably any number of other contributing factors at work. But in
the technology field and particularly where software testers are concerned, I think
these are the main things that will determine whether or not a tester decides to stick
around, or decides to twist and look for a role elsewhere. So let’s take a look at the
problems in more detail.

39
Learning and Development

Testers along with many of our more or less technical colleagues working on software
development projects are a curious bunch. They like to learn as much new stuff as they
possibly can. Which provides employers with a couple of interesting challenges.

If you have a tester who is not curious, you most likely don’t have a very good tester.
All of the best testers I have met are innately curious and have a burning intrinsic mo-
tivation to poke and pry and break or improve things. They may have various degrees
of aptitude for some of the ancillary skills you’ll be looking for in your particular brand
of tester, like coding and automation ability for example, but at their core they will be a
curious little beaver.

If your tester isn’t curious – you hired the wrong tester. Read the earlier article in this
ebook on hiring the best software testers and try to do a better job next time.

If your tester is curious however: You had better be giving them plenty of rope with
which to exercise their curiosity. Otherwise you’re going to end up with a sad little
tester because they have nothing new to learn. And sad testers tend to develop a desire
to move on and explore new pastures instead. Pastures where their curiosity will be
recognised and rewarded, not hampered or curtailed.

I’ll provide you with some strategies for exercising your testers curiosity at the end of
the article. But next, let’s take a look at recognition.

40
Recognition and Rewards

Testers don’t like to be treated as 2nd class citizens. There. I said it. Hopefully this will
not be news to most readers. But just in case you need some further clarification on
this point.

I have seen and been in the position of being treated like the work I’m doing doesn’t
matter. Like it doesn’t add value to the project on which I’m working. It’s not fun. I don’t
like it. I’ve never met a tester who does like it. You wouldn’t like it. So don’t do it.

Here’s an idea instead. If your development team don’t feel like they need a dedicated
testing professional, then don’t hire one. Simple, right?

Or, if you’ve hired a tester who genuinely isn’t adding value, and who is for some reason
unable or unwilling to put in the work to improve themselves to a point where they can
and are adding value – sack them. Try to hire better the next time.

If on the other hand, you have a tester who is adding value -- who is helping your team
to deliver a superior product or service -- celebrate them and their work (along with the
rest of the team, of course). Because delivering great quality products is a team effort
and good testers play a big part in that. How big of a part will likely determine the next
point.

41
Compensation and Pay

How much folk are paid for their efforts is clearly a tricky subject. People don’t generally
like to talk about pay (certainly in the UK where I come from anyway) but they surely do
like to feel as though they’re being remunerated at a level that correlates with the value
they add.

When I was a junior tester I got paid a junior salary. As my experience and skills
increased my remuneration increased accordingly. Now I consult and charge a fee for
my services. Clients are happy to pay that fee because they can see the value that is
being added.

Clearly that path is not for everyone and is most likely not a great model for your team or
organisation if you’re reading this as a test manager. You may also have any number of
constraints around how much you’re able to pay people. But I’m just making the point.
There are plenty of opportunities out there for ambitious and skilled testers. So if you
have a good one and you want to keep a hold of them, you should be prepared at some
point to have a serious conversation about their benefits package.

If they’re maxed out on the payscale, try to think about whether there are some other
things you can do for them by way of perks and bonuses. Some new hardware for
example. Remote working. 20% time. Travel. Etc. Try to think out of the box. It’ll be
worth it.

42
Additional Strategic Advice

If you’ve taken some of those points on board, then you’re most likely looking for some
ways to actually address the issues. Good News! You’re in the right place. Try imple-
menting some of the ideas on the checklist below.

Your Tester Is Curious

• Maintain a testing book library (the books in this list will make a great start)

• 20% of their time can be set aside for skills and interest development (so
long as it can be tied back to work)

• Send them off to learn other areas of the business so they can expand their
skills and increase their domain knowledge

• Pair them with developers so they can learn some coding and development
skills to supplement their testing knowledge

• Start a regular brown bag lunch session so that testers (and anyone else
who is interested) can eat lunch together and share knowledge, skills,
useful experiences, lessons etc.

• Start a meetup and invite people from outside of your company to it

so your tester gets exposed to other ways of thinking and approaching
software testing

• Send your tester for some training

• Send your tester off to a conference

43
 Your Tester Needs Recognition

• Try to observe when they’re doing an awesome job, in the moment,

and praise them for it

• Implement some kind of public recognition program and nominate

them for it

• Credit them publicly for their work, in a standup or show and tell for
example

• Give them some more responsibility and make sure they and others
know why

• Spend time with them one-on-one and tell them how much you
appreciate their work, over lunch for bonus points!

Your Tester Needs Rewarding

• Give them some training budget – even a few books can make a
world of difference

• Send them to a conference or some training

• Upgrade their hardware

• Give them some extra time off or a more flexible working arrangement

• Cut them some slack – even if you can’t promote them, you can provide
them with greater leeway and authority with which to do their work

Bear in mind that these are just ideas to try and get you started. Hopefully as you read
through them, you’ll find some of your own ideas being stimulated. As you do, add them
to the list and try them out. By paying attention to their needs, you’ll be more likely to
keep your best testers on board.

44
Hiring Testers

Making Do When
You’re Understaffed
Erik Dietrich, DaedTech LLC

45
Are you feeling spread thin when it comes time to test your application? Is your team
missing deadlines and pushing back releases to validate your product? This is a common
feeling. It manifests itself in many ways.

For example, picture the following scenario. Your team is understaffed. You can’t
get through testing fast enough to get features out quickly. It’s all hands on deck—
the testers, developers, analysts, and managers are all stepping in to help test new
features. Deadlines slip and morale drops as testing takes center stage.

Or how about this. Your team is writing software with a hard deadline. Regulations are
changing and you need to be ready. But as usual, development took longer than
expected. Your QA team wanted two weeks to test, but they’re only going to get three
days. And even after pulling extra hours, you’re still behind.

So now you know you’re not alone. But where do you go from here? The short answer to
your problem is “automate.” To which your response will probably be, “But we don’t have
time to automate! We don’t even have time to test!”

OK, let’s look at the long answer then.

46
Manage Your Testing

You have your unit tests that run on the developers’ machines, contract tests, and tests
that run in some other system. Acceptance and regression tests are handled by your
offshore team using SoapUI and reported back weekly. The QA team performs manual
acceptance and UI testing based off their Excel sheet.

All the results are collated, reviewed, and shown to management. Plans to enhance
testing and add new feature testing are discussed, put in a separate document, and
reviewed by the appropriate people.

Then when issues are found, you capture that data in yet another system like Jira,
Pivotal Tracker, or Zendesk. Sometimes you track it in more than one system.
Therefore, your team spends hours each week going from one system to another trying
to keep tabs on it all.

I hope that illustrates the point that all these systems and all this back and forth cause
waste. Take a hard look at what’s really going on with each feature and test. Find spots
where things are repeated and recommunicated. When you have a clear picture, look
for solutions that will let you manage all your tests in one place. Make sure it’s visible
and integrates well with your existing systems.

47
Automate Your Tests

If the testing process itself takes a lot of time and effort, one of your first steps might
be to automate it. Therefore, review all manual tests and look for streamlining opportu-
nities. Take a look at what tools are available to help your team get the repetitive grunt
work done quickly and easily.

However, don’t fall into the trap of testing everything from the UI. Keep the testing
pyramid in mind. And note that the acceptance test doesn’t need to be at the UI level. It
can be at the API or boundary level. Having too many tests at the top of the pyramid will
slow you down. The tests will be flakier, run more slowly, and be harder to maintain. If you
notice your tests are slowing down, it’s time to see if the focus is on the right functionality.

Once you automate your tests, the next step is easier. This is a great example of how
you can make the change easy, then make the easy change.

Test Early, Test Often

48
With software testing, the sooner the better. This is why test-driven development (TDD)
pushes testing first. TDD ensures you’re testing before the code is even written. Though
even if you don’t want to take on something like TDD, don’t wait to test everything at the
end. Make sure to test features as soon as they’re ready.

And the more often you test, the more processes you’ll be forced to automate. Making
frequent testing an easy process will ensure you’ve got a streamlined and efficient way
to run these tests.

Prioritize

Prioritization comes in two parts. On one side you’ll have to prioritize which tests need
attention first. On the other, you’ll have to prioritize which issues to fix first.

For the first half, again we have two things to consider:

• What features are a priority

• What types of tests are a priority

After that, prioritize bugs and changes that come out of testing. Make sure everyone
knows the difference between a minor formatting bug and a major “all hands on deck”
bug that needs to be the top priority. Be explicit about prioritization rules. Don’t leave it
to assumptions.

49
Adopt the Right Tools

The QA team needs tools to be more efficient. And sometimes they need training. Don’t
just throw out new tools and expect the team to use them efficiently. Take time to learn
the tools, share that knowledge, and find the most effective ways to test with them.

Fix Small Inefficiencies

Sometimes test teams get into a rut. They perform the same tests day after day.
Eventually, they stop seeing inefficiencies and parts of the operation that could be
improved.

Have someone sit with a tester for a day or two. Better yet, have someone outside the
regular work group watch what testing involves for a few days. The visitor should note
anything that seems repetitive or redundant. Those steps are the first places you can
look for automation opportunities.

50
Also, encourage testers who know all the shortcuts and tricks to the tools they use to
share that knowledge with others. Testing is a skill that requires knowledge of the tools
and practice using them. It can’t be left to those who hunt and peck their way to a result.

Another opportunity for streamlining is test data. From what I’ve seen, setting up test
data often takes longer than the actual test. If this is the case for your team, work on
automating this data entry. Or figure out how to get around it.

Many opportunities exist for improvement and automation. It’s up to you and your team
to find them.

Come Back To The Short Answer

Now let’s come back around to the short answer of “automate.” Is that the silver bullet?
Yes, if you know where to aim and focus your efforts.

You can’t go from no automation to having everything fully automated and efficient
overnight. Take time to review inefficiencies in your process and start working on
improving them.

In addition, when looking at automation, look beyond just the tests themselves. Are
there other processes that can be automated? Consider tests that report and monitor
themselves. Or tickets that close themselves after a successful test. When we talk
about test automation, it shouldn’t just be the tests that we’re automating, but the
whole process of ensuring a quality product.

51
Hiring Testers

How to Hire the Agile

Team You Need
Jeff Langr, Langr Software Solutions, Inc.

52
You’re interested in hiring for an existing team that purports to be “agile,” whatever that
means. Tell me, if you can:

• What’s core to the culture of that team — what do they hold near and dear about how
they work?

• Which one technical element is a must-have — if there are any?

If you can’t answer these questions succinctly, you’re jumping the gun with your hiring.
Here are six major points to consider — three job requirements essentials, and three
important aspects about hiring — before you draft a job requisition for an agile team.

Agile Essentials

I believe the success of agile is predicated on two things: creating a highly collaborative,
cohesive team, and fostering the willingness to keep learning and adapting.

When I say highly collaborative, I mean team members must work with one another
throughout the day, and they must work on as few things as possible. If your team
members spend most of their day in their own cube, or if you have as many current work
items as team members, your team is not working in a highly collaborative fashion.

If collaboration and adaptation are paramount, what should you put into your job requi-
sition (“req”)? Keep it simple: Describe briefly how you currently work and your interest
in making people comfortable with continual learning.

“We mob occasionally, and when we don’t, we prefer to pair. We seek folks comfortable
working in this fashion and with regularly evolving team practices and dynamics.”

53
Technical Essentials

Delivering software, agile or otherwise, still demands expertise in the core technol-
ogies we employ, but we don’t need everyone to be an expert on every technology. In
fact, we can do just fine with a few experienced folks in the core technologies as long as
we are truly collaborating. Techniques like mobbing and pairing can do wonders to help
savvy people rapidly ramp up.

Lest you think I’m discounting expertise, I firmly believe it’s essential for at least a
couple of people to know each technology very well. You might survive without that
expertise, but you will also waste a lot of time as a result. (Hint: You can rent expertise.
Just make sure the person you rent it from excels at helping others ramp up on the
technology.)

The worst thing you can put into a job req is a laundry list of technologies. All it accom-
plishes is chasing off candidates who might adapt well to your world.

At one place I worked, we sought an experienced C++ developer to help with test-driven
development mentoring and coaching. Before I got my hands on the job req, it was an
HR-produced morass of a dozen or more technical demands. I simplified the demands
to the following:

“Our developers predominantly work in C++, so we seek people with solid C++ experience—
and people who still want to work in this challenging language. You must also have experi-
ence helping others learn TDD.”

54
People Essentials

One more characteristic I consider important to software development in general (agile

or not) is good problem-solving skills. While we can all learn these skills, some folks
don’t appear to have much aptitude for it. I’ve seen employment end in tears when we
realized that recent hires weren’t able to contribute sufficiently to development efforts,
no matter how much we tried to help them along.

While you don’t need to ask your candidates to solve complex puzzles, they can be
expected to think and work through mildly interesting problems and mundane, typical
challenges. Doing so is pretty much our minute-to-minute job. You can explain this
expectation in the job req if you like, but you could also wait to reveal it during an initial
conversation.

Ultimately, however, what’s most important from an individual stance is that they want
to work the way your team works.

“In addition to regular pairing, we expect programmers to produce developer-focused

tests and to contribute to BDD efforts.”

55
Selling

While you may view hiring as having candidates sell themselves and their skills, the
first mentality should be that you are selling them on your culture — you are trying to
find people experienced with, and happy about, working the way you do. As such, the
first focus of your req should be on making the job sound appealing. Don’t overdo it, but
remember that every hire is a give-and-take negotiation process, and thus so should be
the job req itself.

I mentioned earlier rewriting the job req for a TDD coach. Before doing so, an external
agency sent me a couple of dozen resumes, most with neither C++ nor TDD empha-
sized or even mentioned. Out of the candidates, only two were anywhere near what we
needed, and neither of those survived an interview.

After simplifying the job req to the essentials, we received resumes for five candidates.
Four of those five looked good enough to interview, and all did well in the interviews. We
ended up hiring a top-notch candidate.

56
Interviewing

To give candidates the best opportunities to shine, you can play out the interview
across a number of forums. I’ve done this a few different ways, but here’s a three-step
approach that worked well:

Initial phone conversation: A 15-minute chat to describe the company and

what we were doing, and to get a feel for what the candidate is interest-
ed in (and capable of) doing. Where I worked, this conversation was run by
business folk.

Initial technical conversation: A 15- to 30-minute chat with development

team members to clarify the technical landscape and get answers to a
question or two about the candidate’s background. More importantly, this call
sets the stage for the core of the technical interview.

We asked developers to build code for a handful of increasingly interesting, related

features. Developers were told to get back to us when done to schedule a follow-up. We
expected them to spend perhaps an hour or two on their homework. We would explain that
they would be expected to pair with us when ready, to add one more interesting feature.

If it took a few days for them to get back to us, that was fine; if it took a couple weeks,
that was fine too. Rushing to hire a candidate is never a good idea.

Importantly, we explained that they were free to use whatever technologies they felt
most comfortable with: operating system, programming language, editor and so on.

57
The goal was on the problem-solving, not learning a new technology. We wanted them
to be able to approach the forthcoming demonstration session with as much comfort
as possible. (It turned out that choosing one of our desired technologies, something
they weren’t so comfortable in, was always a mistake for the candidate. We reminded
them of this reality.)

Demonstration session: A 60- to 90-minute session with at least a couple

of us. The candidate was asked to step us through their solution and explain
choices they made. Sometimes we’d ask what they thought about alternative
approaches; I remember dismissing a candidate almost immediately when
they took offense at the thought that there might be a better way.

We then presented the candidate with a new feature, something that would add an
interesting wrinkle to their existing solution. They were expected both to talk and work
through coding a solution at the same time. Our goal wasn’t to put them on the spot; in
fact, we sought to help them as they worked.

The candidates we dismissed were people who weren’t capable of interacting with us
effectively. Some wanted to turn their back and scribble things for a while; while that’s
perfectly acceptable, the candidates we appreciated more were the ones who could
talk us through their thought processes. We most rapidly dismissed the candidates who
weren’t open to listening to our suggestions.

Apart from information about their ability to solve problems, the most important part of
this interaction was discerning whether we thought we wanted to work with them every
day as part of our team.

We made it clear that either the candidate or we could choose to terminate the process
at any step.

Sometimes we ended up on the fence. For a while, we had a second pair of developers run
another interactive session, but this second session always resulted in the same uncertainty.
We decided to end the interview process, one way or another, after a single session, and I
believe that was the best choice. Either we knew they were a good hire, or we passed on them.

All of the above steps can be accomplished online, though of course, face to face is
usually far more revealing. If at all possible, get the candidate to visit in person.

58
Diversity and Culture

A successful organization embraces diversity of thought and approach. While I firmly

believe this diversity is important, I also believe that building a culture is an important
aspect to building a successful team — which also implies that I believe teams within
the same organization can and should have differing cultures.

The culture represents the shared beliefs, attitudes and preferred ways of working for a
team. A healthy team understands its culture and accepts that its culture will change as
it matures. It also accepts that “immature” new ideas might benefit the team, and that
experimentation might be worthwhile.

While keeping open to new ways of thinking, a team should not feel like it needs to hire
people who will directly subvert their current ways of working: “Oh, you do pairing? I
hate pairing, and here’s why it’s stupid …” No, thanks. We don’t need the drama.

Diversity is great as long as it doesn’t threaten to destroy the team coherence. A team’s
set of core values should be small and strongly held, and it should be open to practices
that improve the team without jeopardizing its values.

The core Agile Manifesto values promote individuals, interactions, collaboration and
responding to change. Your approach to hiring should be no different.

59
About TestRail

We build popular software testing tools for QA and development teams. Many of the
world’s best teams and thousands of testers and developers use our products to build
rock-solid software every day. We are proud that TestRail – our web-based test man-
agement tool – has become one of the leading tools to help software teams improve
their testing efforts.

Gurock Software was founded in 2004 and we now have offices in Frankfurt (our HQ),
Dublin, Austin & Houston. Our world-wide distributed team focuses on building and
supporting powerful tools with beautiful interfaces to help software teams around the
world ship reliable software.

Gurock part of the Idera, Inc. family of testing tools, which includes Ranorex, Kiuwan,
Travis CI. Idera, Inc. is the parent company of global B2B software productivity brands
whose solutions enable technical users to do more with less, faster. Idera, Inc. brands
span three divisions – Database Tools, Developer Tools, and Test Management Tools –
with products that are evangelized by millions of community members and more than
50,000 customers worldwide, including some of the world’s largest healthcare, financial
services, retail, and technology companies.

60