Agile 2008 Conference
Colossal, Scattered, and Chaotic
(Planning with a Large Distributed Team)
Wes Williams
Sabre Airline Solutions
wesley.williams@sabre.com
Abstract
If planning for a large co-located 30-plus
development team is not enough to make you want to
pull your hair out then try a 30-plus development team
located across several time zones, in places with
different cultures and languages. Now, you’ve reached
a level in the Agile planning game that would send
most product owners running home to their mommies.
Planning in Agile Projects requires enabling the
team to have a continuous conversation with the
customer. How can you have meaningful, continuous
conversation and feedback when your team is scattered
across several time zones, cultures and languages?
This paper describes our experience of how one of our
largest teams in Sabre Airline Solutions® was able to
make changes that allowed the continuous
conversations to take place. The paper is written from
the perspective of both development locations the main
office in Dallas, Texas and the “remote” office in
Krakow, Poland.
The team successfully overcame many obstacles
while rapidly growing a small co-located development
team into a large globally distributed team that was
able to efficiently deliver customer valued, quality code
and customer satisfaction with each one-week
iteration. It attributed this success to three main Agile
areas: evolving the Agile Customer Role, making our
meetings valuable and efficient for everyone involved,
and good coordination across the sub teams.
1. Introduction
Sabre Airline Solutions has a real-time data
management application to manage critical processes
for an airline which the cost of maintaining the
978-0-7695-3321-6/08 $25.00 © 2008 IEEE
DOI 10.1109/Agile.2008.25
Mike Stout
Sabre Airline Solutions
michael.stout@sabre.com
technology and support was becoming prohibitive
while impacting the ability to enhance the tool. The
product was supported predominantly by manual tests,
which made the task of continuous refactoring a
challenge. Looking at all options, the decision was
made to do a product refresh from technology
perspective. Although Airline Solutions had many
successful small to medium-size projects that used
Agile values and principles from the beginning, this
would be Airline Solutions’ first opportunity to
implement a large-scale project following an Agile
Methodology starting day one.
The Agile Methodology the team followed was
most similar to eXtreme Programming (XP). When
the team was small and co-located it worked in two-
week iterations. When the team grew in size and
became globally distributed the team decided one-
week iterations would be necessary to enforce a higher
level of collaboration and integration across the team.
The team was committed to using Test Driven-
Development (TDD), both at the unit and functional
level, to verify the completeness of new functionality.
The team performed pair programming but didn’t do it
to the extent XP purists would recommend. The
application was built with a continuous integration
build process using Ant and CruiseControl. The team
used user stories, release planning, iteration planning,
and daily standups to keep everyone informed and
make decisions as quickly as possible.
The project began with a small hand picked team of
co-located developers. The team was put together by
Airline Solutions’ chief architect. His criteria for
choosing team members were excellent technical skills,
many years of business experience, and not only
working experience on Agile projects but advocates of
Agile values and principles.
356
 Although the team was committed to operating the
project based on Agile values and principles as best it
could, its best of intentions were slapped in the face by
reality. The rigid schedule and scope set by a waterfall
contract drove a release schedule that could not be met
by the small team. The team had to grow and it had to
grow faster and larger than was originally expected.
This caused budget restrictions and also meant that the
team needed to grow globally. The team ended up
growing to 30 developers (14 in Dallas, Texas and 16
in Krakow Poland), three testers (one in Dallas and two
in Krakow) and one full time Agile customer (i.e.
proxy customer/product owner) in Dallas and one part
time proxy customer traveling most of the time.
There were many growing pains. The Agile
customer and technical architect were traveling and
rarely available to the team sending the planning,
standup, and retrospective meetings into a tail spin.
The team spent a lot of time going in circles having
lots of entertaining and lively discussions that, at the
end of the day, got them nowhere without the decision
makers.
Something had to change or the project was going
to fail.
2. Evolving the Agile Customer Role to
Work Across Time Zones and Teams
The team understood the number one principle
behind Agile is to satisfy the customer through early
and continuous delivery of valuable software. The
need for a single voice representing our real customers
had to be addressed. The development team needed
someone who could communicate the business needs
and make decisions based on multiple airline customer
needs.
The person originally identified to perform as the
Agile customer was chosen because he had many years
of business domain knowledge and worked closely
selling and marketing the product to airline customers.
Unfortunately, for the development team, people with
these skill sets are in high demand. The sales and
marketing team needed him in front of the customer as
often as possible, selling and negotiating. This meant
he was on the road a lot and unavailable to the
development team.
When the team was small, co-located and contained
developers that also had several years of domain
knowledge, the lack of access to the customer only
caused slight delays and small amounts of re-work.
357
Growth and globalization of the development team
magnified those inefficiencies. Growth meant only a
small portion of the team had the right domain
knowledge to make good “educated guesses” based on
the limited story descriptions and acceptance criteria.
Being globally distributed across several time zones
now meant half of the team was in a location with
limited access to the customer team.
Management agreed to pull two experienced subject
matter experts from other parts of the organization and
added them to the team to help fulfill the Agile
Customer Role. The two new Agile Customers would
work closely with the traveling Agile Customer to
ensure they were communicating to the team with a
single voice. Planning started getting back on track
with better user stories and acceptance criteria.
Development and testing now had a single voice that
could make the business decisions around priorities
and functionality that were needed.
Although having two newly dedicated Agile
customers helped the team tremendously it wasn’t
enough. They quickly became overwhelmed with
creating stories, acceptance criteria, verifying stories
and acceptance test, and answering questions from 30-
plus people. And there was still the issue of not having
an Agile customer at the ‘remote’ location.
The team then invented the role of “feature owner”.
It was introduced to assist the Agile Customers in
creating stories and acceptance criteria, a kind of proxy
to the Agile Customer. This role was intended to be
performed by senior developers with at least some
domain knowledge. These developers would become
proxy customers for a specific feature of the system.
They would break the feature into user stories with
acceptance criteria and be available to answer
developer questions about the feature. The Agile
Customer would verify these stories and acceptance
criteria and answer any questions the feature owners
had about the stories. This role increased the Agile
Customer’s availability to the developers and testers on
a day to day basis.
Obviously, this meant some senior-level developers
were writing less code because they were creating
stories and acceptance criteria and spending a
significant amount of time answering questions. It
might seem that the team would develop less code but
adding this role actually increased the productivity of
the development team. Many times projects add more
developers and are surprised that it does not make the
team faster. Development capacity was not the
constraint for the team in this case. The team had to
look at the whole development process as a system and
not focus on improving one piece of the system. In
this case the Agile Customer had a huge queue of
business information that needed to be turned into user
stories and acceptance criteria. Meanwhile the
developers were idle waiting for work or worse writing
code that was not what the customer wanted. By
having the developers perform a part of the customer
role they were no longer idle and the customer’s work
queue was reduced, increasing productivity.
This allowed the team to create feature owner
representatives in both of the global locations. The
team was able to temporarily relocate a senior
developer with business domain knowledge to the
remote office since those skill sets were limited there.
Having feature owner’s in both locations did increase
the remote office’s productivity.
The ideal solution would have been to have a
permanent full-time Agile customer located in the
remote office as well as the senior technical person.
This would have allowed the domain knowledge of the
team to grow much faster and increase productivity.
The idea was considered but the team was unable to
find someone, with the necessary business domain
knowledge, to relocate or find someone that could be
hired before the project was completed. However, the
team was able to send the Agile Customers from time
to time to the remote location for a couple of weeks.
As Krakow team lead Adam Szczeblewski said
“Feature owner is not the best idea, but the best we
could come up with within constrained resource
situation. In the future I'd like to see more Agile
Customers (decision empowered business people) in a
project of such a scale”. Many people on the team
voiced this concern and believe they functioned best
when all roles were performed in all locations.
The biggest mistake the team made with the
“feature owner” role was asking a developer with
limited domain knowledge to take on the role. This
caused the team to have incomplete stories and
acceptance criteria because the feature owner did not
ask the right questions to the Agile Customer. This led
to some of the same issues the team had prior to
introducing the role.
It would be nice to say that fixing the Agile
customer role was the silver bullet that fixed all the
team problems but it didn’t.
358
3. Overcoming Meeting Inefficiencies
How do you make a meeting with 35 people
effective, efficient, and beneficial for everyone
involved? You don’t! Most meetings do not need
everyone and rarely is there a topic that is relevant to
everyone on the team.
It was very frustrating for the team to have a
planning meeting that lasted a whole day and at the end
of the meeting team members still didn’t really
understand what needed to be done. During
retrospectives it was difficult to get some people to be
open and honest in a large group environment. Even if
everyone felt comfortable speaking in a large group it
would take forever to discuss everyone’s issues and
come to a consensus on what needed to be done.
Retrospectives were reduced to showing a happy face
or sad face on a sticky note as to how you felt and no
real changes to the team processes were made after the
meetings. Standup meetings were also affected by the
large single team, lasting an hour or more. After
operating like this for about a month it was finally
obvious to everyone that a restructuring of the team
needed to take place. The team needed to be broken
into sub teams.
It was decided to initially break the team along
technical boundaries, the GUI, core business logic, and
a team that dealt with back-end integration with
customer systems. For those screaming, “NO” right
now, it is understandable. This was a terrible way to
divide the team and caused more issues than it solved.
The team knew there was a risk in dividing the team
this way. However, they also knew they had a
significant issue with the chosen GUI framework. The
framework met the basic design ideas that were
desired, but the framework for it was not completely
implemented. Several key features were missing and
the team was working with the owner of the framework
to implement these and stay with the frameworks
design as best as possible. Therefore, the team went
ahead and split the sub teams along the technical
boundaries with the understanding it would be fixed as
soon as there was enough of a technical framework to
do so.
In the beginning things were much better than they
were on the large monolithic team. Meetings were
more meaningful and valuable and the team started
making good progress again. Iteration planning
meetings became very productive and took only one to
two hours depending on the sub team and difficulty of
the stories in the iteration. Standup meetings were
taking 15 minutes or less and were beneficial for
everyone. Retrospectives also became useful and real
issues were not just talked about but changes were
made to resolve them.
Although things on the individual sub teams were
better it wasn’t long before we started to see problems
across sub teams. Stories weren’t complete from a
user perspective. Teams were working on different
features at different times with different acceptance
criteria. There was a lot of re-work coming back into
the system. Teams were blaming each other for
incomplete functionality, failing builds, test, etc. It
was obvious that the team ended up making the classic
mistake of sticking to the technical division too long.
In hindsight, the technical split should have been
avoided when dividing the teams and should have been
structured along functional or feature lines. Dividing
along functional lines allowed user-valuable pieces of
functionality to be completed and delivered at the end
of each iteration. Eventually, the teams evolved to
work on customer functionality and features that cut
across all the technology layers in the system but this
took too long.
Once the structural issues with dividing the teams
were dealt with the communication across the sub
teams was much more productive. This gave each of
the sub teams a common overall team goal every
iteration. The teams collaborated more and paired with
each other more often, fixing many of the build issues,
broken test problems, and eliminated the finger
pointing.
Dividing the teams wasn’t a silver bullet either.
The Agile customers and testers needed to be available
and work with all teams including attending all team
meetings. In order for the limited number of Agile
customers and testers to attend all meetings, the
meetings for each of the sub teams needed to be
staggered instead of concurrent. The feature owners
helped with this issue because the feature owners were
part of the team and attended the entire meeting.
Another issue that comes up is how do you have an
effective and efficient meeting when the team is in
different locations? The team tried three different
methods for meetings: the first was video conferences,
the second was each location in a conference room,
and the third was conference calls with headsets from
individual desks.
Video conferences seemed like a great idea because
the team could see each other which allows for
359
familiarity and being able to read body language as
well as hear what the person is saying. The problem is
technical issues slow things down significantly. If you
have a 30-minute issue, which is not uncommon, and
the meeting was scheduled for one hour you have
wasted a lot of time. Day-to-day and weekly meetings
such as standups and iteration retrospectives and
planning were not efficient with the video conference.
The video conference was best for long four-plus hour
meetings such as release planning and release
retrospectives.
A conference call from a conference room phone
was the next attempt at making the daily and weekly
meetings more efficient. Each office site would get
together in a conference room and call each other.
This seemed like the next best thing to everyone in the
same room. But oddly enough technology got in the
way again. Using the best conference phone available
to the team provided terrible coverage of the room.
This caused conversations to be difficult for everyone
to follow. The main problems with conference phones
were sound quality and following etiquette of only one
person talking at a time. Poor sound quality lead to a
lot of repeating and misunderstandings. Fixing the
sound quality is possible by providing more external
microphones. However, the etiquette issue is more
difficult to solve.
If one person from each location tried saying
something at the same time you would only hear the
person in the room with you. If several people in the
same conference room are talking at the same time,
that conversation can be followed by the people in that
room but the remote location can not. Even the
slightest overlap caused others on the phone to loose
what was said. Most conference phones are designed
to eliminate background noise and if two people are
talking at the same time usually the only person heard
is the one closest to the phone’s microphone.
The solution that worked best, but still had issues
was using headsets for all team members and doing a
conference call from each member’s desk, a kind of
virtual conference room. The benefit of the virtual
conference room, with everyone on the phone, was
everyone had to follow the same etiquette of only one
person at a time talking. With everyone on the phone,
with a headset, each person had the same experience
and quickly learned to allow another person to finish a
statement before speaking themselves.
The team implemented the virtual conference room
using a teleconference facility with U.S. and
international dial-in numbers to keep the costs down.
Each meeting had an assigned dial-in number and
code. Team members all called in from their desks to
the conference number. The phones used by the team
had headsets that connect directly to the phone.
Having the meetings on the phone was less of a
disturbance to nearby teams than a physical standup
meeting would have been.
The biggest issue team members faced with
conference calls from their desks was that it takes more
discipline, like most Agile practices. There was a
temptation to read e-mail, instant messages, or browse
the Internet. The team had this problem at times, and
we simply let the team police itself. Today, the virtual
conference room idea has been adopted by many teams
across Airline Solutions and has been very successful.
4. Scrum of Scrums to Get Sub-Teams
Working as One Team
It was known that dividing the team into sub teams
would require good coordination. There was a desire
to avoid silo teams working in a vacuum. This is
where a page was taken from the Scrum methodology
playbook. One of the primary ways to scale a large
Agile team in Scrum is to create small 6-10 people
Scrum teams. Each of those teams contributes one
person who attends “Scrum of Scrum” meetings to
coordinate the work of each of the Scrum teams.
As described above, the large team was divided into
three sub teams with 8-12 developers per team. Each
team had developers and testers who were globally
distributed between Krakow and Dallas. Each team
identified a team lead (Scrum Master) who would
represent their team during Scrum of Scrum meetings.
Each sub teams conducted a short standup meeting
every morning at 8:30 a.m. in Dallas (3:30 p.m.
Krakow). Immediately after the individual team
standups, the Scrum Masters would have a Scrum of
Scrums standup meeting to discuss cross-functional
issues. This same process was also followed for
retrospective and pre-iteration planning meetings.
Pre-iteration planning meetings were used to pre-
plan each of the sub team’s next iterations. Only the
Agile customers, Agile coach, sub team technical leads
and Scrum Masters attended the pre-planning meeting.
The effectiveness of the whole iteration depended how
well this meeting went. You’re probably thinking, “Of
course well-planned things should run much smoother
than those that aren’t”. That’s true, but for a small co-
located team you can get away with and recover from
360
lacking great planning. However, for a large globally
distributed team you have to have a great pre-planning
meeting every iteration.
Prior to the pre-iteration meeting, the Agile
customer would pull the highest priority items from the
backlog that would meet the capacity of each of the
three sub teams. The Agile Customer would then build
the high-level user stories for each of the sub teams.
He would then identify each of the “cross cutting”
stories for the three sub teams. During the pre-iteration
meeting, the Scrum Masters would discuss the details
of those cross-cutting stories to ensure they understood
exactly how the teams needed to interact on that story
during the iteration. Then each of the Scrum Masters
would assign “feature owners” to each of their stories.
Feature Owners would then take their user stories and
begin to break them down into tasks and tests by
working with the Agile Customer, testers and
developers on their sub team. All of this had to be
complete before the sub teams iteration planning
meetings on Monday morning.
Every Monday morning, each of the three sub teams
would go off concurrently to conduct their own
iteration planning meeting. Teams would verify
completeness of stories from the last week’s iteration,
they would calculate the team velocity based on the
past iteration, and then conduct a team retrospective.
After the retrospective, the team would usually take a
short 15 minute break and then come back to review
and discuss the new iterations stories, the tasks break
down of those stories, and agree on how much work
could be completed in the new iteration. If this process
was successful there was a good chance the iteration
would also be a very efficient and successful iteration.
6. Conclusions
A large distributed team is not something to aspire
too. Many problems would have been completely
avoided with a small co-located team. However, if you
are in a situation where you can not avoid this,
planning for a large globally distributed team is
challenging but not impossible. We dealt with the
biggest issues that the team had to overcome while
planning. The team needed a customer who was
involved and available to the developers and testers
when they needed them. The first thing done to
address this was adding an Agile Customer who
understood the business and communicated clearly to
the testing and development teams. This Agile
Customer needs to be available to the whole team in
both time zones, and the ”feature owner” concept was
added to assist the customer in this area. An Agile
customer located in both locations would obviously be
the ideal way to handle this.

There’s nothing more defeating to a team than

attending lots of unproductive meetings. The team
found meetings with more than 7-10 people were not
efficient. Smaller teams, focused on delivering
business value with every iteration, were critical
elements of the team’s success.

Breaking the team into smaller teams means you

have to have a way to keep good communication
flowing between those sub teams. The Scrum of
Scrums idea of having a point person from each sub
team coordinating efforts across teams worked very
well.

If everyone is not co-located the team found that

meeting in a virtual conference room, which makes
everyone follow the same meeting etiquette, was the
most productive meeting format. Over engineered
communication techniques usually did not work well
for reoccurring daily or weekly meetings.

Agile teams sometimes forget how important

planning is. Hopefully, they continue to have
meaningful retrospectives that help get them back on
track. Good planning provides teams the capacity to
always be proactive instead of chasing their tails
reacting.

Planning is very important, but it is only one part of

an Agile methodology. The Agile customer role,
making meetings efficient, and scrum of scrums all
helped team members address the planning issues that
were stopping them from delivering value to their
customer. The team followed many other agile
practices that were also critical to its success.

7. Acknowledgements
We would like to acknowledge Ben Hogan, Angela
Martin, Scott Hunt, and Evie Williams for reviewing
and correcting our mistakes that no word processor
could find.

We would also like to acknowledge Adam

Szczeblewski, Divyesh Sheth, John Sigler, Madhav
Ayyagari, and Kamal Singhee for comments they made
for this paper. Your thoughts and opinions are greatly
appreciated and valued.

361