This action might not be possible to undo. Are you sure you want to continue?
All copyright and trademarks on the Material remain the property of The City of Calgary. This document and information (the "Material") is provided solely for general illustration purposes and does not create a business or professional services relationship. Laws and regulations vary by jurisdiction and change from time to time; compliance with such standards depends on the particular circumstances. Any reliance on this Material provided is solely at the user's own risk. The City is not responsible for any liability for any direct, indirect, incidental, consequential or other damages resulting from the use, misuse or misinterpretation of the Material. Before making business decisions, please consult a professional advisor.
Executing Agile in a Structured Organization: Government
Janette Scott, Robyn Johnson, & Michael McCullough The City of Calgary Janette.Scott@calgary.ca Robyn.Johnson@calgary.ca MichaelM@quadrus.com
The City of Calgary recently completed a business process reengineering of their application development process. As a result, an Agile Pilot was born. This experience report shows our Agile journey from conception, through adaptation and execution.
1. Agile Here?? No WAY!
Would Agile practices fit in The City of Calgary’s application development environment? Would our team be willing, able and authorized to make decisions? Would we be able to adapt processes? We have followed structured processes for so long, would the corporate culture be able to adjust to this much change? Calgary, the fourth largest city in Canada, has over one million citizens. This dynamic municipal government has 28 Business Units, and employs approximately 14,000 civil servants. As would be expected, our organization’s culture is similar to other government institutions and large, well established corporations. We are characterized by: Constraining policies, procedures and processes Rigorously defined roles and responsibilities Hierarchical authorization and reporting structures
Communication and collaboration are hampered when conforming to the hierarchical structure Decision making is slow due to the top down single direction flow of information Delays in approvals often result in unauthorized work Authority to make work related decisions is limited due to the hierarchical structure Team dynamics are focused on managing the team as apposed to leading it Planning for the entire project is done up front with a focus on task management Development teams (IT staff) are expected to follow a traditional waterfall application development process The complexity of our process can obscure the end goals and diminish motivation of the team’s effort Many Project Managers have 3 or more projects at varying stages of development, so they are continually switching context from one project to another The development team is made up of union and non-union employees whose roles and responsibilities are carefully defined and adhered to
The four hundred IT professionals provide technical services ranging from maintaining infrastructure to application development and support. The reality of the environment we work in is: Client engagement varies. Their participation may be challenging to get and to keep past requirements sign off
The City of Calgary IT Business Unit is in the midst of change, focusing on process and quality. A business process reengineering of the application development process resulted in the recommendation to assess the viability of Agile within our environment.
2. Why me?
We were so excited to Pilot, and then panic hit! culture? Could we find a differently with us? Could focused team? Did we be selected for the Agile Would this work in our client who would work we really get a “single” really understand what
management was asking for? Did management truly understand what they were asking for? Could there be flexibility in the processes? What did we know about Agile? Investigation into Agile led us to believe that our clients and IT could benefit in several ways, but the confidence from initial research was wavering. Based on our culture it might be too much. The impact to our careers could be positive or negative! We felt there was no middle ground. What had we gotten ourselves into and where would we start?
IT staff was curious. Some had past experience and wished to share, others had questions and concerns. There was primarily skepticism and doubt about Agile. Our culture is apprehensive and resistant to change and this was viewed as big change. We heard the full spectrum of comments ranging from: A large corporation in Calgary had tried and backed out of Agile, so surely it would fail here too Some Agile components such as Test Driven Development (TDD) are really hard and would be impossible to incorporate here We didn’t know enough about Agile and TDD to make it work Agile is only a development practice and applies to design, develop, test phases, but it is not relevant to project management Agile will never fit with our current processes I hope this doesn’t work TDD promotes high quality (the only word of encouragement we received)
3. This will never work
We settled down and focused on getting everything to launch the Agile Pilot in place; we needed a mentor, a workspace, a client, a project, and a team. We were now in our comfort zone of project planning. In order to try Agile and make any sort of recommendation on viability at The City of Calgary we needed to try it with a team of certain size and makeup. There were several meetings with Managers to present our needs and goals for resources. Client selection was based on several factors; support of the general concept and objective of the Agile Pilot, with a clear understanding and agreement to a 100% daily availability. While the client as a team member would be knowledgeable of business requirements, they would also be empowered and authorized to make decisions. The project had to have the potential to demonstrate a solid set of Agile components within the constraint of six months. The team, now considered to include client and IT staff, was brought together for the first time in a training class to meet and familiarize everyone with the approach, tools, and techniques of Agile development. More importantly, it was an opportunity to allow for team building. There was a mixture of excitement and confusion; excitement to be part of the team and confusion over roles and responsibilities. Developers came from different sections within Application Development and Application Support. Many didn’t even know each other. One class exercise that particularly resonated with the team was the development of a list of team values. Rather than simply listing values, the team was challenged to describe how they would demonstrate these values to one another. Physical space was an enormous problem, but we were able to get a temporary location for the team to be co-located, including a dedicated meeting room.
We realized that we needed to communicate! Communicate! Communicate! A great deal of effort was spent with IT staff that would be directly or indirectly impacted by the Agile Pilot. Communication included the goals of the Agile Pilot, an overview of Agile, and the candidate Agile process. We knew at this point we were not going to change anyone’s mind, so our goal was to dispel fear and rumors. Service groups, including Enterprise Application Integration, Database and Server Administration, required additional working sessions. This was necessary since these groups were used to scheduling services for projects that were developing systems using a traditional approach. It was important to establish a potential working relationship with these groups since development and deployment of each release depended upon it. Additionally, the Agile Pilot needed to help them understand how they could be structured to help future projects adopting Agile.
4. This is really going to happen
Somehow everything fell into place; we had a mentor, a willing client, a project, a team, and a really great workspace. The team had been sent on training and was mostly familiar with the terminology and approach for what we were going to do. The gap between theory and practice had to be filled with the confidence that only experience can bring.
The greatest uncertainty for the development team was over TDD. Could they make it work? Would everyone adopt it? The project kickoff was the first opportunity for the team to discuss the actual product they were going to build together. The meeting itself was not remarkable except that it was the first opportunity to discuss the working relationship we would have. Our culture had been one of command and control, so the development team would typically expect that they would be largely spectators to the kick off. In fact, most of the requirements, analysis and planning would have been completed before they were involved. Additionally, given our hierarchical nature, most of this information would have been disseminated in a top down fashion. The development team sensed they were in for something different since they were being involved so early. They were not used to this sort of early and open engagement with their client and so they were quiet and reserved in the initial stages of the project. As time went on the team became vocal and active participants in project discussions. They then became vocal within the organization and people started to recognize that something very unique was going on with this team. We set a goal at the kickoff to complete our envisioning activities and to start development within a week and a half. This meant we needed a product backlog, development tools and environments set to go in 8 working days. This goal was a bit of a shock to some at first but we accomplished it with little trouble. We started the following day with a user story workshop. The team approached it with hesitant optimism, believing they could do it, just not knowing how. This was a clumsy and awkward affair at first. The team was still used to reading requirements; they had not been actively engaged in an activity like this before. The team had just been formed, so the lines of communication and trust were just being established. The mentor had each person write at least one user story in that first meeting to ensure that everyone understood they need not be spectators. To add to the awkwardness of this first meeting, several external service group representatives attended as they were expecting the typical project requirements kickoff. We then had a large and confused audience for the team as they tried to write their first set of user stories. For the client, there was uncertainty over how to communicate their requirements. How do we tell them what we need without writing down all the details? This uncertainty was tempered with the persistence and
influence of the mentor who assured all that we were on the right track. By the second meeting, the team had gotten past the initial hesitation and was easily and naturally creating user stories. They had developed sufficient user stories to talk about iteration and release planning. Estimation, prioritization and planning followed. The mentor was with us full time through the envisioning period helping to run the meetings and get things moving on the project. He promised to step out of the way as soon as we could walk. The introduction of Agile relied on the mentorship of the team and team leadership. This intense mentoring continued through the first three iterations.
5. The world didn’t end
The first iteration started and things got off to a reasonable start. This was the first time for daily stand up meetings, managing tasks and applying TDD. Nothing really broke, but then again we were not blazing through developing our user stories either. In fact, as the first few iterations came to a close, our pace was looking glacial. Iteration planning meetings and task decomposition was slow and laborious at first. Historically the team had not been involved in these activities. Typically the tasks and assignments would be determined by someone else. This was a real cultural adjustment for the developers. It was through this period that the team and the client had to hold on to their belief that this could work. Our mentor assured us that this was perfectly normal but it was hard to see that at this point. As we looked at the list of user stories that still needed to be built it seemed like we were facing a monster. The major accomplishment through this period was the application of TDD. The team worked in pairs making a conscious effort to work in this new way. By the end of the third iteration everyone saw the value and believed they were as fast at developing with TDD as without. In the end, this was a period characterized by a small degree of confidence gained. The team had proven they could work this way. Whether it was the best way to work had not yet been clearly shown.
6. This might work
On the fourth iteration things started to turn around. Based on prior progress, the team had chosen a small number of user stories. They had these completed
in just half the iteration. Now they were going back to the client looking for more. Not only was the team’s productivity improving but the process itself was evolving. The team was actively reviewing the way they worked at the iteration review and implementing changes that made a real impact. Changes included improvements to the wall where tasks were being tracked, creating testable tasks and managing project risks on the wall with the release and iteration. This was a striking change to the culture where the way people worked and their roles had been rigorously defined. The team was stepping out of this mold and challenging their expectations of themselves and their work processes. The mentor started to step back at this point to allow the team the opportunity to gain confidence. As our mentor explained, if he did not step back and allow the team to start solving their own problems, they would develop an unhealthy reliance on him. The goal was to have the team realize that they could make the decisions themselves. The team at this point was running on its own. Not without feedback though, as the mentor was still available and in touch but at arm’s length. The mood had changed. The process was not only okay but actually worked. Work was being accomplished and the quality of the product was very high.
Progress reporting was openly and visually displayed whereas before reporting was summarized and sometimes unclear
In the end our clients reflected on the experience and stated this showed that: “IT really wants to listen!” The development team also observed the dramatic cultural change that Agile presented. Before the Agile Pilot, projects were run with a top down command and control approach. Developers were assigned their work and were generally not in regular communication with the client. The team’s observations of the Agile Pilot were: They had control over what they did and how they did it whereas before tasks were created and assigned by someone else They knew their client through face to face conversations whereas before they didn’t feel as comfortable speaking with them Given control of their work they were better able to focus on their tasks whereas before there was less of a sense of control when work was assigned There was a stronger focus on quality and testing with the application of Test Driven Development and a Test Analyst on the team whereas before testing and quality was inconsistent
7. It really works
We continually checked in with the team. We wanted to know their thoughts on working with Agile. Everyone was eager and thrilled about the new work environment. A noticeable cultural difference was observed. Historically there was a sense of distance from the project from the client’s perspective. Our highly structured culture had created barriers to visibility, transparency and communication with our clients. The client’s observations of the Agile Pilot were: They felt they owned the product and were part of the team whereas before they were an observer They saw working software early and often whereas before they did not see the product until much later in development They felt they had the ability to direct the project plan whereas before the Project Manager made all the planning decisions
We continually heard from the developers: “I want to keep working this way”. At time of writing there is sense of disbelief among them that they will be able to. They recognize the serious challenge that Agile presents to the predominant culture here. The Senior Management at The City of Calgary has continually demonstrated their support, openness and willingness to change throughout the Agile Pilot, so we are confident that the team will be able to continue working with Agile.
8. Everyone is excited!
There was keen interest on how the project was going. The buzz that Agile could really work at The City of Calgary was out there. Individuals wanted to know what was making this such a fun project; the team’s enthusiasm was spreading! The excitement and enthusiasm of the Agile Pilot team was noticed throughout all of IT!
9. Where do we go from here?
Today the culture at The City of Calgary is still command and control, structured and hierarchical. However, the culture and dynamics on this team has been noticed. Many individuals and groups are now starting to consider the impact of Agile. There will be many challenges to overcome. We have only scratched the surface of change. How do we facilitate change within the client community? Due to the Agile Pilot, we now have a respected champion! We need to draw on this to promote Agile. Self-organizing teams will be a challenge for some Project Managers and team members, each having to let go of some apprehension and uncertainty. Roles and responsibilities will change. The roles and responsibilities for team members, particularly for the Project Manager and Technical Lead, need clarification. The Technical Lead was a new role within the organization and some of the responsibilities were traditionally the Project Manager’s. This led to some confusion on who should take ownership of certain activities. For instance, there was an issue with the quality of testing and visibility into progress on testing. The Technical Lead and Project Manager were both unsure of who should be addressing this. Functional testing was executed within the iteration and the Test Analyst was located with the team. The test effort, however, was largely a black box to the development team and leadership. There was inadequate visibility into what was being tested and how. This led to uncertainty on the progress of testing and quality of the product. Test Analyst responsibilities will be defined by another initiative underway: the creation of a focused test centre. The reporting and tracking of progress for the Agile Pilot was highly effective for the development team and client. The visibility into status and progress for Senior Managers, however, was not adequately addressed. Management reporting will need to be defined within the broader rollout for Agile at The City of Calgary. The Program Management Office (PMO) was developing project management processes and best practices while the Agile Pilot was running. Initially the PMO did not think there was project management in Agile. Over time, they recognized the need to engage with the team to create convergent processes and approaches to managing Agile projects.
The Agile Pilot was successful in demonstrating how Agile can work at The City of Calgary. It has proven that a structured organization can incorporate Agile, and in addition, IT has a higher degree of confidence in Agile. Executive sponsorship and endorsement of the Agile Pilot was critical to the project’s success. The Management team demonstrated support, encouragement, and openness to change. Service groups were very willing to collaborate and adapt. We now need to implement Agile as an accepted and valued practice at The City of Calgary.