In order to be truly successful in Agile, you must have a feedback loop that enables you to validate whether or not the software you are building is meeting the mark of the end consumer. The most ideal situation is getting access to the actual end consumer of the software. The less translation layers the better!

If you can’t get access to the end the consumer, you have to rely on the next best thing – a very capable product owner who can accurately convey the needs of the end customer. Without this, you run the danger of writing software that can change rapidly based on the product owner’s direction but ends up missing the mark of the end consumer anyways.

Even if you have a good proxy, you have to be able to release software frequently enough so that the end consumer gets their hands on it. No matter how great your proxy is, as we all know, communication in verbal or written format never replaces working software and hands-on feedback.

So… what happens if you have to integrate your agile work with systems that don’t follow the same release cadence or the same methodology? What happens if your current system isn’t modularized enough to enable loose coupling? What happens if you don’t have the ability to deliver production quality agile driven software independently of the systems you are integrated with?

Unfortunately for us, all of the items just listed haven’t been solved yet. We are trying to come up with solutions as fast as possible but in our case, we ran out of time and were forced to adhere to a joint agile and waterfall release cycle. We shoot for two agile releases within a single waterfall release. The last embedded agile release has to stop three months prior to the waterfall release in order to allow for sufficient time to test the agile and waterfall together.

We have not lost sight of reaching a point where our agile software is mature enough to release into production independent of our waterfall code. Unfortunately, it’s pointless for us to fight this specific battle until we actually solve the issues that would enable us to deploy into production after a release even if we weren’t embedded within a larger waterfall release cycle.

Before we fight the battle of being able to release agile software into production without the waterfall timeline constraints, we must

• Modularize – we can’t be constrained by the systems we integrate with. Being tied to a downstream waterfall system on a completely different product life cycle forces us to comply with the weakest link
• Dedicated Teams – you must leverage team members that are dedicated to the effort vs. team members that can quickly be pulled off for other efforts. More often than note, this tends to be QA resources because in a waterfall world, resource planning occurs really far in advance. Switching to agile is often driven by the development organization so resource dedication generally gets solved in development first.
• Environments – we must have the flexibility to test our software rapidly and repeatedly. This includes the ability to test integrations at least partially through stubbed out interfaces and effective test harnesses. With a lack of mature deployment and configuration process, you suffer long environment ramp-up cycles that inhibit your ability to effectively reach done, done, done
• Quality – the entire agile release train must embrace the concept of done, done, done. Our goal is working software – that includes development, unit and functional testing automation, documentation and product owner acceptance. Not reaching this level of quality forces you to carry technical debt into subsequent release cycles
• Team Commitment – everyone is responsible for the items I just mentioned. Not just a few developers or the scrum master or a tester. Everyone must stand behind these important concepts. If you don’t have access to QA resources or documentation resources, don’t sacrifice quality. Sacrifice scope in order to force the right difficult discussion on dedicated resources vs. being hampered by a side effect conversation on quality. Most importantly, the leaders sponsoring change must believe in the same team commitments. This leads me to my final point
• Leadership who Understands the Big Picture – If you think switching to agile initially is hard, realize that staying agile is 10x harder. After the newness of a new process wears off, people start to revert back to their original safety nets. Only certain leaders embrace the ever changing industry and are therefore willing to stay abreast of recent trends. These types of leaders reach out to the industry for answers because they embrace change and they embrace the fact that they don’t know everything. Other leaders will start to make process changes based on what they previously knew. Recognizing you don’t know what you don’t know is critical trait of a leader embracing agile

The morale of this story is three fold…

1. If you strive to be agile, beware of all the complications that will prevent you from becoming agile
2. If you falter, choose your battles wisely – there is always a time and it’s best to lose a battle vs. losing the war
3. Transformation takes time – especially at scale. Recognize that upfront or you will end up frustrated each and every day you go to work

In order to be truly successful in Agile, you must give each team the autonomy, empowerment and desire to own the work they commit to. Agile is about making small teams effective and you make agile scale by enabling many small teams in parallel. Waterfall leverages a different technique for scale – one based on structured hand-offs were more often than not, the skilled architects lead the design and planning aspects of development.  This creates an ownership and accountability issue because the teams planning the work don’t own the implementation of the work. When you are able to convert an entire organization from waterfall to agile, the transformation is easier because everyone is involved in the change and you can address the ownership and process issues at the same time. 

But what if you can’t convert the entire organization to agile? Perhaps part of the overall system is mainframe based. Perhaps you have a culture that is predominantly not accepting of agile. Perhaps you don’t have the executive support to do a full switch. There are many reasons why an entire organization can’t convert to agile all at once.

For this post, I am going to declare a few key assumptions:

• For whatever reason, the entire organization can NOT convert to agile – you are stuck with a waterfall process and an agile process
• The work the agile team is doing IS important and is NOT a prototype or side project. The work must be integrated into the main system and it must be production quality

First and foremost, no mater how hard you try, waterfall and agile don’t mix. It’s not the methodology itself but the life cycle that’s the issue. If you had a “waterfall” process that went from start to finish in 4 weeks, you would be able to manage it with an agile process based on 4 or even 2 week cadences. It’s the life cycle that causes the issue – not the “methodology” itself.

Now for the SOA part. Service Oriented Architecture is the latest buzz word in the software industry. The reality is that good architects have been designing software like this for 10s of years. The difference now is that focus on a well defined XML based contracts that if done right are truly agnostic. A well implemented SOA based system in theory supports interchangeable implementations for the same interface contract. SOA enables loose coupled systems that actually work well together.

What does this have to do with waterfall and agile? Everything – because it’s actually the solution to two major obstacles people face when converting to agile. Here’s how SOA based thinking can be leveraged to make the agile transition much easier;

• You can decouple yourself from complex integrations by managing to a well defined contract. This gives you the ability to stub out functionality until you have the opportunity to integrate with the external system. This is important because the external system may follow a different life cycle
• Teams are given full ownership of the “black box” behind each component. They must abide by the contract of the interface but how they implement the work behind the interface is completely in their control
• You can scale agile more effectively by defining teams for each component. You can leverage the contract definitions to manage the dependencies and relationships between the teams
• Finally, you have the ability to mitigate life cycle discrepancies between methodologies by using the interface itself as the mechanism to allow different methodologies. The process each team uses to develop within their black box is irrelevant to the other teams. 

The last point is key. If you try to leverage two processes within a single component boundary or for two or more components that are tightly integrated, you will fail. In order to implement agile effectively, you must have the ability to remain loosely coupled from components that leverage other processes such as waterfall. If you don’t, you will fight every obstacle that makes the processes distinctly different including

• Life cycle discrepancies
• Resource management differences including availability and skill set
• Environment availability 
• Inability to have a feedback loop due to lack of availability of edge systems
• Politics related to design reviews, design approvals and even statistical measurement such as the number of design changes within a release (remember – in waterfall, a change means a bad design more often than not)

Where I work, the conversion to agility started with the acquisition of a small start-up software company. In our case, the acquiring company was a case book study on waterfall – numerous hand-offs with each hand-off generally crossing organizational boundaries. One group made a business request. Another group wrote requirements void of any hint of solution. Another group created the functional design for the solution and yet another group created the technical design. The technical design included pseudo code for all logic including DB queries. A DB group converted the word version queries to real DB queries and a separate development group coded the C++ logic. You think this is bad? It gets worse… Same approach for testing (test planners, test automators and test executers), documentation & training (planners, writers, training planners, trainers), etc…

It was a nightmare but for some reason, I never gave up. I continued to push the two teams I was responsible for and who were using agile to show the rest of the company that we were special. We continued to prove ourselves and over the course of two years, we were the only group that really turned heads. There were a lot of reasons for this and to be fair, they weren’t all a result of our performance. For example;

• We were developing the “new” thing – what everyone was interested in and cared about
• We didn’t have to conform to existing functionality and backwards compatibility constraints (at least not originally)
• We were considered innovators and so upper management was excited – plus they just spent a lot of money to purchase our small little company
• We were experienced in the new technologies – in our case web services, .NET, etc.
• We had the skills – we had already gone through the evolution of growing a larger number of architects vs. being bottled necked by an elite analysis team. We didn’t need one or two skilled architects telling everyone what to do. We didn’t need specialized DB administrators – everyone had the skills and we mentored people well

As you progress down the path of truly becoming agile, you will pass several tipping points that validate you are succeeding. At the time, they may not appear like tipping points but in hindsight, you will always see them. For example

• Let’s Try It – The point where you get enough executive support to try agile at a smaller scale
• Let’s Scale It – Enough momentum exists to force a larger effort. In our case, we jumped to 10 teams
• The Build’s not Broken – A monumental achievement when builds are no longer the most discussed issue in a large scale agile effort
• Visibility Past Your Nose – The roadmap exists beyond one release in a sufficiently accurate manner to communicate it clearly to everyone
• Fist of 4+ – The point where the large majority of your iteration teams commit with confidence
• And the list goes on…

When you go through the process of converting an existing non agile development process into using Agility, you have to realize that culturally, folks aren’t going to be used to the visibility agility provides. In fact, it might even frighten folks. Hence the title of this article – Zero to Fear in 60 Seconds.

As part of rolling out agility, we made it very clear that accountability and visibility were top priority. We stressed the importance of empowerment at the iteration team level. We raved about the ability for the team to truly own their work beginning to end. At the same time, we reminded folks that in order to be empowered, they had to be accountable.

We kicked off our first agile release with roughly 10 teams all executing in parallel within 2 week iterations. We held our first 2 day release planning session to establish our release commitment. We then ran our iterations using iteration planning days and iteration demo days – the basic release planning and iteration planning approach that Dean Leffingwell and others describe so well.

Everything was working great – or at least it seemed. Everyone would come to the daily scrum of scrum and communicate their status appropriately – pretty much always green with a few yellow’s from time to time. It was a smooth start – lots of excitement and lots of empowered folks.

Unfortunately, it was a bit of a facade. In hindsight, what really happened is that folks weren’t used to speaking up. They weren’t used to the visibility. They were scared to say things weren’t going well. Especially when every other team was doing so well. What we experienced was a fear of accountability so strong that it actually prevented our ability to truly show visibility. Even though we enabled everyone to be empowered and we had multiple mechanisms in place to drive visibility, we actually lived in a dream world until our hardening iterations came along.

Now – you are probably asking yourself how could this be? How could everyone have missed the signs? Why didn’t Product Owners or Scrum Masters speak up and state the obvious? It’s not that simple – it was our first agile release of that magnitude. With the exception of 2 teams, the remaining teams had little agile experience. In addition, not a single team had run in 2 week iterations before. We were all new to this format.

So… the morale of the story is that it takes time to figure out visibility. You can’t assume that a new agile process will bring about visibility and all of a sudden eliminate the struggles non agile processes have. For us, we are just now reaching a point where visibility is truly king. Where people know how to communicate and express the the true state of their team. Where our daily scrum of scrums really drive out cross team coordination, dependencies and impediments. We got there by coaching our scrum masters and product owners. We enhanced our ability by structuring our release planning and iteration planning in a manner that fed the updates within our daily scrum of scrums. Here are some basic tips that really helped us

• Release Planning – Make it clear to everyone what epics & stories have cross team interest. For example, Team A is committing to a story that Team B and C are dependent on. Everyone should understand this and the person running the Scrum of Scrums should remind folks regularly
• Iteration Planning – In addition to stating iteration commitments, ask teams to restate their commitment to the release commitments. For us, we use Epics as the “release commitment” and stories that comprise the epic as the “iteration commitment”. This gives us the ability to track release commitments at a higher level vs. having to track every single story. It also gives product owners autonomy to adjust stories post release planning without the fear of missing a release commitment. It’s a bit nebulous – you have to have faith in your product owners to protect the spirit of the commitment
• The reiteration of whether or not we are committed to our release planning commitments occurs numerous times through the release – once per week. Each iteration planning update as stated above. In addition, we also do a commitment checkpoint at each iteration mid point. The daily scrum of scrum that occurs during the middle of each iteration is actually run a bit different. Scrum masters provide a feeling on the their team’s projection for the end of iteration and the release. 
• Mentor scrum masters and product owners to provide valuable updates during each daily scrum of scrum. Use feedback often to ensure your scrum of scrum is viewed as valuable. People shouldn’t see it as just another meeting – if your updates or attendance is poor, question your format and not your people. Meetings should be interesting and debate and discussion should exist
• Leverage a few star scrum masters and product owners to set the standard. Call on them first during your release planning, iteration planning and scrum of scrums to provide updates. If you pick someone who doesn’t provide good updates, you may cause a domino effect where everyone follows with a poor update
• Provide a status update that goes out to everyone – not just executives. Every member of the entire agile release train should have access to the same status report that goes out to executives. We do a 2 week roll-up status report that occurs after each iteration planning event. This also includes an update of our release plan of record. No one from the team provides a status report – they provide enough updates in their daily scrum of scrums, commitment meetings, etc. The individual that runs our daily scrum of scrums for all 10 teams does it solo
• Provide metrics on Done, Done, Done results of user stories – however, do so for their growth and not for external purposes. Missing a user story in an iteration is bad but it’s not the end of the world. In fact, it may be the best thing that happened since slice bread if the team learns from their mistake. The metrics give them the ability to improve. Hold them accountable through their retrospectives and let them know that you expect improvement. If it continues to be a problem, then address it properly. 

The requests and increased pressure from clients to move from producing a cabernet sauvignon that takes several years into making a sauvignon blanc that can be turned out quickly are coming from all angles.  The promises of the new process cause a lot of excitement and buzz.  Short term successes start to build the momentum and the agile train is growing with passengers.  The problem, there will be bumps.  All of a sudden, you find those from the previous process are quick to request expectations and depth comparable to a cabinet sauvignon being produced in the time of a sauvignon blanc.  This isn’t to say you can’t produce a complex sauvignon blanc, but it takes practice – so don’t expect this right away.

The challenges associated with this become increasingly difficult when you enter a world where both processes have to co-exist in a single development organization (Waterfall and Agile).  Some are working on the new agile methodology while others continue to work on the way things used to be.  It is divided by teams and by applications, which causes a natural chasm.  However, these teams are expected to integrate and blend when it is all said and done.  When the heat is turned on, those that were on the fence or just wrapped up in the excitement of agile quickly revert back to wanting all that was waterfall… thorough design documentation, pseudo code, and painstaking approval processes.

This is not the recipe for a good blend.

You all of a sudden find yourself in a situation where the agilest are making compromises to the new process, scrutiny from the highest levels is increased, and those that were on board are starting to wonder which side the company is going to end up supporting.

How do you keep people from going sour?  How do you ensure that they keep the faith in agile?

We have been faced with those questions at our company. In business for over 20 years, we have served our telecom and cable service based clients well, as they successfully support over 40,000,000 subscribers on our systems. However, as happens with all business, the competitive and technological landscape is calling for a new way of doing business. In order to meet the system demands, we introduced the agile methodology. Due to the fact that not all resources could be converted at once and the trepidation around a new process, only a portion of the core development team was converted and the balance of the organization remains in a waterfall methodology. The excitement level, both internally and from our clients, was high when we first rolled out the process. As time has marched on, challenges have arisen up and support has waned.

Just like any process, things will go wrong.  Lessons are to be learned.  Unanticipated challenges crop out of nowhere.  Above all this, there are those that are trying to ruin the crop before it is even harvested in the first place.  Whether out of fear or comfort, the want the old to continue to succeed and the new to never come off the vines.

There are three key ingredients that set you up for a chance at success:  Communication, Executive Sponsorship, and Agile leadership with conviction.

The first answer to most challenges starts with communication.  Up front and honest communication that outlines that this is new, this is unchartered territory and there will be sour grapes in the bunch.  It is a constant battle to ensure those on the fence that the process can make a difference and that we will get there.

This is where the next two ingredients come in.  With executive sponsorship it makes it possible to knock down walls that previously stopped progress.  The fear people have with a new process is somehow alleviated by just saying “Executive so and so” is behind this initiative and it is critical for the company’s success.  Keep in mind, that knocks down the wall initially, but then as time goes on those that have their doubts will retreat, and likely go to the executives with their complaints.  It is at that time you need to see the true strength in executive management when they stand by their decision.

Finally, conviction in the leader of the agile process has to be there, and that conviction has to trickle its way through the agile side of the process and ultimately to those that are on the fence.

There is a saying that says to “keep your friends close and your enemies closer”.  In an environment that is introducing agile into a blended world, I don’t think anything could be more true.  While it is important to ensure your own house is in order, it is doubly important to make sure the teams around you that are watching with skepticism are included in what is going on and the pressure is reapplied to what they are responsible for with regards to making the overall company successful.  The agile leader(s) with conviction and confidence will help navigate these waters.

Product Management is a good place to continuously use as a litmus test.  If they continue to support what is going on, then you continue to have a fighting chance.  Keep them in the loop, make sure they are constantly reminded of why agile is good and the benefits associated with the methodology.  They are there, and the company can get there – you just have to be sure to recover from the bumps.

It is not an easy formula, but none of the good blends ever are.

Agile relies on team accountability and team empowerment to deliver consistently amazing results. High performers are attracted to this type of environment because there is no place to hide. If you can’t perform at the level expected by your team, you’ll be answering to the entire team vs. a single manager.

There is no place to hide. Nowhere. The average performer will feel their skin burning under the bright spot light. It won’t just be the team that notices – even the individual will quickly realize they aren’t meeting the team’s expectations. 

In a healthy environment, the average performer will be quickly addressed. Perhaps via a new role that leverages their skills more effectively. Perhaps they will self select out. Perhaps they will be coached out of the organization. While this may seem harsh, it’s the best decision for the team, the individual and the company. It’s expected and it has to happen fast. Because there is no place to hide. Especially when you run in short iterations and the criteria for success is clearly defined within the user stories committed to by the team. The average performer is accountable to the team and it’s so much tougher than being accountable to one manager. As a result, high performers know they will be working with other high performers who will provide mentoring, enjoyment and a great sense of team accomplishment. A players attract A players. It’s that simple.

But what if the culture you live in doesn’t believe in high performance? What if it’s customary to give average performers second, third and even fourth chances? In the agile world, you are dead. The key thing gained by agility is compromised – team accountability and team empowerment. If the team is lucky, the average performer will leave on their on accord. Even if the team is that lucky, they will lose faith in management’s ability to eliminate impediments. The ability to truly leverage the utmost power of agility may be lost forever.

The problem becomes significantly worse at scale. What if six teams are highly effective but one isn’t? What if the scope committed in a release requires all seven teams to be highly effective? All of a sudden, the overall agile release train suffers because they are only as strong as the weakest link. What if executive sponsors are still tentative about agile? Will you let a single team of 6-8 people destroy the ability for 40+, 100+ to become truly agile?

While there are many reasons why agility may not work, I personally believe no other killer is more powerful than ignoring the spotlight agile shines so well. If you truly want to experience the power of agile, you must heed the advice of the spotlight and you must do it quickly. Coupled with a solid retrospective strategy, heeding the advice of the spotlight will send you down the path of success faster than any other advice I can offer.

Agility leverages the power of team accountability and team empowerment. Why even try if you aren’t willing to use it fully???

For now, let’s focus on the product owner. Until we tried to scale agility beyond 1-2 teams, I never truly realized how powerful the product owner role was. At scale, the product owner role becomes pivotal to the success of the overall effort. No other role has the ability to clearly articulate the vision of where each team needs to go. No other role enables each iteration team to be truly autonomous while also ensuring the vision is preserved.

In addition, an interesting dynamic occurs when a team is fortunate enough to have an effective product owner. Even though the product owner represents the traditional product management organization (and may even be part of the PM organization), they really are a member of the iteration team.

Sounds pretty simple but don’t underestimate the behavioral transformation the overall organization goes through. Traditionally PM asks for 10 items and development says they can only deliver 5 items. This constant tension generally leads to a clear organizational boundary.

That changes with an effective product owner because a) they become aware of what the iteration team can really deliver, b) they develop a strong and powerful bond with the team and c) they commit and operate as part of the team and not as separate requester.

The tension that previously existed disappears. Even if it doesn’t disappear 100%, it shifts from development vs. product management to product management vs. product management. Sounds odd and to some extent simple. It’s a very powerful transformation that ultimately leads to better planning and much, much less frustration. Everyone becomes more knowledgeable and in doing so, more of a family.

Conversely, not having an effective product owner will destroy the ability for an iteration team to be effective faster than any other individual or impediment. We’ll cover some of these issues in future posts.

So what does diversification have to do with Agile? Well… in terms of release planning and vision at an enterprise scale, it’s everything! The standard approach the industry has traditionally used for software development is to define a vision and road map that in turn is used to accurately plan the development, testing, documentation, deployment, etc. of a new set of functional features.

This “rigid” plan is then locked and execution starts. Any shift in execution is viewed as a bad thing which in turn leads to the “next” plan being adjusted because the current plan obviously can’t change – it’s locked! In the agile world, you still have something similar – it’s just in the form of iterations and shorter duration releases. While you embrace change, you still want to define a solid plan that is just long enough to avoid constant shift distractions.

The power of diversification in agile comes from a thinking about planning a few different ways

• Minimize the duration of the release to the least time possible – this enables you to still lock the release in order to avoid distractions and constant shifts while still enabling time to market adjustments
• Define your road map in a manner that enables flexibility without giving up visibility. You don’t want to error on the side of not defining a road mapbecause you still need to be able to communicate commitments that extend beyond a single agile release to your sales force and clients. You can accomplish this by adjusting the number of external commitments (we call them MUST epics) based on the maturity or evolution of the plan of intent. The further out the POI, the less you commit to.
• Define your client commitments in terms of Epics. An epic is a larger grained story that is subsequently broken down into smaller grained stories. If you have capable product owners, you can communicate the spirit of the “epic” to your clients while still giving the team the autonomy to adjust via the proper definition of stories during the actual release execution
• Embrace change – let product managers shift the contents of a POI. The key guideline is that the closer the POI is to being converted to a true Plan of Record (POR) as part of the release planning process, the less it should change. This is a guideline that will improve your velocity and release planning efforts. It is not a hard rule.

So… why on earth would a product management group that is used to 1 to 2 year road maps ever agree to something like above? Well they won’t initially. However, if you involve them in the agile process and some of them actually dedicate a good portion of their time to successfully enabling the product owners of the iteration team, you will quickly see an almost reverse effect. Very quickly, they will recognize that the value of being able to adjust to the unknown is desirable and powerful. This combined with the ability to lock in a small subset of important features vs. the entire release is exactly what they needed. It’s just not a natural thing to do until you experience it first hand.

OK – diversification as a topic may have been a stretch. It’s just something I thought about as I was playing around with stocks and watching Jim Cramer’s Mad Money. Hopefully they can give you something to think about in the agile world!

• What it takes to actually develop working software
• The quality of the people involved in the development process
• The issues and impediments that arise from execution