The End of the Waterfall as We Know It

Traditional waterfall methods remain the leading delivery vehicle for organizations, despite inconsistent delivery results. This research helps application leaders uncover the advantages of agile, iterative or incremental project delivery.

Key Challenges

• Traditional waterfall practices result in inconsistent delivery results.
• The longer the project's duration, the more requirements change and results vary.
• Waterfall projects carry significant risk of change through to the testing phases of the project.

Recommendations

Application leaders:

• Transition from traditional waterfall methods to agile, iterative or incremental delivery methods where possible.
• Constrain projects where waterfall must be used to less than 90 days.
• Put in place stringent risk management and project review processes if duration constraints are not possible.

Introduction

Waterfall development processes remain the most commonly used methods in client organizations. According to Gartner's IT Key Metrics Data, waterfall methods were employed on 56% of development efforts in 2015, with iterative methods used in 21% and agile in 23%. These numbers vary slightly over the past three years, and actually show a bump in 2015 for waterfall compared to the previous two years. (There is, of course, selection bias here, since only certain organizations do formal benchmarks. Via inquiries, conference one-on-ones and client visits, we believe that agile – including lean and Kanban – may be under-represented here.) The same data shows that 39% of projects have a duration of less than three months, and that 20% of projects exceed a year in duration. It's no wonder to us, therefore, that according to the same research, the success of delivery was somewhat less than impressive. Only 60% of projects were completed on time, and only 71% finished on budget. The average schedule variance was 23%, and the average budget variance was approximately 16%.

At first glance, this may not seem particularly dispiriting (although for some it is). However, nine-month planned projects that actually take 11 to complete can be more disruptive to an organization's resource and demand management processes than might be expected. In addition, the numbers above are averages. They don't reflect the long-tail outliers, which can have immediate and dire consequences for businesses, and quite often for application leaders and CIOs (as in, employment-related dire consequences).

Quite simply, waterfall methods, when used in the traditional, project-based manner, are inconsistent and risky. Since there are other choices available that have the potential to be more consistent and less risky, it's time to begin the transition to these methods.

Analysis

Understand the Basic Method Types

Gartner divides software development methods into four broad categories:

• Waterfall methods take a set of (allegedly) clear and stable requirements, project how much the project will cost and, by using a sequential work breakdown structure (WBS), decide how long the project will take and how many resources it will require.
• Incremental waterfall methods also address a set of clear and stable requirements, but the work is broken into a set of fixed-time, fixed-resource and fixed-cost waterfalls that are repeated with discipline concurrently or purely sequentially, with no discipline overlap. Each increment includes design, construction and testing activities. If the project is large enough, there may be multiple teams all working in an incremental method, with cross-team dependencies planned for and mapped.
  
  Organizing work in this manner produces more-transparent progress than waterfall because the completeness of the software from each increment can be assessed. Testing is performed throughout the software development life cycle (SDLC) instead of being pushed to the end, thereby reducing risk and improving quality. This means that a project scoped to nine months would be delivered using three-month increments (or another variant, and since each increment may not stand alone, there may be work necessary to complete software created in one increment later in the project).
• Iterative methods build on incremental waterfall by adding an explicit opportunity for feedback after each increment or iteration. This allows for the early detection and resolution of problems in the requirements. Done correctly, the iterative approach assumes that requirements cannot be frozen before construction begins. Known requirements are realized and implemented using short cycles of analysis design and implementation, enabling the system to evolve. This opportunity for feedback allows some Mode 2 projects to use these methods, as long as the iteration is short enough. Testing is performed throughout the SDLC instead of being pushed to the end, thereby reducing risk and improving quality. True iterative development is done on a regular release basis (that is, every x weeks or months).
• Agile methods are fixed-time, fixed-resource methods that plan delivery based on function decomposition rather than time sequence (WBS). Agile projects do not require a clear set of requirements at the beginning, as the feedback from the customer is used to determine the best solution during the project. Because of the need for constant feedback, the timeboxes here are shorter than for iterative (usually two to four weeks). Such rapid feedback loops allow this functionality to easily handle Mode 2 projects, where the solution is not known upfront. This class of methods may also include lean/Kanban approaches, which don't focus on iterations but do share the same planning and cultural needs.

Traditional waterfall and incremental methods are primarily project delivery models in which a project is a collection of business capabilities. Iterative or agile methods can be used in project mode, but are designed as product delivery methods, where a certain product (application) is released to production on a regularly scheduled basis.

The Impact of Mode on Method

Many clients have begun to take Gartner's advice to explicitly segment their work. This matters a lot when it comes to choosing a method.

Mode 1 describes the more traditional work that application organizations do. The presumed goal is to provide a clearly defined set of functionality in a stable, secure and predictable way. It's tempting, therefore, to presume that waterfall would work well here, since that's the method that's been traditionally employed. However, reading the data above and based on our conversations with clients, it's pretty clear that if the traditional waterfall is used, it's not very stable or particularly reliable. Using any incremental delivery approach will improve the reliability and quality of a Mode 1 project, and the agile method's very short increments provide the best results. Since requirement inaccuracies are still the most common and expensive form of defect in a Mode 1 project, incorporating a constant feedback loop from the project stakeholders can catch and fix these issues early in the project.

Mode 2 describes work that is more exploratory, is less known and demands quick results. Mode 2 requires constant feedback from the project stakeholders in order to evolve to the best solution to a problem, constrained by the time and money available. Agile development is an obvious choice here.

In conclusion, there are two key observations when it comes to the combination of mode and method:

• Mode 1 work does not demand waterfall – it can be delivered by any recommended method.
• Mode 2 work demands agile.

Of course, where large-scale efforts are involved, there may be multiple methods used. This means that strong program management techniques must be employed to manage dependencies between the work efforts.

Hybrid Agile Forms

There are also many hybrid methods, most of which combine agile and iterative practices. They're commonly referred to as "wagile," "scrumfall" or "waterscrum." Many fail because they rely on establishing the requirements upfront. The attempt to lock down requirements over a series of iterations eliminates the early stakeholder feedback that is one of the primary benefits of an agile approach. Even in Mode 1 projects, requirements need to be revisited over the life of the project or product effort. This isn't just a trap for hybrid methods – it can occur in iterative methods as well.

In some projects, there's a temptation to mix and match methods in a heterogeneous fashion, working some of the tasks in a time-sequenced manner while others are performed in a true agile manner. Although this may be successful on a single project, it's not recommended as a long-term strategy.

Transition From Traditional Waterfall Methods

The emergence of waterfall methods was an attempt to apply engineering principles to software development. The traditional waterfall is based on two seemingly simple premises, both of which must be present for delivery success:

1. There is a single set of requirements and delivery options that describe what the business wants to do (requirements) and how it will be delivered (analysis and design).
2. The specifications do not change or change only minimally (and perhaps predictably) during the construction and test phases of the project, subject to a strong change management process.

The second premise is a source of amusement and dismay to those who develop and/or deploy software solutions. There are always changes. The key to a reliable delivery method lies in minimizing the number of these changes when a set of functions is under construction and testing. This is a fundamental issue, and the difference between waterfall and agile. Waterfall is based on defined process control, which works well when all the variables are known and controllable, such as in mechanical engineering. However, waterfall fails when there is a degree of uncertainty; in such cases, empirical engineering processes are superior because they use continuous feedback to adjust the process as needed.

The first premise is highly debatable. Things change within the business requirements (new regulatory or process needs), requirements are wrongly specified (defects of commission), and some are not apparent or are forgotten (defects of omission). There are multiple options for the actual delivery, especially regarding usability, look and feel, that aren't apparent until the software is actually developed. An engineering solution to requirements would typically involve building prototypes to determine the suitability of business or technical requirements, or further exploring the proposed solution before actual construction has occurred. Although those are best practices for requirements management, most projects simply use a business requirements document in text-only formatting. No wonder this partial solution doesn't work.

Furthermore, traditional waterfall projects carry increased risks over longer timeframes. The risk of finding a requirements defect in testing always exists, and the longer the duration of the build/test phases, the higher the risk that requirements will change. Because iterative and agile methods use short, tight timeboxes, this risk is minimized.

The traditional waterfall method simply breaks down under the fallacy of its assumptions. Certain techniques can be employed to lessen the impact; however, there will still be adverse impacts on scheduling and budgeting unless the durations of the construction and test phases of the project are constrained. Hence, avoid traditional waterfall methods without constraints on duration. There are three potential solutions here:,/

1. Limit the duration of any waterfall project. We'd recommend no longer than 90 days (and we understand that multiple waterfall efforts may be required to deliver interdependent, large groups of functions).
2. Move to agile or iterative, or a mix of both methods, for a project. If the collection of functions in the project is estimated to take 12 months, break those 12 months into four three-month timeboxes, or 12 one-month "sprints" (incremental waterfall versus true agile). Be cautioned that the shift to a true agile method requires substantial cultural changes. In any event, put software in the hands of the business (or let them look at it, at least) more frequently to avoid the big-design-upfront trap. Requirements are revisited on a regular basis, enabling the development teams to learn through the process.
3. Move to a product-based delivery structure using agile or iterative, or a mix of both methods. This requires a substantial amount of organizational change, not only to the applications team, but to the way software is viewed and used by the business. Although it was considered maverick four years ago, this is an approach that a number of clients are contemplating or have begun using during the past 12 to 18 months.

These approaches require changes in the interaction patterns between the application team and their business colleagues, and none of them can be forced on the business. Understand the facts and use them to craft a story for change. In most organizations, there is fertile ground in the business for changing the way applications are delivered, so the transition can begin with a few application teams without requiring all teams to change at the same time.

It's time for application leaders to recognize what the facts tell us: All other things being equal, traditional waterfall approaches are less reliable and more risky than a more duration-constrained approach. Moving to an agile, iterative or hybrid approach should be a goal for any development organization.