Why Agile Emerged (And Why It Is Harder Than It Looks)

“Requirements are guesses. Designs are guesses. Estimates are guesses. What matters is how fast you can find out.”  Mary Poppendieck

“The problem isn’t that we don’t know how to build software. The problem is that we don’t know how to build software together.”  Gene Kim

Waterfall works very well when you know exactly what you are building. If you are constructing a house, pouring a foundation, or installing a system with fixed specifications, defining the work in detail up front makes sense. The materials are known. The interfaces are stable. The sequence is predictable.

Technology work rarely behaves that way.

For much of my early career, however, we tried to treat it as if it did.

A project would be funded and the next step was to engage the IT service provider. We started, reasonably enough, by trying to explain what we were aiming for. We wrote vision statements. We developed detailed project charters. We described outcomes, constraints, and measures of success as clearly as we could.

It did not take long for the conversation to shift.

None of that was sufficient to begin the work. What was needed was a comprehensive list of requirements, written in a precise format that signaled completeness and control. Each one began the same way:

The system shall…

So we produced them.

Teams spent months creating statements that sounded definitive. Workshops focused on wording. Use cases were debated. The resulting artifact allowed schedules to be built, estimates to be justified, and governance approvals to move forward.

What we were really doing was committing to an approach before we understood how the system would need to interact with other systems, before technical constraints surfaced, and while the underlying technology continued to evolve in parallel.

Once approved, the requirements became the reference point for everything. Contracts, plans, and reviews all anchored to a description of a system that existed only on paper. When development began, reality asserted itself quickly. Interfaces were more complex than expected. Dependencies surfaced late. Technology options changed. What looked clear during documentation became ambiguous during implementation.

The response was rarely to pause and rethink the approach. The response was to adjust the artifacts. Requirements were revised. Change requests were issued. Schedules were updated. Each revision restored order in the documentation, even as the gap between what was being built and what was needed continued to widen.

Eventually, delivery happened. The system aligned with the approved requirements and passed formal testing. Whether it aligned with the organization’s current needs was often a separate question.

Agile entered the conversation as a practical alternative to that pattern.

The promise was not that uncertainty could be eliminated, but that it could be managed differently. Instead of defining everything up front, teams would deliver small pieces of working software, observe how they behaved, and adjust the next increment based on what they learned. Decisions would be informed by evidence rather than defended by documentation.

On paper, it was a better approach.

In practice, it was difficult to execute—even at the team level.

Scrum provided structure: short iterations, clear roles, a prioritized backlog, and regular checkpoints to inspect results and adapt. But organizations tried to apply Scrum inside operating models that still expected fixed commitments, detailed forecasts, and early certainty.

Teams adopted the mechanics of Scrum but remained constrained by governance expectations that had not changed. Backlogs existed, but scope was still treated as fixed. Iterations occurred, but deadlines were immovable. Reviews happened, but learning was limited by decisions already made elsewhere.

Scrum was not the problem. It did exactly what it was designed to do. It gave teams a disciplined way to manage work in the presence of uncertainty. What it did not do was change how the rest of the organization planned, funded, and coordinated that work.

And that distinction matters.

Scrum was never intended to solve the entire delivery problem. It was designed to help small teams learn and adapt as they built software. Understanding what Scrum does well—and what it deliberately leaves unsolved—is the necessary next step.

Next: Scrum: What the framework is, and what it was never trying to be.

Coda

Someone posted a job announcement on LinkedIn.  It was for a government agency that has been in the spotlight lately and has become the center of some controversy.  I was stunned by the reactions!  They were unprofessional and rude. LinkedIn is supposed to be a professional network, and it is fine to disagree with people. But be professional. The internet is forever.

#Agile #Scrum #SoftwareDelivery #TechnologyLeadership #DigitalTransformation #EnterpriseIT #ProductDevelopment #SystemsThinking #ProgramManagement #ProjectManagement #ITLeadership #OrganizationalChange