One of the benefits of being an agile organization is the elimination of IT shadows: the functions and activities that crop up in response to the inadequacy of the plans, competency and capacity of captive IT.
IT shadows appear in a lot of different forms. There are shadow IT teams of developers or data engineers that spring up in areas like operations or marketing because the captive IT function is slow, if not outright incapable, of responding to internal customer demand. There are also shadow activities of large software delivery programs. The phases that get added long after delivery starts and well before code enters production because integrating the code produced by dependent teams working independently is far more problematic than anticipated. The extended testing phases - or more accurately, testing phases that extend far longer than anticipated - because of poor functional and technical quality that goes undiscovered during development. The scope taken out of the 1.0 release resulting in additional (and originally unplanned) releases to deliver the initially promised scope - releases that only offer the promise to deliver in the future what was promised in the past, at the cost of innovation in the present.
None of these functions and activities are planned and accounted for before the fact; they manifest themselves as expense bloat on the income statement as no-alternative, business-as-usual management decisions.
The historical response of captive IT to these problems was to pursue greater control: double down on big up-front design to better anticipate what might go wrong so as to prevent problems from emerging in the first place, supplemented with oppressive QA regimes to contain the problems if they did. Unfortunately, all the planning in the world can’t compensate for poor inter-team collaboration, just as all the testing in the world can’t inspect quality into the product.
Agile practices addressed these failures through teams able to solve for end-to-end user needs. The results, as measured and reported by Standish, Forrester, and others, were as consistent as they were compelling: Agile development resulted in far fewer delays, cost overruns, quality problems and investment cancellations than their waterfall counterparts. With enough success stories and experienced practitioners to go round, it’s no surprise that so many captive IT functions embraced Agile.
But scale posed a challenge. The Agile practices that worked so well in small to midsize programs needed to support very large programs and large enterprise functions. How scale is addressed makes a critical distinction between the truly agile and those that are just trying to be Agile.
Many in the agile community solved for scale by applying the implicit agile value system, incorporating things like autonomous organizations (devolved authority), platforms (extending the product concept into internally-facing product capabilities) and weak ownership of code (removing barriers of code ownership). Unfortunately, all too many went down the path of fusing Agile with Waterfall, assimilating constructive Agile practices like unit testing and continuous build while simultaneously corrupting other practices like Stories (which become technical tasks under another name) and Iterations (which become increments of delivery, not iterations of evolving capability), ultimately subordinating everything under an oppressive regime pursuing adherence to a plan. Yes, oppressive: there are all too many self-proclaimed "Agile product organizations" where the communication flows point in one direction - left to right. These structures don’t just ignore feedback loops, they are designed to repress feedback.
If you’ve ever worked in or even just advocated for the agile organization, this compromise is unconscionable, as agile is fundamentally the pursuit of excellence - in engineering, analysis, quality, and management. Once Agile is hybridized into waterfall, the expectation for Agile isn’t excellence in engineering and management and the like; it is instead a means of increasing the allegenice of both manager and engineer to the plan. Iteration plans are commitments; unit tests are guarantees of quality.
Thus compromised, the outcomes are largely the same as they ever were: shadow activities and functions sprout up to compensate for IT’s shortcomings. The captive IT might be Agile, but it isn’t agile, as evidenced by the length of the shadows they cast throughout the organization.