Agile DW/BI in a Nutshell

Agile DW/BI is based upon the classic agile approach called scrum that revolutionizes the way software professionals collaborate within the project room. We put business and coders in the same room, repeatedly giving them two or three weeks to deliver the next increment of shippable code. We provide them only a high-level design to work from, letting them work in the fastest way they can devise, figuring out the remaining details as they go. We ask them to accumulate and utilize a reference model to guide their detailed designs, and have them work to a tough definition of "done" that's applied daily, using only two, light-weight, paper-based tracking tools that all parties can use to monitor progress within and between iterations.
We have to adapt generic scrum in two important ways for data integration projects. First, the solution architect must work with the project's business partner to draft a short vision document, partition the development work into "bite-size" chunks, and sequence the list of desired features for business priority and technical dependencies. Second, we utilize a second agile technique called Kanban to organize the work of the data modeler and systems analyst so that their specs for each module are 80 percent complete when the developers start building each module of code. At that point, however, the developers proceed with largely generic Scrum methods, gaining tremendous speed through a combination of self-organized teams, information-rich programming, and quality-driven development -- all of which allow them to fail fast and fix quickly. Let's consider these aspects to see where "fail fast and fix quickly" begins.

Self-Organized Teams

Whereas waterfall projects prepare a detailed work breakdown structure before coding starts, agile leaves most of that detail for teams to figure out for themselves. Activity within the project room during the first few iterations may look chaotic for a new team, but scrum requires teams to end each cycle with a discussion of how to improve their work habits. Agile places a tough burden upon the team: demo new, shippable code to your business partner every few weeks. Teams quickly devise for themselves exhaustive standards for module development, including traditional quality techniques such as peer reviews and thorough "as-built" documentation for operations. At the end of the cycle, the embedded business partner either accepts or rejects the new modules based upon their functional capabilities.

By placing such tough objectives upon the team and repeating the process every few weeks, agile forces the developers to get good at delivering increments of new value and keeps them good at it. They quickly eliminate every inefficiency in their process, evolving their work habits to be many times more effective than the highly-managed approach offered by waterfall methods.

Fail Fast and Fix Quickly

All three aspects we've described save labor, but more important, they form a synergy that further doubles a team's development velocity. Software professionals make mistakes during projects -- mistakes in requirements, design, and coding. By generating a big, detailed design before coding starts, waterfall methods ossify these mistakes, so they cost a tremendous amount of time to resolve once the application reaches system test.
Through regular demos of shippable code and continuous integration testing, agile forces a team's mistakes to the surface quickly, allowing business and IT to have informed, adult conversations about what can be done with the resources available. The early discovery of major issues means the developers still have time to correct misconceptions in requirements and design. They can keep the project on track, avoid large-scale rework, and consolidate designs so less must be built. Building less, avoiding major defects, preventing rework -- this is where agile DW/BI gets its tremendous speed.

By Ralph Hughes, MA, PMP, CSM.