How Kanban-style development gives us another way to deliver on Agile values
Years ago — Feb 25th 2008 to be exact — I wrote this draft article. At the time Kanban development was a cool new thing — bleeding edge Agile. Due to a series of unfortunate events the article wasn’t published in the magazine it was originally target for. I’ve been sitting on it for quite some time, so it’s time to open-source it.
Ramblin’ Preamble
Recently I’ve been seeing lots more about Kanban development in discussion groups, articles, and at conferences. The surprising thing for me is that many smart Agile people — people I know to be intelligent insightful people seem bugged by Kanban — seem to see it as threat to Agile thinking. Others see it as using new trendy words to describe best practices we already understand. Basically, I keep running into a lot of grumpy agilistas and a few Kanban fanatics. And, the reason I think this is weird, is that I’m seeing Agile people behave as strangely about Kanban as traditional process folks behaved about Agile. They seem threatened. They see Kanban as a fad. For me I find great value in Lean and Kanban thinking. I do use Kanban ideas in all the Agile teams I coach. And, I do use strict Kanban, WIP limits and all, with a couple teams. No one’s been hurt by it yet. And if it was the rebranding of already known best practice, possibly I was too dense to get it before hearing it clearly described in the Kanban metaphor. Great thanks goes to Aaron Sanders and Joe Arnold who helped me write this article over a year ago — back when I had high hopes of breaking the story. Great thanks goes to Karl Scotland who apologetically described to me what his team was doing back in 2007 — apologetic because he wasn’t using time-boxes as was seen to be the “right” way to do agile. Today I’m hoping this long-ish article helps clear up some misconceptions for a few folks, and simplify it for those scratching their heads about it. As with many of my blog essays, feel free to skim by just reading the section headings. You’ll get the idea.
Agile development is a value system — not a process
The Agile Manifesto is a short, 4-line, set of phrases that indicate what’s important to those practicing Agile development. These statements are the “mom and apple pie” statements for agilists. There’s no specific processes specified — however over the last several years common practice has emerged. (For more on agile value system and culture, read Agile is Culture, Not Process.)
Common Agile Practice takes the best from the buffet
Specific processes call themselves “Agile” when they believe the value system that motivated their creation is aligned with an Agile value system. Scrum and Extreme Programming currently dominate named Agile processes. Less well known, but equally important, Agile processes include Crystal, Dynamic Systems Development Methodology (DSDM), Feature Driven Development (FDD), and likely a few others. Each one of these processes knits together several valuable practices and a typical process flow. If you dumped all these good practices out onto a table, you’d have quite a buffet of very good practices with which to tailor your own process. And that’s exactly what most organizations do. It’s common to take a Certified Scrum Master course where originally you’d learn the ins and outs of Scrum’s Scrum Master role, but today you’ll learn the basics of Scrum, all its roles and process flow, and a lot of additional practices that weren’t in the original process described by Ken Schwaber and Mike Beedle. (Is that gold-plating or evolution?) It’s common to include practices for managing requirements using XP’s user stories and perform XP style release planning. It’s common to shrink Scrum’s original sprint (development cycle) recommendation of one calendar month to 2 weeks. It’s common to use XP style estimation. It’s common to conclude a sprint with a reflection session or retrospective as used in approaches like Crystal, or DSDM. I personally add in practices for UI design and testing, story mapping, and more rigor in the whole product owner/customer role. Once simple Scrum runs the risk of becoming down right Baroque in its ornamentation. Today’s typical Agile process, no matter what name you call it, takes the best from the buffet of Agile practices to create a typical process where:
- Project needs or requirements are expressed in user stories placed in a backlog, and ideally written by Product owners (in Scrum), or customers (in XP) in collaboration with the development team. (Sometimes they magically appear.)
- Developers give high-level estimates saying how long user stories will take to complete.
- Product owners arrange user stories into incremental releases that take typically 6 weeks to 6 months.
- Product owners choose the next stories, highest value first, for each development time-box. The stories chosen need to “fit” into the time-box based on how quickly the team can produce software.
- At the end of each development time-box the team should have incrementally built some of the product. The team (proudly) demonstrates the finished product to product owners and other stakeholders.
- The team adds up the development estimates for the user stories completed during the time-box. This is the velocity (from XP) that’ll be used to estimate the amount that can be completed in the next time-box.
- The team holds a retrospective to evaluate how well they’ve done and what changes could be made to the process to allow things to go better, then the next time-boxed development cycle is planned.
- Time-boxed development continues through to release — which is a short way of saying “rinse and repeat.”
There’s more common practices, such as daily standup meetings to synchronize the team, and burn-down charts to show development progress, but that’s enough to make my point: Today’s Agile process is a mash-up of a variety of good ideas from a variety of sources.
Shift happens: slow change in common Agile practice causes common problems
In many organizations this flow works pretty well. However, there are a few common complications that cause trouble. The problems I came up with seem to be about size. Seems size always matters. (See the old but still relevant Mystery of the shrinking story.)
Shrinking stories makes Agile development easier
There’s pressure in organizations practicing Agile development to shrink user stories — to tell “smaller” stories so to speak. There’s lots of good reasons for this. It’s a lot easier to estimate a story that’s small — which can lead to more accurate estimates, and better predictability. It’s easier to plan with smaller stories. With big stories — stories that might take weeks for a developer to implement — it becomes difficult to plan a development time-box — particularly when the iterations are only a couple of weeks. It seems that only a couple stories fit — and there’s often room for half a story — but how do you build half a story? Splitting them into smaller stories makes it easier to plan those time-boxes.
Shrinking stories makes Agile development harder
Once you’ve shrunk your stories down, you end up with different problems. User story backlogs become bigger and more difficult to manage. When user story writing was first described as a practice by Kent Beck in the first edition of Extreme Programming Explained, a user story was “estimable between one to five ideal programming weeks.” These days best practice calls for stories that can be completed in a few days or less — easily a 5th their original size. Originally an original list of stories for a project might number in the dozens. Today a user story list, or product backlog from Scrum, can easily climb into the hundreds of stories. Prioritizing, managing, and planning with this sort of backlog can be a nasty experience. Shrinking stories forces earlier elaboration and decision-making. Where product owners could write their stories fairly generally and consider many of the details later, now breaking them down into smaller stories forces more thinking earlier in a planning lifecycle. Managing little stories forces us to keep better track of how they fit together. Product owners are often asked to break down stories to a level where a single story becomes meaningless. To keep track of what’s meaningful to them and other stakeholders, they often need to keep track of bigger items such as the features of the product and how many stories contribute to building up that feature.
Short development time-boxes make Agile development easier
It’s faster to plan a short time-box. Originally XP time-boxes, or iterations were suggested to be 2-4 weeks in length. Original Scrum recommended a sprint be a calendar month so that the development time-box could better sync in with business month and quarter-driven cycles. Anyone who’s attended an Agile planning meeting knows they can often last about an hour longer than you can stand it. Short iterations — say a week or two — allow planning to be quicker. These days it’s common for a typical Agile time-box to be two weeks. We get quicker feedback from a shorter development time-box. At the end of an Agile time-box we have a chance to review running software and make changes to your product plan or product backlog. We can also measure the team’s development speed, or velocity. And using a team retrospective, we can suggest and implement process improvements. These frequent opportunities to evaluate our product, the pace of development, and the process we’re using and then respond with changes are what make Agile development agile.
Short development time-boxes make Agile development harder
Short time-boxes herniate. Elaboration of story details pops out of the time-box: During an ideal Agile time-box we’ll have frequent discussions between developers, testers, and those on a product owner team — like business analysts, user experience people, and business people. We’ll do this to understand what we need to build and describe what we’ll do to validate the story was really done. When time-boxes are short, there’s less time for this conversation. It’s common to move many of the conversations to detail the story and describe acceptance to the time-box before so we can be ready to really get moving with development when the time-box starts. Some Scrum practitioners are beginning to put some rigor around defining what ready means — what work needs to be done on a story before it can be placed into a sprint. Testing pops out of the time-box: It’s difficult to fit thorough validation of the story into a short time-box as well. So, often testing slips into the time-box after. Which leaves the nasty problem of what to do with bugsÉ which often get piped into a subsequent time-box. The result of these herniated time-box activities is a cycle that’s actually 3-4 times longer than our time-box. To get work done, we’ll use a time-box to elaborate stories, one to develop them, another to more thoroughly test them, and if there are bugs, possibly another to fix them. Communication drops and exhaustion sets in. As time-boxes shrink those on the product owner team and testers find themselves in a constant mode of getting ready for a next time-box and evaluating past time-boxes. They work long hours, attend lots of meetings, and seem to have less time to be available to help developers with the current time-box. Since their focus is on a future or past time-box, questions about this time-box seem like interruptions. Collaboration decreases and tensions increase. Their work load is heavy, bumpy, not smooth or even.
Applying Lean thinking to Agile breaks the rules?
Over the last several years a large number of Agile practitioners have begun to include thinking from Lean Manufacturing in their Agile approaches. Initially led by Tom & Mary Poppendiek in their book Lean Software Development, you’ll now find practitioners world wide talking about and making lean process changes to their Agile approaches. Recently I’ve heard lots of discussion about Kanban style development — an approach that breaks that breaks the primary rule of today’s common agile practice: the fixed development time-box. While lots have been writing and talking about it, David Anderson lead early charges into this space and still works hard has a thought leader.
Kanban in Lean manufacturing distilled
You’ll find a lot of terminology in Lean software development comes from Japan and from the Toyota Production System in particular. Kanban translated literally: “Kan” means visual, and “ban” means card or board.
Picture yourself on a Toyota production line. You put doors on Priuses. You have a stack of 10 or so doors. As you keep bolting them on, your stack of doors gets shorter. When you get down to 5 doors, sitting on top of the 5th door in the stack is a card — a Kanban card — that says “build 10 doors.” Well it may not say exactly that — but it is a request to build exactly 10 more Prius doors. You pick the Kanban card up, and run it over to the guy who builds doors. He’s been waiting for you. He’s been doing other things to keep busy while waiting. The important thing here is that he’s NOT been building Prius doors. He takes your Kanban card and begins to build doors. You go back to your workstation, and just a bit before your stack of doors is gone, the door guy comes back with a stack of 10 doors. You know that Kanban card is slid in between doors 5 & 6. You got the doors just in time. The whole thing sorta works like magic. Only you wish you had the door-building job. That guys seems to have a lotta free time on his hands.
Kanban cards are used to limit the amount of inventory the factory builds. It doesn’t do the Toyota factory any good to build doors faster then they can assemble cars. It just wastes money on excess doors, and parts of doors. Excess work in progress is considered to be waste in Lean manufacturing. (It’s probably waste in non-Lean manufacturing too.) In the above completely made up example, you’ll never have more than 15 finished doors hanging around. (Mudha is Japanese for waste. Learn it to impress your Lean friends.)
 |  |
Using Kanban cards, we’ve set the limit at 15. So Kanban cards work a bit like a currency. Just like there’s only a fixed amount of money in a system limited by the amount of currency in print, there’s only a fixed amount of inventory that can be in the system limited by the Kanban cards. Kind of a simple way to manage things when you think of it. Corey Ladas explains this well in his Scrumban article and book.
Kanban development limits work in progress
Kanban thinking in software development attempts to do a similar thing. We want to limit unnecessary work in progress to be no higher than it needs to be to match the throughput of the team. Kanban thinking applied to Agile development results in sweeping changes that throw out much of what Agile practitioners consider necessary practice.
In Kanban development:
- time-boxed development is out
- stories are larger and fewer
- estimation is optional or out completely
- velocity is replaced by cycle time
By now you should be scratching your head a bit. Exactly what’s left of Agile if we get rid of time-boxes, change the meaning of stories, and stop measuring velocity. And, exactly what do car doors and Kanban cards have to do with software development? Don’t get hung up on process. Remember, agile development isn’t a process.
Kanban development distilled
Kanban development revolves around a visual board we use for managing work in progress. (Not surprising since Kanban means visual board). It’s common for Agile teams to put the stories in a current time-box on a board with columns for the stages stories go through such as “development” and “testing.” The Kanban board is similar — with a few more rules.
The basic idea is stories start on the left side of the board and race across the board through the phases of development they need to go through to be considered “done”. Finished stories, ready to release into production pile up at the end. And, because this is a Kanban board and we’re going to limit the amount of work in progress, so we’ll limit the number of stories we allow on the board. The numbers written on the bottom limit the number of stories allowed at each station. This example Kanban board was built on a whiteboard with stickies, whiteboard markers, and blue painters tape.
The Kanban board – left to right:
1. goals
On the far left side of the Kanban board is a column labeled “goals.” Here we’ll find large goals for the product — the big things we’re driving at that compel us to build software to achieve these goals. Keeping them there on the left side of the board keeps everyone on the same page — focused on achieving those goals. This idea was stolen from Arlo Belshee who found that by placing goals on the left, the amount of “thrash” on the board goes down. Thrash is when item priority changes frequently.
2. stories queue
The next column contains a queue of stories ready to be started. It’s a queue because the stories are “standing in line” — the story at the top will be the first to be started. When started it’ll leave the queue and all the others will shift up.
3. story process steps
Immediately to the right of the stories queue is a series of columns for the process steps a story needs to go through to be called “done.” The first column is often used for elaborating the story. In development teams at Yahoo it’s labeled UED for user experience design. If a story is here, the UI is being prototyped or describe sufficiently so that development can proceed. You might have a column where business analysts spend time tracking down technical details that developers need to understand to write code. The next column is often used for development. And, the column after that for testing. These columns aren’t set. You should discuss with your team the phases that stories go through to be completed. Some organization may use columns for writing documentation, or preparing customer service people to support the feature in production. These aren’t job titles. While it may be true that different people’s job titles focus on different columns, focus on the story and what it needs to mature. For example if your organization doesn’t use business analysts or user experience people, use the first column for the place in story elaboration where developers sit down and talk to business people and others to describe the acceptance tests for the story — describe what “done” means. It’s generally a bad idea to start writing code (the next phase) if you don’t know how to tell when you’re done. Each process step column is divided into two parts: The top is used for stories currently in progress in that phase. The bottom is the buffer. When work for that phase of the story is completed, it moves from “in progress” to the “buffer” where it’ll wait to be pulled into the next phase. The last process step won’t have a buffer — because when that phase is complete the story is done. When we set limits for work in progress, we’ll set a total number for the process step that includes both “in process” and the “finished buffer” for that process step.
4. done
When the story reaches the “done” row it’s ready, really ready to be deployed into production.
Stories must be minimal marketable features
The term “minimal marketable feature” was first used in the book Software by Numbers. This book placed emphasis on the smallest portion of work that could be release into production and have value to its resulting customers or users. To be marketable the feature needs to be large enough to be useful — probably larger than the teeny stories that take a couple days to build and seem to be best practice in Agile development today. A MMF may take weeks to build. But the important thing isn’t how long it takes to build, but that it be understandable and valuable to those who’ll receive it. To identify a MMF some folks ask the question “Would I announce it in my company’s product blog?” If it’s too tiny to mention, then it’s not a MMF. This is an important part of Kanban development where the focus is on increasing value released from the development process. Each thing released must be valuable by itself.
Set limits for work in progress
To be lean, we’ll limit the number of stories we allow onto the board. A common formula is to add up all the members of the team in all roles and divide by two. All roles includes developers, analysts, user interfaced designers, testers, deployment people — anyone immediately responsible for getting features to market. For example, if team members total 20, we might limit the number of MMF-style stories on the board to 10. We then need to set limits for each column. For the “stories” queue — the place where the stories wait patiently in line before going into progress, set this limit at about half the “in progress” limit. For instance if the in progress limit is 10, limit this queue to 5. For the limits of the story process steps, the limit is often half the number of people that can perform the work for that phase of development. For instance if you have 6 developers, you might limit the development in progress column to 3. Now, this will force developers to work together on stories. I do find in practice that this may not work out for all teams — so I often see limits that equal the number of developers (or those that can perform the process step) or often 1.5 * the number of people in a role. Of course if you do this, it’ll raise the overall work in progress — and as you might expect, items will take longer to finish. When you total the limits for each “in progress” slot on the board, they should add up to your work in progress limit. You may need to jiggle the limits in each process step to get them to work out right. Or, you could raise the work in progress limit — which is a little like printing more money when you run out. In the example Kanban board “Kanbans” are actually the blue tape lines that mark where we can put a sticky note. Each tape line in the “in progress” and “buffer” areas represents one item allowed in progress. You can get the same effect by just writing the limit above the column.
The life of an MMF style story on the Kanban board
This story starts with a queue of MMFs patiently waiting on the left side of the Kanban board. They sit next to the goals on the left edge where everyone can see that these MMFs will help the product reach these goals. A user interface designer and business analyst working as a pair pull the top story from the queue and slide the others in the queue up. They put the story in their “elaboration in progress” column. They track down necessary stakeholders and have discussions necessary to describe acceptance for the story. They meet with developers to make sure the acceptance criteria is understood and complete enough to proceed. Everyone agrees it is. The BA says “I’ll move it into the buffer then.” “No need says the developer, my in progress slot is empty, so I’ll pull it directly in there and get started.”
Notice how the BAs and UI designers collaborated with stakeholders and developers. They were responsible for getting the work done in the slot. But, it didn’t mean they didn’t collaborate with others in different roles. You’ll find in a healthy Kanban practice that there’s lots of collaboration across steps. As the we continue working on the story we’ll see is move across the board from left to right as those working on each phase of the stories lifecycle do the work they need to move it on. They’re cautious not to throw anything “over the wall.” They know if they move something into their buffer that’s not really ready, those responsible for the next phase won’t pull it in — they’ll kick it back. This makes collaboration and communication critical.
Time passes. Stories are elaborated, moved into development, then into testing. But, then a wrinkle occurs. Today developers have finished a story, and s they walk to the Kanban board to move it out of development, they notice their single buffer slot is full — and the “testing in progress” column is filled to its limit. What now? The developers talk to the testers. “We’re really struggling to keep up here. It’ll be till tomorrow morning before we can get some of these stories moved out.” “HmmÉ” says a developer “Can we help test?” “Of course you can!” says the tester. “With your help we can get these cleared out by the end of the day.“ The tester grins “I just don’t want you validating a story you implemented.”
Limits point out bottlenecks
When a column in a Kanban board is full, we know that group is at capacity. We also know that if this keeps happening that that process step is likely where a bottleneck is. Using a Kanban board we learn pretty fast what’s slowing stories down from being completed and we can take steps to improve performance at the process point where it really matters since improving it elsewhere won’t really help throughput. In fact if people upstream continue to work at their regular pace, the people in the bottleneck role will continue to get further behind. Ultimately we want to figure out how to keep development and throughout smooth — to find consistent bottlenecks and by changing the way we work or the number of folks in a particular role eliminate them. This smoothing of work is what Lean people called Heijunka with the goal of eliminating unevenness or Mura. On first seeing Kanban boards, and least one expert I know said it was actually a Heijunka Box — the thing we put up to show work in progress with the goal of smoothing. I’m not expert and don’t want to get too hung up on the terminology. All of it as strictly defined in manufacturing is a bit of a force-fit in software development anyway.
Expedite using the “expedite” track
There are always urgent feature requests or issues in production that need immediate handling. Above the stories queue and each of the story process steps columns is an expedite slot. The row of expedite slots forms a “fast track” across the top of the Kanban board. When an item is placed in this fast track slot, it becomes the immediate highest priority for the team. It’ll be pulled from that slot, and moved through the story process steps as soon as possible, and ahead of all other work in progress.
Measure performance in cycle time
It’s common to measure “velocity” in Agile development as the sum of the estimates of stories completed during the development time-box. But, since there’s no development time-box in Kanban development, we’ll measure story-by-story how long they took to complete — the “cycle time” of the story.
- As a story enters the story queue, write on it the entry date.
- As the story enters the first story process step column record on it the day it was started.
- Then finally, as it reaches the last “done” column at the end of the board, record on it the date it’s finished.
Now you’ve got the numbers you need to compute cycle time. The difference between the entry date and done date is the cycle time that includes the wait time in the queue. To estimate cycle time for new stories, take the average cycle time from the stories completed. Calculate also the time between “started” and “done” this is the working cycle time. It’ll include time when the story is sitting idle in buffers — but that’s what cycle time means. If you notice that relatively trivial pieces of work have a two week cycle time, that should be telling you something. If you’ve ever waited in line for the Pirates of the Caribbean in Disneyland you might remember signs along the way that say “Your wait time from here is 30 minutes” — something like that. Now you can post your own wait times on your Kanban board. At the bottom of your story queue post the average cycle time with wait time. It’ll say something like “Your wait time for a story here is approximately 18 days.” At the top of the queue post the average working cycle time. It might say “your wait time from here is 14 days.”
Who needs estimation?
When you place focus on how quickly you can get functionality done, and have the ability to measure just that, then the estimates don’t much matter. In fact, many using a Kanban approach have simply stopped estimating at all. Yes story sizes vary, but being able to give a wait time plus or minus a few days is sufficient for many organizations’ concerns. Some do still estimate stories. Then use those estimates in conjunction with cycle time. Using a spreadsheet we can calculate the average cycle time for stories with a given estimate. If you do this, consider placing a handy chart next to your Kanban board showing estimate in one column, and wait times in adjacent columns. With this you’re answering the real question stakeholders are asking for when they get estimates: “when am I going to see this functionality in the software?” If your stakeholders are like mine, they don’t want to know when they’re going to get this functionality, the want to know when they’re going to get all this functionality. I find that if I place stories into a spreadsheet with start and end dates, and calculate cycle time, if I select an arbitrary time period — say a two or three week time period — I can see how many stories where completed during this time period. For instance I might see the team finished 22 stories in 3 weeks — that’s about 7.3 stories per week. Given a backlog of 100 stories I can reasonably infer that it’ll take between 13 and 14 weeks (100/7.3). That’s yesterday’s weather for Kanban — at least the way I calculate it. If I know that during three week time period there where 15 working days and that 5 developers worked the entire time, that’s 75 developer days. Knowing that lets me calculate the average number of developer days per story: 3.4 (75/22) — Which is darn close to pi — which makes me believe it has to be right. 😉 This number, 3.4, is what XP practitioners referred to as load factor.
Evaluation cycles, not development time-boxes
You’d still find the “heartbeat” of regular cycles in a team using Kanban boards — at least if they’re doing agile right you should. The only difference is the cycles aren’t used to plan and commit to stories any longer.
Synchronize the team every day
The daily standup or daily scrum meeting occurs as normal, but now it occurs in front of the Kanban board. Instead of the regular meeting ritual of checking in with each person to find out what they worked on yesterday and will work on today, the discussion revolves around the Kanban board and what will likely move on and off the board today, where “traffic” seems the heaviest, and what we could do to clear bottlenecks.
Reflect every few weeks
Every few weeks the team still holds a retrospective to look at the process they’re following, discuss successes and ways it could be improved.
Demonstrate every few weeks
To show work to stakeholders outside the team, a regular product demonstration is scheduled to show the product and get feedback from stakeholders.
Getting back to Agile ideals
Getting back to Agile ideals is the motivation for those practicing Agile development using a Kanban approach. The Agile manifesto described the things Agile practitioners value, but doesn’t prescribe time-boxed development as the way to get it. Lean practices help teams increase throughput. They don’t make developers type faster, rather they draw attention to bottlenecks that slow things down, help you see them and respond to them quicker. Using a Kanban board lets you easily visualize work in progress across different roles and lets you see when someone is taking on too much work simultaneously. A task board as it’s commonly used in an agile approach can give you the visualization too. But, widening the task board to separate testing from development from acceptance or other process steps helps me better visualize where things are clogging up — helps me better diagnose problems. And, setting hard limits for process steps and respecting them really makes me deal with the problem in a way that dropping a pile of stories into a sprint or iteration didn’t. But, maybe it’s just me who’s lazy and avoids dealing with tough problems. I’m sure you’d never run into a situation where you and your team let lots of finished development work pile up waiting to be tested.
Ideal Kanban can pay off big
Ideally your Kanban board moves MMFs across and actually deploys them into production when they reach done. For organizations that can leverage this high frequency of delivery, this is a big financial benefit. The simple math described in Software by Numbers shows the huge increase in rate of return.
There’s no one as zealous as the newly converted
There’s a lot of folks pretty excited about Kanban out there. I am too. Sometimes that zeal takes the form of telling people practicing common agile time-boxed development that they’re wrong. But, I guess I’m crusty enough to know that there’s lots of right ways to succeed and anyone who believes they’ve found the best ways is likely wrong. Don’t let those voicing opinions strongly for, or against, Kanban approaches stop you for digging in deeper and understanding the ideas behind it.
Further reading:
Many links are laced through the essay — but here are a bunch in no particular order: