High Velocity Innovation

Indicators to Assess the Strength of Your Rapid Learning Cycles Program 

Key Takeaways

• Monitor the health of your program with a combination of observations and a handful of quantitative metrics.
• Five metrics will provide the most important information to assess your team’s effectiveness and efficiency in the Product Development Process and allow comparison across programs.
• Your own eyes and ears are the best tools to assess the health of a running team.

I celebrated a major milestone in the project to write my second book by purchasing a small desktop fountain. It makes a nice gurgling sound that helps my concentration. Every time I empty it out to put in fresh water, I need to remove the rocks that decorate it, and then rearrange them until the fountain sounds right. If it’s too loud, it’s distracting and if it’s too soft, it does no good. When it’s just right, the room brightens up with its cheerful song.

I can tell when a Rapid Learning Cycles program is going well. The team is engaged. The Events are dynamic. The Program Leader knows how much to expect from the team, when to push a little and when to back off. The Knowledge Gap and Key Decision reports are clear, even to me as an outsider. The external evidence I see is that Knowledge Gaps receive careful thought and attention, and Key Decisions get made on time.

When your team is working well, you will know. But if the team is not working well and something needs adjustment, it will be easier to spot and remove the barrier if you have been doing some regular observation and reflection.

Things to Observe

We’ll start with the things to notice. When you are attending team meetings or walking around your office, these are some things to look for to give you a sense of the overall program. They are not quantitative metrics – but they are leading indicators of the results that will show up in the program metrics if you are not paying attention. It’s easier to make adjustments when the team is new and the program is not very far along than it will be later, after team norms have crystallized and the program is in its busiest phases.

The team’s general morale and stress level. Are people relaxed and calm, or are they a little frantic? Do they seem to be overloaded? How much tension is in the office? Do people seem to be working well as a team?

What happens at Status Events. Are there a lot of Activities that need help? Is the group able to keep the meetings short and efficient? Is the team beginning to hold themselves accountable for following good Status Event practices?

What happens at Learning Cycle and Integration Events. Are Knowledge Gaps closing? Are Key Decisions being made? Do the participants actively participate in the discussion or do they mostly listen passively? Do the reports share conclusions and recommendations or just data and information? How is attendance? 

The Knowledge Gap and Key Decision Reports. Do they capture what you need to know now? Do they capture what you will need to know for the next version of the product so that you won’t have to do the experiments over again? Do they contain links to the back up information that is important to you? Do they have a home that makes it easy for them to be found?

Stakeholder Feedback. Are the stakeholders engaged? How much visibility does this program have outside the team? Does it seem like this program has the priority that it merits? Do the stakeholders provide valuable feedback for the team?

Team Feedback. What do the team members say about the process when you speak to them 1:1? Are they getting used to it? Do they like it? If not, why? What are the sources of friction for them?

Early Look Customer Engagement. If you have customers engaged to help you close Knowledge Gaps, how engaged are they? How effective are your methods for working with them? How good is the information you’re getting from your work together?

Personal Observations. What do you notice about yourself? What are the sources of friction for you? Has the method helped you become a better leader, coach and partner, or is it making things hard for you? How worried are you about the team and the product right now? Does it seem like the team is building the product you have envisioned? If not, why?

© 2016 Whittier Consulting Group, Inc. All Rights Reserved.

Things to Measure

Even if you sense that your program is going well, you need to prove that to others, especially the sponsor who is supporting you and the other stakeholders whom you depend upon to get the product out the door. You need to demonstrate that the Rapid Learning Cycles framework is an improvement over traditional product development as practiced at your company.

Some groups try to measure everything, but this just adds a lot of work for you as the Program Leader, without adding much value to the program. You need to be selective about the metrics that you monitor. If you are in a process-heavy organization that measures everything, you will need to focus even more. Instead, consider graphing these five measures over the life of the program:

Days of Delay from Key Decisions Made After the Last Responsible Moment. This is the only conformance-to-schedule metric I recommend for early-phase programs, because if Key Decisions get made on time, then the team will reach their gates on time, and the program will stay on track for its launch date. The team gets no credit for Key Decisions made early, and receives no penalty for Key Decisions that were scheduled earlier than they needed to be made. If a Key Decision needs more time, teams should be encouraged to rearrange the flow to push the Last Responsible Moment to a later date.

Percent of Closed High Priority Knowledge Gaps that close before the related Key Decision. This is a measure of the team’s learning effectiveness and efficiency. The metric will encourage teams to focus on the most important Knowledge Gaps first. If the team is not closing many Knowledge Gaps, they could be poorly defined or the team’s cadence could be off. 

Budgeted vs. actual COGS. For me, this is a better proxy of design quality than validation test results. When a team gets into trouble or doesn’t have time to learn, they develop more expensive solutions. They don’t know the limits of the system and so they overengineer. They choose a tried-and-true material over a low cost alternative that requires some investigation.

Number and impact of late design changes. This will be a lagging metric – by the time you have seen a loopback, it’s too late to prevent it. If you want to demonstrate that Rapid Learning Cycles made a difference and should be considered for future programs, this will be an important metric to compare between Rapid Learning Cycles programs and the ones that don’t use them.

Projected vs. actual launch date, as compared to similar programs. If the typical product is six months late, and you are only three months late, you have saved three months. The team should get credit for that.

Notice that I do not recommend measuring schedule conformance for Knowledge Gaps or Activities – they should change in response to program needs. The time or money required to reach any given phase isn’t helpful to optimize when we push decisions backward and pull learning forward. Most teams make their early gates. It’s better to measure slips at the end by measuring the design changes.

What Do You See?

Your best tools for making these observations are your own eyes and ears. What do you see and hear? What observations do you have? What’s not as smooth as you’d like it to be? What’s working better than you expected? When I visit a client site, I like to walk through the team workspaces and visit their team rooms. I’m usually not looking for anything specific. I visit the spaces as quietly as I can to avoid interrupting the work, and see what I notice:

What’s on the walls or in the team rooms? Are they using visual models to communicate, or a lot of text and numbers? Do IT and Legal policies get in the way of their ability to share information (if the walls are bare, that’s usually why)? Does it look like they inhabit the space, or is it a bit sterile? Does the space support collaboration and creativity?

What are people doing when they’re not in an Event? Are they at their desks working alone? Are they working in pairs? Are there a lot of impromptu meetings in conference rooms or the coffee area? Is it a noisy or quiet space (either is OK, just different)? Does the room seem energized or tired?

What time do people come in and when do they leave? Relative to the company norms, are people eager to get to work? Do they stay late? Do they perhaps stay later than they should or work weekends? Where do they take lunches or breaks?

How easy is it for non-colocated team members to participate fully in the program? Is the audio or video quality OK? Does the team manage timezone differences well so that team members don’t have to disrupt their sleep? Are the key elements of the Rapid Learning Cycles framework – the Key Decision / Knowledge Gap logs and Learning Cycles plan – available online for remote teams to view? Has the team had a face-to-face with their remote colleagues?

What keeps you up at night? Finally, what are you most worried about right now? I knew we were on to something special with the Rapid Learning Cycles framework when a program manager told me that it’s the first time she could sleep over the Christmas break – the program was on track and she had nothing to worry about. If you had trouble sleeping over the holidays, why?

I know it’s time to recharge the battery or change the water when my fountain stops making its happy sound. What are your personal indicators that your Rapid Learning Cycles process needs some adjustment?