Last summer, Carla wrote a blog post in which she expressed amazement that she is still repeatedly asked, "Can you measure the impact of knowledge management? And, if so, how?" APQC's response (as well as Carla's) is always the same: "Of course you can!" In that post, Carla went on to cite some of the tactics that APQC has found most successful in measuring KM impact.

Now, I admit that measuring things is not a glamorous or rewarding process.  There is no shortcut for defining, developing, collecting, analyzing, monitoring, and reporting measures that track performance. (Process flow that!)  Furthermore, just following the steps will not always yield the results you want or expect. 

Despite this, I constantly tell APQC's KM measurement customers to just trust the process--because I know it works.  After all, a process is like a recipe, in that it gives you the basic steps. Anyone can follow a recipe, but expert chefs know how to select the best ingredients and equipment, fine-tune the instructions, and trouble-shoot when things don't go as planned. The same goes for measurement: The process provides the framework, but high-quality inputs, effective tools, and process knowledge are needed to ensure you get the best result every time.

Let me share three pieces of advice that, when combined, go a long way to ensuring the success of a KM measurement effort.

1. Use tools that provide a framework for thinking and promote dialog.
2. When designing measures, engage people who are involved in the workflow itself.
3. Prepare for the long haul, and take steps to align measures with change management initiatives.

Why and how to measure the impact of knowledge management are questions that never go away. A savvy reporter called me about this the other day. She said that she keeps hearing that people don't know how to measure KM. I admit that I find this upsetting given the amount of time APQC spends helping KMers do just that. It isn't always easy, but few things worthwhile are.

The holy grail of knowledge management measurement is to tie participation to outcomes. The more people participate in communities of practice, share information, use information, adopt practices, and so forth, the more we expect to see a correlation with business and mission outcomes. And we do.

But there is more to it than that.

Measuring return on investment for KM has always been a challenge. Recently, I received an e-mail from one of our members--Kevin Gannon of U.S. Navy Carrier Team One--asking about this topic.

One of our process masters ... found [an] article on KM ROI--very interesting.  There are a couple of "estimates" such as "average cost of turnover is 1.5 times the annual salary of the job," "it takes 13.5 months for a new employees to maximize their efficiency," "ROI for a brain-drain orientation project is 10:1," etc.  My question is: As we start trying to nail down ROI calculations, do you think it is fair to quote numbers like this?  That question might be too subjective--a better one might be: Do these numbers seem to match your experience in this area?

What I told him was this: Over the years, APQC has had the opportunity to do deep research as well as hands-on work with organizations that have succeeded in measuring the impact of their KM efforts.  These efforts were measured via targeted projects where knowledge flow and KM approaches (e.g., communities of practice, best practice replication) were embedded into the business workflow.  The results have ranged from a 50 percent return on investment within the first nine months up to a 5:1 return on continuous efforts. 

APQC is always looking for more effective ways to calculate ROI. If your organization has experienced an ROI from KM, we'd love to hear about the focus of your efforts and how you performed your calculations.

... after speaking quite succinctly about the futility of using traditional measures and biased surveys to measure KM impact in complex environments, he suggested using Sensemaker and the various methods from Cognitive Edge / Dave Snowden's work for narrative elicitation, capture and analysis to uncover the real impact of KM work.

Since assessing and proving the value of KM seems to be one of the problems we hear about most often at APQC, I thought it would share this with our audience. For more general information about measuring your KM program, check out our articles Can You Measure the Value of Knowledge Management? and Five Tips for Effective KM Measurement Systems. We also have a list of KM Measurement Tools and Resources to guide you to more detailed information on this topic.

Dale Arseneault has posted an in-depth discussion of the pre-conference training session he attended--"Measuring the Impact of Knowledge Management," led by APQC's Cindy Hubert--on his own blog. I encourage everyone to read and comment on his post at Reflections on Knowledge Management and Organizational Innovation.

When it comes to evaluating the success of CoPs, efforts tend to revolve around two measurement categories: assessing health and measuring impact.

Depending on the objectives of a particular community or community structure, measures may include the investment and resources required to sustain communities, the level of activity occurring within the CoPs, and or the degree to which knowledge and/or practices are transferred. Some of the standardized measures that APQC uses include:

• frequency of visits to CoP Web sites,
• number of reads per month,
• number of postings per month,
• number of downloads per month, and
• member satisfaction ratings.
  

For more information on measuring various aspects of your KM program, see our Measurement and ROI section.