13. Learning Lessons with Knowledge Audits

Most development agencies have committed to become learning organizations. But the use of evaluation for learning may be less important than that of other inputs, such as self-evaluation and training, and evaluation results may only marginally support policy, strategy, and operational changes. The Independent Evaluation Department of the Asian Development Bank (2006) determined to apply knowledge management to lesson learning. In 2007, it formulated a strategic framework to improve the organizational culture, management system, business processes, information technology solutions, community of practice, and external relations and networking for that. These Knowledge Solutions explain the strategic framework. They also describe the knowledge audit methodology developed to tie in with the department’s audiences. The online, questionnaire-based survey of perceptions conducted as a first exercise that year provided ready and multiple entry points against which the department can take measures to that intent, as well as a comprehensive baseline assessment against which to judge progress. Fundamentally, these Knowledge Solutions contend that evaluation agencies should move from “make and sell,” at the simplest level, to “sense and respond” in ways that are increasingly satisfying to stakeholders. Knowledge from evaluations will not be used effectively if the specific organizational context, knowledge, and relationships of evaluation agencies, and the external environment they face, are not dealt with in an integrated and coherent manner. Knowledge management can shed light on possible operating frameworks for this and knowledge management initiatives can be applied to catalyze and facilitate identification, creation, storage, sharing, and use of lessons. That would be knowledge utilization indeed.

Knowledge must not be seen as something supplied from one person to another or from better-off countries to developing countries, but as something that can flow back and forth and be continually improved, adapted, and refreshed using knowledge management tools. What is more, knowledge management tools are more effective where the specific knowledge, relationships, and context of organizations and the external environment they face are dealt with in an integrated and coherent manner.

Evaluations are conducted to find out what results are being achieved, what improvements should be considered, and what is being learned. In ADB, this is done with systematic and impartial assessment of policies, strategies, partnerships, programs and projects, including their design, implementation, and results. Sharing lessons¹ also demonstrates good governance and advances understanding of what an organization aims to accomplish, thereby generating support for it. The principal audiences for evaluations, using ADB as an example, include the Board of Directors, Management, the operations departments, ADB’s developing member countries, the international evaluation community, and of course ADB’s Independent Evaluation Department itself.

In the case of ADB, inter- and intra-organizational relationships encompass ADB’s Independent Evaluation Department, other departments,² developing member countries, and the international evaluation community. Figure 13.1 shows these primary interfaces with the specific organizational context, knowledge, and relationships of ADB’s Independent Evaluation Department and the external environment it faces to structure entry points for lesson learning.

Knowledge management must be embedded into an organization’s business processes. It is not an activity delivered exclusively by a distinct business unit or a particular process. An architecture must be built to initiate and implement organization-wide knowledge management initiatives. Here, four pillars are critical to success. They are (i) leadership, (ii) organization, (iii) technology, and (iv) learning. The below table outlines the core functions, typical activities, and implementation elements of a stable architecture for lesson learning.

Learning lessons is contingent on improving organizational performance in five areas of competence. They are (i) strategy development, (ii) management techniques, (iii) collaboration mechanisms, (iv) knowledge sharing and learning, and (v) knowledge capture and storage (Collison and Parcell 2001).³ Sundry knowledge management tools can support endeavors in each area, including, for example, knowledge audits, activity-based knowledge mapping, action learning sets, peer assists, and exit interviews. Conspicuously, the advent of the Internet has brought information technologies that complement and supplement the knowledge management tools at hand to make knowledge flow more effectively around and across organizations. The technologies include e-learning, web conferencing, collaborative software, content management systems, Yellow Pages, e-mail lists, wikis, and blogs. Where an organization might aim to be in specified time and the priority areas of competence that it might therefore decide to focus on can be investigated by means of such diagnostic tools.

Drawing the elements of knowledge, relationships, context, and external environment; audiences; interfaces; architecture; and knowledge management tools in a conceptual structure generates the operating framework within which decisions on knowledge management initiatives can be taken and implemented. Figure 13.2 depicts the operating framework within which knowledge management tools were leveraged by ADB’s Independent Evaluation Department for lesson learning in ADB.

Organizations looking to knowledge management develop business plans that are aligned with their goals and objectives. To raise knowledge vigilance to the point where attitudes are realistic and automatic, and tacit knowledge is internalized, such plans usually identify needs and issues within the organization and are couched against a framework for addressing these. Needs and issues, as well as the business processes associated with them, are typically determined by (i) the external environment; (ii) the mandate, vision, goal, and objectives of the organization; (iii) the overall strategic direction; (iv) the size and spread of the organization; (v) organizational history and culture; (vi) staff skills and experience; and (vii) available resources.

The elemental steps of business planning are (i) identify key staff groups in the organization; (ii) conduct comprehensive and holistic analyses with the key staff groups to identify needs and issues and barriers to organizational performance; (iii) supplement the analyses with inputs from managers and organizational strategy documents to determine an overall strategic focus; (iv) develop findings and recommendations to address the needs and issues and to tackle the barriers identified; and (v) implement a series of knowledge management pilots based on the findings and recommendations, leveraged by suitable knowledge management tools, and with concern for measuring the effectiveness of outreach. Figure 13.3 illustrates the process commonly followed to develop a business plan for knowledge management.

Learning is a process, not an attainment. Hence, in ADB, the Independent Evaluation Department’s knowledge management business plans are aligned against ADB’s to set in train the drive for continuous improvement that is at the heart of strategic frameworks. Moreover, the annual business planning process specifies that regular annual knowledge audits linked to annual business plans will deliver outputs steadily against each interface based on operational needs and priorities but also resources, with flexibility and adaptability.

Knowledge audits help organizations identify their knowledge-based assets and develop strategies to manage them.

In 2007, IED particularized a knowledge audit methodology, its principal means, and associated time frame, to be applied in four phases spanning about 5 months. The methodology draws on the elements of knowledge, relationships, context, and external environment; interfaces; and architecture deemed most relevant to the department. The four phases are (i) knowledge audit preparations, (ii) knowledge audit analysis, (iii) knowledge audit review, and (iv) business planning. Box 1 enumerates possible related steps and activities and Box 2 sketches an indicative time frame for implementation. Since knowledge management is a process for optimizing and leveraging the stores of knowledge in an organization, the accent placed (concurrently or in turn) on each constituent of a knowledge audit will depend on where an organization is and where it wants to be. Boxes 1 and 2 should be interpreted in view of that. A second important caveat is that the following section on the survey of perceptions conducted by IED in 2007, which emphasized identification of knowledge needs, should not be taken as all that a knowledge audit can be.

To underpin future knowledge audits, IED formulated in 2007 survey questionnaires that drew out perceptions of the performance of independent evaluation across the four interfaces. The questionnaires were designed against the Five Competencies Framework. The framework of organizational competence for knowledge management comprises (i) strategy development, (ii) management techniques, (iii) collaboration mechanisms, (iv) knowledge sharing and learning, and (v) knowledge capture and storage.⁷ The questionnaires were comprehensive, organized, systematic, and inclusive; they provide the framework within which IED can search for continual opportunities to ameliorate the independent evaluation function and its feedback mechanisms. The responses to the questionnaires also revealed rich seams of “as-is,” baseline information, which will be mined vigorously. Box 3 shows for each interface the area of competence on which the questionnaires centered.

Knowledge surveys The survey that opened IED’s first knowledge audit aimed to gain insight into how people within the four interfaces perceive the department’s knowledge management activities. From the results, IED measured awareness of and identified gaps in the department’s knowledge products and services. The Five Competencies Framework was used to assess the department’s organizational performance. This framework identifies these areas of organizational competence for knowledge management as (i) strategy development, wherein tools are used to help an organization achieve a particular goal in knowledge management through a long-term plan of action; (ii) management techniques, which cover a range of practices from assessing the forces for and against desired organizational changes to assessing managerial approaches to mistakes, in order to do things right; (iii) collaboration mechanisms, which pertain to facilitating effective practices in working with others; (iv) knowledge learning and sharing, which means using techniques to learn from and improve future activities; and (v) knowledge capture and storage, wherein routines are applied to ensure that an organization retains essential knowledge. For each interface, survey questionnaires covered only the areas of competence deemed most relevant.

The survey adopted a variant of the Likert scale⁸ to evaluate the perceived level of organizational performance per competence area, wherein respondents specify the extent of their agreement with a statement. Five choices were given per question to gauge perception of IED’s competences: (i) never, (ii) seldom the case, (iii) sometimes the case, (iv) often the case, and (v) always the case. Two methods were used to determine overall perception of organizational performance in each area of competence. The first was based on the responses of the majority per question. The second established an objective measure by computing the weighted average score to account for the perception of the entire sample.

Box 4 gives a snapshot of the perception of the performance of IED in each area of competence by respondents from each interface.⁹ Respondents from IED thought that the department is doing well in the areas of strategy development, collaboration mechanisms, and knowledge capture and storage. But the department is “on the fence” in knowledge sharing and learning, and its competence with management techniques must get better. Respondents from other departments felt that IED does well only in strategy development. They were ambivalent with regard to collaboration mechanisms. They recommended that the department should deploy more efforts in the areas of knowledge sharing and learning, and knowledge capture and storage. Respondents from the international evaluation community felt that the department is doing well in all three areas of competence regarding which their opinions were sought.

The survey of perceptions substantiated the basis of the knowledge management initiatives that IED introduced from 2007. It clarified the need for others. Among the new knowledge products and services developed, Learning Curves are handy, two-paged quick references designed to feed findings and recommendations from evaluation to a broader range of clients. Success Rates present condensed information on successful ADB projects. The Evaluation Information System is an online database of lessons, recommendations, and ADB Management responses. The department hosts the secretariat of the Evaluation Cooperation Group.¹⁰ It has also overhauled ECGnet, the group’s communication tool. Evaluation Alerts are targeted information bytes delivered to personal mailboxes. Methods and guidelines for using plain English, disseminating findings and recommendations, and conducting exit interviews have been prepared. The evaluation pages were refurbished from top to bottom. They are updated daily and are now one of the most accessed first-level directories in adb.org. IED formulated regional technical assistance for capacity development for monitoring and evaluation, expected also to suggest a strategy for evaluation capacity development.¹¹ The department advertises its knowledge products and services on ADB Today ¹² and adb.org (and through other channels and at several venues) with one-time, near-term, and continuous efforts. The survey of perceptions suggested other opportunities. They included Evaluation Chats, a communication tool that would facilitate the establishment of an evaluation community of practice focused on the conduct and dissemination of strategic evaluations, harmonization of performance indicators and evaluation methodologies, and development of capacity in evaluation and evaluative thinking. Evaluation News and Evaluation Presentations were introduced too. They offer, respectively, reports on events in monitoring and evaluation and short photographic or PowerPoint displays on evaluation topics. IED’s knowledge management initiatives are framed deliberately to increase value-added from operations evaluation, and are managed with knowledge performance metrics. Client feedback is sought regularly by various means.

Box 5 identifies the that might be leveraged to fill the remaining knowledge management gaps identified, and emphasizes with shading the areas of organizational competence found wanting at the time of the survey.