Keywords

1 Introduction

The dissemination of digital technologies causes profound changes in the education sector. For professional teaching and learning, digitalization phenomena like the substitution of jobs and occupations through technology (Dengler and Matthes 2018), the rapid evolution of occupational profiles and job descriptions (Conein and Schad-Dankwart 2019), and digital workplace learning (Harteis 2019; Ifenthaler 2018) are gaining more and more importance. Teachers, trainers, and pedagogical professionals not only need to embrace digital technology as a powerful tool for administration and communication but also as a meaningful asset for teaching and learning. Digital technology shapes knowledge and, quite often, is a subject to teaching and learning in itself (Gibson and Ifenthaler 2018). Current models of media competencies for pedagogical professionals reflect those aspects, be it for trainers in adult and continuing education (Rohs et al. 2019), teachers in vocational schools (Seufert et al. 2019), or company-based training personnel (Breiter et al. 2018).

Likewise, digital transformation is now regarded as critical and relevant to the survival of organizations of all kinds (Kenney et al. 2015). Educational organizations, however, do not seem sufficiently well prepared for the challenges of digitalization, often struggling with bad infrastructure or staff shortages (Bernhard-Skala 2019). The few existing models of technology integration in educational organizations predominantly focus on school settings as well as on individual factors explaining the assimilation process (Niederhauser and Lindstrom 2018). Still, organizational perspectives on digitalization especially in the education sector are scarce (Ifenthaler and Egloffstein 2020). Within most organizations, the initial phase of a digitalization process – the “fuzzy front end” – is perceived as ill-defined and chaotic (Berghaus and Back 2017). This makes it all the more important to determine a common status or starting point from which the transformation process can be controlled or – at least – supported. If interventions or measures are then undertaken with respect to digitalization, it must be clear what criteria will then be used to decide on success or failure.

In this chapter, we illustrate a systematic approach for supporting the digital transformation of a large educational organization in a joint research project in Germany. After a short description of the research context, we follow a case study approach and outline a maturity model of technology adoption for educational organizations that served as a baseline for training and organizational development. Then, we describe the evaluation concept of a professional development program that focused on individual and organizational aspects of digitalization.

2 Research Background

2.1 The Joint Research Project #ko.vernetzt

#ko.vernetzt is a joint research and development project for the promotion of digital media competence and media education in the field of vocational education and training (VET). The Institute Youth Film Television Berlin-Brandenburg e.V. (short: JFF-BB) is responsible for the coordination of the research network and project management. Further partners in the network are the University of Cologne, the University of Mannheim, and the Kolping-Bildungswerk Paderborn gGmbH (short: KBW). Coming from diverse research backgrounds, these members constitute a multi-perspective research network for dealing with the overall project challenge: the implementation and supporting of digital transformation processes in a networked educational organization from the field of VET. Direct project goals at KBW are:

  • Development and dissemination of an organization-wide understanding of digitalization processes within an educational institution.

  • Development and implementation of demand-oriented qualification modules to promote professional media competencies.

  • Organizational implementation of media development projects.

The scientific objectives point beyond the field of practice:

  • Development and testing of a heuristic for dialogue-oriented organizational development in the context of digitalization (Hofhues et al. 2018).

  • Development of a generic maturity model of digitalization for educational institutions (Ifenthaler and Egloffstein 2020).

  • Transfer of the findings to other companies in the education sector and beyond.

The joint research project is divided into three project strands with different responsibilities between the partners in the network: qualification, organizational development, and evaluation.

2.2 Research Context: Kolping-Bildungswerk Paderborn

The KBW represents a complex field for the implementation of measures of digitalization and application-oriented research. As a holding organization, the Bildungswerk is operating in various locations with 25 subsidiary organizations. With over 5000 participants daily, it is one of the largest providers of vocational education and training and adult education in the state of North Rhine-Westphalia. In 2018–2019, around 2000 employees generated a turnover of approx. 88 million euros. Following the motto “Education with value,” the activities of the KBW cover a large part of the education chain. The 11 business areas of the KBW are adult education/vocational further training, securing skilled workers, education consulting, training and occupation, school, inclusion, internationalization, work with refugees, child and youth welfare, nursing and care services, as well as vocational rehabilitation.

With respect to these diverse business segments, multi-professional teams with heterogeneous target groups often work at the KBW on non-standardizable tasks. Different and sometimes ambivalent approaches to digital technologies result in a non-uniform understanding of digital transformation processes and diverging requirements for media-related educational work. The KBW thus exemplifies the diverse challenges VET providers are facing in the light of the digital transformation. On this basis, the approaches and experiences from #ko.vernetzt are to be tested for transferability and made available to other VET providers within and beyond the field.

Professional development and staff training at the KBW have traditionally been organized on a decentralized basis. It is only since 2017 that a central unit has systematically established a holding-wide training program. To this end, requirements from external stakeholders such as the Job Centre are taken up and translated into training courses, which are then usually carried out by external trainers. The focus of the courses for the first half of 2020 has been on prevention. Special courses such as the additional qualification in rehabilitation pedagogy will continue to be organized by the subsidiaries on their own responsibility. Digital media-related training is currently limited to introductory and advanced courses on the spreadsheet software MS Excel (tool training), as requested by the participants. Employees need to apply for participating in further training measures and seek approval by their respective superiors.

Hence, it seems obvious that any structured support for implementation of digitalization processes within the KBW needs an exploration of the field at first. It was necessary to assess the current state of digitalization for being able to set up, implement, and support adequate measures.

3 Tracing the State of Digitalization: Development and Application of a Maturity Model

3.1 Background and Rationale

In order to trace the state of digitalization within KBW, a maturity model of technology adoption in educational organizations (MMEO) was developed and applied (Ifenthaler and Egloffstein 2020). Maturity models are an “established means to identify strengths and weaknesses of certain domains of an organization” (Lahrmann and Marx 2010, p. 522) that serve to identify discrepancies between the actual and the intended organizational designs which subsequently can be overcome by development activities. “Digital maturity,” thus, is understood as the state of an organization’s digital transformation, that is, what the organization has already achieved in terms of performing transformation efforts (Chanias and Hess 2016). The goal of MMEO was to get an overview of existing individual competencies and organizational capabilities with regard to the digital transformation. In line with current, comparable approaches (Gramß 2020), the model was developed from a synopsis of six maturity models as a hierarchical model with six specific dimensions. The approach is mainly quantitative, allowing for a scoring on five maturity levels with the following descriptors: digitally minimalist (0–30 points), digitally conservative (31–50 points), digitally pragmatist (51–70 points), digitally advanced (71–90 points), and digitally trailblazing (91–100 points). Table 3.1 provides an overview of the maturity model of technology adoption for educational organizations, its dimensions, and respective indicators.

Table 3.1 Dimensions of the maturity model for educational organizations (MMOE)
Full size table

3.2 Operationalization

The MMEO model dimensions (see Table 3.1) were operationalized and administered in a quantitative survey on digital transformation among the employees of KBW. This digital transformation survey covered five areas: conceptions of digitalization (eight items), use of information technology (IT) and digital media (ten items), evaluation of IT and digital media (two items), digitalization in job-related contexts (eight items), and general attitudes toward digitalization (seven items). Most items were answered on a six-point Likert scale.

The study was conducted between June and August 2018. In total, N = 222 employees (58% female, 34% male, 8% n/a) participated in the two waves using the digital transformation survey. The first wave was administered online, and the second wave as a paper-and-pencil survey to reach additional employee groups. More than half of the participants were involved in teaching, while 30% worked in a leadership position. Participants’ work experience ranged from 1 to 46 years, with an average of 18.5 years. In the application of the MMEO, the following questions were addressed:

  • How do employees use IT and digital media, and is there a difference between private and job-related usage?

  • What are employees’ attitudes toward work-related aspects of digitalization?

  • What is the maturity level of technology adoption within the educational organization?

4 Results

With regard to the use of IT and digital media, participants were asked to differentiate between the private and the professional contexts. As highlighted in Fig. 3.1, there are highly significant differences between private and job-related usage for all the technologies and tools with medium to high effect sizes, except for “email.” The use of conventional media seems to dominate within the professional context.

Fig. 3.1
A bar graph depicts high professional usage versus private usage, where specialized software, standard software, computer notebook, email, cloud services, gaming software and apps, tablet, smartphones, streaming services, video, and messenger services are plotted.

Professional vs. private us of digital media and IT (∆ M, Cohen’s d; 208 < n < 215)

Full size image

Findings focusing on attitudes toward digitalization in job-related context are shown in Fig. 3.2. Participants report that IT and digital media already introduced changes in the work environment and that a further integration of IT and digital media could help to achieve further improvements of the work environment. However, participants also report issues with regard to support for technology integration, less autonomy in IT and digital media use, as well as a lack of IT and digital media for learning and teaching purposes (see Fig. 3.2).

Fig. 3.2
A bar graph depicts the digital transformation work context, with improvements achieved through the use of I T and digital media reaches a peak of 4.68 and sufficient support for the use of I T and digital media has a minimum of 3.29 in my work environment.

Digitalization in work-related contexts (M / SD; 184 < n < 201)

Full size image

In order to determine the maturity level of technology adoption with KBW, the maturity level of technology adoption for each dimension was calculated. After a weighting of the dimensions, the overall maturity score of technology adoption was determined, and the semantic label for the maturity level was assigned.

As shown in Table 3.2, the highest subscore was calculated for the dimension culture, and the lowest subscore was calculated for the dimension organization. With an overall maturity score of 59.51, the educational organization is classified on maturity level 3, labelled as “digitally pragmatist.”

Table 3.2 Subscores in the MMEO dimensions and maturity score
Full size table

4.1 Discussion

Results reveal an average level of digital maturity among the employees surveyed, so that the organization as a whole can be classified as “digitally pragmatist” (Ifenthaler and Egloffstein 2020). However, there is still considerable space for improvement. While employees appear to be open-minded about digitalization, there is a clear need of support for the use of IT and digital media. Likewise, the scope for decision-making concerning IT adoption as well as management commitment and support could be improved. Concerning the use of IT and digital media in professional and private contexts, the results of the digital transformation survey show that it’s mostly traditional tools being used at work. Video, for example, which is gaining more and more importance in educational contexts (Poquet et al. 2018), and messaging services including enterprise social networks remain largely underrepresented (Ifenthaler 2018). Here, the operationalization of MMEO in KBW points toward a clear demand for technology-oriented qualification modules in the professional development program.

MMEO provides a static picture of the state of digitalization within an educational organization. As such, it can be compared to the environmental analysis as carried out in human performance technology models (Foshay et al. 2014). Together with a more dynamic perspective as derived from an organizational development process (see, e.g., Helbig, Hofhues, & Lukács in this volume), this picture can provide various reference points for curriculum, program, and intervention design (Gosper and Ifenthaler 2014). MMEO can be the starting point for internal benchmarks, between employee groups and organizational units, helping uncover blind spots and areas of improvement. MMEO can also be used for benchmark comparisons between different organizations, and, of course, for tracing dynamic developments over longer periods of time, when used iteratively.

However, in its present form, the model still has some drawbacks. The operationalization and implementation are complex and should thus be facilitated. Furthermore, there is an overemphasis on the employee dimension that also increases complexity. Future applications of the model should aim to balance the dimensions by, for example, better integrating the strategy and leadership and the culture dimension. A complimentary survey among managers and executives could provide additional insights here.

5 Tracing Developments in the Field of Digitalization: A Multi-Perspective Evaluation of a Professional Development Program

5.1 Background and Rationale

Qualification is the central strand of the #ko.vernetzt project. Starting from the level of digital maturity as assessed with MMEO, professional development in #ko.vernetzt is implemented through the qualification series “digital education with value” by JFF-BB (see Bröckling, Behr, and Erdmann in this volume), which has been the main instrument to support and develop digital transformation activities within the regarded context. Organizational development processes were being triggered as direct and indirect effects of this program with the help of special contact persons, the so-called disseminators. In order to trace the individual and organizational outcomes of the program, for accountability reasons, and to fulfil the overall scientific requirements, an adaptive, multi-perspective evaluation concept with four segments has been put into practice. Thus, the evaluation not only focuses on short-term effects but also on long-term outcomes, and it tries to integrate the individual and the organizational perspective. Figure 3.3 illustrates the target concepts and the respective instruments for the four segments of the #ko.vernetzt evaluation concept.

Fig. 3.3
A model illustrates a four-segment digital transformation, which comprises long-term learning offering module polls, long term learning success module polls, and short term competence development initial and final surveys, organizational implementation final survey, and interviews.

Four-segment evaluation concept

Full size image

The evaluation was implemented as an external evaluation within the project network. The University of Mannheim acted as third-party evaluator to the professional development program, and neither KBW nor JFF-BB were involved in data collection, analysis, and reporting. As the program was open to anybody interested in the topics with a complete freedom of choice concerning modules or starting points, an adaptive pre-post-evaluation design had to be implemented. The evaluation consists of an initial pre-series survey that every “first-timer” was asked to fill out before his/her first module, and a final survey that will be implemented several weeks after the program has ended. In between, every participant was asked to fill out a module poll for every module attended. So the number of module polls per participant can vary between 1 and 12 at maximum. Furthermore, participation in the evaluation was not compulsory. Likewise, participants were kindly asked to assign themselves a unique code for data coupling purposes, which also worked on a completely voluntary basis.

With regard to the widespread “Four-Level Evaluation Model” (Kirkpatrick and Kirkpatrick 2006), the short-term effects in segments 1 and 2 can be attributed to the reaction and the learning level. Long-term competence development in segment 3 also applies to learning. The organizational effects in segment 4 would manifest on the behavior level. The business level from Kirkpatrick’s model is not addressed in the evaluation concept at hand, as it is not possible to calculate the monetary outcomes of the professional development program. Although the basic assumption of causality within the “Four-Level Evaluation Model” has been widely challenged (Gessler and Sebe-Opfermann 2011), the evaluation concept in #ko.vernetzt still follows the idea that perceived learning success can lead to long-term competence development and to a further implementation of digital technology on the organizational level. With regard to the CIPP evaluation model (context, input, process, product) by Stufflebeam (2003), the evaluation concept addresses learning products as well as the learning process, as the module polls were being iterated over a longer period of time. There was no need to explicitly address context or input aspects, as the inputs of the program (e.g., the topics of the modules) were jointly developed with the learners in a participatory approach (see Bröckling, Behr, and Erdmann, in this volume).

5.2 Operationalization

The initial survey covered three areas: motives for participation (five items), “digital” self-efficacy (eight items), and personal and professional background (five items). The items for digital self-efficacy were derived from psychometrically validated instruments on professional self-efficacy (Schyns and von Collani 2014) and uncertainty tolerance (Dalbert 2002) and adapted to the context of working with digital media and IT. Except for the socio-demographics, all items were scored on a 6-point Likert scale. The paper-and-pencil survey was handed out to every new participant at the beginning of each module of the qualification series between fall 2017 and early 2020.

The module polls consisted of four areas: self-assessment of competence (five items) following the “Evaluation in Higher Education: Self-Assessed Competences” (HEsaCom) instrument (Braun and Leidner 2009), emotional-motivational reactions (four items), assessment of the instructional quality, and the quality of learning (nine items) and I like/I wish (two items) and overall rating (one item). The items were answered on a 6-point Likert scale, except for I like/I wish (open format) and the overall verdict (German school grade scale, from 6 = insufficient to 1 = very good). The closed items were repeated in every poll for all the different modules for comparison, while the open-ended questions enabled a topic-based content-specific feedback.

The final survey which will repeat the measurement of the initial survey’s constructs and the final interviews have yet to be carried out.

The overall sample consists of 59 distinctive participants of the qualification series (59% female, 39% male, 2% n/a). About 63% among them were involved in teaching, while 25% were in a leadership position. The average work experience was 17.9 years, with a range from 2 to 40 years. Over 60% of the participants held an academic degree, while the others had a background in the (German) VET system.

Over the first five qualification modules, 35 learners took part in the evaluation. Table 3.3 gives an overview on those modules, their contents, and the participants in the evaluation.

Table 3.3 Q-modules in the evaluation
Full size table

In the module polls, the following questions were addressed:

  • How do participants perceive the instructional design of the modules?

  • How do participants perceive their learning success in the modules?

For the initial survey alone, a research question was:

  • What were the motives for taking part in the course series?

6 Results

For the first four Q-modules evaluated, participants rated the instructional design as follows (Fig. 3.4):

Fig. 3.4
A scatter plot depicts the evaluation of instructional design features, with Q 1, Q 3, and Q 4 plotted at plus plus plus entirely true, and Q 2 plotted at plus plus.

Evaluation of instructional design features

Full size image

On a generally positive level, Q-module 2 falls behind the other modules in every category. Especially the media usage seemed to leave room for improvements.

Concerning the perceived learning success, results are shown in Fig. 3.5.

Fig. 3.5
A scatter plot depicts the learning success, with Q 1, Q 3, and Q 4 plotted at plus plus plus entirely true, Q 2 plotted at plus plus, and W 1 plotted at minus.

Learning success

Full size image

Again, Q2 falls behind the other seminars in every category. However, the webinar W2 has notably weaker ratings with regard to learning success. A very similar profile applies to the additional four noncognitive aspects of learning success not depicted here.

Looking at the initial survey and the larger sample of all participants of the qualification series, the motives for taking part become clear (Fig. 3.6):

Fig. 3.6
A bar graph depicts the motivation for participation, with I use this qualification series to acquire job qualifications has a maximum value of 4.95, and I will put considerable effort into this qualification series has a minimum value of 4.24.

Motivation for participation (M / SD; n = 59)

Full size image

It becomes clear that learners mainly take part because of job-related motives. In doing so, participants did not intend to put too much effort into the professional development program. Additional results from the entry survey reveal a good internal reliability of the digital self-efficacy scale (M = 3.89; SD = 0.805; Cronbach’s alpha = 0.83; eight items), so that the scale can be used for tracing possible competence gains in a pre-post design. However, the mean value of digital self-efficacy is rather low among the sample.

6.1 Discussion

The evaluation concept in #ko.vernetzt is based on three premises: First, openness with respect to access to the qualification series, and openness regarding topics, as the professional development program has been developed in a participatory, demand-oriented way. Second, voluntariness with respect to the participation in the evaluation and the sharing of personal information. Participants can decide whether they can assign a unique identifier to their survey answers, so that the different data can be combined. Third, multi-perspectivity, as the concept aims to address individual and organizational as well as short-term and long-term aspects.

In the first evaluation segment, the quality of the learning offering is analyzed. Results from this segment can provide formative feedback for instructional design and program development. Different topics, trainers, and delivery modes can be compared or benchmarked. The first evaluations show that the webinar, at least back in 2017, was not a feasible option that could replace the face-to-face modules. This, however, might have changed in the meantime, as in 2020 synchronous web-based seminars certainly have become more common.

In the second evaluation segment, individual learning success is the main concept. However, this is only measured via short self-reports, thus in a very subjective manner. Despite all their shortcomings, self-reports are the dominant instruments in educational evaluations in professional learning and development, as other options – let alone objective assessments (Gibson and Ifenthaler 2018; Gibson et al. 2019) – are mostly infeasible due to business reasons. However, alternative options like peer assessment or 360-degree feedback could be taken into account.

The third evaluation segment tries to address the development of individual competencies. From a purely scientific point of view, this, of course, can only be an approximation. Due to the restrictions in the field, the evaluation concept does not allow for strong research designs. Without a comparison group and a valid competence test, and with little or no control over interventions, causal attributions are hardly possible to be made. However, linking the development of digital self-efficacy to the professional development program on digitalization might still provide valuable insights. Thus, a concluding evaluation should be carried out.

The fourth evaluation segment, finally, addresses the organizational implementation of digital media and IT with a long-term perspective. Here, additional follow-up interviews with disseminators of digitalization or other qualitative data are necessary.

7 Conclusion and Outlook

This chapter reported on the quantitatively oriented part of the joint research and development project #ko.vernetzt. The aim of these research efforts was to support and, where necessary and possible, stimulate the process of digital transformation within a networked educational organization. To this end, diagnostic measures and evaluations were implemented, aiming at both the individual (employees) and the organizational (holding, areas, subsidiaries, programs, locations) levels.

In order to carry out an organizational diagnosis on the current state and potential future directions of digital transformation processes within KBW, a maturity model of technology adoption in educational organizations (MMEO) has been developed and operationalized (Ifenthaler and Egloffstein 2020). To assess the effects of the #ko.vernetzt professional development program, a multi-perspective evaluation concept has been implemented. With MMEO and the evaluation concept, we have combined practicability (openness, adaptability) and a scientific approach (accuracy, rigor) in a fruitful way. In the first place, this provided the necessary empirical grounding for qualification measures within KBW. However, the approach can also be transferred to other organizational contexts. For example, selected aspects of the digital transformation survey are taken up in a larger study among various institutions of adult and ongoing education and VET in the state of North Rhine-Westphalia currently.

However, it must be clearly stated that the implementation of a maturity model alone cannot guarantee a successful process of digital transformation. Additionally, a clear digitalization strategy should be formulated and implemented. Such a strategy should not be restricted to the employees and their competencies and the organization and its internal structures and processes. A wider perspective on the market, competitors, and other stakeholders involved is necessary when dealing with the complex phenomenon of digitalization. Moreover, such a strategy involves planning and targeting. With MMEO, different levels of digital maturity can be defined, but nothing is said about the progression from one level to another. The necessary steps, of course, must be an integral part of a coherent digital strategy.

Finally, this surely is one of the of most basic (and simple) findings of #ko.vernetzt: it is the people who put digitalization into practice. Without committed trailblazers, no real progress can be made. Disseminators need to be true change agents (Vey et al. 2017), and they need to have room for initiative, adequate resources and management support. In such a prolific setting, the digital transformation can be supported and actively managed. On the other hand, organizations that fail to provide such an environment run the risk of falling behind.