Measuring learning that is hard to measure: using the PECSL model to evaluate implicit smart learning

The focus of this paper considers smart learning as learning that may be possible in autonomous, citizen orientated, ‘smart technology’ enhanced activities situated in public space real-world locations. Much of this learning might be that which is considered as implicit—suggested though not directly expressed (Implicit, 2020). Placing context distinctly within smart learning literature, Liu et al. provide helpful precedent by defining explicit and implicit smart learning in ‘Characteristics and Framework of Smart Learning’ (2017a, pp. 38–43). They declare that a “smart learning system includes two aspects: school smart learning system and social smart learning system” (p. 38). Defining explicit learning as ‘what people normally think of as formal learning’ that often happens in school, and implicit learning as social learning, often happening ‘in an environment of community learning, enterprise (work) learning, and learning in public places’. They additionally note that the implicit learning happening at home or in social situations may not always be thought of as learning (2017a, p. 38). Liu et al.’s descriptions offer a relevant foundation for considering the learning referred to in this paper, and the difference between explicit and implicit learning within contexts of smart learning environments.

Social learning activities in public places are the focus of this paper, and have ‘manifold opportunities for learning’ within the concepts, ideas, topics, technologies and interactions of smart learning environments, further described in some depth in Lister (2021b). Activities “can be scoped and designed for a wide range of purposes, and learning can play a part either as an explicit aim, or as an implicit or even covert goal (Lister, 2020)”, (Lister, 2021b, p. 2). For example, some activities would have learning as a specified purpose, while other activities may not be explicitly about learning, but rather about citizen engagement, feedback gathering or creative discovery and content creation (for fun or creative pursuit). Some gamified activities may have learning as a hidden or covert aim, similar to that indicated in Fang (2013), or work from Gee (2007) or Prensky (2003), cited in Hense and Mandl (2014). In these kinds of smart learning activities, learning is present in more general terms of expanding communication skills, self-agency, broadening cultural understanding, and advancing digital skills of participant learners (Lister, 2020, 2021b). This is similar to how Hense and Mandl summarise Gee or Prensky’s argument in relation to digital learning games (2014, p. 182).

Quoting the 16^th party congress of the Chinese Communist Party¹ report, Liu et al. (2017a) remark “to ‘form a learning society in which all people learn and pursue life-long learning, which in turn facilitates people’s all-round development … it is an important path to realize the visualization of learning society… to highlight the role of implicit learning…” (2017a, p. 43). This shared aim with the UNESCO Sustainable Development Goal 4 (SDG4²), echoes an understanding of smart cities to emphasise the importance of enhancing citizen quality of life (e.g. De Lange & De Waal, 2017), and civic learning (Sacré & De Visscher, 2017). Carroll et al. highlight the challenge of “enhancing awareness, engagement, and interaction pertaining to individual and collective human experiences, meaning making, activity, intentions, and values” (2017, p. 2). These challenges relate well to ideas about smart learning activities in public spaces, with concepts such as writing the smart city (Jordon, 2015; Sacré et al., 2017) ambient literature (Koehler, 2013), pyschogeography concepts for smart learning (after ‘Dérive’, Debord, 1958), e.g. Pinder (2005), Kazil and Hou je Bek (2010) and Hou je Bek (2002), or community arts projects utilising smart technology such as Wood Street Walls³ (Lister, 2021b). These activities embrace Sacré and De Visscher’s (2017) “cultural understanding of civic learning” that focuses on “citizens’ assemblage of the social, the material and the symbolic, as a kind of wayfinding in society” and complement Vinod Kumar’s “paradigms of well-being” for smart learning environments, of relationships with self and community, understanding self more clearly, seeing ourselves in others, and self-realisation (2020, p. 43). The authors own research (e.g. Lister, 2021a) into smart learning activities conceptualised as real-world journeys are part of these kinds of activities, considered to potentially enhance citizen community life and understand more about the motivation and engagement of learner-participants (Lister, 2020, 2021c). Dron’s “complex conversational process that can and usually does lead to much that is of value beyond what is planned” (Dron, 2018, p. 3) and Siemens’ ideas about designing environments in which motivated learners acquire what they need (2006, p. 119) closely align with concepts of social implicit smart learning.

Social implicit learning in smart learning environments is not reliant on large-scale data driven technical infrastructures, embedded beacons, sensors or personalised notifications (e.g. in Uskov et al., 2016, cited in Badie, 2018). It does not require complex (some would say intrusive) personal learning ontologies (Rezgui et al. 2014), it is simply available in anyone’s phone at any time. An availability of WiFi and smartphone apps to provide context-aware content delivery via geo-tagged coordinates or augmented reality triggers, to create or provide maps, or perhaps to use other apps in smart ways such as indicated above could all be described as forming smart learning environments. This is an interpretation of the 'smart enough cities’ of Green (2019), where individual smartphone adaptive technology conjures potential ad-hoc smart learning experiences as and when a person requires and interacts.

In writing about virtual reality implicit learning, Slater (2017) provides helpful historic context, citing Reber (1989) to define implicit learning as “the process whereby individuals learn complex information unconsciously and gain abstract knowledge” (Slater, 2017, p. 24). Reber cites his own much earlier work, stating “(s)ome two decades ago the term implicit learning was first used to characterize how one develops intuitive knowledge about the underlying structure of a complex stimulus environment…” and that “… implicit acquisition of complex knowledge is taken as a foundation process for the development of abstract, tacit knowledge of all kinds” (1989, p. 219). Kaufman et al. (2010), citing a body of other work, assert that implicit learning “takes place on a daily basis without our intent or conscious awareness”, and “plays a significant role in structuring our skills, perceptions, and behavior” (2010, p. 321). Eraut (2000) compares types of learning utilising a ‘Time of Stimulus’ typology of informal learning. Categories of ‘Past episode(s)’, ‘Current experience’ and ‘Future behaviour’ define implicit learning as ‘implicit linkage of past memories with current experience’, ‘selection from experience enters the memory’, and ‘unconscious effect of previous experiences’ (Eraut (2000, p. 13). Slater (2017) additionally refers to Seger (1994) for further defining characteristics of implicit learning (2017, p. 25, from Seger, 1994, p. 164), summarising these later as “what is learned is non-conscious, it is complex, it is not as a result of hypothesis testing, and it is not based on episodic memory” (p. 29). Though Slater’s (2017) work differs from smart learning activities in that it discusses virtual reality embodiment, there are similarities in the sense that “(t)here is clearly no hypothesis testing or deliberate attempt to learn something based on episodic memory. People simply have an experience” (2017, p. 30). It is this experience that is of most interest to the work discussed in this paper and others by the author, to understand more about how self-reported participant experiences can contribute to planning for learning in smart learning environments.

In the social smart learning context of this paper, the challenge of defining implicit learning is that the term ‘implicit' can describe many kinds of incidental learning occurring in a wide range of learning contexts. For example, in the unplanned but related aspects of learning in formal education as well as the haphazard citizen learning of everyday life. Implicit learning can be consciously intentional, or much less consciously aware unintended choices that catch the attention or motivated interest of the learner. Intertwined variations of implicit learning can be the billions of learning journeys that start every day with a Google search (Dron, 2018) or the implicit learning that "takes place on a daily basis without our intent or conscious awareness” (Kaufman et al., 2010).

It is useful to further explore implicit learning in smart learning environments in relation to participant experience, and experience variation as understood in the context of the methodology of phenomenography. Phenomenography is a qualitative methodology that investigates learner (participant) experience, and uses responsive emergent interviews to discover a range of possible ways of experiencing a phenomenon. Phenomenography analyses interviews at collective level, though individual context is retained, to establish what is known as an outcome space consisting of categories of description (CoD) for how phenomena are experienced (e.g. Marton & Pong, 2005; Reed, 2006, p. 8). Categories of description attempt to discover the range of relational and often hierarchical categories of experience in commonality and then variation within categories. Initially a methodology specifically orientated toward formal classroom type learning (Marton & Säljö, 1976 in Svensson, 1997), it has expanded to include other fields, such as learning with technology (Souleles et al., 2014) and user experience of digital applications (Kaapu & Tiainen, 2010). Phenomenography is a non-dualist (Reed, 2006), second order analysis perspective (Sjöström & Dahlgren, 2002, p. 340), attempting to see from the perspective of the interviewees themselves by only looking at manifest transcript content, rather than make any latent assumptions about why interviewees say things, described by Bowden as “if it is not in the transcript, then it is not evidence” (Bowden, 2005, p. 15). Phenomenography is novel within smart learning research, though has been referred to in relevant contexts, for example in Badie (2018) for understanding conceptions of learning (e.g. Säljö, 1979a, 1979b), and “to put ourselves into the learners’ shoes and observe the phenomenon of ‘learning’ from their perspective” (Badie, 2018, p. 394).

From this self-reported participant perspective, implicit learning can perhaps benefit from phenomenographic approaches to investigating it because implicit learning is that which is (sometimes unconsciously) selected by the learner themselves, not based on an intended set of learning outcomes. This can be defined as a learner’s ‘object of vital interest’, as described by Greeno and Engestrom (2014), in response to the phenomenographic concept of three aspects of an ‘object of learning’. These are (i) the intended object: what should be learned, (ii) the enacted object: what is possible to learn, and (iii) the lived object: what is actually learned (Marton et al., 2004). Greeno and Engestrom asserted that the “intended object is depicted as a monopoly of the instructor. However, learners also have intentions…” (2014, p. 133), emphasising the issue of learner agency. The intended object of learning—a fact, skill or aspect of knowledge that can be further applied—would under usual circumstances be assessed with formal assessment criteria. Phenomenographers subsequently developed Variation Theory (VT) to support pedagogical approaches for this kind of intended learning, and assumes there are ‘critical aspects’ of an intended object of learning, so as to learn and apply it correctly (Orgill, 2012). However, this is not relevant in the context of this paper, as implicit smart learning is motivated by learners’ objects of vital interest, and is the domain of the learner (activity participant), not the tutor. Reflecting on ways to evaluate this kind of implicit smart learning to perhaps enhance design of more engaging and effective activities is therefore worth further consideration.

Learning evaluation frameworks are plentiful, with an overload of possible choices available in the literature. Social constructivist concepts dominate (Kivunja, 2014), rooted in sequential instructional design, explicit learning outcomes, and ‘reward and punishment’ (e.g. Bruner, 1966). Kivunja makes a convincing argument for the need of new learning paradigms based in twenty-first century skills, listing amongst others communications; creativity and innovation; initiative and self-direction; flexibility and adaptability, and social and cross-cultural interaction. These skills are all part of what this paper touches upon in ideas about evaluating flexible learning based in the concept of phenomenographic experience variation, as it might relate to smart social implicit learning. The author sets out a number of learning evaluation possibilities in relation to phenomenography, which assumes an interpretive non-dualist paradigm (e.g. Reed, 2006) of the learner's person-world relationship in a context of an intersubjective co-constitutive lifeworld (Sandberg, 2005, p. 56). This attempts to highlight potential significance of the learner's lifeworld of lived experience, and its impact on learning in the ad-hoc messy setting of citizen smart social learning activities such as those described in this paper. What learners want to learn, often without thinking about it as learning, is what is under discussion. The ideas and possible choices are offered from this perspective, attempting to plan for and evaluate implicit learning utilising the range of experience complexity that a phenomenographic study can uncover.

The PECSL is considered as a thinking and planning model of design considerations, inspired by user experience and user centred design, for example in Garrett (2010). Applying this model to the design of smart learning activities would be an iterative cyclical process (after Gibbons, 2016), and seek to plan for experience complexity as a way of considering what learner-participants might focus on with intrinsic motivation for value and richer engagement (Lister, 2021b, 2021c). By interpreting experience variation and complexity as experience relevance structures, ‘good fit’ pedagogical approaches could be selected to acknowledge each CoD in learning design. Planning for experience variation with related pedagogical relationships are further elaborated in Lister, 2021b (pp. 5–7), using descriptive examples for different kinds of activity concepts similar to those indicated in the introduction to this paper.

Three mechanisms of implicit learning evaluation are outlined here, to potentially work alongside the PECSL model. These conceptual and cognitive domain equivalence mechanisms offer possible ways to describe levels and types of learning and may assist in developing criteria to evaluate implicit smart learning according to relevance and nature of activity.

Figure 1 shows an overview of pedagogical alignment for PECSL experience CoD, levels of complexity and related pedagogies, along with concepts of surface to deep learning, supplemented by ‘arrangement, viewpoint and argument’ terms as ways to think about levels of learning and experience complexity. Bloom’s Revised and SOLO equivalences offer further ways to consider experience complexity and surface to deep learning using familiar learning taxonomies. ‘Pedagogical relevance’ factors are added to attempt to illustrate how each CoD and associated evaluation might relate to learning design.

To assist in clarification of CoD experience complexity and evaluation equivalences, Table 3 combines Table 1 with the descriptive range of surface/deep learning including arrangement, viewpoint and argument terms, and Bloom's Revised & SOLO taxonomies.

Badie (2018) argues that “in order to propose more analytic descriptions of smart learning, we need to put ourselves into the learners’ shoes and observe the phenomenon of ‘learning’ from their perspective”, referring to Säljö’s “seminal studies on learners’ conceptions of ‘learning’” (Saljo, 1979a, in Badie, 2018, p. 394). Though Säljö’s, 1979a work is unfortunately unavailable to this author, other work by Säljö (1979b), Marton et al. (1993) and Richardson (1999) cite the same or similar studies, reliably describing the hierarchical conceptions of learning that the study defined. In broad terms, the conceptions of learning begin at a simple acquisition of facts that can be memorised, extending to application of facts and procedures, developing towards abstraction and further interpretation. Focusing on adult learners, Badie ‘sketches’ on Säljö’s conceptions to re-imagine them, noting that the model could be reinterpreted as layered, with inner/deeper layers supported by outer/shallower layers. Badie’s six conceptions are: (1) Knowing more; (2) Keeping in mind; (3) Selecting; (4) Meaning Constructing; (5) Interpreting the Reality; (6) Self Realising (Badie, 2018, p. 394). Webb (1997) draws attention to ideas about surface/deep learning that pre-date the phenomenographic surface/deep ‘metaphor’, citing Bloom’s original taxonomy (1956), Gagné (1970) and Pask and Scott (1972), amongst others. However, Webb notes that the surface/deep metaphor differs as develops “the importance of ‘context’ in opposition to the ‘innateness’ of a cognitive psychology steeped in the tradition of individual difference … towards the idea that the learning environment, the curriculum and in particular the assessment regime, informed the approach to learning which individuals would adopt” (Webb, 1997, pp. 196–197). In relation to smart learning and autonomous learning environments, context is argued here as highly significant, impacting many aspects of potential experience complexity and depth of learning that might be possible.

It is useful to note that Säljö’s seminal work on conceptions of surface/deep learning (noting also Säljö’s work with Marton, 1976, 2005) has continued to be featured in numerous pedagogical discourses (e.g. Biggs & Collis, 1982; Schmeck, 1988 (various); Biggs, 1995; Selwyn, 2011). Within the PECSL model these qualitative differences of learning are highlighted through expressions of gaining value and motivation for participation in a smart learning activity, resulting in varying outcomes for depth and quality of understanding (e.g. Lister, 2021c, p. 238–239).

Within a context of phenomenography, Hounsell (1984, 2005) developed ‘arrangement, viewpoint and argument’ as ways to describe levels of complexity in essay writing. Hounsell’s descriptive terms describe complexity of understanding, shown in Table 4, and are an alternate complementary terminology that can be adapted and modified to describe the complexity of experience in a smart learning activity (SLA), described in Table 5.

Hounsell’s conceptions included ‘sub-component’ terminology (1984, p. 21) providing further detail, described as data, organisation and interpretation. Likewise, adapting these for application in SLA’s, we can see a potentially useful correlation with Hounsell’s initial definitions and further interpretation for the SLA. Table 6 combines both the original and SLA interpretation.

Reasoning for utilising an equivalency of Bloom’s Revised and/or SOLO learning taxonomies is based in practical and research literature contexts. Tutors are familiar with what Bloom’s and SOLO are, they are well known in learning communities, and use of these taxonomies can aid in communicability of evaluation methods. Additionally, Bloom’s and SOLO have precedent to be used in both phenomenographic contexts (Biggs, 1995; Marton & Svensson, 1979; Newton & Martin, 2013; Taylor & Cope, 2007) and smart learning contexts (Lorenzo & Gallon, 2019; Badie, 2018; Nikolov et al., 2016).

Learner generated content (LGC) (Pérez-Mateo et al., 2011) may often form part of explicit (directed) learning and be assessed against specified criteria. However, here I argue that LGC offers a wealth of opportunity to evaluate implicit learning, in terms of unspecified LGC being created and shared by participants in smart learning activities. Though some prompts might be offered, the nature of the content is not specified, encouraging active participation and engagement in self directed ways.

In the previously referred to author’s research, participants were invited (it was not mandatory) to create content such as photographs, written notes or comments to respond to participating in the smart learning journey activity. Figures 2 and 3 provided below show some of the content uploaded to an online group area. In the context of PECSL experience relevance structures, these images illustrate how image content might be evaluated as experience variation complexity, that could then be further supplemented by associated learning evaluation mechanisms. By relating surface/deep learning equivalence to the experience complexity being shown it becomes possible to evaluate the quality of this informal content. Learning taxonomies can be additionally used to enable equivalence with other factors or to combine with more formal learning if relevant.

Experimenting with this concept, this technique was applied to the content and images uploaded by the participants in the research, using letter and number combinations to indicate categories and levels of experience complexity. The reader may find it helpful to refer to Tables 1 and 3 to follow what is outlined below:

This concept could potentially be further exploited to contribute to interpretation of content for demonstrated experience variation and levels of complexity using image recognition machine learning techniques. Early ideas were presented by the author as part of the AI in Education series at the University of Oxford in November 2019 (Lister, 2019).

Thinking about the PECSL experience relevance structure of ‘Discussing’, activity design approaches could accommodate discussion topics to support exploration and sharing, probing for deeper reflection and engagement, and further follow up if in relation to community activities. This supports Wegerif (2022), who emphasises the development of dialogic self and community, to ‘talk together better’, of openness and self-reflection, expanding self-identity and creating communities of mutual trust (2022, pp. 7–9).

In activities that may often be autonomous and voluntary, discussion may occupy an important central role amongst groups of participants, with simpler topics of discussion including routes, locations and digital apps if they are being used. Topics expand in complexity to include location or theme related opinions, memories, reflections, and wider relevance between aspects of activity to participant experience and life, demonstrating a deeper enriched participation and sense of value in the discussion itself as well as the content of it. These kinds of conversations are recalled in interviews with participants in the previously described research, and perhaps indicate that the ways participants talk to each other highlights the value they may be experiencing in their activity participation.

Discussion topics in autonomous informal smart learning can be difficult to capture to evaluate for quality and value, however perhaps utilising more established reflection techniques may offer solutions. Real time note making, comments online or post-activity discussion sessions can provide content and opportunity for evaluation through experience variation and complexity, interpreted as surface/deep learning equivalence. Reflective community discussion in civic initiatives (e.g. techniques in Lin et al., 2011, p. 55), or the multimedia project e-portfolios of Paterson (2019, p. 401) might be utilised as informal aspects of activity process or design, depending on nature and purpose of activity.

Post activity focus groups using emergent reflective discussion can uncover much about what participants have learned during their participation. The author has employed this technique in class discussion sessions after students participated in smart learning journey activities as part of a curriculum of study. Student group discussion unpacked what happened and what students thought they had learned, resulting in reporting they had learned more by discussing it with each other afterwards than they had been aware of before discussion took place. They produced sets of group notes, offering further opportunity for evaluation of their learning. These sessions are explored in more detail in Lister (2022b). As stated previously, participant/learner reflection and evaluation of an activity could include learners evaluating their own learning as part of a designed process of measurement for activity effectiveness. Attempting to capture this kind of ‘talking’ and engaging participants in their own learning process may lead to effective ways of evaluating the quality of implicit learning, and alert the consciousness of a participant learner towards their own act of learning itself (Marton & Svensson, 1979, pp. 473–474).