Mixed Reality and Gamification for Cultural Heritage

Undoubtedly today, 3D technology (either virtual reality, augmented reality or mixed reality) helps us in the representation and interpretation of reality, present and past, so it has become a powerful ally of the social sciences and humanities, especially cultural heritage and archaeology. The digital 3D representation of reality around us has opened a world of possibilities—possibilities which grow each day with the emergence of new challenges and concepts such as 4D or 5D. Emerging technologies applied to the field of cultural and archaeological heritage have resulted in the emergence of new concepts such as virtual heritage, digital heritage, digital archaeology, virtual museums, cyberarchaeology or virtual archaeology, amongst others. New concepts to describe new realities, which in turn generate new challenges.

In this scenario, we are witnessing the first steps of what may soon be a new discipline, for which there is still no consensus on its name. This new field of knowledge demands a theoretical corpus to support it but also new recommendations and guidelines, internationally accepted and able to guide research and ensure the quality of new projects.

This chapter is divided into five sections. In the first introductory section, the geometric documentation of cultural heritage is defined, while its necessity is also stressed. In addition, the various products which could be included in a geometric documentation are also presented. Moreover, the standards and specifications accepted nowadays are mentioned. In the second section, the passive data acquisition methods are presented. They include those sensors and methodologies which collect data based on the radiation emitted from the objects and have an external—usually natural—source, e.g. the sun. In the third section, the active methods are presented. They include sensors and devices that emit their own radiation and record the part radiating back from the objects of interest. In the fourth section, the contemporary processing methods of the acquired data are presented. They include processing of all kinds of raw data, irrespective of their origin or method of acquisition. Finally, in the last section, three examples are presented in order to enlighten the readers with the various methodologies of acquisition and processing of the data for three representative cultural heritage objects of varying size and properties.

This chapter describes the image-based 3D reconstruction of the Priscilla catacombs in Rome, as carried out in the European ROVINA project. The 3D reconstruction system was mounted on a small mobile robot, which could autonomously roam the labyrinth of the catacombs’ corridors. The 3D reconstruction system was designed to cope with the specific challenges posed by the narrow passages found there. It consists of multiple cameras and light sources, mounted on spherical arcs. Also the structure-from-motion (SfM) software needed adaptation to optimally cope with the particular circumstances. Firstly, the information coming from the different cameras is handled jointly. Secondly, the feature matching needs to withstand the negative effects of the strongly changing illumination between different robot positions—moreover the environment is mostly dark and humid. Thirdly, for the same reasons, the usual texture mapping techniques would cause strong seams between the textures taken from different robot positions, and these were avoided through a more sophisticated analysis of surface reflectance characteristics. The chapter includes visual examples for parts of the 3D reconstruction.

In the heritage field, the demand for fast and efficient 3D digitization technologies for historic remains is increasing. Besides, 3D has proven to be a promising approach to enable precise reconstructions of cultural heritage objects. Even though 3D technologies and postprocessing tools are widespread and approaches to semantic enrichment and storage of 3D models are just emerging, only few approaches enable mass capture and computation of 3D virtual models from zoological and archeological findings. To illustrate how future 3D mass digitization systems may look like, we introduce CultLab3D, a recent approach to 3D mass digitization, annotation, and archival storage by the Competence Center for Cultural Heritage Digitization at the Fraunhofer Institute for Computer Graphics Research IGD. CultLab3D can be regarded as one of the first feasible approaches worldwide to enable fast, efficient, and cost-effective 3D digitization. It is specifically designed to automate the entire process and thus allows to scan and archive large amounts of heritage objects for documentation and preservation in the best possible quality, taking advantage of integrated 3D visualization and annotation within regular Web browsers using technologies such as WebGl and X3D.

Intangible cultural heritage (ICH) is a relatively recent term coined to represent living cultural expressions and practices, which are recognised by communities as distinct aspects of identity. The safeguarding of ICH has become a topic of international concern primarily through the work of United Nations Educational, Scientific and Cultural Organization (UNESCO). However, little research has been done on the role of new technologies in the preservation and transmission of intangible heritage. This chapter examines resources, projects and technologies providing access to ICH and identifies gaps and constraints. It draws on research conducted within the scope of the collaborative research project, i-Treasures. In doing so, it covers the state of the art in technologies that could be employed for access, capture and analysis of ICH in order to highlight how specific new technologies can contribute to the transmission and safeguarding of ICH.

From the ancient library of Alexandria 2300 years ago, the objectives of the collection of information has a common fundamental base: to gather, preserve, and promote knowledge, thus helping in the intellectual and cognitive evolution of humanity. Today, the information revolution has given the ability to scientists, educators, researchers, and individuals not only to use a variety of digital libraries as an information source but also to contribute to these libraries by uploading data that they create, leading to a massive production of knowledge that we need to verify, manage, archive, preserve, and reuse. Cultural heritage data is a category in digital libraries that needs much attention, because of its crucial role in helping us to interact with the past and learn, promote, and preserve our cultural assets. Digital documentation of tangible and intangible heritage, data formats and standards, metadata and semantics, Linked Data, crowdsourcing and cloud, use and reuse of data, and copyright issues are the rising challenges that we try to address in this chapter, through literature review and best practice examples. At the end of this analysis, this chapter tries to predict the near future of digital heritage libraries, where 3D digital assets will be part of augmented, virtual, and mixed reality experiences.

In the last decade, a lot of effort has been put by the cultural community around the world into digitization and aggregation activities. The main outcome of these was the development of portals like Europeana, DPLA, DigitalNZ, and National Library of Australia, which are collecting and providing access to the public digitized cultural assets from Europe, America, New Zealand, and Australia, respectively. Their main objective, however, is not only to bring the public closer to culture but also to efficiently represent information about cultural objects that will make them useful to various target groups like teachers, students, and developers by also permitting their creative reuse. The best practice for fulfilling this requirement is the publication of such information according to the Linked Open Data (LOD) principles. In this chapter, we present the tools developed and the methodology adopted through the participation of our group in aggregation activities for enriching and publishing cultural heritage as Linked Open Data.

Starting from different applications in 3D integrated laser scanner survey of heritage buildings and sites, this chapter will illustrate advanced procedures in 3D surveying and modelling of complex buildings, focusing on the ongoing European project “INCEPTION—Inclusive Cultural Heritage in Europe through 3D semantic modelling”, funded by the EC within the Programme Horizon 2020. The project methodology proposes the enhancement of efficiency in 3D data capturing methodologies, procedures and devices beyond current standards, especially as regards their suitability and aptitude for tangible cultural assets, characterized by nonconventional features, locations and geometries. Within the overall documentation process, the semantic enrichment of 3D models will enable an inclusive approach for accessing and understanding European cultural heritage through digital documentation.

The Church of the Holy Sepulchre (Church of the Resurrection) is one of the most important historical sites of Christianity. The current Aedicule structure is the result of various construction phases, damages and destructions, reconstructions, and protection interventions, and as such, it serves as an emblematic case study for five-dimensional (5D) modelling. The innovative and interdisciplinary approach adopted for the modelling of the Holy Aedicule of the Church of the Holy Sepulchre utilizes data from the following: (a) architectural documentation: Description of the current form and structure, as well as its evolution through the ages, based on historic documentation; (b) analysis of construction phases: The construction phases were revealed by a ground-penetrating radar (GPR) survey that was implemented within an integrated methodology, which enabled the technique to identify the various interfaces; (c) geometric documentation: Generation of a 3D high-resolution model, through an automated image-based method and through using terrestrial laser scanning; (d) materials documentation: A wide range of analytical and nondestructive techniques have been used in order to characterize the building materials and extract data for fusion in 5D modelling; and (e) 5D modelling: visualization of the historic construction phases of the Holy Aedicule of the Church of the Holy Sepulchre. The integrated modelling which, after the above analysis, includes enhanced information covering all aspects of the Aedicule structure, geometry, and materials and forms the basis for the creation of an innovative tool that induces mixed reality (MR) with the focus on the Aedicule’s structural evolution (time factor—4D) and on its materials (5D).

This chapter presents the latest advances in historic building information modeling (HBIM) that have been transformed into models for mobile apps based on augmented and virtual reality. The chapter aims to demonstrate that a complex model based on HBIM can be used in portable devices to extract useful information not only for specialists in architecture, engineering, and construction industry and cultural heritage documentation and preservation but also for a wide user community interested in cultural tourism.

Cultural heritage assets, the bearers of historic evidence, are under continuous pressure from change, deterioration, and destruction. Therefore, there is a need to identify and monitor the related risks and to develop appropriate measures for increasing the resilience of cultural heritage. The activities for establishing a European system for data collection and its application in the field of preventive conservation are an ongoing process, where the issue of risks and resilience is well addressed. Recently, there has been an interest in developing a model of built heritage resilience related to mitigation and reaction on sudden environmental impacts, following the resilience models of contemporary buildings. However, these models cannot be simply extended to heritage buildings because of their specific character. In this chapter, a contribution to an acceptable resilience model of heritage buildings is presented.

3D reconstructions have always been an important medium for teaching, illustrating and researching historical facts and items, especially architecture. While 3D reconstructions in academic contexts aim at an accurate virtual representation of a historic original, various knowledge communication effects influence a creation and understanding of virtual representations. From a temporal point of view, architecture usually lasts beyond a human lifespan, and concepts, ideas and messages of deceased builders are available only via sources—either through the architectural object itself or by descriptions or depictions of it. While a creational process of virtual representation is often performed by cross-disciplinary workgroups, an exchange of knowledge between involved individuals is characterised by the need for a synchronisation of personal mental models and organisational and cooperational learning. Moreover, architectural representations address a wide and heterogeneous audience. All described processes are highly supported by visual media, such as images, virtual models or the architectural object itself. To explore knowledge-related phenomena, the authors performed four stages of investigation using qualitative and quantitative research methods. While a first research stage focuses on the scope and overall relevance of virtual architecture within the field of digital heritage, a second stage investigates phenomena due to a creation of virtual architectural representations. A third stage examines how skills and competencies for creating virtual architectural representations evolve during a project and if teaching facilitates the development. Finally, a fourth stage evaluates design approaches for virtual building representations to make them comprehensible for an audience.

In recent years, the popularity of mixed reality (MR) environments has increased as they provide attractive and immersive experiences for educational, entertainment, and training purposes. The increasing advances of Augmented Reality (AR) and Virtual Reality (VR) hardware and software technologies also constitute an interesting area of research. In this work, we present the main pipeline followed for creating virtual character-based AR experiences, specifically in cultural heritage environments. Our main goal in this chapter is to compare different software methodologies for creating VR environments and present a complete novel methodology for authoring life-sized AR virtual characters and life-sized AR crowd simulation using only modern mobile devices. One important aspect of these environments that we focus on is creating realistic and interactive virtual characters via procedurally generated body and facial animations which are illuminated with real environment light. Virtual characters' transformations are handled efficiently using a single mathematical framework, the 3D Euclidean geometric algebra (GA), and the conformal geometric algebra (CGA) which is able to handle translations, rotations, and dilations. Using such a single algebraic framework, we avoid conversions between different mathematical representations; as a result, we achieve more efficient performance. We also compare the efficiency of different GA code generators—(a) the Gaigen library, (b) libvsr, and (c) Gaalop—so that a future user of GA can choose the most appropriate, currently available s/w library that will provide the most optimal and efficient results. Our main research involves the following questions: (a) Are novel, low-cost, modern HMDs suitable as VR platforms? (b) Which are the most appropriate s/w platforms needed to realize such VR digital heritage gamified experiences? (c) Can we achieve more efficient AR scene authoring? (d) Can we achieve more efficient AR animation interpolation and skinning using a single mathematical framework employing GA?

The world is multisensory. All five major senses, visuals, audio, smell, feel and taste, are important to how we, humans, perceive the world around us. Furthermore, the interaction of the senses—cross-modal effects—can significantly affect our understanding of an environment, with one sensory stimuli even being ignored in the presence of other more dominant stimuli. This is as true now as it was in the past. So if we are to attempt, through computer simulation, to understand how a past environment may have been experienced, it is essential that these virtual reconstructions are multisensory, otherwise we run the very real risk of misrepresenting the past.

This chapter describes how multimodal serious games can create an immersive experience to enhance the visitor’s experience. The creation of more engaging digital heritage exhibitions by seamlessly integrating technologies to provide a multimodal virtual and augmented reality experience is presented. In engaging exhibitions, participants can switch between different modes of exploring the physical artefacts at the museum and can explore these artefacts further through serious games, user interfaces, virtual reality and augmented reality. Different types of interaction paradigms are also illustrated. Moreover, a framework for multimodal cultural heritage is proposed based on the above technologies. Finally, future research directions for creating new opportunities for scientific research are presented.

This chapter describes how to create, animate and interact with virtual humans in the context of cultural heritage. Our case studies are previous European projects that our lab participated in, such as ERATO, LIFEPLUS, EPOCH and CALVIN. Our contribution to these projects was to show the state of the art of virtual humans and interactive applications. First we talk about modelling and how to generate avatar components (head, body and clothes) based on physic, spatial and external information: picture, 3D scans and measurements. Then we move on to the intangible part, where we explain how to capture and use motion to animate the avatar in various situations and semantic contexts, via different forms of gesture, expression and lip synchronization. Finally, we focus on the interaction loop between the detection of visitors and the reactions/expressions of the avatar, the dynamics and the integration of metadata such as pictures, storytelling and information that enrich user experience through an exhibition presented at the world’s biggest museum of computer science: Lady Ada.

This chapter reviews an example of preservation and gamification scenario applied to traditional sports. In the first section, we describe a preservation technique to capture intangible content. It includes character modelling, motion recording, and animation processing. The second section is focused on the gamification aspect. It describes an interactive scenario integrated in a platform that includes a multimodal capturing system, a motion comparison and analysis, and a semantic-based feedback system.

Recent advances in technology have provided new ways to present information and interact with visitors in public spaces. Relevant implementations range from the simple utilization of standard PCs that present minimal multimedia content to large theater halls that immerse visitors in virtual worlds or display fancy 3-D representations of exhibited items. Mobile robots provide an attractive alternative to the above, and, as technology becomes progressively available, the number of paradigms of robots in museums is steadily increasing. For this class of robots, human-like interactivity is an essential part of their functionality, aiming at correctly perceiving and understanding natural human behavior and also acting in ways that are familiar to humans using a variety of modalities. Motivated by the latest advances in mobile robots as museum tour guides, this chapter provides insight in how to develop robotic avatars with advanced navigation capabilities, vision-based tracking technologies for human–robot interaction, and personalized, choice-driven access to information in heritage sites.

This chapter presents recent advances in ambient intelligence technologies (AmI) for digital heritage experience in the context of virtual museums (VM). This is delivered under the light of an integrated and scalable approach towards the creation of AmI heritage environments. More specifically, this chapter presents a framework for representing knowledge and the appropriate mechanisms for personalizing content and user interfaces (UIs) to each individual user of a VM application. Based on the presented framework, a number of enabling technologies are reviewed as employed in alternative application contexts to explore (a) the provision of personalized interaction with artworks, (b) the provision of mixed reality technologies for blending the physical with the virtual world, (c) gamification techniques for educational purposes within the museum context, (d) the usage of gamification as a means to produce interactive art installations and (e) the usage of portable mobile and custom hardware devices as a museum guide and as a means for exploring a museum in the context of a treasure hunt game.

Carefully curated digital collections, structured with rich metadata sets and accessible via search engines and APIs, are not enough for users anymore. Multimedia narratives on the web and other digital “wayfindings” help a wider audience access the content of digital collections and also familiarize them with the research products that are published online. Digital humanists, then, face a twofold challenge: how to create scientific-oriented resources that serve the need of both scholars and general users and how to introduce nonspecialists to the digital collections produced by academics. The case of epigraphy is interesting, as there are already several examples of how niche content can be introduced to a wider public using multiple tools. This chapter illustrates the effort made by the Europeana network of Ancient Greek and Latin Epigraphy (EAGLE) in both integrating the largest collections of digitized inscriptions in Europe in a single database and providing users with tools for research, interaction, and fact finding. In particular, we will focus on the web-based storytelling tools that help users build engaging multimedia narrative based on inscriptions and ancient monuments and on a virtual exhibition that showcases some of the most spectacular items in the EAGLE collection.

Modern imaging and scanning processes and digital manufacturing methods like additive manufacturing (AM, also known as rapid prototyping) are very important tools for documentation and the support of research in the field of cultural heritage artifacts. This chapter presents the data handling and the capabilities of the most important AM processes and materials used. Many samples linked to medical applications that are closely related to cultural heritage artifacts show successful applications for both documentation and scientific research. Having once obtained the 3D data set, AM opens up business cases for marketing and customized production of cultural heritage artifacts. Examples are giveaways made from plastics as well as precious one-offs made from metal.

Narratives spun around a series of physical objects in a virtual environment may produce a truly compelling story, but does their potency draw on the fact that in spite of their intangible delivery their provenance is clearly deeply rooted in their museum provenance? We trust, either by what we witness with our own eyes or perhaps by what we instinctively know to be true. An institution, such as a museum, is devoted to the custodian care to their collections, and the publics’ certainty of this responsibility cues us to willingly suspend our own disbelief, even when encountered online where we cannot see the physical objects.

Based on a methodology, developed by an EU project for heritage education, we evaluate a specific cultural heritage platform both quantitatively and qualitatively in order to explore how online visitors experience virtual heritage. We will also explore the ways in which educational methodologies can now be enhanced by these narratives and how material objects and physical environments can forge meaningful connections for students as well as museum visitors to explore.

From the ancient library of Alexandria 2300 years ago, cultural collections have a common fundamental base; to gather, preserve and promote knowledge helping the intellectual and cognitive evolution of humanity. Nowadays the information revolution has given scientists, educators, researchers and individuals the ability not only to use a variety of digital libraries as a source of information but also to contribute to these libraries by uploading data that they create, leading to a massive production of digital data that we need to verify, manage, archive, preserve and reuse. Cultural heritage (CH) data is a category in digital libraries that needs our attention the most, because of their crucial role in helping us to interact with the past and learn, promote and preserve our cultural assets. Digital documentation of tangible and intangible heritage, data formats and standards, metadata and semantics, linked data, crowdsourcing and cloud, the use and reuse of data and copyright issues are the rising challenges that we try to address in this chapter, through literature research and best practice examples. At the end of this analysis, this chapter tries to predict the future of Digital Heritage Libraries, where 3D digital assets will be part of augmented, virtual and mixed reality experiences.