Abstract
Immersive technology for human-centric cyberphysical systems includes broad concepts that enable users in the physical world to connect with the cyberworld with a sense of immersion. Complex systems such as virtual reality, augmented reality, brain-computer interfaces, and brain-machine interfaces are emerging as immersive technologies that have the potential for improving manufacturing systems. Industry 4.0 includes all technologies, standards, and frameworks for the fourth industrial revolution to facilitate intelligent manufacturing. Industrial immersive technologies will be used for smart manufacturing innovation in the context of Industry 4.0’s human machine interfaces. This research provides a thorough review of the literature, construction of a domain ontology, presentation of patent metatrend statistical analysis, and data mining analysis using a technology function matrix and highlights technical and functional development trends using latent Dirichlet allocation (LDA) models. A total of 179 references from the IEEE and IET databases and 2,672 patents are systematically analyzed to identify current trends. The paper establishes an essential foundation for the development of advanced human-centric cyberphysical systems in complex manufacturing processes.
1. Introduction
Industry 4.0 (I4.0) is the latest standard for data and computation oriented advanced manufacturing [1, 2]. The emphasis is placed on the deployment of technologies such as cyberphysical systems (CPS), Internet of Things (IoT), and big data analytics to achieve cost and quality benefits in the manufacturing sectors. The deployment helps overcome issues such as scalability, distribution, timeliness, reliability, security, and fault tolerance. The wide array of differential and affordable sensors at the physical level, the increased processing at the computational level, and the variety of data available for analytics at the data level form a three-pillar foundation for the transition from Industry 1.0 to Industry 4.0 [3–8]. Table 1 shows the industrial evolution transition that leads to technology shifts from Industry 1.0 to the Industry 4.0 [9]. Further advances from the basic user interface (UI) to VR-enabled CPS are explained in [10].
The current generation of products has short lifespans due to dynamically changing consumer demand. When coupled with the need for increased product quality and increasing labor costs for mass customization, smart factories become the goal of I4.0 [3]. In principle, CPS gives industrial objects microintelligence. IoT provides the ability for things to connect to the Internet and to combine big data analytic solutions to optimize production systems with continuous learning capabilities. Immersive technologies help visualize the data and actions in real-time facilitating dynamic responses to demand [11]. Immersive technologies create a human-centric virtual world where humans and machines interact seamlessly.
Immersive technologies provide an environment for objects that are otherwise virtual in nature through the integration of vision, sound, and tactile feedback [12]. There are three levels for immersive technologies [13]. The nonimmersive level is a desktop computer experience where the virtual environment is generated without the need for specialized hardware. The semi-immersive level uses elements of the real world to construct virtual reality applications for applications such as construction modeling, flight simulators, and robotic navigation. The user navigates a representation of themselves within the virtual environment. The fully immersive or neural-direct mode achieves the highest level of virtual reality. This level provides an immersion into the virtual world where the human brain is directly interlinked to the database system and the viewer’s current orientation and position in the virtual world are experienced as it unfolds.
This research is a continuation of earlier research in the area of Industry 4.0 [14], industrial CPS, and IoT linking towards human-centric Industry 4.0 [15, 16]. The research objective is to integrate and propagate industrial immersive technologies (IIT). The study provides technical specifications and an ontology for IIT. Quantitative and qualitative analyses of technical publications and patents form the basis for the current IIT development. The datasets for analytics are extracted from global governing bodies including the Institute of Electrical and Electronic Engineers (IEEE), the International Organization for Standardization (ISO), the Institution of Engineering and Technology (IET), the International Electrotechnical Commission (IEC), the Advancement of Medical Instrumentation (AMI), the United States Patent and Trademark Office (USPTO), the World Intellectual Property Organization (WIPO), and the Guobiao Standards (Standardization Administration of China, GB). The research findings and analysis identify market potential by outlining growth benefits for embedding immersive technologies into advanced manufacturing.
2. Domain Definition and Motivation
The domain of this research includes virtual reality (VR), augmented reality (AR), and brain-machine interface (BMI) that is interchangeable referred to as brain-computer interfaces (BCI). Background studies show brain research increasing in the virtual reality area [5, 10, 11, 13, 17–20]. The motivation for this research comes from an immersive technology background review covering IEEE and IET online databases. The methodology developed in this research is generic; additional literature databases can be added to enhance the comprehensive background study. The review points towards BMI related research which will act as an enabler to translate virtual world interactions into real world actions. The background information helps form hypotheses that BMI will play a key role in the next industrial revolution. Milgram’s reality-virtuality continuum is remodeled in Figure 1 [21]. This assumption is supported by Lexinnova’s generic VR patent landscape analysis report [22]. The domain definitions are explained in the following paragraphs.
VR provides innovative ways for designers and engineers to interact and collaborate which accelerates creativity and productivity. VR is a host of technologies that mimic interactive 3D environments. This virtual world is designed so that users find it hard to distinguish the differences between real and virtual. The VR world can be created by wearing VR-enabled helmets or goggles [23]. Users see events from all angles in immersion and can manipulate virtual elements or constructs in the virtual world.
Augmented reality (AR) combines Mixed Reality (MR) or Substitutional Reality (SR) where the virtual world and the real world are blended in the immersive settings. AR helps designers and developers create images within applications that blend elements of the real world. Users are able to interact with real world virtual content and make distinctions [23]. The brain-machine interface is a framework that helps to create a communication channel between the human brain and the machine. There are three categories, that is, invasive, semi-invasive, and noninvasive BMIs. Invasive BMIs are microelectrode arrays surgically placed into the cortex area of the brain. Semi-invasive BMIs are electrodes placed on the exposed surface of the brain using electrocorticography (ECoG). Noninvasive BMIs use sensors and circuits placed on the scalp to measure the electrical potentials produced by the brain electroencephalography and the magnetic fields of the brain called magnetoencephalography. Noninvasive BMI using electroencephalography shows significant advancements in signals and systems [11]. Steady-state visually evoked potentials are based on the brains electrical signals generated when the retina is excited by a visual stimulus. This technique is preferred in brain interfacing research because of good signal-to-noise ratio [24]. The focus of our current research is an evaluation of noninvasive BMI, which is viewed as the technical evolution of VR and AR which enables users to translate action conceived in the virtual world into actions in the real world.
3. Research Methodology and Structure
A systematic review cross-references technical publications and essential patents. The detailed methodology is presented as a research structural flow in Figure 2. The domain definition is the primary building block for this research and reflects the current state of the science. A principal domain technology review (for domain definition) is followed by key term identification and ontology generation. The domain ontology and schema key terms serve as query input for further literature and patent searches. The result of query execution is a high volume of publication and patent data. The publication data is manually reviewed, classified, and organized on the basis of citation count. This approach helps ensure high coverage of publications related to the current state of IIT. The patent documents are text and data mined, using computer-assisted algorithms, to depict the underlying patent landscape. The results are cross-referenced to improve review accuracy. Further, the results act as input to identifying key technical development trends.
The goal of this research is to learn the structure and opportunities for technology, standards, and intellectual property in the domain of IIT. The methodology begins with domain definition to identify related terms. The domain scope considered for this research is immersive technology in industries and manufacturing. The domain definition is followed by the literature review and the creation of the key term corpus. The IEEE and IET web explorer is used as the search platform to collect literature and key terms and the review are based on the order of citations in descending order. A top-down and bottom-up approach is used to build the key term corpus, organize technological specifications, and create a domain ontology. The ontology is enhanced iteratively every time a relevant key term is identified. The ontology generated is tested and refined using expert review. A patent search is executed using intellectual property search interfaces on the web. Conventional analysis transforms patent volume information into basic inferences such as top assignees and patent codes. The cross-referencing of these results helps validate the direction of research. Further, the analysis uses a technology function matrix (TFM) and latent Dirichlet allocation (LDA) to model IIT patent groupings.
A TFM is a patent map that helps visualize quantitative patent information with respect to the technical and functional features in the patent landscape. TFM consists of key technology terms on one axis and key function terms on another. Normalized Term Frequency (NTF) values are calculated for the key terms. The higher the NTF value, the more important the term. A one hundred key term limit is applied to each term library to ensure accuracy. The patent text mining is executed where the frequencies of terms in each patent are used to calculate the NTF value. The patent document NTF vector is compared with the term libraries NTF vector to determine if the patent belongs to a specific technology or function and assigned to the corresponding cell in the TF matrix. The final TFM is ready when all patents in the patent dataset are fully iterated.
Topic modeling is a statistical approach for finding topics that occur in an archived corpus. LDA is an unsupervised algorithmic approach for proficient information examination [25]. Topic modeling is utilized widely in numerous industries for different mining functions [26–32]. The results are used to formulate business objectives and core strategies where understanding patent dynamics are beneficial. LDA application allows identification of current industry trends and emerging applications useful for additional research and commercialization. A consistency check is performed by cross-referencing technology specifications with the patent analytics results.
4. Immersive Technology Ontology and Key Terminologies
Ontology is a collection of terms in a domain linked to visualize properties, relationships, and associations. An ontology structures domain knowledge, enables reuse of domain knowledge, and makes domain assumptions explicit [33]. The technology review combined with expert evaluation is used to generate the ontology represented in Figure 3. Since many subtopics in VR and AR are highly correlated, they are merged into one technology group. This approach increases query performance and reduces redundancy. There are some abbreviations commonly applied as the domain terminologies. The following abbreviations used to represent the ontology schema for IIT are shown in Figure 3:(1)3D: three-dimensional space(2)EEG: electroencephalogram(3)SBCI: self-paced brain-computer interface(4)CNC: computer numerical control(5)PLC: programmable logic controller.
The ontology represented in Figure 3 has immersive technologies as the top most layer followed by VR, AR, and BMI. Key terms that fall under each domain are arranged alphabetically under each section. Knowledge from heterogeneous sources is combined to form a single schema for a consolidated view. The key terms for VR and AR are derived from [10, 13, 34–47]. The key terms for BMI are derived from [17–20, 24, 48–52]. The key terms are preprocessed to eliminate redundant values and are reviewed by subject matter expert before ontology integration. Ontology offers a perspective towards solving interoperability problems brought about by semantic obstacles [53]. The results represent explicit knowledge contained within VR, AR, and BMI domain types software applications within the industrial and manufacturing domain. Ontology validation is explained in Sections 6, 7, 8, and 9 by cross-referencing patent and technology analysis results.
5. Technology Review and Specifications
During the 1990s, many companies failed to deliver VR products and services to the marketplace. Nintendo Virtual Boy was discontinued a year after market introduction [54]. The increase in the processing power of computers has enabled the current market success of immersive technologies. VR, AR, and BMI are being introduced across a number of market channels that include industrial and consumer applications [23]. The theoretical and methodological contributions and current knowledge from IEEE and IET publications are used to collect data for immersive technologies. IEEE and IET databases represent the research of the broad scientific community along with tools to facilitate advanced search optimization and downloads. The diverse coverage of research journals, conference, eBooks, and technical standards helps users collect data related to the current state of the art. The search reflects the tabulated publication data. The 2010–2016 dataset, row 2, Table 2, is used to derive the preliminary background and inferences. The dataset from row 4, Table 2, is used for the literature review and ontology generation for IIT.
Literature classification requires a subject matter expert (SME) to cluster the publication dataset as shown in Figure 4. Publications belonging to the application group encapsulates research of IIT integration for product/concept design, robotics control, and equipment for manufacturing. Publications in the training group encapsulate research on professional training systems for manufacturing, skill transfer, and human resource development using 3D manipulators. Publications in the case studies group encapsulate IIT studies conducted on industrial robotics, semiconductor chip forming and breakages, perception mechanism for assembly line environments, interaction with Computer-Aided Design (CAD), and visual communication methods. Publications in the hardware group encapsulate research on IIT customizations for motion capture devices, brain wave capture, customized displays for vehicular environments, and specialized head mounted displays. Publications in the simulation group encapsulate research on escape vehicles, precision welding, power cell production, and fixtures for grinding wheels. Publications in the algorithm group encapsulate research for feature extraction for EEG signals, correction of geometric distortions, and genetic algorithms for structure optimization. Publications in the review group consist of literature reviews related to VR production lines in China. Publications in the software group encapsulate research on motion tracking and eye positioning in VR displays. Publications in the movement group encapsulate research on navigation methods and kinesiology for human and robotics in IIT environments. Publications in the interfacing technology group encapsulate research for IIT interoperability for the brain with 3D applications for hardware devices and software platforms. Publications in the brain study group encapsulate research on a wheelchair, unmanned aircraft control, a virtual avatar for gait adaptation, temporal perception using VR neurocognitive, and brain imaging for concussion assessment.
An overlap across topics such as VR, application, research, and the interface is observed from manual review and clustering. The overlap strengthens the assumption (Figure 1) that Milgram’s reality-virtuality continuum is an enabler for future industrial VR and AR products. The following sections present a consolidation of key points from literature and technological specifications. The dataset is ordered by citations and time in descending order to ensure maximum coverage of key technology specifications.
Technology analysts predict that immersive technologies such as AR and BCI/BMI will be commercial successful [23]. Immersive VR and AR are used to train pilots to overcome fear of height as well as disorientation from flight maneuvers. With the help of VR, scientists better visualize models and comprehend chemical reactions with enhanced visual details. Architects use VR to visualize and change the angle of view of structures and simulate walking through prototype buildings. VR also offers the potential for museums and art galleries to allow visitors greater access from remote distances. The VR computer is the portal for access to physical experiences and visualization. The leading immersive frameworks include Oculus Rift and Microsoft HoloLens. These cost efficient applications have gained considerable market share as Computer-Aided Design (CAD) interface solutions. Immersive technologies enable designers and engineers to collaborate on projects from multiple locations. Microsoft, Apple, Samsung, HTC, Facebook, and Sony are leading investors in these technologies [23]. Immersion, when combined with the Internet of Things (IoT), promises a new era of technologies for Industry 4.0 called the Internet of Presence (IoP) or the Internet of Experience (IoE). The SME classification results represented in Figure 4 show that the application related literature has the highest number of publications followed by training, case studies, and simulations. The technology specifications from industrial VR publications with high citation are provided in the following paragraphs.
The industrial virtual reality literature shows that industrial training has a high demand for innovation in higher education. 3D visualization and computer hardware advancements lead to a growing range of training materials. Companies, professionals, and researchers are creating interactive VR educational environments [36], such as the Mathematical Virtual Learning Environment (MAVLE) using VR [55]. The Microsoft Visual C++ programming platform has been used to identify opportunities, reduce cost, and increase competitiveness in many parts of the product development life cycle [56]. A VR based aircraft carrier management system for the French navy (GASPAR) demonstrates further industrial VR’s capabilities. GASPAR simulates environment modeling, procedure organization, and validation using VR applications [57]. Virtual walk around for factories increases productivity. Motion-capturing sensors record the user’s indoor movements and the algorithm digitizes these movements for the virtual world. Virtual walk around reduces health problems, decreases labor turnover, and reduces costs [38]. VR manufacturing reduces costs and the time needed to go from product concept to production. Advancements in processing, optical visualization, power management, and data sciences are making VR a viable solution for traditional manufacturing and product realization [37, 58].
Industrial augmented reality literature has the highest number of publications followed by case studies, hardware customization for industrial machines, and movement related studies. SME classifications show that all publications with high citations are application related that support industrial processes. Industrial AR is applied in areas such as design, planning and production, abd service and maintenance. The German manufacturer Volkswagen has continuously worked on AR-based factory planning processes. Volkswagens reports that AR system application increases reliability [39]. AR applied in aeronautical maintenance allows for the efficient transfer of knowledge by offering new perspectives with dramatic improvements in maintenance tasks [41]. AR applied in acceptance testing helps real-time comparison planned for a given project. The comparisons are realistic and use an expert heuristic for automatic validations [42]. Operator safety in industrial material management and processing is critical. AR systems deployed in radioactive material processing can superimpose virtual digital models onto video streams which increase the operator’s viewing scope and decrease fatigue. The concept is validated by superimposing radiation information on real data streams [46].
The IIT brain interface literature is divided into invasive, semi-invasive, and noninvasive classes. The industrial BMI area is noninvasive-centered. SME classification shows that noninvasive industrial BMI publications are concentrated in the VR area, followed by algorithms and case studies. Top citations are distributed across a spectrum of case studies, simulations, mobility research, EEG research, monitoring, software, algorithm, and hardware. Key points denoting technology specifications are presented below. BCI as a support system for video game control offers a new means of navigation and interaction in 3D environments. The interactions are supported by the noninvasive capture of cerebral activity, recorded using EEG electrodes. This helps to create a “think-and-play” user experience for games of the future and has been successfully applied to robotics control [18, 48]. Another widely used technology is called functional magnetic resonance imaging (fMRI), a neuroimaging procedure based on magnetic resonance imaging. fMRI brain activity is measured by detecting changes in blood flow. fMRI technology results in higher activation and improves task performance [51]. BCI in VR allows individuals to interact directly with an environment rather than a computer monitor ensuring higher immersion and interaction with the computer [19]. BMI applications for wheelchair control is a case analogous to driving a car. Motor execution and motor imagery simulations indicate control and accuracy [20, 49]. Technology specifications indicate the potential for industrial brain interface applications for robotic arm control. The future growth in the area requires collaboration among physical scientists, neuroscientists, engineers, and social and behavioral scientists [17]. Finally, the geographical analysis of publication metadata indicates that China has the largest number of publications in the area of industrial VR and AR, followed by Germany. The USA has the largest number of publications in the area of BMI/BCI research and publications followed by Japan and Canada.
6. Patent Search and Statistical Analysis
The ontology represented in Figure 3 is used to generate a search query for the Derwent Innovation (DI) system. The DI system automatically converts the search query to Boolean format. The DI search ensures an accurate and broader set of data for analysis and inference generation. The search scope includes patent databases from prominent global manufacturing economies. The dataset is filtered using a nine-year range from 2007 to 2016.
The ontology in Figure 3 represents the IIT hierarchy. An ontology is a higher dimension visual representation which needs dimensional reduction and simplification. The ontology in this step is converted into a search query that the DI patent search system uses as input to generate the patent dataset. This conversion is carried out using prior experience in patent search query optimization. High-frequency words identified during the literature review are subject to independent expert review. Table 3 represents the key terms represented in the DI system input format: column 2 represents a topic, keyword filtering criteria, and scope filtering while column 3 shows the generated Boolean search query.
The result of the above queries is split into 2 categories according to the derived ontology. VR and AR contain 1,995 patents in 629 Derwent world patents index (DWPI) classification families and BMI contain 677 patents under 427 DWPI families. When analyzing patent data for landscaping, data are assessed from a qualitative and a quantitative perspective. Quantitative analysis is statistical and helps determine metrics and ratios. The quantitative information in this section represents top assignees and international patent classification (IPC) analysis for VR, AR, and BMI.
Table 4 presents a statistical meta-analysis that identifies the top 10 assignees of technology in the field of IIT. Top assignees analysis shows that Seiko Epson has the most patents in the industrial VR and AR domain with 28 patents. Seiko Epson is one of the largest Japanese owned imaging related manufacturers, followed by Korea Electronics Telecomm with 11 patents. LG Display and Hyundai Motor have 10 patents each. From the technology specification in Section 5, VR technology is extensively used in the automotive industry [39]. The placement of Hyundai Motor as the third is a cross-reference between literature review and patent data analysis. Table 5 presents Samsung Electronics as the largest assignee in the BMI area with 26 patents and accounts for 15.76 percent of the market. Panasonic is the next largest assignee with 13 patents. Thalmic labs has 13 patents and is the creator of wearable technology. Microsoft has patents in VR, AR, and BMI. Microsoft holds 8 patents in VR and AR which represent around 6 percent of all patents. Microsoft holds 8 patents in the BMI alongside extra 9 patents in brain interfacing technologies through its auxiliary Microsoft Technology Licensing.
The top IPC classes for global VR, AR, and BMI utility patents are shown in Table 6. There is an overlap between VR, AR, and BMI in terms of IPC classes. IPC category analysis shows the top three IPC in both categories are G06F referencing electrical digital data processing. The G06F subclass contains patents dealing with processing or transporting of data and data processing equipment such as electric digital data processors. The G02B categorizes optics and the subclasses contain patents dealing with lens or prisms (an optical system designed to produce realistic images); G06T categorizes imaging and subclasses containing patents dealing with image acquisition, feedback, watermarking, and compression. Table 6 shows that IPC classes A61B, H04W, and G06K are specialized for brain interfacing. The A61B class is focused on invasive and semi-invasive brain interfacing. This class refers to diagnosis and surgery and its subclasses deal with prostheses, orthopedic, nursing appliances, and treatment or protection of eyes or ears. The H04W class is focused on wireless communication needed for noninvasive interfacing with focus on protocols, connectivity, and information. The G06N class deals with algorithms, platforms, and architectures required for biological computing [59].
An application on a year-by-year basis for VR and AR patent domains shows exponential growth from the year 2014 with the publication counts of 339, 394, and 579 against the application count of 118, 138, and 79 for years 2014, 2015, and 2016. BMI patent data shows low volumes until 2014 with steady growth in patenting. There are a total of 81 grants against 24 applications during the year 2016. This ratio is due to a reduction in the cost of sensors, actuators, and controller technology coupled with advanced manufacturing demand [14].
A geographical patent distribution analysis shows China companies as the leading assignees in industrial VR related patents. This growth results from mobile platform companies including Xiaomi, Huawei, Pico VR, Baofeng Mojing, and Deepoon. Followed by the USA, Korea is in the 3rd place. The US companies are the leading assignees in BMI mind-machine interface (MMI) and direct neural interface (DNI) patents. The US has a large number of industrial companies and research centers for brain research combined with governmental policies and incentives. The following sections are text mining techniques applied to construct the TFM and group patents based on LDA to map insights and the inferences from IIT data set and external cross-references.
7. Patent Technology Function Matrix Construction
A technology function matrix (TFM) is a unique approach to patent data analytics. A patent map visualizes quantitative patent information with respect to the technical and functional features in the patent landscape [15]. A TF framework consists of technology columns and function columns. The conventional method of constructing a matrix is to read the patent documents manually and fit them into a matrix. IIT TFM uses text mining for a qualitative overview of underlying IP dynamics. The TFM is constructed using the ontology represented in Figure 3. The steps to generate a TFM are presented as a flow chart in Figure 5. TFM is constructed using VR, AR, and BMI technologies and industrial engineering function key terms from the ontology.
The TFM construction uses the patent dataset and IIT ontology as the primary input. The ontology is split for technology and function term libraries. The TFM structure is created with technology keywords related to immersive building blocks assigned to the technology axis. Industrial application functionality keywords are assigned to the function axis. Normalized Term Frequency (NTF) based text mining terms are applied to extract the key terms represented in is the multiplication of , the reciprocal of , and the summation of divided by , where is one of the terms in all document collections, is one of the documents in all document collections, is the total number of all the document collections, is the frequency of term appearing in the document , is the total number of words in document , and is the total number of words in all document collections. Figure 6 is an example of the TFM generation process. The NTF value of each word is the weighted value and a term library comparison is made for each indicator for patents in the IIT data set. The assumption is that patents with high-term repetitions belong to a certain technology or function. The degree of association between the patent and the term library is represented by the sum of NTF value that appears in the keywords of patent and term library simultaneously. Since patent size influences term frequencies, matrix standardization needs to be carried out on the total NTF value as shown in is the dependent variable (NTF value), is the population mean, and is the population standard deviation. A threshold value is applied to remove less relevant patents for a given term. The threshold used is the third quartile value which is the middle value between the median and the highest value of the given dataset. The logic for assignment into the cell for the TFM is to determine whether the words under each indicator and the words below the patent match. If equal, the assigned value for the cell is 1; otherwise, it is 0. The term comparison table for each indicator with patents is constructed with a summation value of each patent word NTF value under each indicator term library. The keywords represent each patent and each technology or function indicator. The summation value is considered to be the similarity level between each patent and each technology or function indicator. The patent is then regarded as belonging to the set of functions or the technology set. The results of the technology and the function patent counts are computed independently and the two matrices are multiplied to obtain the final TFM.
The IIT TF Matrix is presented in Table 7. TFM suggest growth in AR, and sensors are favouring patenting across industrial functional domains. Demands for a diverse array of sensors in logistics and prototype design functions are triggers for the growing volume of 653 patents in sensor technology. The role of integration and convergence of engineering tools, methods, and the need for processing in the context of interfacing, brain-computer, and virtual reality technologies occupies consecutive positions with 447 and 384 patents. Compared to the sensor technology, there are fewer patents for electrode technology which shows IIT favouring noninvasive sensing methods, against invasive or semi-invasive methods. Correlating the result of this TFM with the top IPC analysis presented in Section 6, it can be inferred that optics, sensing, and processing are the most competitive areas. Since there are fewer patents in the industrial safety area this could be a future area of patenting expansion. Optical sensing and processing are crucial elements for immersive systems because human eye limitations for the display are placed 3 to 4 cm away from the eye. Advancements in optics will make lenses in immersive technologies more human-friendly. A reduction in the number of core VR patents can be seen from the TFM numbers. This represents VR technology reaching maturity and leveraging technologies like AR and BMI. The finding further supports the Milgram’s reality-virtuality continuum expansion hypotheses proposed in Section 2. AR technologies progressive patenting in prototyping, e-commerce, logistics, and manufacturing design functions in the industrial ecosystem suggest the need for future standardization research for sustainable industrial interoperability.
8. LDA Patent Topic Modeling
LDA is a generative statistical model for topic modeling. LDA is used in modeling descriptions for given documents called topics [27, 30, 31]. LDA was initially introduced for textual data modeling and uses words in a document. Topics generated are based on the probability distribution of words in a document compared to conventional stripped word frequencies. The models assumption is that words carry strong semantics information and documents with similar topics will use similar groups of words. Hence, documents are seen as a probability distribution over latent topics and topics are probability distributions over words. LDA model has been extended to various other applications such as semantic annotation in satellite imaging, social media tagging, natural language processing, object recognition, spam filtering, web-mining, and fault identification [26–32]. LDA is chosen for the probabilistic patent analysis because of prior application and validation against various bag of words model and proof of higher efficiency in unstructured literature data clustering for systematic reviews [60]. This section is an implementation of LDA model using online method to model topics for IIT patent data [61].
LDA is a generative model for sematic words of a patent based on generic continuous latent parameters for mixtures of hidden topics. Each topic is characterized by words in vocabulary and represented with per word topic distribution. A mixture of topics is used to represent a corpus of a patent using its per topic patent distribution. Assume that there are words in vocabulary and topics in the patent corpus with patents. Let represent the number of words in the th patent in corpus and the prior parameter vectors for Dirichlet distribution are and . In per topic patent distribution, let be the topic that the th word in th patent may belong and let be a multinomial distributed random variable with parameter vector drawn from . Since a word structure stands for a topic, per word topic distribution is needed. For per word topic distribution, let be the th word in th patent and the word may be assigned to a topic, namely, . Since a topic is composed by words, let be a random variable with multinomial distribution , where is sampled from [25]. The structure relation among the above parameters of the model is represented in the plate notation shown in Figure 7. The joint distribution of per word topic and per topic patent distributions is described inwhere is a matrix with dimension spanned by row vectors . Further, by integrating and , the joint distribution in (3) is derived as . Thereafter, the prior parameter vectors and are approximated by using the online method [61, 62]; the final patent topic assignment is determined. For example, for a patent with words, to determine whether it contains latent topic , (4) is utilized. For any ,The flowchart presented in Figure 8 gives an overview of the steps involved in the LDA application. IIT patent information from sections title, abstract, summary, and claims is extracted, filtered by cleaning, preprocessing, and normalization to form the patent data corpus. A document term matrix is then generated on which LDA object is trained for term generation. A selection criterion is used to filter terms of interest. Terms surpassing a minimum number of patents are considered as resultant patent topics. Extracted LDA topics are grouped and analyzed.
LDA application models the VR, AR, and BMI dataset of 2,672 patents across 467 topics with an average of 22 patents per topic. Patent documents contain high volumes of technological legal information that is subjective. The automatic selection of the optimal number of topics to represent the corpus in consideration is still a problem to be solved. Since an insufficient number of topics render the patent groups as too coarse, an excessive number of topics are selected. Exhaustive subject matter expertise is further used to make interpretations and validations. The subject matter expert uses prior experience in the patent pooling and licensing management area to determine the best fit. A similar validation approach was applied in validating human mobility region discovery using LDA [63]. An IPC class code cross-verification is applied manually to further verify the results obtained. The outcome demonstrates that while topics with higher patent frequencies represent a broader technological domain, topics with lower patent frequencies represent very specific technological inventions. Further, 188 topics having comparative semantics are reduced to common terms. The SME driven manual term reduction yields a specific IIT patent topic dictionary (Table 8).
The reduced terms obtained from the analysis form six groups. The optics group has an average of 36 patents and contains patents related to inventions related to the behavior of light and displays and its association with three-dimensional planes in IIT context. The ownership summary below provides a snapshot of termwise institutional holdings. The summary encompasses sample technology based on the analysis of details corresponding to the reduced terms. Seiko Epson leads with patents for manufacturing light guide devices, virtual image display apparatus, optical elements, image display devices, and polarized light separation devices. Denso and companies like LG, Samsung, Motorola, and Qualcomm have patents in display apparatus and heads-up display manufacturing. The hardware group is a collection of physical components that constitute IIT including sensors, actuators, circuits, and computation elements. The group has an average of 26 patents. Top assignees vary in the group depending on the subtopic term. Companies such as BOE and LG have considerable presence in the hardware level touch screen technology. Examples of inventions in this group are the Korea Research Institute of Standards and Science’s patenting in the elastic tactile sensors and Samsung’s physical button display module. The software group contains patents consisting of information, data, and computer programs. The group has an average of 37 patents. The Advanced Telecommunication Research Institute International’s patent on brain information processing software, Apple’s programmable tactile touch for interfacing, Baidu on line network technology patent on transaction information systems, and Guangdong Coagent Electronics gesture and voice control systems are some grants in this group.
The interaction group contains patents relating to user’s communicating with a system or other users. The group has an average of 30 patents. BOE Technology patents are in the wearable interaction technologies, Commissariat’s patents are in neural control, and Ethicon Enco Surgery Inc.’s patent targets eye glance guidance. General Motor, Huawei Device, and Guangdong’s patents cover head mounted display interaction technology. Hyundai Motor Company’s patents present examples of patenting under the topic group. The anatomy group contains patents related to the structure, characteristics, and functions of human anatomy roles and implication in IIT. Sensory feedback patents of AliphCom and BOE Technology face motion-capturing method for the head mounted display systems, the center of Human-Centered Interaction for Coexistence patents related to mobile terminal and training method for a person, and finger operation of a vehicle by Hyundai Motor Company. The industrial group represents patents applied in the context of industrial engineering including assembly, manufacturing, welding, automobile, and tools. The group has an average of 17 patents. The group contains patents related to self-adaptive interaction systems by Anhui Agricultural University, three-dimensional GIS system by Beijing Huadian Tianyi, intelligent lens rotation by Beijing Yunshi, and VR welding education systems by Lincoln Global.
The leading country reduced term analysis can be analyzed using LDA. The results are represented graphically in Figure 9. China has good topic coverage in the IIT domain with an average distribution of 7 patents per topic, followed by the USA, Korea, WIPO, and Japan with 6, 2, and 1 patents per topic. These findings are in line with Lexinnova’s prior VR patent landscape analysis reports for geographical coverage [22]. A comparative reduced term graph is shown in Figure 7 and shows that the market leaders in IIT patent representations have uniformly high patents followed by followed by groups dealing with hardware, software, and interactive optics.
Further evolutionary trends of topics over a period of years from 2007 are calculated. Figure 10 demonstrates a graphical layout of a topic trend across the reduced topic spectrum. The evolution graph shows that patents related to IIT software and interaction have dynamic progressive trends since 2007. These reduced term groups show exponential growth since the year 2011 with patent grants doubling yearly. The topic evolution graph identifies the direction of promising patents for frequent transfer transactions for future industrial use. Technology transfer and licensing are mechanisms for industrial collaboration and helps secure advances in technology.
LDA as modeling technique achieves a platform for a host of patent analytics functions. LDA is a probabilistic, generative model, with patents grouped on top of LDA semantics. Patent groups identified in this section promote technological opportunity and identify areas of risk, opportunity within a given technology area.
9. Standardization Outline and Future Research Scope
Standardization is a systematic approach to propagate innovation, interoperability, and knowledge [64]. The lack of clarity in generic immersive technology standards limits the rate of adaptation for designers and engineers [35]. Virtual reality markup language (VRML) is one such standard. The University of Kiel has created a tool for Volkswagen that produces point clouds in VRML 2.0 format [39]. A report on Devices and Radiological Health workshop in brain-computer interfacing summarize the need for regulations, privacy, and patient protection. Regulation deals with the development of new tools, standards, and approaches to assess the safety, efficiency, quality, and performance. The need for creation of standards in engineering adhering to established medical standards ensures consistency in product development and clinical trial design. The need for medical device standards organizations, such as Association for the Advancement of Medical Instrumentation in collaboration with engineering associations such as Institute of Electrical and Electronics Engineers (IEEE) and International Organization for Standardization (ISO), are needed to develop BMI/BCI standards [65]. Patent data citation analytics play an important role in standards outline mapping. Forward citations (patents that cite a given patent), backward citations (patents that a given patent cites), and nonpatent citation analytics help determine essential patents via a citation map. A preliminary forward citation analysis in the IIT patent dataset reveals that USPTO patent in the BMI area with patent ID US20090312817A1 published in the year 2009 by Wicab has the highest count of citing patents with 136 citations. Wicab’s patent deals with systems and methods for altering brain and body functions and for treatment. Patent ID US20100238161A1 in the VR area refers to a computer-aided system for 360° heads-up display published in the year 2010 with 127 citations. Patent ID US20090273563A1 on programmable tactile touch screens has 115 citations. An average citation analysis shows that industrial VR and AR technology Japan’s semiconductor energy lab (SEL) has the highest average citation. The companies’ 14 patents have been cited 135 times giving it an average citation value of 9.64. BMI’s InteraXon has key patents in the area of brainwave-controlled computing technology and applications has 9 patents cited 58 times giving it an average citation value of 6.44. Since the immersive technology adoption pace in the area of manufacturing depends on the creation of standards and a broader range of licensable IPs, standardization outline is an urgent requirement.
Research findings show that immersive technologies in manufacturing are reaching an inflection point. Industries today approach new products by simulations and analysis instead of assumptions or conjecture [38]. Technology research firm Superdata predict VR reality will grow in areas other than gaming. This report predicts that the initial growth wave in VR related to the gaming the market share will shrink and social media, entertainment, automobile, aerospace, and manufacturing will see exponential growth [41, 66]. According to latest WIPO IP Facts and Figures, 2016 consolidation report that trademarks are the fastest growing segment with 27% growth in the past one year [59]. This research shows that the volume of patents in each competitor segment has grown exponentially in the past 12 to 18 months. This growth curve shows similar characteristic as the smartphone growth curve. Historically, such growth curves lead to litigation. Thus, standardization for litigation avoidance and mitigation along with trademarking is required.
10. Conclusions
This paper presents a systematic study of IIT for improving manufacturing systems using collective intelligence. The idea of “smart” factories is no longer a futuristic vision but a commercial necessity. Independent publication, reports, and white papers from leading manufacturers report a planning reliability increase, higher shop floor utilization, and testing outsourcing benefits achieved using immersive component’s application such as VR and AR [37–39, 58]. Manufacturing industries, however, lack confidence in new immersive investments to upgrade their existing infrastructure due to concerns of underlying market fragmentation and costs involved. This research presented a novel review to the problem of immersive inclusion using exploratory discovery over a large corpus of unstructured technical literature and patents. A collection of 179 publications and 2,672 patents between years 2007 to 2016 is subjected to statistical, text mining and probabilistic analysis. The key contributions are outlined as an ontology for IIT and technology review specifications presented in Sections 4 and 5. An organized patent query generation method and preliminary statistical analysis representing top assignees in each category, IPC classes, the time evolution of patent publications, and geographical indicators are presented in Section 6. Section 7 presents word frequencies mined technology function matrix that allowed visualization of the technofunctional direction and current industry trend. A probabilistic topic modeled patent grouping is presented in Section 8 identifying topics terms in underlying IIT spectrum. The results are validated using subject matter expertise and cross-referencing with external sources to further find associations and evolution across geography and time. The algorithmic discussion and simplification presented in this paper towards analytic ensure transparency for IP professionals and industry practitioners who usually find it hard to accept results of black‐box engines in generic landscape reports. An introduction to future standards research using standard essential patents and patent citation analytics is presented in Section 9. These research maps and benchmarks prove the market potential and growth benefits for embedding immersive technologies in advanced manufacturing. Industry 4.0 promises an increase in efficiency, flexible resource management, and the individualization of mass customization. The resulting intelligent collaboration of all technical components presented involved in immersive technology spectrum has high potential in advanced manufacturing technologies and smart factories for the realization of Industry 4.0 [22, 47, 67, 68].
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This research is partially supported by the Taiwan Ministry of Science and Technology (Grant no. MOST 106-2218-E-007-012 -MY2).