3. Automatic Detection and Validation of Smart City Events Using HPC and Apache Spark Platforms

High performance computing (HPC), big data, and artificial intelligence (AI) technologies are playing a key role in enabling smart society systems to sense the cities and other environments at micro-levels, detecting events, making intelligent decisions, and taking appropriate actions, all within stringent time bounds. Social media have revolutionized our societies and are gradually becoming a key pulse of smart societies by sensing the information about the people and their spatio-temporal experiences around the living spaces. The aim of this work is to develop data management and analysis techniques for smart societies. Specifically, we use big data, machine learning, and other platforms including Spark, MLlib, Tableau, and Google Maps Geocoding API, to study Twitter data for the detection and validation of spatio-temporal events in London. We empirically demonstrate that physical, virtual, and conceptual events can be detected automatically by analyzing data. We find and locate congestion around London. We detect the occurrence of multiple events including “Underbelly Festival,” “The Luna Cinema” and “London Notting Hill Carnival 2017,” their locations and times, without any prior knowledge of the events. An architecture of our big data analytics tool based on Spark for the detection of spatio-temporal events is provided along with the details of the main system components using six algorithms. The event detection pipeline has been enhanced using a methodology to automatically validate the factuality of the detected events. We also provide a comparison of three machine learning methods, support vector machine, logistic regression, and Naïve Bayes for event detection.