Building a Data Pipeline for the Management and Processing of Urban Data Streams

Urban data streams (UDS) originate from various sensors and Internet of Things (IoT) devices deployed in smart cities as well as social media sources such as Twitter and Facebook. The large volumes of urban data need to be harnessed to help smart city stakeholders and applications make informed decisions on the fly. Furthermore, effective management and governance of smart city components relies on the ability to integrate and federate their data, process urban data streams locally, and use big data analytics. Data integration and interoperability is a challenging problem that smart cities are facing today. Successful data integration is crucial for improved services and governance. This chapter describes a framework that aims to serve in building a data pipeline for the acquisition and processing of urban data streams, urban data analytics, and creation of value-added services. The framework relies on latest technologies for data processing including IoT, edge computing, data integration techniques, cloud computing, and data analytics. The proposed platform will facilitate real-time event detection, notification of alerts, mining the opinions of citizens regarding the governance of their city, and building monitoring dashboards. A prototype of the platform is being implemented using the Kafka messaging platform.