A Sequence-Oriented Stream Warehouse Paradigm for Network Monitoring Applications

Network administrators are faced with the increasingly challenging task of monitoring their network’s health in real time, drawing upon diverse and voluminous measurement data feeds and extensively mining them. The role of database systems in network monitoring has traditionally been that of data repositories; even if an application uses a database, the application logic is implemented using external programs. While such programs are flexible, they tend to be ad-hoc, opaque, inefficient and hard to maintain over time. In this paper, we propose a new way of implementing network monitoring applications: directly within a database as continually updated tables defined using a declarative query language (SQL). We also address a crucial technical issue with realizing this approach: SQL was designed for set-oriented data transformations, but network monitoring involves sequence-oriented analysis. To solve this problem, we propose an extension to SQL that makes sequence-oriented analysis easier to express and faster to evaluate. Using a prototype implementation in a large-scale production data warehouse, we demonstrate how the declarative sequence-oriented query language simplifies application development and how the associated system optimizations improve application performance.