Data-aware optimizations for efficient analytics over geo-distributed data

Abstract

Modern applications process large volumes of data, which are analyzed and used to improve user experience, guide business decisions, and drive innovation. This data is increasingly generated and persisted across multiple geographical locations, which presents several challenges for traditional analytics systems designed to operate in a centralized fashion. First, the wide-area network links that connect these locations can be exceedingly slow and less reliable than high-speed networks in the cloud. Second, the ubiquity of smart devices has driven an increase in the volume of data available for processing. Third, several application domains process real-time data and require low-latency responses for queries. Finally, data sovereignty laws can further constrain the ability to transfer data for analysis. New approaches are required to deliver low-latency analytics over large data volumes distributed across the country or the globe. This thesis proposes utilizing data-awareness--the ability to observe properties of the data being processed--to improve the efficiency of geo-distributed analytics systems. If the system has some knowledge of the underlying data distributions, queries can be optimized to reduce latency and improve resource efficiency. For example, we show that identifying devices with similar data distributions can accelerate model training in federated learning. Knowledge of data similarity across geo-distributed data sources can also be exploited to improve wide-area network efficiency and reduce query latency. We also show that inference serving systems can produce higher-accuracy inferences by dynamically selecting a model based on the data. By considering the properties of geo-distributed data sources, systems can optimally navigate trade-offs between resource usage, accuracy, and latency.