Mekky, Hesham2016-12-192016-12-192016-09https://hdl.handle.net/11299/183348University of Minnesota Ph.D. dissertation. September 2016. Major: Computer Science. Advisor: Zhi-Li Zhang. 1 computer file (PDF); x, 104 pages.The popularity of online services, such as social networks and online banking, has made them a popular platform for attackers. Cybercriminals leverage them to spread malicious software (malware) and steal personal information. In a cybercriminal operation, miscreants infect their victims'’ machines with malware that performs malicious activities. This occurs due to poor security measures implemented by enterprise networks, and the complexity of network management tools. By studying existing malware distribution networks and enterprise networks management tools, we aim to understand the techniques used to infect victims such as drive-by downloads, study malware families and design better detection methodologies, and seek solutions towards an improved network management framework. Towards these goals, this thesis studies three orthogonal problems aiming at addressing security and management problems in modern networks. First, we study malware infections due to drive-by downloads using a large ISP dataset. We show that attackers employ redirections which automatically redirect users' requests through a series of intermediate websites, before landing on the final distribution site. To detect these malicious redirections, we developed a machine learning framework that relies on a distinctive set of features to label the malicious redirections and block them. Second, we study malware network traces for infected hosts in an enterprise network using real malware traces, and we show that malware traffic comes mixed up with legitimate user traffic such as browsing traffic. To improve malware detection, we developed a novel system that decomposes the traffic into separate components, and applies the detection system to the suspected malware component only, and consequently improve detection rates. Third, we postulate native network functions within the Software-defined Network (SDN) data plane, where the same logical controller controls both network services and routing. This is enabled by extending Software-defined Networking to support stateful flow handling based on higher layers in the packet beyond layers 2-4. As a result, network functions (a.k.a middleboxes) can be chained on demand, directly on the data plane. We present an implementation of this architecture based on Open vSwitch, and show that it enables popular network functions effectively and addresses the management problems in enterprise networks. In summary, this thesis addresses these three closely related problems by: (1) protecting enterprise networks from drive-by downloads launched using redirections via a data-driven approach; (2) detecting existing malware activity on the network by decomposing the end-host traffic into a benign component and a suspected malware component, then classifying the malware into its malware family; and (3) building a flexible network architecture that enables managing network functions (e.g. such as systems in (1) and (2) and others like firewalls and load balancers) within the data plane along with the routing using a unified control plane.enMalwareNetworkingNFVSDNSecurityThesis or Dissertation