Filtering for Malice Through the Data Ocean: Large-Scale PHA Install Detection at the Communication Service Provider Level

As a key stakeholder in mobile communications, the communication service provider (CSP, including carriers and ISPs) plays a critical role in safeguarding mobile users against potentially-harmful apps (PHA), complementing the security protection at app stores. However a CSP-level scan faces an enormous challenge: hundreds of millions of apps are installed everyday; retaining their download traffic to construct their packages entails a huge burden on the CSP side, forces them to change their infrastructure and can have serious privacy and legal ramifications. To control the cost and avoid trouble, today’s CSPs acquire apps from download URLs for a malware analysis. Even this step is extremely expensive and hard to meet the demand of online protection: for example, a CSP we are working with runs hundreds of machines to check the daily downloads it observes. To rise up to this challenge, we present in this paper an innovative “app baleen” (called Abaleen) framework for an on-line security vetting of an extremely large number of app downloads, through a high-performance, concurrent inspection of app content from the sources of the downloads. At the center of the framework is the idea of retrieving only a small amount of the content from the remote sources to identify suspicious app downloads and warn the end users, hopefully before the installation is complete. Running on 90 million download URLs recorded by our CSP partner, our screening framework achieves an unparalleled performance, with a nearly 85\(\times \) speed-up compared to the existing solution. This level of performance enables an online vetting for PHAs at the CSP scale: among all unique URLs used in our study, more than 95% were processed before the completion of unfettered downloads. With the CSP-level dataset, we revealed not only the surprising pervasiveness of PHAs, but also the real impact of them (over 2 million installs in merely 3 days).