ABSTRACT
In recent years, researchers have developed a number of tools to conduct taint analysis of Android applications. While all the respective papers aim at providing a thorough empirical evaluation, comparability is hindered by varying or unclear evaluation targets. Sometimes, the apps used for evaluation are not precisely described. In other cases, authors use an established benchmark but cover it only partially. In yet other cases, the evaluations differ in terms of the data leaks searched for, or lack a ground truth to compare against. All those limitations make it impossible to truly compare the tools based on those published evaluations.
We thus present ReproDroid, a framework allowing the accurate comparison of Android taint analysis tools. ReproDroid supports researchers in inferring the ground truth for data leaks in apps, in automatically applying tools to benchmarks, and in evaluating the obtained results. We use ReproDroid to comparatively evaluate on equal grounds the six prominent taint analysis tools Amandroid, DIALDroid, DidFail, DroidSafe, FlowDroid and IccTA. The results are largely positive although four tools violate some promises concerning features and accuracy. Finally, we contribute to the area of unbiased benchmarking with a new and improved version of the open test suite DroidBench.
- Maqsood Ahmad, Valerio Costamagna, Bruno Crispo, and Francesco Bergadano. 2017. TeICC: targeted execution of inter-component communications in Android. In SAC, Marrakech, Morocco, 2017, Ahmed Seffah, Birgit Penzenstadler, Carina Alves, and Xin Peng (Eds.). ACM, 1747–1752. Google ScholarDigital Library
- Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick D. McDaniel. 2014.Google Scholar
- FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps. In PLDI, Edinburgh, United Kingdom, 2014, Michael F. P. O’Boyle and Keshav Pingali (Eds.). ACM, 259–269. Google ScholarDigital Library
- Kathy Wain Yee Au, Yi Fan Zhou, Zhen Huang, and David Lie. 2012. PScout: analyzing the Android permission specification. In CCS, Raleigh, USA, 2012, Ting Yu, George Danezis, and Virgil D. Gligor (Eds.). ACM, 217–228. Google ScholarDigital Library
- Hamid Bagheri, Alireza Sadeghi, Reyhaneh Jabbarvand Behrouz, and Sam Malek. 2016. Practical, Formal Synthesis and Automatic Enforcement of Security Policies for Android. In DSN, Toulouse, France, 2016. IEEE Computer Society, 514–525.Google Scholar
- Dirk Beyer. 2017. Software Verification with Validation of Results - (Report on SV-COMP 2017). In TACAS (ETAPS), Uppsala, Sweden, 2017 (LNCS), Axel Legay and Tiziana Margaria (Eds.), Vol. 10206. 331–349. Google ScholarDigital Library
- Armin Biere, Tom van Dijk, and Keijo Heljanko. 2017. Hardware model checking competition 2017. In FMCAD, Vienna, Austria, 2017, Daryl Stewart and Georg Weissenbacher (Eds.). IEEE, 9. Google ScholarDigital Library
- Amiangshu Bosu, Fang Liu, Danfeng (Daphne) Yao, and Gang Wang. 2017. Collusive Data Leak and More: Large-scale Threat Analysis of Inter-app Communications. In AsiaCCS, Abu Dhabi, United Arab Emirates, 2017, Ramesh Karri, Ozgur Sinanoglu, Ahmad-Reza Sadeghi, and Xun Yi (Eds.). ACM, 71–85. Google ScholarDigital Library
- Stefano Calzavara, Ilya Grishchenko, and Matteo Maffei. 2016. HornDroid: Practical and Sound Static Analysis of Android Applications by SMT Solving. In EuroS&P, Saarbrücken, Germany, 2016. IEEE, 47–62.Google ScholarCross Ref
- Xingmin Cui, Jingxuan Wang, Lucas Chi Kwong Hui, Zhongwei Xie, Tian Zeng, and Siu-Ming Yiu. 2015. WeChecker: efficient and precise detection of privilege escalation vulnerabilities in Android apps. In WiSec, New York, USA, 2015. ACM, 25:1–25:12. Google ScholarDigital Library
- William Enck, Peter Gilbert, Byung-Gon Chun, Landon P. Cox, Jaeyeon Jung, Patrick D. McDaniel, and Anmol Sheth. 2010. TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. In OSDI, Vancouver, Canada, 2010, Remzi H. Arpaci-Dusseau and Brad Chen (Eds.). USENIX Association, 393–407. Google ScholarDigital Library
- Yu Feng, Isil Dillig, Saswat Anand, and Alex Aiken. 2014. Apposcopy: automated detection of Android malware (invited talk). In DeMobile, Hong Kong, China, 2014, Aharon Abadi, Rafael Prikladnicki, and Yael Dubinsky (Eds.). ACM, 13–14. Google ScholarDigital Library
- Gartner. 2017. Gartner Says Worldwide Sales of Smartphones Grew 9 Percent in First Quarter of 2017.Google Scholar
- https://www.gartner.com/newsroom/id/3725117.Google Scholar
- Michael I. Gordon, Deokhwan Kim, Jeff H. Perkins, Limei Gilham, Nguyen Nguyen, and Martin C. Rinard. 2015. Information Flow Analysis of Android Applications in DroidSafe. In NDSS, San Diego, USA, 2015. The Internet Society.Google Scholar
- Wei Huang, Yao Dong, Ana Milanova, and Julian Dolby. 2015. Scalable and precise taint analysis for Android. In ISSTA, Baltimore, USA, 2015, Michal Young and Tao Xie (Eds.). ACM, 106–117. Google ScholarDigital Library
- William Klieber, Lori Flynn, Amar Bhosale, Limin Jia, and Lujo Bauer. 2014. Android taint flow analysis for app sets. In SOAP, Edinburgh, UK, 2014, Steven Arzt and Raúl A. Santelices (Eds.). ACM, 5:1–5:6. Google ScholarDigital Library
- Patrick Lam, Eric Bodden, Ondřej Lhoták, and Laurie Hendren. 2011. The Soot framework for Java program analysis: a retrospective. In Cetus Users and Compiler Infrastructure Workshop (CETUS). http://www.bodden.de/pubs/lblh11soot.pdfGoogle Scholar
- Li Li, Alexandre Bartel, Tegawendé F. Bissyandé, Jacques Klein, Yves Le Traon, Steven Arzt, Siegfried Rasthofer, Eric Bodden, Damien Octeau, and Patrick D. McDaniel. 2015. IccTA: Detecting Inter-Component Privacy Leaks in Android Apps. In ICSE, Florence, Italy, 2015, Antonia Bertolino, Gerardo Canfora, and Sebastian G. Elbaum (Eds.). IEEE Computer Society, 280–291. Google ScholarDigital Library
- Li Li, Tegawendé F. Bissyandé, Mike Papadakis, Siegfried Rasthofer, Alexandre Bartel, Damien Octeau, Jacques Klein, and Yves Le Traon. 2017. Static analysis of android apps: A systematic literature review. Information & Software Technology 88 (2017), 67–95. Google ScholarDigital Library
- Adam P Fuchs, Avik Chaudhuri, and Jeffrey S Foster. 2009. SCanDroid: Automated security certification of Android applications. Technical report, University of Maryland (2009).Google Scholar
- Felix Pauck. 2017. Cooperative static analysis of Android applications. Master’s thesis. Paderborn University, Germany.Google Scholar
- Lina Qiu, Yingying Wang, and Julia Rubin. 2018. Analyzing the Analyzers: FlowDroid/IccTA, AmanDroid, and DroidSafe. In ISSTA, Amsterdam, Netherlands, 2018. Google ScholarDigital Library
- Siegfried Rasthofer, Steven Arzt, and Eric Bodden. 2014. A Machine-learning Approach for Classifying and Categorizing Android Sources and Sinks. In NDSS, San Diego, USA, 2014. The Internet Society.Google ScholarCross Ref
- Bradley Reaves, Jasmine Bowers, Sigmund Albert Gorski III, Olabode Anise, Rahul Bobhate, Raymond Cho, Hiranava Das, Sharique Hussain, Hamza Karachiwala, Nolen Scaife, Byron Wright, Kevin R. B. Butler, William Enck, and Patrick Traynor. 2016. *droid: Assessment and Evaluation of Android Application Analysis Tools. ACM Comput. Surv. 49, 3 (2016), 55:1–55:30. Google ScholarDigital Library
- Henry Gordon Rice. 1953. Classes of recursively enumerable sets and their decision problems. Trans. Amer. Math. Soc. 74, 2 (1953), 358–366.Google ScholarCross Ref
- Alireza Sadeghi, Hamid Bagheri, Joshua Garcia, and Sam Malek. 2017. A Taxonomy and Qualitative Comparison of Program Analysis Techniques for Security Assessment of Android Software. IEEE Trans. Software Eng. 43, 6 (2017), 492–530.Google ScholarDigital Library
- Yannis Smaragdakis, Martin Bravenboer, and Ondrej Lhoták. 2011. Pick your contexts well: understanding object-sensitivity. In POPL, Austin, USA, 2011, Thomas Ball and Mooly Sagiv (Eds.). ACM, 17–30. Google ScholarDigital Library
- Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie J. Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java bytecode optimization framework. In CASCON, 1999, Mississauga, Canada, Stephen A. MacKay and J. Howard Johnson (Eds.). IBM, 13. Google ScholarDigital Library
- Fengguo Wei, Sankardas Roy, Xinming Ou, and Robby. 2014. Amandroid: A Precise and General Inter-component Data Flow Analysis Framework for Security Vetting of Android Apps. In SIGSAC, Scottsdale, USA, 2014, Gail-Joon Ahn, Moti Yung, and Ninghui Li (Eds.). ACM, 1329–1341. Google ScholarDigital Library
Index Terms
- Do Android taint analysis tools keep their promises?
Recommendations
Scalable and precise taint analysis for Android
ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and AnalysisWe propose a type-based taint analysis for Android. Concretely, we present DFlow, a context-sensitive information flow type system, and DroidInfer, the corresponding type inference analysis for detecting privacy leaks in Android apps. We present novel ...
Together strong: cooperative Android app analysis
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringRecent years have seen the development of numerous tools for the analysis of taint flows in Android apps. Taint analyses aim at detecting data leaks, accidentally or by purpose programmed into apps. Often, such tools specialize in the treatment of ...
The impact of tool configuration spaces on the evaluation of configurable taint analysis for Android
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and AnalysisThe most popular static taint analysis tools for Android allow users to change the underlying analysis algorithms through configuration options. However, the large configuration spaces make it difficult for developers and users alike to understand the ...
Comments