ABSTRACT
Security of smart contracts has attracted increasing attention in recent years. Many researchers have devoted themselves to devising testing tools for vulnerability detection. Each published tool has demonstrated its effectiveness through a series of evaluations on their own experimental scenarios. However, the inconsistency of evaluation settings such as different data sets or performance metrics, may result in biased conclusion.
In this paper, based on an empirical evaluation of widely used smart contract testing tools, we propose a unified standard to eliminate the bias in the assessment process. First, we collect 46,186 source-available smart contracts from four influential organizations. This comprehensive dataset is open to the public and involves different code characteristics, vulnerability patterns and application scenarios. Then we propose a 4-step evaluation process and summarize the difference among relevant work in these steps. We use nine representative tools to carry out extensive experiments. The results demonstrate that different choices of experimental settings could significantly affect tool performance and lead to misleading or even opposite conclusions. Finally, we generalize some problems of existing testing tools, and propose some possible directions for further improvement.
- Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In 2011 33rd International Conference on Software Engineering (ICSE). 1–10.Google ScholarDigital Library
- Roberto Baldoni, Emilio Coppa, Daniele Cono D’elia, Camil Demetrescu, and Irene Finocchi. 2018. A survey of symbolic execution techniques. ACM Computing Surveys (CSUR), 51, 3 (2018), 1–39.Google ScholarDigital Library
- Lexi Brent, Neville Grech, Sifis Lagouvardos, Bernhard Scholz, and Yannis Smaragdakis. 2020. Ethainter: a smart contract security analyzer for composite vulnerabilities. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation. 454–469.Google ScholarDigital Library
- Lexi Brent, Anton Jurisevic, Michael Kong, Eric Liu, Francois Gauthier, Vincent Gramoli, Ralph Holz, and Bernhard Scholz. 2018. Vandal: A scalable security analysis framework for smart contracts. arXiv preprint arXiv:1809.03981.Google Scholar
- Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S Pasareanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: preliminary assessment. In 2011 33rd International Conference on Software Engineering (ICSE). 1066–1071.Google ScholarDigital Library
- Jialiang Chang, Bo Gao, Hao Xiao, Jun Sun, Yan Cai, and Zijiang Yang. 2019. sCompile: Critical path identification and analysis for smart contracts. In International Conference on Formal Engineering Methods. 286–304.Google ScholarDigital Library
- Jiachi Chen, Xin Xia, David Lo, John Grundy, Xiapu Luo, and Ting Chen. 2020. DEFECTCHECKER: Automated Smart Contract Defect Detection by Analyzing EVM Bytecode. arxiv:2009.02663.Google Scholar
- Jiachi Chen, Xin Xia, David Lo, John Grundy, Xiapu Luo, and Ting Chen. 2020. Defining Smart Contract Defects on Ethereum. IEEE Transactions on Software Engineering.Google ScholarDigital Library
- Ting Chen, Xiaoqi Li, Xiapu Luo, and Xiaosong Zhang. 2017. Under-optimized smart contracts devour your money. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). 442–446.Google ScholarCross Ref
- Ting Chen, Zihao Li, Yufei Zhang, Xiapu Luo, Ting Wang, Teng Hu, Xiuzhuo Xiao, Dong Wang, Jin Huang, and Xiaosong Zhang. 2019. A large-scale empirical study on control flow identification of smart contracts. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–11.Google ScholarCross Ref
- ConsenSys. 2018. Mythril. https://github.com/ConsenSys/mythril-classicGoogle Scholar
- A Day and E Medvedev. 2019. Ethereum in BigQuery: a public dataset for smart contract analytics.Google Scholar
- B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti, W. Robertson, F. Ulrich, and R. Whelan. 2016. LAVA: Large-Scale Automated Vulnerability Addition. In 2016 IEEE Symposium on Security and Privacy (SP). 110–121.Google Scholar
- Thomas Durieux, João F Ferreira, Rui Abreu, and Pedro Cruz. 2020. Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 530–541.Google ScholarDigital Library
- Etherscan. 2019. Etherscan. https://etherscan.io/Google Scholar
- J. Feist, G. Grieco, and A. Groce. 2019. Slither: A Static Analysis Framework for Smart Contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB). 8–15.Google Scholar
- Asem Ghaleb and Karthik Pattabiraman. 2020. How Effective Are Smart Contract Analysis Tools? Evaluating Smart Contract Static Analysis Tools Using Bug Injection. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2020). Association for Computing Machinery, New York, NY, USA. 415–427. isbn:9781450380089 https://doi.org/10.1145/3395363.3397385 Google ScholarDigital Library
- Google. 2018. Fuzzer Test Suite. https://github.com/google/fuzzer-test-suiteGoogle Scholar
- Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th International Conference on Software Engineering. 72–82.Google ScholarDigital Library
- Neville Grech, Lexi Brent, Bernhard Scholz, and Yannis Smaragdakis. 2019. Gigahorse: thorough, declarative decompilation of smart contracts. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 1176–1186.Google ScholarDigital Library
- Neville Grech, Michael Kong, Anton Jurisevic, Lexi Brent, Bernhard Scholz, and Yannis Smaragdakis. 2018. Madmax: Surviving out-of-gas conditions in ethereum smart contracts. Proceedings of the ACM on Programming Languages, 2, OOPSLA (2018), 1–27.Google ScholarDigital Library
- Ilya Grishchenko, Matteo Maffei, and Clara Schneidewind. 2018. A semantic framework for the security analysis of ethereum smart contracts. In International Conference on Principles of Security and Trust. 243–269.Google ScholarCross Ref
- Jingxuan He, Mislav Balunović, Nodar Ambroladze, Petar Tsankov, and Martin Vechev. 2019. Learning to fuzz from symbolic execution with application to smart contracts. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 531–548.Google ScholarDigital Library
- Bin Hu, Zongyang Zhang, Jianwei Liu, Yizhong Liu, Jiayuan Yin, Rongxing Lu, and Xiaodong Lin. 2020. A Comprehensive Survey on Smart Contract Construction and Execution: Paradigms, Tools and Systems. arXiv preprint arXiv:2008.13413.Google Scholar
- Sungjae Hwang and Sukyoung Ryu. 2020. Gap between theory and practice: An empirical study of security patches in solidity. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 542–553.Google ScholarDigital Library
- Bo Jiang, Ye Liu, and W. K. Chan. 2018. ContractFuzzer: fuzzing smart contracts for vulnerability detection. Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering - ASE 2018, isbn:9781450359375 https://doi.org/10.1145/3238147.3238177 Google ScholarDigital Library
- Sukrit Kalra, Seep Goel, Mohan Dhawan, and Subodh Sharma. 2018. ZEUS: Analyzing Safety of Smart Contracts.. In NDSS.Google Scholar
- George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating fuzz testing. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 2123–2138.Google ScholarDigital Library
- Jun Li, Bodong Zhao, and Chao Zhang. 2018. Fuzzing: a survey. Cybersecurity, 1, 1 (2018), 1–13.Google ScholarCross Ref
- Chao Liu, Han Liu, Zhao Cao, Zhong Chen, Bangdao Chen, and Bill Roscoe. 2018. Reguard: finding reentrancy bugs in smart contracts. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion). 65–68.Google ScholarDigital Library
- Han Liu, Chao Liu, Wenqi Zhao, Yu Jiang, and Jiaguang Sun. 2018. S-gram: towards semantic-aware security auditing for ethereum smart contracts. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 814–819.Google ScholarDigital Library
- Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016. Making Smart Contracts Smarter. IACR Cryptology ePrint Archive, 633.Google Scholar
- Fuchen Ma, Ying Fu, Meng Ren, Wanting Sun, Zhe Liu, Yu Jiang, Jun Sun, and Jiaguang Sun. 2019. Gasfuzz: Generating high gas consumption inputs to avoid out-of-gas vulnerability. arXiv preprint arXiv:1910.02945.Google Scholar
- Valentin Jean Marie Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering.Google Scholar
- Anastasia Mavridou and Aron Laszka. 2018. Tool Demonstration: FSolidM for Designing Secure Ethereum Smart Contracts. In Principles of Security and Trust, Lujo Bauer and Ralf Küsters (Eds.). Springer International Publishing, Cham. 270–277. isbn:978-3-319-89722-6Google Scholar
- MITRE. 2018. Common vulnerabilities and exposures. https://cve.mitre.org/Google Scholar
- Mark Mossberg, Felipe Manzano, Eric Hennenfent, Alex Groce, Gustavo Grieco, Josselin Feist, Trent Brunson, and Artem Dinaburg. 2019. Manticore: A user-friendly symbolic execution framework for binaries and smart contracts. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1186–1189.Google ScholarDigital Library
- MythX. 2019. Smart Contract Weakness Classification and Test Cases. https://swcregistry.io/ Accessed November 4, 2019.Google Scholar
- Tai D Nguyen, Long H Pham, Jun Sun, Yun Lin, and Quang Tran Minh. 2020. sFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts. arXiv preprint arXiv:2004.08563.Google Scholar
- Ivica Nikolić, Aashish Kolluri, Ilya Sergey, Prateek Saxena, and Aquinas Hobor. 2018. Finding the greedy, prodigal, and suicidal contracts at scale. In Proceedings of the 34th Annual Computer Security Applications Conference. 653–663.Google ScholarDigital Library
- Robert Norvill, Beltran Borja Fiz Pontiveros, Radu State, and Andrea Cullen. 2018. Visual emulation for Ethereum’s virtual machine. In NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium. 1–4.Google ScholarDigital Library
- Trail of Bits. 2018. ethersplay. https://github.com/crytic/ethersplayGoogle Scholar
- Reza M Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, and Amritraj Singh. 2018. Empirical vulnerability analysis of automated smart contracts security testing on blockchains. arXiv preprint arXiv:1809.02702.Google Scholar
- Daniel Perez and Benjamin Livshits. 2019. Smart contract vulnerabilities: Does anyone care? arXiv preprint arXiv:1902.06710.Google Scholar
- Anton Permenev, Dimitar Dimitrov, Petar Tsankov, Dana Drachsler-Cohen, and Martin Vechev. 2020. Verx: Safety verification of smart contracts. In 2020 IEEE Symposium on Security and Privacy, SP. 18–20.Google ScholarCross Ref
- Theofilos Petsios, Jason Zhao, Angelos D Keromytis, and Suman Jana. 2017. Slowfuzz: Automated domain-independent detection of algorithmic complexity vulnerabilities. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2155–2168.Google ScholarDigital Library
- Purathani Praitheeshan, Lei Pan, Jiangshan Yu, Joseph Liu, and Robin Doss. 2019. Security analysis methods on Ethereum smart contract vulnerabilities: a survey. arXiv preprint arXiv:1908.08605.Google Scholar
- Raine Revere. 2018. solgraph. https://github.com/raineorshine/solgraphGoogle Scholar
- Sunbeom So, Myungho Lee, Jisu Park, Heejo Lee, and Hakjoo Oh. 2020. VeriSmart: A highly precise safety verifier for Ethereum smart contracts. In 2020 IEEE Symposium on Security and Privacy (SP). 1678–1694.Google ScholarCross Ref
- Matt Suiche. 2017. Porosity: A decompiler for blockchain-based smart contracts bytecode. DEF con, 25 (2017), 11.Google Scholar
- Sergei Tikhomirov, Ekaterina Voskresenskaya, Ivan Ivanitskiy, Ramil Takhaviev, and Yaroslav Alexandrov. 2018. SmartCheck: static analysis of ethereum smart contracts. In the 1st International Workshop.Google ScholarDigital Library
- Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, and Radu State. 2020. Towards Smart Hybrid Fuzzing for Smart Contracts. arXiv preprint arXiv:2005.12156.Google Scholar
- Christof Ferreira Torres, Julian Schütte, and Radu State. 2018. Osiris: Hunting for integer bugs in ethereum smart contracts. In Proceedings of the 34th Annual Computer Security Applications Conference. 664–676.Google ScholarDigital Library
- Christof Ferreira Torres and Mathis Steichen. 2019. The art of the scam: Demystifying honeypots in ethereum smart contracts. In 28th $USENIX$ Security Symposium ($USENIX$ Security 19). 1591–1607.Google Scholar
- Petar Tsankov, Andrei Marian Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Buenzli, and Martin T. Vechev. 2018. Securify: Practical Security Analysis of Smart Contracts. In ACM Conference on Computer and Communications Security.Google Scholar
- Patrick Ventuzelo. 2018. Octopus. https://github.com/pventuzelo/octopusGoogle Scholar
- Mingzhe Wang, Jie Liang, Chijin Zhou, Yuanliang Chen, Zhiyong Wu, and Yu Jiang. [n.d.]. Industrial Oriented Evaluation of Fuzzing Techniques.Google Scholar
- W. Wang, J. Song, G. Xu, Y. Li, H. Wang, and C. Su. 2020. ContractWard: Automated Vulnerability Detection Models for Ethereum Smart Contracts. IEEE Transactions on Network Science and Engineering, 1–1.Google Scholar
- Valentin Wüstholz and Maria Christakis. 2019. Harvey: A greybox fuzzer for smart contracts. arXiv preprint arXiv:1905.06944.Google Scholar
- Jiaming Ye, Mingliang Ma, Yun Lin, Yulei Sui, and Yinxing Xue. 2020. Clairvoyance: Cross-Contract Static Analysis for Detecting Practical Reentrancy Vulnerabilities in Smart Contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings (ICSE ’20). Association for Computing Machinery, New York, NY, USA. 274–275. isbn:9781450371223 https://doi.org/10.1145/3377812.3390908 Google ScholarDigital Library
- Pengcheng Zhang, Feng Xiao, and Xiapu Luo. 2019. SolidityCheck: Quickly Detecting Smart Contract Problems Through Regular Expressions. arXiv preprint arXiv:1911.09425.Google Scholar
- Ence Zhou, Song Hua, Bingfeng Pi, Jun Sun, Yashihide Nomura, Kazuhiro Yamashita, and Hidetoshi Kurihara. 2018. Security assurance for smart contract. In 2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS). 1–5.Google ScholarCross Ref
- Yi Zhou, Deepak Kumar, Surya Bakshi, Joshua Mason, Andrew Miller, and Michael Bailey. 2018. Erays: reverse engineering ethereum’s opaque smart contracts. In 27th $USENIX$ Security Symposium ($USENIX$ Security 18). 1371–1385.Google Scholar
- Xiaogang Zhu, Xiaotao Feng, Tengyun Jiao, Sheng Wen, Yang Xiang, Seyit Camtepe, and Jingling Xue. 2019. A feature-oriented corpus for understanding, evaluating and improving fuzz testing. In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security. 658–663.Google ScholarDigital Library
Index Terms
- Empirical evaluation of smart contract testing: what is the best choice?
Recommendations
ReSuMo: a regression strategy and tool for mutation testing of solidity smart contracts
AbstractMutation testing is a powerful test adequacy assessment technique that can guarantee the deployment of deeply scrutinized and more reliable Smart Contract code. However, regularly evaluating the test suite during the evolution of a project can be ...
Smart contract testing: challenges and opportunities
WETSEB '22: Proceedings of the 5th International Workshop on Emerging Trends in Software Engineering for BlockchainBlockchain technologies have found important and concrete applications in the real world. Active solutions leverage Smart Contracts for the management of cryptocurrencies, sensitive data, and other valuable assets. One of the core objectives of ...
Can Neural Networks Help Smart Contract Testing? An Empirical Study
Internetware '23: Proceedings of the 14th Asia-Pacific Symposium on InternetwareSmart contracts are one of the most successful applications of blockchain technology. In order to guarantee the security of smart contracts, researchers have successively introduced various testing methodologies, including static analysis, symbolic ...
Comments