Authors : Porfirio Tramontana , Domenico Amalfitano , Nicola Amatucci , Anna Rita Fasolino Authors Info & Claims
Pages 149 - 201 Published : 01 March 2019 Publication History 23 citation 0 Downloads Total Citations 23 Total Downloads 0 Last 12 Months 0 Last 6 weeks 0 Get Citation AlertsThis alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Context Testing is a critical and costly activity in the life cycle of a mobile application, due to the growing request of new applications and to the rapid evolution of mobile devices and frameworks. Testing automation may represent an effective solution to improve the quality of mobile applications and to reduce testing costs. Objective We have performed a systematic mapping study to find, analyze, and classify papers in the scientific literature that are related to the automation of functional testing of mobile applications with the aim to provide a classification scheme useful for researchers and practitioners to have a clear view of the state of the art and to easily find existing solutions to their issues. Method We have conducted the study on the basis of a set of 18 research questions. Search queries have been formulated and applied to 7 search engines and the resulting papers have been filtered by considering sets of inclusion and exclusion criteria. The selected papers have been systematically classified and, in addition, a bibliometric analysis has been performed. Results A systematic map including 131 papers has been obtained and is publicly available. The papers have been classified on the basis of the supported testing activities, the characteristics of the techniques and tools they present, and the evaluation methodologies adopted to validate them. The bibliometric analysis has allowed the identification of the most active researchers, the most attractive venues, and the most influential papers. Conclusions The analysis of the systematic mapping has allowed the identification of some research trends and gaps in this field of study. For example, we have observed a strong prevalence of Android-based approaches, a lack of contributions from industry, and the absence of specific venues and journals focused on mobile testing automation.
Adamsen, C.Q., Mezzetti, G., Moller, A. (2015). Systematic execution of android test suites in adverse conditions. In Proceedings of the 2015 International Symposium on Software Testing and, ISSTA 2015 (pp. 83-93). New York: ACM.
Ahmad, A., Li, K., Feng, C., Asim, S.M., Yousif, A., Ge, S. (2018). An empirical study of investigating mobile applications development challenges. IEEE Access, 6, 17:711-17:728.
Akanksha Ashok Magare, M.D.L. (2016). Automated gui testing for android application. Imperial Journal of Interdisciplinary Research, 2(8), 884-888.
Amalfitano, D., Fasolino, A.R., Tramontana, P. (2011). A gui crawling-based technique for android mobile application testing. In ICST Workshops (pp. 252-261).
Amalfitano, D., Fasolino, A., Tramontana, P., De Carmine, S., Imparato, G. (2012a). A toolset for gui testing of android applications (pp. 650-653). cited By 16.
Amalfitano, D., Fasolino, A.R., Carmine, S.D., Memon, A., Tramontana, P. (2012b). Using gui ripping for automated testing of android applications. In ASE '12: Proceedings of the 27th IEEE international conference on Automated software engineering. DC: IEEE Computer Society Washington.
Amalfitano, D., Fasolino, A., Tramontana, P., Amatucci, N. (2013a). Considering context events in event-based testing of mobile applications. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 126-133).
Amalfitano, D., Fasolino, A., Tramontana, P., Robbins, B. (2013b). Testing android mobile applications: Challenges, strategies, and approaches. Advances in Computers, 89, 1-52.
Amalfitano, D., Amatucci, N., Fasolino, A., Tramontana, P. (2015a). Agrippin: A novel search based testing technique for android applications (pp. 5-12), cited By 1.
Amalfitano, D., Amatucci, N., Fasolino, A.R., Tramontana, P. (2015b). A conceptual framework for the comparison of fully automated gui testing techniques. In Sixth International Workshop on Testing Techniques for Event BasED Software (TESTBEDS) (pp. 2015).
Amalfitano, D., Amatucci, N., Fasolino, A.R., Tramontana, P., Kowalczyk, E., Memon, A. (2015c). Exploiting the saturation effect in automatic random testing of android applications. In The Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) (pp. 2015).
Amalfitano, D., Fasolino, A., Tramontana, P., Ta, B., Memon, A. (2015d). Mobiguitar: Automated model-based testing of mobile apps. IEEE Software, 32(5), 53-59. cited By 29.
Amalfitano, D., Amatucci, N., Memon, A., Tramontana, P., Fasolino, A. (2017). A general framework for comparing automatic testing techniques of android mobile apps. Journal of Systems and Software, 125, 322-343. cited By 0.
Amatucci, N. (2016). Automated gui testing techniques for android applications. PhD thesis, Ph.D. course in Information Technology and Electrical Engineering.
Anand, S., Naik, M., Yang, H., Harrold, M. (2012). Automated concolic testing of smartphone apps. No GIT-CERCS-12-02.
Anbunathan, R., & Basu, A. (2015). A recursive crawler algorithm to detect crash in android application (pp. 256-267), cited By 0.
Anbunathan, R., & Basu, A. (2016a). Automatic test generation from uml sequence diagrams for android mobiles. International Journal of Applied Engineering Research, 11(7), 4961-4970. cited By 0.
Anbunathan, R., & Basu, A. (2016b). Data driven architecture based automated test generation for android mobile. cited By 0.
Arnatovich, Y.L., Ngo, M.N., Kuan, T.H.B., Soh, C. (2016). Achieving high code coverage in android ui testing via automated widget exercising. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC) (pp. 193-200),
Arzensek, B., & Hericko, M. (2014). Criteria for selecting mobile application testing tools (vol. 1266, pp. 1-8), cited By 0.
Azim, T., & Neamtiu, I. (2013). Targeted and depth-first exploration for systematic testing of android apps. SIGPLAN Not, 48(10), 641-660.
Baek, Y.M., & Bae, D.H. (2016). Automated model-based android gui testing using multi-level gui comparison criteria (pp. 238-249), cited By 0.
Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S. (2015). The oracle problem in software testing: a survey. IEEE Transactions on Software Engineering, 41(5), 507-525.
Basili, V.R., Caldiera, G., Rombach, H.D. (1994). The goal question metric approach. In Encyclopedia of Software Engineering. Wiley.
Behrouz, R.J., Sadeghi, A., Garcia, J., Malek, S., Ammann, P. (2015). Ecodroid: An approach for energy-based ranking of android apps. In 2015 IEEE/ACM 4th International Workshop on Green and Sustainable Software (pp. 8-14).
Bielik, P., Raychev, V., Vechev, M. (2015). Scalable race detection for android applications (vol. 25-30-Oct- 2015, pp. 332-348). cited By 5.
Canfora, G., Mercaldo, F., Visaggio, C.A., DAngelo, M., Furno, A., Manganelli, C. (2013). A case study of automating user experience-oriented performance testing on smartphones. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (pp. 66-69).
Choi, W., Necula, G., Sen, K. (2013). Guided gui testing of android apps with minimal restart and approximate learning. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages and applications (pp. 623-640). ACM.
Choudhary, S.R., Gorla, A., Orso, A. (2015). Automated test input generation for android: Are we there yet? (e). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 429-440).
Coelho, T., Lima, B., Faria, J.P. (2016). Mt4a: a no-programming test automation framework for android applications. In Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation (pp. 59-65), A-TEST 2016. New York: ACM.
Corral, L., Sillitti, A., Succi, G. (2015). Software assurance practices for mobile applications. Computing, 97(10), 1001-1022.
Costa, P., Paiva, A., Nabuco, M. (2014). Pattern based gui testing for mobile applications (pp 66-74). cited By 6.
Crispin, L., & Gregory, J. (2009). Agile Testing: A Practical Guide for Testers and Agile Teams, 1st edn. Addison-Wesley Professional.
de Sousa Santos, I., de Castro Andrade, R.M., Rocha, L.S., Matalonga, S., de Oliveira, K.M., Travassos, G.H. (2017). Test case design for context-aware applications: Are we there yet? Information and Software Technology (pp. -).
De Cleva Farto, G., & Endo, A. (2015). Evaluating the model-based testing approach in the context of mobile applications. Electronic Notes in Theoretical Computer Science, 314, 3-21. cited By 2.
Delamaro, M., Vincenzi, A., Maldonado, J. (2006). A strategy to perform coverage testing of mobile applications. In Proceedings of the 2006 international workshop on Automation of software test (pp. 118-124). ACM.
Dev, R., Jaaskelainen, A., Katara, M. (2012). Model-based gui testing. case smartphone camera and messaging development. Advances in Computers, 85, 65-122. cited By 2.
Do, Q., Yang, G., Che, M., Hui, D., Ridgeway, J. (2016). Regression test selection for android applications (pp. 27-28). cited By 0.
Dubinsky, Y., & Abadi, A. (2013). Challenges and research questions for testing in mobile development: Report on a mobile testing activity. In Proceedings of the 2013 ACM Workshop on Mobile Development Lifecycle, MobileDeLi '13 (pp. 37-38). New York: ACM.
Dutia, S.N., Oh, T.H., Oh, Y.H. (2015). Developing automated input generator for android mobile device to evaluate malware behavior. In Proceedings of the 4th Annual ACM Conference on Research in Information Technology, RIIT '15 (pp. 43-43). New York: ACM.
Fazzini, M., Freitas, E.N.D.A., Choudhary, S.R., Orso, A. (2017). Barista: A technique for recording, encoding, and running platform independent android tests. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST) (pp. 149-160).
Gao, J., Bai, X., Tsai, W.T., Uehara, T. (2014). Mobile application testing: a tutorial. Computer, 47(2), 46-55.
Garousi, V., Mesbah, A., Betin-Can, A., Mirshokraie, S. (2013). A systematic mapping study of web application testing. Information and Software Technology, 55(8), 1374-1396.
Garousi, V., & Mantyla, M.V. (2016). When and what to automate in software testing? a multi-vocal literature review. Information and Software Technology, 76, 92-117.
Gomez, L., Neamtiu, I., Azim, T., Millstein, T. (2013). Reran: Timing- and touch-sensitive record and replay for android. In Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, Piscataway, NJ, USA, ICSE '13 (pp. 72-81).
Gomez, M. (2015). Debugging of Mobile Apps in the Wild Guided by the Wisdom of the Crowd. In 2nd International Conference on Mobile Software Engineering and Systems - ACM Student Reasearch Competition. Florence.
Gomez, M., Rouvoy, R., Adams, B., Seinturier, L. (2016). Reproducing context-sensitive crashes of mobile apps using crowdsourced monitoring (pp. 88-99). cited By 1.
Griebe, T., & Gruhn, V. (2014). A model-based approach to test automation for context-aware mobile applications. In Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC '14 (pp. 420-427). New York: ACM.
Griebe, T., Hesenius, M., Gruhn, V. (2015). Towards automated ui-tests for sensor-based mobile applications. Communications in Computer and Information Science, 532, 3-17. cited By 0.
Griebe, T., Hesenius, M., Gesthuesen, M., Gruhn, V. (2016). Test Automation for Speech-Based Applications. In Fujita, H., & Papadopoulos, G.A. (Eds.) New Trends in Software Methodologies, Tools and Techniques, Frontiers in Artificial Intelligence and Applications, (Vol. 286 pp. 85-100). 15th International Conference on New Trends in Intelligent Software Methodology Tools, and Techniques (SoMeT), CYPRUS, SEP 13-15, 2016.
Gudmundsson, V., Lindvall, M., Aceto, L., Bergthorsson, J., Ganesan, D. (2016). Model-based testing of mobile systems - an empirical study on quizup android app (vol. 208, pp. 16-30). cited By 0.
Hao, S., Liu, B., Nath, S., Halfond, W.G., Govindan, R. (2014). Puma: Programmable ui-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '14 (pp. 204-217). New York: ACM.
Harrison, R., Flood, D., Duce, D. (2013). Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interaction Science, 1(1), 1.
Hesenius, M., Griebe, T., Gries, S., Gruhn, V. (2014). Automating ui tests for mobile applications with formal gesture descriptions (pp. 213-222). cited By 4.
Holl, K., & Elberzhager, F. (2016). Quality assurance of mobile applications: A systematic mapping study. In Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, MUM '16 (pp 101-113). New York: ACM.
Hsiao, C.H., Pereira, C., Yu, J., Pokam, G., Narayanasamy, S., Chen, P., Kong, Z., Flinn, J. (2014). Race detection for event-driven mobile applications. ACM SIGPLAN Notices, 49(6), 326-336. cited By 3.
Hu, C., & Neamtiu, I. (2011a). Automating gui testing for android applications. In Proceedings of the 6th International Workshop on Automation of Software Test (pp. 77-83).
Hu, C., & Neamtiu, I. (2011b). A gui bug finding framework for android applications (pp. 1490-1491). cited By 6.
Hu, G., Yuan, X., Tang, Y., Yang, J. (2014). Efficiently, effectively detecting mobile app bugs with appdoctor. In Proceedings of the Ninth European Conference on Computer Systems, EuroSys '14 (pp. 18:1-18:15). New York: ACM.
Hu, Y., & Neamtiu, I. (2016). Fuzzy and cross-app replay for smartphone apps (pp. 50-56), cited By 0.
Hu, Y., Azim, T., Neamtiu, I. (2015). Versatile yet lightweight record-and-replay for android (vol. 25-30-Oct- 2015, pp. 349-366), cited By 4.
Hu, Y., Neamtiu, I., Alavi, A. (2016). Automatically verifying and reproducing event-based races in android apps. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016 (pp. 377-388). New York: ACM.
Imparato, G. (2015). A combined technique of gui ripping and input perturbation testing for android apps (vol. 2, pp. 760-762). cited By 0.
ISO 29119 Software Testing Standard (2013). Software and systems engineering software testing part 1: concepts and definitions. ISO/IEC/IEEE, 29119-1(E), 1-64.
Jaaskelainen, A., Takala, T., Katara, M. (2012). Model-Based Gui Testing of Android Applications, Addison-Wesley Professional (Pearson Education) (pp. 253-275).
Jamrozik, K., & Zeller, A. (2016). Droid mate: A robust and extensible test generator for android (pp. 293- 294). cited By 0.
Jensen, C.S., Prasad, M.R., Moller, A. (2013). Automated testing with targeted event sequence generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA (pp. 67-77). New York: ACM.
Jha, A., Lee, S., Lee, W. (2015). Modeling and test case generation of inter-component communication in android (pp. 113-116). cited By 2.
Jiang, B., Long, X., Gao, X. (2007). Mobiletest: A tool supporting automatic black box test for software on smart mobile devices. cited By 1.
Jiang, B., Chen, P., Chan, W.K., Zhang, X. (2016). To what extent is stress testing of android tv applications automated in industrial environments? IEEE Transactions on Reliability, 65(3), 1223-1239.
Jiang, B., Zhang, Y., Chan, W.K., Zhang, Z. (2017). Which factor impacts gui traversal-based test case generation technique most? a controlled experiment on android applications. In 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS) (pp. 21-31).
Joorabchi, M., Ali, M., Mesbah, A. (2016). Detecting inconsistencies in multi-platform mobile apps (pp. 450-460). cited By 0.
Kaasila, J., Ferreira, D., Kostakos, V., Ojala, T. (2012). Testdroid: Automated remote ui testing on android. cited By 5.
Kirubakaran, B., & Karthikeyani, V. (2013). Mobile application testing 2014; challenges and solution approach through automation. In 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) (pp. 79-84).
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE Technical Report EBSE-2007-01, Software Engineering Group School of Computer Science and Mathematics. Keele University and Department of Computer Science, University of Durham.
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., Linkman, S. (2009). Systematic literature reviews in software engineering - a systematic literature review. Information and Software Technology, 51(1), 7-15.
Kochhar, P.S., Thung, F., Nagappan, N., Zimmermann, T., Lo, D. (2015). Understanding the test automation culture of app developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) (pp. 1-10).
Li, A., Qin, Z., Chen, M., Liu, J. (2014a). Adautomation: An activity diagram based automated gui testing framework for smartphone applications (pp. 68-77). cited By 1.
Li, L., Bissyande, T., Papadakis, M., Rasthofer, S., Bartel, A., Octeau, D., Klein, J., Le Traon, Y. (2016a). Static analysis of android apps: A systematic literature review. Technical report, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg, Fraunhofer SIT Technische Universitat Darmstadt. Germany: University of Wisconsin and Pennsylvania State University.
Li, Q., Jiang, Y., Gu, T., Xu, C., Ma, J., Ma, X., Lu, J. (2016b). Effectively manifesting concurrency bugs in android apps. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC) (pp. 209-216).
Li, X., Jiang, Y., Liu, Y., Xu, C., Ma, X., Lu, J. (2014b). User guided automation for testing mobile apps. In Software engineering conference (APSEC), 2014 21st asia-pacific (Vol. 1, pp. 27-34).
Li, Y., Yang, Z., Guo, Y., Chen, X. (2017). Droidbot: A lightweight ui-guided test input generator for android. In Proceedings of the 39th International Conference on Software Engineering Companion, ICSE-C '17 (pp. 23-26). Piscataway: IEEE Press.
Liang, C. J. M., Lane, N. D., Brouwers, N., Zhang, L., Karlsson, B. F., Liu, H., Liu, Y., Tang, J., Shan, X., Chandra, R., Zhao, F. (2014). Caiipa: Automated large-scale mobile app testing through contextual fuzzing. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (pp. 519-530). New York: ACM.
Lin, Y.D., Rojas, J., Chu, E.H., Lai, Y.C. (2014). On the accuracy, efficiency, and reusability of automated test oracles for android devices. Software Engineering. IEEE Transactions on, 40(10), 957-970.
