A large scale analysis of mHealth app user reviews

Mobile applications have become an essential aspect of our daily lives. mHealth apps are one of the most rapidly evolving categories of mobile apps, with millions of people downloading and using some form of mHealth mobile app frequently from all over the world. According to the World Health Organisation (WHO), more than 30% of adults are deemed insufficiently physically fit due to a lack of activity and having an unhealthy lifestyle (Higgins 2016). Inactivity kills over 3.2 million people each year, according to Higgins (2016) and Joseph et al. (2014). An unhealthy lifestyle is characterised by insufficient exercise, excessive stress, poor diet, and inconsistent sleeping habits (Organisation WH 2020; Joseph et al. 2014; Spittaels et al. 2007). Solving these issues has become an essential goal for healthcare organisations and providers.

Interest in mHealth apps has grown dramatically in recent years, especially in the aftermath of the COVID-19 global pandemic and country lock-downs (Timmers et al. 2020; Pal et al. 2020; Dwivedi et al. 2020; Ventola 2014; Saadat et al. 2020). As a recent example, a whole new category of mobile apps known as“Contact Tracing” has emerged in the mobile app market.

Since a great many people use smartphones worldwide, mHealth apps have the chance to change this behaviour by motivating and encouraging people to exercise and stay healthy in their daily lives. According to WHO, 61% of the people who downloaded and started using mHealth apps have managed successfully to increase their physical activity. Moreover, the usage of mHealth apps in self-monitoring provides excellent control for people with chronic health conditions such as the elderly, which decreases the burden on health resources (Williams et al. 1998; Baker et al. 2002; Baker et al. 1998; Williams et al. 1995).

The mHealth app global market is growing very fast, which includes high revenues and downloads count, as shown in Fig. 1¹. According to WHO, mHealth apps should be accessible to everyone, regardless of their gender, culture, age, and language. mHealth app developers should allow their apps to be accessible to the elderly, disabled, and those who are consistently identified as high-risk groups (Thelwall and Levitt 2020). Elderly and disabled people face numerous challenges and have unique needs, and it is unfair to make it difficult for them to use mHealth apps (Morey et al. 2017). Since mHealth apps save lives and improve the lives of elderly and disabled people, mHealth apps need to be designed and developed in a particular way to be fully accessible by all types of users. According to the authors of (Morey et al. 2017), it is worthwhile to invest in developing mHealth for elderly users because older adults are likely to be the largest community using mHealth apps as these apps can bring significant benefits for them. While interacting with mobile apps, users from diverse cultural backgrounds have different preferences (Oliveira et al. 2016; Reinecke and Bernstein 2011; Reinecke and Gajos 2014). Feedback from users of the same app differs across different cultures and languages (Guzman et al. 2018). Many mHealth apps are not highly rated, which identifies significant problems with these apps, and this explains why users have lots of issues and trouble while using these apps.

Despite the existence of over 318,000 mHealth apps in the mobile app market, there is little evidence of them working as they are meant to/intended for a variety of reasons (Byambasuren et al. 2018; Liew et al. 2019; Institute 2020; Nordrum et al 2016). Many mHealth apps are buggy, as demonstrated by (Athilingam and Jenkins 2018; Anderson et al. 2016), causing users to lose interest and abandon the app after a short period of time. By looking at the user reviews of mHealth apps, millions of users are raising and reporting significant issues and problems that they have experienced while using these apps. For a variety of reasons, app developers typically disregard and ignore these user reviews (Vasa et al. 2012; Packard 2015). Many user reviews are written in different languages, which make it a complex and highly challenging task for the developers of mHealth apps to manually translate, classify, and analyse all of these reviews and, indeed, to respond to those reviews by fixing any highlighted issues. Mobile app user reviews and star ratings contain much information that is useful, not only for app developers but also for other app users (Vasa et al. 2012). This information helps mHealth app developers and designers to understand and analyse user behaviour, expectations, and feedback. Furthermore, it enables them to regularly update their apps to improve the user experience (Li et al. 2018). In addition, it allows them to gain accurate insights into their users, which enables them to refine their marketing strategies and increase publicity (Vasa et al. 2012; Qiao et al. 2018).

In this paper, we analyse and categorise mHealth app user reviews and discuss the aspects of compliments and criticism mobile users have towards these apps. We present the relationship between these aspects and the apps’ rating. In addition, we manually analyse and present the most common issues and problems raised by the users for each analysed aspect. We then give some recommendations for app developers based on our deep analysis.

The key contributions of this research include:

The rest of this paper is structured as follows. We first provide a motivation for our research and then detail our study approach and methodology. This is followed by a presentation of the key findings and then a discussion of implications for research and practice. We then summarise key related work to our study. We finally summarise key conclusions and directions for future research.

There are several ways mHealth app users can submit their feedback. In our research, we chose to work on user reviews since they have several advantages (Maalej et al. 2016). Moreover, they are considered the primary source of any available user feedback, directly linked to app features. Many problems and issues are reported and documented through mHealth apps’ user reviews. Those reported issues contain various issues in the UI (User Interface), sign-up, bugs, accessibility, privacy, and UX. Despite the huge revenue of mHealth apps, many of these issues seem to be replicated and not fixed by the released updates (Jusoh 2017; Adhikari et al. 2014; Chatzipavlou et al. 2016). As a result, some mHealth apps have low ratings (Heffernan et al. 2016). Mobile users generally have the right to access and use any mHealth apps they wish to under safe conditions. They should not expose themselves or their personal data to a risk that could lead to severe harm or consequences later on. Furthermore, mobile app engineering organisations are obliged by law and ethical agreements to ensure that their apps are accessible by all types of people (Organisation WH 2020). Since mHealth apps are essential for saving and improving the lives of people, we could potentially save billions of dollars per year by resolving those issues/problems and improving the engineering design and development practises of mHealth apps (Organisation WH 2020; Joseph et al. 2014; Spittaels et al. 2007; Higgins 2016).

In Chevalier and Mayzlin (2006); Mcilroy et al. (2017), the authors stated that user reviews could influence users’ decisions on downloading or purchasing a specific mobile app. Users tend to gain more confidence in downloading or purchasing an app if it has a high number of user reviews. If the majority of these reviews are positive, the mobile users’ trust increases. This explains why an app with few reviews or negative ratings may not succeed right away, as the majority of users will be hesitant to download or purchase this app (Khalid et al. 2014; Mcilroy et al. 2017). Moreover, user reviews can increase or decrease people’s interest and inspiration to download or engage with these apps based on previous ratings. On the other hand, from the mobile app developers’ side, user reviews allow them to have the power to enhance the reputation and credibility of their mobile apps by releasing updates that fulfil the needs of users based on their feedback and requests. Mobile app businesses and organisations that do not have many good reviews are definitely missing an excellent opportunity to grow their business and increase app downloads. While app store reviews provide potentially very beneficial information about how and what people expect from mHealth apps, comprehensive analysis of a large number of reviews that increase every day remains a major time-consuming challenge (Huebner et al. 2018).

Some research has already been done in this area by analysing the user reviews for other types of mobile apps (Guzman and Maalej 2014; Iacob and Harrison 2013; Chen et al. 2014; Gomez et al. 2015; Gu and Kim S 2015; Maalej and Nabil 2015). To the best of our knowledge, no work has yet been done to analyse mHealth apps’ user reviews on a large scale. Our research aims to provide insights to developers in identifying key issues raised by the users of mHealth apps through their user review feedback and submitted ratings. This will help them to proactively identify previously reported users’ issues before releasing new apps or updates and prevent these issues from happening in the future. Furthermore, it will assist mHealth app developers and designers in conducting better mHealth app engineering practices. By developing an automated tool that can extract, translate, and classify millions of mHealth app users’ reviews into different aspects.

By classifying the users’ reviews into different aspects and analysing them, we will be able to understand the significant issues reported by the users of mHealth apps. Moreover, we will be able to set some recommendations for app developers based on each analysed aspect. In this work, we investigate user reviews relating to fourteen mHealth app aspects – advertising, payment and subscription, compatibility, UI/UX (User Interface/ User Experience), resources, stability, connectivity, account and logging, privacy and security, user requests, notifications and alerts, uninstallation, updates, and internationalisation issues on the ratings of mHealth apps. This analysis will provide mHealth app developers comprehensive insights into the expectations and perceptions of their users when using these applications. The mHealth apps used in our study are categorised into twelve different subcategories. These mHealth app subcategories are diet and weight loss, emerging, education, women’s health, fitness activity tracking, fitness and workouts, lifestyle planner and goal tracker, scheduling and reminder apps, sleep and meditation, and mental health. This categorisation gives us the opportunity to analyse our results at an additional level of depth.

We investigated the following two research questions:

Using our automated analysis tool, we have automatically extracted, translated, and classified over 5.1 million user reviews sentences for different subcategories of mHealth mobile apps into 14 different aspects. These reviews were extracted from both Apple and Google Play stores belonging to 278 mHealth apps. Our tool uses a ”bag of keywords” classification approach to classify and investigate how different aspects of mHealth apps can affect their ratings. In addition, all mHealth apps used in our study were manually sub-categorised into 12 different subcategories. We used the star rating of the app and the app review classification to identify strengths and weaknesses in the apps. From our manual analysis of 23,000 app reviews, we identified that most app reviews mention only 1 or 2 app aspects, allowing us to correlate app rating and app aspect(s) mentioned. Then we did a manual analysis of 14,000 user reviews (1,000 reviews for each aspect) to identify the major issues that were found in each analysed aspect and to provide evidence-based recommendations for the developers of mHealth apps.

The distribution of the star ratings of each analysed aspect in our study is depicted in Figure 4 and analysed further below. The top raised aspects in all mHealth apps included in our study were UI and UX (28.81%), User Requests (13.44%), Payments/Subscription (8.54%), Stability (7.55%), Compatibility (5.77%), Updates (5.44%), and Connectivity (3.48%). The uninstallation aspect reviews were the worst rated, followed by login and then privacy. On the other hand, the UI/UX aspect was the best-rated, followed by multimedia and then user requests.

The uninstallation aspect was the worst-rated, with 55.82% of the users who raised uninstallation issues in their review rated the app with only one star. Followed by account and logging issues where 52.2% of the users rated the app with only one star. Then, privacy issues where 37.36% of the users who raised privacy issues rated the app with only one star. On the other hand, UI/UX was the best-rated aspect, where only 6% of the users rated it 1 star, followed by internationalisation issues, where 19.46% of the users rated it 1 star, and then user requests, where 19.33% of the users rated the app 1 star.

As the most highly rated subcategory, women’s health app users were the most satisfied. In contrast, fitness activity tracking was the lowest-rated app subcategory in our assessment of all the mHealth subcategories. Due to several common issues and problems, users gave fitness activity tracking applications low ratings on both the App Store and Google Play.

In the following subsections, we identify the most positive and negative aspects across different sub-categories of mHealth apps based on classified user reviews. We also determine what are the key issues and problems raised by the users for each analysed aspect.

mHealth apps are downloaded and used by millions of people around the world on a daily basis. There are currently more than 318,000 fitness and health apps in the mobile app market (Liew et al. 2019; Institute 2020). When interacting with mobile apps, users from diverse cultural backgrounds have different preferences (Oliveira et al. 2016; Reinecke and Bernstein 2011; Reinecke and Gajos 2014). That is why the feedback from users across different cultures and languages differs for the same app (Guzman et al. 2018). As shown in Anderson et al. (2016), many mHealth apps are buggy, in which their users lose engagement and discontinue using them after a short period of time. Many users are experiencing some issues while using the mHealth apps and want to submit some feedback to the developers of these apps. There are many ways for users to submit their feedback to mobile app developers for development and research purposes such as contact forms and mobile surveys, emails and social media feedback, and user reviews. On the other hand, mobile app development organisations are always seeking users’ feedback. This feedback allows the developers of these organisations to enhance their apps with their regular updates. Surprisingly, 42% of organisations and companies fail to analyse users’ feedback due to some challenges. Not listening to users’ feedback and experiences leads to missing an excellent opportunity to improve apps and enhance users’ satisfaction and experience.

A number of studies have investigated and analysed user reviews for mobile apps. Several studies looked into different methodologies to analyse user reviews for extracting app features (Guzman and Maalej 2014; Iacob and Harrison 2013). Other studies developed some methods to classify user reviews into predefined aspects (Haggag et al. 2022; Fazzini et al. 2022; Chen et al. 2014; Haggag et al. 2021; Gomez et al. 2015; Gu and Kim S 2015; Maalej and Nabil 2015; Haggag 2022). Moreover, some research has been conducted on mHealth apps, but they concentrate only on user experience, neglecting the users’ dedication to the apps. In Di Sorbo et al. (2021), the authors developed a tool called NEON that use a natural language processing approach. They did a small study on 100 reviews across two categories (feature request and problem discovery). The authors picked up around 35% of the rules inferred by their tool are relevant for identifying app review sentences, reporting bugs or requesting enhancements. The authors indicated that the effectiveness of NEON would degrade when dealing with review sentences containing mixtures of code elements and natural language or incomplete sentences. In our study, we had many user review sentences that were not complete, had spelling errors, were written in different languages, etc. In Pimenta et al. (2020), the authors tried to design mechanisms to improve and introduce a healthier behavioural change through mobile devices. They detected the experience satisfaction using BCT Taxonomy and user feedback. They had done a systematic review of the applications, which was followed by sentiment analysis. Then finally, they came up with the design implications, which are related to the implementation of BCT features. That interactive system has shown various relationships, such as being physically active and having the habit of eating healthy, while negative behaviours were mostly triggered by rewards and threats. In Alqahtani and Orji (2020), the authors were focusing on Mental health applications, as it reflected a great intervention on mental issues. Their findings showed that the mobile interventions’ effectiveness led to low engagement, and they failed to understand the reasons for this behaviour. However, the authors discovered two very important factors, user usability and experience. They were trying to analyse user reviews to understand the reasons behind users’ ceasing the use of these applications. They have reviewed 106 mental health, 13,549 reviews. They pointed out that the main reasons for quitting the usage of apps are the user interface and user-friendliness. A long-term commitment to using the app does not necessarily mean involvement (Zichermann 2011; Pandey et al. 2013). Since the mobile apps market is highly competitive, apps’ developers need to take their users’ feedback seriously, not to lose any of their loyal users. This can be done by carefully analysing users’ reviews and working on resolving their issues and fulfilling their needs (Khalid et al. 2014). Ratings are always associated with the quality and popularity of the mobile app (Khalid et al. 2014). Mobile users generally see a mobile app with a low rating as non-popular, which badly affect the number of downloads and revenues (Khalid et al. 2014). The number of user reviews increases when the number of downloads increases (Mcilroy et al. 2017). Moreover, the number of user reviews is directly correlated with the number of apps updates done by the developers of mobile apps (Mcilroy et al. 2017). This leads to a spike in the number of reviews after each update as users are either experiencing new problems while using the app, or they are fully satisfied with the new update (Mcilroy et al. 2017). In Vasa et al. (2012), an exploratory study was performed on what could impact the users to write an app review. In Iacob et al. (2013), they investigated different factors that affect user reviews, such as the correlation between the rating, price, and the number of downloads. The authors of (Chen et al. 2014), proposed a computational structure by adopting a semi-supervised algorithm in order to extract and rank insightful reviews. In Vu et al. (2015), the authors developed a semi-automated keyword-based approach for mining user opinions in their reviews. In Mcilroy et al. (2017), the authors indicated that further studies are needed to better understand and analyse user reviews through Apple and Google app stores. More research has been done on data reviews, in Wu et al. (2021) they tried to identify the key features from app user reviews. In Di Sorbo et al. (2021), they have built an NLP-based tool for software artefact analysis. In Di Sorbo et al. (2016) and Panichella et al. (2015), the authors came up with a list of recommendations for software changes, including maintenance and evolution. Also, some research has been done on profiling users via their reviews and doing that via a systematic approach as shown in Dong et al. (2021) and Genc-Nayebi and Abran (2017).

The global market for mHealth apps is increasing at a rapid pace, resulting in significant revenues and downloads. However, as reported in user reviews, many of these mHealth apps have significant problems. App developers would benefit from a better understanding of major user concerns in order to improve the quality and adoption of their apps. While many previous studies have employed app reviews to investigate users’ opinions about mHealth apps, many of them are limited in scope, scale, and/or analysis. In this paper, we analysed over 5 million user reviews of 278 mHealth apps. We classified these reviews into 14 different aspects/categories of issues reported using an automated tool that we developed based on a bag of keywords. User satisfaction levels were compared amongst several mHealth app subcategories to investigate the impact of different aspects of mHealth apps on their ratings. Based on our findings, women’s health apps were recognised as the highest-rated subcategory. In contrast, fitness activity tracking was the lowest-rated app subcategory in our review of all mHealth subcategories due to various issues and problems in multiple investigated aspects. Over half of the users who reported uninstallation issues in their reviews rated the app 1-star. Half of the users rated the account and logging aspect 1-star each due to several problems and issues they faced while signing up or logging in to their account. Over a third of users who raised privacy concerns rated the app a 1-star review. However, only 6% of users gave the app a 1-star rating for UI/UX. 20% of users reported issues with the handling of user requests and internationalisation concerns. By manually analysing a sample of 1,000 user reviews from each investigated aspect/category, we validated our findings. We developed a list of recommendations for mHealth app developers for each investigated aspect based on our user review analysis.