Curated apps providers can recommend, and patients can use.
The explosion of medical, as well as health & fitness, apps in the marketplace has made it near impossible for patients or providers to actually determine the credibility, quality, and effectiveness of these apps. At HMI, our goal is to critically evaluate mobile medical apps based on their clinical relevance, credibility, accuracy, and effectiveness, as well as usability and functionality, so that providers can recommend, and patient can use, condition-specific mobile apps with confidence.
Apps were selected for evaluation via a review of the currently available apps on the most popular mobile operating systems on the market: Android and iOS. We conducted our initial search strategies by concentrating on chronic conditions common in the United States, and those targeting issues impacting large numbers of people:
In order to identify apps related to the above categories, we reviewed apps currently listed in the top 100 positions in the Paid & Free sections of the Android and iOS app stores. We then selected apps related to each of the categories listed above to be reviewed. If not enough apps were available in the top 100 apps, we considered apps with lower rankings. We took this approach because the top paid and free apps are most likely to be of current interest to patients and have a significant download history, thus warranting closer inspection for medical validity. While there are thousands of apps currently on the market, we believe by starting with the most popular apps (defined by download history) is a mechanism to help identify apps of highest risk/benefit to users. In doing so, we additionally opted to not review apps which we felt were unlikely to be used in medical care due impracticality or lack of clinical soundness. Such determinations were made based upon the apps’ descriptions within the relevant app stores.
The apps we have reviewed to date are meant to demonstrate our process across a broad sample of apps. As such, our initial app review is based on 15 apps across five categories: mental health, diabetes, heart disease, medication adherence, and sleep aids. These apps were selected from our initial search process. Figure 1 depicts the selection mechanism which was used. Additionally, we have also elected to showcase a number of apps that we believe are substantially dangerous or vehemently incorrect, to demonstrate to the public the dangers of the current app market.
Our goal is to create reviews which are both useful and reliable. To achieve this, we incorporate multiple experts into our review process, as much as possible. Reviewers independently review each app on a set of predetermined criteria. The resulting reviews are examined by an editor, with particular attention paid to discrepancies between the reviews. The two reviewers discuss any discrepancies until agreement is reached. This process aims to increase the reliability of reviews by shining a spotlight on areas of the reviews in which there is poor agreement between the reviewers. Doing so lessens the influence of any individual reviewer on the findings of a review.
The reviews themselves are designed to contain both objective information about the apps and the subjective determinations by the experts reviewing them. Objective details in the reviews include identifying information, the platform of the app, the version of the app, the provenance of the app, and the app’s intended use. The subjective portion of the reviews include consideration of clinical relevance, clinical credibility, usability, functionality, security, and privacy. Reviews conclude with a brief summary of the app’s intended use and key features/ functionality.
Of note, as much as possible, the reviewers and editors try their best to base their reviews on the app’s intended use and claims made by the app developers (on the app or their website). For instance, an app that is intended to be “a digital diary” for logging food/caloric intake cannot, and should not, be judged on it’s ability to induce weight loss. However, a similar mobile app that claims to be a great weight loss app by allowing users to “enter daily food/ caloric intake”, even though absolutely beautiful and easy-to-use, will be judged based on it’s ability to induce weight loss via published evidence that substantiates it’s claims.
Effectiveness: Clinical Relevance/ Credibility/ Evidence-base
Functionality: Features, data sharing, integration with other apps or HealthKit or electronic medical records
Usability: user interface, user experience, easy of use, look & feel, etc.
Not recommended: <40
Future Goals and Landmarks
While our initial app selection is meant to demonstrate how HMI will evaluate apps, our ultimate goal is to expand beyond the immediate selection of apps identified. We intend to review the remaining apps in the top 200 we had previously identified and those in the remaining categories not covered. Furthermore, we will periodically reconsider the best-in-class apps, as new apps are introduced and updated. We will additionally expand into other health categories, and may cover apps addressing rare diseases in the long-term. Lastly, wearable devices and health platforms will be added to our review list as well. To achieve this scale, we will recruit both members of the medical community and patients to assist us in the review process, bringing hacking medicine to the public.