Unique Challenges of Monitoring & Evaluating mHealth

Monitoring and evaluating the process and impact of any health intervention presents challenges, but there are several unique factors to consider when planning for M&E of an mHealth program.

  • Indicators—how to measure impact? It is important to remember that mHealth interventions are not stand-alone projects, but rather technologies that enhance larger programs. As Table 2 illustrates, some of the indicators used to measure the impact of an mHealth intervention will be the same as those used for the larger program.
  • Structure of the evaluation—what is feasible? Though considered the gold standard for evidence, randomized controlled trials are often not feasible for evaluation of mHealth interventions due to time and funding constraints, and the fact that once launched, the service might be available to anyone in the country who owns a mobile phone. Given the constantly changing nature of technology and infrastructure, innovative research methods are needed to evaluate the process and measure the impact of mHealth implementation.
  • Timelines and integrating feedbackwhat is the balance? Data collection should start early in the planning process to ensure that key decisions and action steps are documented from the beginning of the project. Tracking these elements of the planning and implementation process can not only inform later efforts to scale up or replicate the program but can also help you understand what about the mHealth intervention worked well, what did not, and why. It is crucial to collect and incorporate feedback from end users on the acceptability and usability of the application before it is launched and on an ongoing basis once the program is rolled out. That said, the planning and implementation team must balance the need to improve the mHealth application and the implementation process based on user feedback with the need to move forward with the project. Setting timelines and limits for the submission of feedback and tempering expectations for the capacity of the solution to change or evolve will help avoid delays and disagreements.
  • Data overload—how to manage? Additionally, as mentioned elsewhere in the Guide, technology platforms that enable mHealth solutions often have the capability to collect data in real time. This means that every interaction between the technology and the end user can be recorded. To avoid data overload, the mHealth project team needs to assess what kinds of data to monitor and evaluate and at what intervals.

THE EXPERTS SAY...

“Look at the indicators the health service is using, and use those same indicators.” – Isaac Holeman, Medic Mobile

“If you are asking a lot of questions [via SMS], people stop responding. We found that four questions worked well. We saw that with every extra question beyond that the respondents dropped off.” - Bhupendra Sheoran, YTH (formerly ISIS), SexInfo and The Hookup, USA