Designing M&E into mHealth Programs

© 2012 Okwen Patrick Mbah, Courtesy of Photoshare

To effectively monitor and evaluate an mHealth intervention, use qualitative and quantitative research methodologies. As the mHealth application is developed, consider whether and how the platform can collect data, in what format, and at what intervals (refer to Technology Design in the Solution Design & Testing section for key considerations). Aside from electronic use data that can be collected in real time, consider reaching end users on their mobile devices to ask survey questions (see the m4RH case study example). Once you have a sense of the kind of data collection that can be built into the mHealth service or tool, you can design other qualitative and quantitative techniques into your M&E plan.

In addition to collecting and analyzing M&E data on the process and outcomes of the mHealth intervention, cost data must also be tracked and studied to determine whether the mHealth solution is cost-effective and should be scaled up. If the intervention is worth scaling up, stakeholders and financial backers will want to know the initial costs of scaling up the program, as well as the potential cost savings—to the particular health program and to the health system at large—of operating the program at scale.

Table 2 summarizes illustrative program elements worth monitoring and evaluating, and provides sample indicators for each element. The indicators in bold print could potentially be measured with data generated by the program’s mobile platform.

Table 2. Program elements and sample indicators for monitoring and evaluation

Program Elements

Sample Indicators

Health outcomes

Changes in:

  • Contraceptive prevalence rate (CPR)
  • Maternal mortality rate
  • Rate of new HIV infection
  • Nutritional status (rates of anemia, stunting)

Health behaviors

Changes in:

  • Demand for health services related to mHealth program
  • Percent of women breastfeeding
  • Adherence to antiretroviral therapy
  • Contraceptive continuation

Acceptability of the program

  • Percentage of subscribers retained
  • Percentage of users who unsubscribe
  • User-reported satisfaction

Quality and accessibility of mHealth services

  • Number of regular mHealth program users
  • Profile of mHealth users (gender, age, location, economic and social status)
  • User-reported satisfaction
  • Messages to target audience delivered in a timely manner
  • Quality of data collected via mHealth application
  • Level of use of health services related to mHealth program
  • Percentage of messages vetted by technical experts

Health worker performance

  • Change in volume of clients served
  • Client-reported satisfaction
  • Supervisor-reported performance observations

Capacity of local government and implementing organizations responsible for program and health outcomes

  • Number and duration of program-related trainings
  • Number, nature, and duration of successful partnerships

Program sustainability, program costs, and cost effectiveness

  • Number and nature of funding sources
  • Total cost of ownership
  • Average cost of mHealth program per beneficiary
  • Money saved per beneficiary by change in target health behavior or health outcome
  • Willingness of user to pay for mHealth services

Capacity of target beneficiaries

  • Increased use of related health services
  • Demonstrated understanding of health concept(s) addressed by mHealth program
  • Change in target health behavior

Extent to which the program operates in an enabling environment

  • Existence of supportive policies and procedural guidelines
  • Adherence to interoperability standards and program procedures
  • Ongoing allocation of resources in budget for mHealth program
  • Action steps taken by program management team to enforce implementation-related changes

For guidance on the development of national M&E plans, see the Mobile Alliance for Maternal Action (MAMA) Global Monitoring and Evaluation Framework. The MAMA Framework focuses on monitoring and reporting on progress when implementing mobile messaging programs for mothers, but the information in this document can be useful to other kinds of mHealth programs as well.