Research & Evaluation Steps in an mHealth Intervention

A group of mHealth experts in New Zealand have established and implemented a process for the design, testing, and evaluation of mHealth interventions, which they describe in detail in their paper A Development and Evaluation Process for mHealth Interventions: Examples from New Zealand. Table 3 summarizes the authors’ research and evaluation steps in an mHealth intervention. While it might not be possible to complete all of these steps for all mHealth interventions, the table presents an example of a high-quality mHealth research strategy, with an emphasis on monitoring and evaluation in later stages. The table offers helpful illustrative measures for each research activity.

Table 3. Summary of the research and evaluation steps in the development of an mHealth intervention

(Source: Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: Examples from New Zealand. Journal of Health Communication: International Perspectives. 2012; 17 Suppl 1:11-21.)

 

 

 

Research methods

 

Purpose

 

Example measures

Formative research

 

 

 

  • Focus groups
  • Online surveys

 

 

 

To inform the development of the intervention content and process

 

 

  • How do the target audience use their phones?
  • How does evidence and theory behind intervention fit with mobile phone usage?
  • What would attract target audience to program?

Pretesting

 

 

 

  • Online surveys
  • Focus groups
  • Individual interviews

 

 

  • To determine acceptability of proposed intervention to target audience
  • To improve and refine intervention on basis of feedback
  • What styles/content/language do they prefer?
  • Are messages useful and understandable?

 

 

Pilot study

 

 

Small and nonrandomized

 

 

  • To test content and regimen of intervention
  • To test processes
  • Is intervention acceptable?
  • Are there any technical or process issues?
  • Baseline measures for sample size calculations if necessary

Randomized control trial

 

Pragmatic community-based randomized control trial

To test the effect of the intervention in comparison with a control group

  • Health-related outcomes
  • Objective measures
  • Participant satisfaction
  • Adverse/unintended effects

Qualitative research

 

 

 

Semi-structured interviews

 

 

 

  • To improve the intervention further
  • To determine implementation issues and methods
  • How can we improve the intervention?
  • What aspects were un/helpful?
  • Was theory conveyed?
  • Were messages acted upon?
  • What are the best methods for rollout and promotion?

Evaluation of implementation impact

 

  • Phone/online surveys
  • Semi-structured interviews

 

To determine the effect of the intervention once scaled up

 

Reach, utilization, health-related outcomes, flow on effects on health/other services, unintended consequences, effect of different promotional methods