Evaluating WhatsApp as a Migrant Survey Tool

What’s up with WhatsApp?

Migrants WhatsApp for Surveys

Our phones – like it or not, they have become an integral part of our lives. As well as being our connection to loved ones, our cameras, our alarm clocks and general portals to the wider world, smartphones increasingly represent an important data collection tool. This is particularly true in the context of sudden contact restrictions, like the COVID-19 pandemic.  

Two billion people, that’s more than a quarter of humans on the planet, use WhatsApp every month. It is the most popular messaging app worldwide, however, its use as a data collection tool has only recently been explored. The ‘old fashioned’ survey tool is a self-administered, paper-and-pencil; Quicker and cheaper web surveys have been around for a while, but along with cost and coverage, WhatsApp surveys offers additional advantages in low-income countries: messages do not use too much data and voice notes are popular in countries where literacy rates are low and internet connection is unreliable.

image

At IOM, we have tapped into the potential of smartphones as a survey tool for migration researchers aiming to study cross-border migration behaviour. Migrants are evidently on the move; this complicates the logistics of more traditional face-to-face surveys and heavily increases costs. You might think that when moving country, people change their phone numbers, however many of us do not change our phone numbers linked to apps like WhatsApp when we move, so WhatsApp provides a useful way to survey migrants.  

One thing that has so far not been clear is the effectiveness of phone surveys, generally, and in migration research specifically. The IOM Global Migration Data Analysis Centre (GMDAC) collaborated with the Media, Communications and Awareness Raising Unit in West and Central Africa to launch a field study in Senegal and Guinea that took place between 2020 and 2021. This study aimed to address the gap in the evidence base for phone surveys using WhatsApp, especially in the context of geographic mobility.

In this field study, we compared the response rates between WhatsApp and interactive voice response (IVR) modes, using a sample of 8,446 contacts in Senegal and Guinea. In doing so, we sought to answer three main questions:  

1. Is WhatsApp more effective in terms of unit non-response than IVR survey modes in the context of follow-up phone surveys in low-income countries?

2. How do WhatsApp and IVR compare in terms of cost-effectiveness (i.e. relative unit costs for completed surveys in the follow-up survey)? 

image

What did we find?

  • Firstly, not everyone wants to take part in phone surveys. Both WhatsApp and IVR survey modes yield modest response rates (6-20% depending on the country), so incentivizing participation could be a good idea.  
  • In general, people engaged less with WhatsApp surveys, compared to IVR: Nearly 8% fewer people engaged with the WhatsApp surveys.  
  • On the other hand, WhatsApp survey costs were substantially lower, meaning more could be administered for the same cost.  
  • Also, of the surveys started, the survey completion rates were higher for WhatsApp, and we saw no difference in sample selection bias related to age, sex or education level, compared to IVR.  
  • There were mixed results between countries, with WhatsApp performing worse in Guinea. 

Given how new research into using WhatsApp for migration related surveys is, we hope that this study’s findings help to inform future research and programming.

The full paper, entitled “Effectiveness of WhatsApp for Measuring Migration in Follow-Up Phone Surveys. Lessons from a Mode Experiment in Two Low-Income Countries during COVID Contact Restrictions”, was published in July 2022 in the Social Science Computer Review.

Do you have experience using WhatsApp as a survey tool? Has this sparked any ideas in your awareness raising work? Feel free to contact us at MCARteam@iom.int

3. Do different survey modes introduce different selection bias in survey participation relative to the base sample?

COVID-19
evaluation
online