With mobile phone penetration rates exceeding 80 percent in many developing countries, development organizations have a new and exciting way of communicating with and collecting information from beneficiaries and stakeholders. Paired with the growing emphasis on evidence-based decisions, collecting data through SMS surveys has generated a significant following and an array of tools for design and deployment. For many good reasons, the development community is jumping on the bandwagon. But before the bandwagon gets rolling too fast, it is also important to abide by fundamental data collection best practices and recognize that SMS surveys are not a panacea to all survey challenges. Let’s take a look at some strengths, limitations, and best practices for SMS surveys.
SMS surveys are attractive for many reasons. Most importantly, they greatly reduce the level of effort, time, and cost as compared to in-person enumeration. Whereas conducting a survey in person can require training survey implementers (known as data enumerators) and significant effort to collect responses, SMS surveys can be deployed instantaneously and data can be available for analysis within hours or days. This is a tremendous asset in fluid environments where there is a need to make quick and informed decisions. When the target population is geographically inaccessible or remote, the logistical advantages of SMS versus in-person enumeration are amplified assuming there is cell service in these remote areas. Proportionate to these logistical advantages, the cost of SMS surveys is dramatically less than in-person enumeration. Pricing schemes vary, but SMS surveys are often five to ten times cheaper than an in-person survey.
Survey by SMS has other advantages for data quality as well. Even trained enumerators operating in a system with data quality controls can be susceptible to data quality risks, such as inconsistency in survey delivery, an increased risk of transcription error, and even fabrication of data. Studies have shown that survey respondents are more honest when responding to online or SMS surveys as opposed to phone or in-person surveys. Therefore, SMS surveying can contribute to higher levels of transparency, consistency, and accuracy in data collection.
The most inherent limitation of SMS surveying is that not everyone has a cell phone, and those without cell phones are often the most marginalized — women, the poor, individuals with disabilities, and the elderly. Illiteracy and unreliable mobile networks depress response rates among lesser educated and rural communities, respectively. A 2015 study by the Center for Global Development found that the demographic composition of SMS survey respondents in four developing countries differed by 14 percent on average from the general populations in those countries. Rural populations were underrepresented by 30 percent, women by 25 percent, those who haven’t completed primary education by 38 percent, and the elderly by 12 percent. Certain surveying and statistical techniques can mitigate some degree of skewing, and face-to-face surveying is vulnerable to skewed sampling in other ways. However, as a general rule, surveys that seek to be representative of the general population are best conducted by in-person enumerators.
SMS surveys are also limited by the amount of information you can convey to and request from respondents, a trade-off for its ease of deployment. There is little space for elucidation in a 160-character text message. Questions that require contextual preludes or nuance are unlikely candidates for SMS surveys. Responses should be similarly concise, typically multiple choice. Efforts to capture narrative through SMS surveying are largely in vain, prone to low response rates and nonsensical responses. For these reasons, SMS surveys are not ideal for exploratory data collection; instead, they are most practical for highly structured surveys when the range of responses is limited and known.
Finally, there is value to using a human enumerator to deliver a survey because they serve as the initial data quality control point against inadvertent errors and intentionally misleading responses. For instance, an SMS respondent might accidentally enter a two instead of a three, which could be avoided by conducting the survey in person. Intentional misrepresentation is also a possibility for SMS responses (for example, a man who responds on behalf of his wife, or a child who claims to be an adult). Other SMS respondents known as “speeders” respond repetitively without reading the question in order to finish the survey quickly, and still others simply decide to stop the survey midway through. While face-to-face surveys don’t eliminate all data errors, some of these possibilities are greatly reduced when using an in-person enumerator.
The Bottom Line
How we choose to deliver a survey can have a significant impact on cost, timing, and data quality. When considering SMS surveys, it is important to take into account the following factors and best practices.
- Timeliness. SMS surveys can be deployed instantaneously, and data can be available within hours or days.
- Cost. SMS surveys are significantly cheaper (by a factor of 5 to 10) than in-person enumeration.
- Structure. SMS surveys are most practical for highly structured surveys when the range of responses is limited and known, rather than more exploratory, qualitative studies.
- Representativeness. SMS survey sampling is not fully representative of the population, and it under-represents marginalized populations, in particular.
- User Experience. SMS surveys can be frustrating to navigate and susceptible to deceptive responses.
Ultimately, SMS surveying is a powerful and innovative data collection tool that should be in every monitoring and evaluation specialist’s toolbox. Its high efficiency and low cost open new doors for monitoring, and facilitate evidence-based decision-making. However, like any effective tool, SMS surveys must be deployed selectively in recognition of their limitations, and a number circumstances still call for face-to-face surveying.
Gabriel Pincus is a manager in Chemonics' Monitoring, Evaluation, and Learning Department.