​​A Guide to Remote Data Collection

​
​Overview

​​
​​In light of the COVID-19 pandemic, the security of survey participants and enumerators is of utmost importance, which has led to suspension of all field activities, at least for the foreseeable future. 
​​
​​However, if you do decide to transition to a remote method of collecting data, this document by DIME Analytics provides guidelines on best practices for transitioning an existing field survey to  a remote survey. Ideally, we believe it is best to postpone data collection in such situations, as remote surveys usually have lower response rates, and respondents are less likely to provide reliable information (especially on sensitive topics). 
​​
​​Among the various alternatives to in-person data collection, phone-based surveys are best suited for the typical research population. This guide focuses on explaining the multiple steps involved in transitioning to a phone survey in particular- including updates in training, data quality norms, and project timelines. We conclude this guide by providing a quick overview of alternatives to phone surveys such as web surveys and recorded surveys. We do this because we realize that in certain cases you might not want to use phone surveys, or use a combination of methods to collect your data.  
​​

​
​Note to readers

  • ​​This document assumes that you already have an intermediate-to-advanced understanding of the survey process.
  • ​​ This document draws largely on the previous experiences of the authors as well as resources compiled by various research organizations. The aim is to ensure that quality data collection can continue unrestricted despite limitations on collecting data through conventional methods. 
  • ​​In case you are looking for information on a particular aspect of the transition process, click on the section-specific links in the list of contents below to skip directly to that section.

​
​Contents

​
​Phone surveys

​​A phone survey or a computer-assisted telephone interview (CATI) is one where the enumerator calls the respondent and asks them the questions over telephone. The enumerator enters these responses onto a programmed digital survey, which is then shared electronically with you and your team. In the context of this guide, we refer to a specific case, that of work-from-home phone surveys. In these surveys, enumerators conduct the interviews from their own homes to avoid any human contact.
​​
​​It is important to note that it is not always possible to transition to a work-from-home (WFH) phone survey. There are many serious considerations to keep in mind before considering a transition and be extremely cautious when transitioning. In some cases you might have to suspend data-collection activities entirely. In the following section, we discuss the conditions under which the transition to a phone survey is feasible. 
​​

​
​Feasibility 

​​In cases where you believe it is feasible to conduct data collection over the phone, you must keep in mind certain considerations. These include:
​​
  • ​​Is it practical to ask these questions over a telephone?
  • ​​This is an important consideration because in some cases asking questions over the phone can drastically alter the nature of your data, or the manner in which respondents interpret your questions.
  • ​​
  • ​​While there are a broad range of questions that you can ask over the phone as well, there are a few exceptions like:
  • ​​Questions on sensitive topics (such as gender-based violence), and 
  • ​​Questions with test components (such as a Math test for a school-based survey).
  • ​​
  • ​​How long is the survey? Can I reduce the average duration without affecting the outcomes of interest? 
  • ​​General guidelines and best practices suggest that phone interviews should be short and longer surveys result in lower response rates. If your original questionnaire (for a field survey) took longer than 20 minutes to answer (on average), you might need to reduce the length of your questionnaire or the number of questions when transitioning to a phone survey.
  • ​​
  • ​​In cases where this is not possible, you might consider the feasibility of conducting the survey over multiple phone calls.
  • ​​
  • ​​ Does a majority of your target population have access to telephones? 
  • ​​This is important because if your target population does not have access to telephones , the survey will introduce a selection bias into your sample. Further, it will lead to lower response rates, which can reduce the cost-effectiveness of your survey.
  • ​​
  • ​​Do you have contact details of your target population?
  • ​​If this is a follow-up survey, hopefully your team has collected contact information about the respondents. But if this is a baseline survey, and you do not have access to the contact numbers, you can consider random-digit sampling. However this is only possible when you access to a long list of phone numbers from the population.
​​
​​If your answers to the above questions are yes, that means you can begin the transition process. In the next section, we discuss the logistics of data collection for a WFH phone survey. 
​​

​
​Before collection

​​As you might expect, the logistics of conducting a WFH phone survey are different from those for a field survey. You and your team should carefully work through some additional steps before you can start the phone interview. At each stage of this process, you must communicate clearly with everyone involved in the data collection, such as the government agencies, survey firm, enumerators, etc.
​​
​​These steps are listed below:
​​
​
​Update the survey timeline
​​The first step of the transition should be to update the survey timeline. When you transition to a phone survey, each step leading up to, and including, the data collection will take additional time. You must make sure you account for this time, and allocate a reasonable window for each step.
​​ 
​​DIME Analytics can review the updated timeline for a project.
​​
​
​Amend your IRB approvals 
​​If the project has any existing IRB approval(s), you will need to submit an amendment to each IRB, and receive approval for each of the following changes:
  • ​​Changes in the survey instrument
  • ​​Addition of any team members handling PII data
  • ​​Changes to incentives provided to respondents 
​​
​​While most IRBs are cognizant of the current crisis and are fast-tracking amendments, you should allocate enough time for the IRB amendment to come through before you can (re)start data collection.
​​
​​If your project does not have an IRB approval yet,  we recommend that you apply to an IRB immediately.
​​
​
​Update procurement-related documents
​​In cases where the survey firm has been procured by the World Bank, your team will have to make certain changes to procurement-related documents. This includes updates to the:
​​
  • ​​Survey firm Terms of Reference
  • ​​Make sure you update the survey terms of reference (TOR) and the survey firm’s technical proposal to include all relevant changes in protocols viz. mode of data collection, change in survey duration, change in timeline, and so on.
  • ​​
  • ​​Survey budget / proposal
  • ​​The survey firm conducting the data collection should submit an updated budget accounting for the changes in costs as a result of the transition - including equipment, incentives, and survey costs. 
  • ​​In case  additional funds need to be disbursed by the survey firm, the contract will have to be modified. This typically requires approval from the organization funding the survey firm, or from Corporate Procurement in the case of World Bank funding.
​​
​​DIME Analytics can review these updated documents.
​​
​
​Update the survey instrument(s)
​​You may have to radically overhaul the survey instrument, in order to meet the following requirements:
  • ​​Keep the survey short.
  • ​​
  • ​​It’s absolutely essential to keep the survey super short and the maximum length of a survey can not be over 40 minutes.  A pilot study by IDInsight 10-15 minutes as the ideal length for a phone survey. 
  • ​​The average survey length should not be greater than 20 minutes.
​​
  • ​​Note that reducing the length of a survey takes considerable amount of time. Make sure to account for this in the timeline update. You can try the following to reduce survey length:
  • ​​
  • ​​Drop modules which can not be asked over phone (for example GPS coordinates)
  • ​​Identify the most essential outcomes of interest and keep questions related to these outcomes 
  • ​​Ask about aggregates instead of individual items.
  • ​​For example, an in-person consumption module which goes over 25 individual items should be cut down in a phone survey to as few categories as possible (such as durable items, food, etc). When aggregating it is essential to define what items fall under which category and this should be made clear to the respondent during the survey.
  • ​​Break the survey into 2 or 3 shorter surveys. 
  • ​​You can do this if key outcomes of interest can not be measured in one survey.  However, you must remember to account for differential attrition across these instruments as all respondents may not complete the entire set of surveys. 
​​
  • ​​Update the consent to a verbal consent.
  • ​​A short and easy to understand verbal consent should be included in the survey.  This should also inform the participants about the audio audits.
  • ​​
  • ​​Add identifiers at the start of the survey.
  • ​​This helps to ensure the correct respondent is interviewed.
  • ​​
  • ​​Collect as many telephone numbers as possible.
  • ​​This is particularly helpful in case you are planning a follow-up survey.
  • ​​
  • ​​Make the questions as concise and clear as possible.
  • ​​Add hints and definitions throughout.
  • ​​
  • ​​Code and repeatedly test the updated survey for electronic data collection
  • ​​SurveyCTO has built useful resources for an easy transition to a phone survey. 
  • ​​Note: The CATI starter kit is not applicable on the current version (v2.6) of the World Bank SurveyCTO server. It will be available on the World Bank SurveyCTO server as soon as it is upgraded to SurveyCTO 2.7.
  • ​​
  • ​​This adds an additional layer of check on data quality, which is necessary for such a drastic transition in the method of data collection.
​​
​​DIME Analytics can review the coding of your survey and provide further inputs on reducing survey length.
​​
​
​Update data collection protocols
​​The majority of the protocols set up for in-person data collection will need to be revised. Make sure allocate enough time for these updates to be put in place. These changes are as essential as the survey instrument updates.  You will have to either change the following protocols, or confirm that they still work in the new context:
​​
  • ​​Set up of the team.
  • ​​Ensure that the set up of the survey team is such that there are not too many enumerators reporting to 1 supervisor (no more than 5). You should consider adding more supervisors if needed (remember, you’re already saving on budget with no travel to the field!) 
  • ​​
  • ​​Hiring.
  • ​​If you require to hire enumerators/supervisors, make sure to hire locally and find candidates who have previous enumeration experience. 
  • ​​
  • ​​Survey manuals.
  • ​​The survey manuals will have to go through a detailed update to ensure it accounts for all changes put in place. See here an example of survey protocols from Tavneet Suri’s Kenya UBI project.
  • ​​
  • ​​Procuring equipment.
  • ​​Additional equipment will ideally have to be procured and made available to the enumerators. This includes, telephone or mobile devices, SIM Cards, money to recharge SIM cards, tablets/computers to enter data on, and internet access for all parts of the process - from training to sharing data.
​​
  • ​​In the current work from home situation it is probably going to be a difficult process to get these equipment to the field team. A few ways to do this are
  • ​​ship the equipment to each enumerator
  • ​​use the enumerator’s smartphone for data collection instead of tablets
  • ​​let the enumerators buy the equipment needed (making sure this doesn’t break any of the the local lock down protocols) and reimburse them for the costs
  • ​​
  • ​​While there is no one-stop solution as each context will have different challenges, teams will have to be creative while trying to ensure data security in procuring and transferring equipment and data to the field team. 
  • ​​
  • ​​Communication within the team.
  • ​​Decide with your team about communication flows within the team. This should include details about :
  • ​​who is the first point of contact for each person in the team’ (for all enumerators, supervisors, and above)
  • ​​how enumerators will be provided with the sample they have to call on a daily basis
  • ​​who will ensure enumerators have enough phone credit to call respondents
  • ​​what data will the enumerators submit on a daily basis (or a regular previously decided upon frequency)
  • ​​how will enumerators share the data collected (submit forms at the end of each day)
  • ​​who will monitor enumerator performance (e.g. number of phone calls completed per day)
  • ​​how will issues identified in the data quality checks be conveyed to the enumerators
  • ​​whom the enumerator should reach out to in case of any questions or if a respondent requests an escalation. It often works well to create a WhatsApp group with all enumerators to stay connected
  • ​​Always have a plan B (and C) in place in case the decided upon plan fails for any reason (like connectivity issues)
  • ​​Tracking sheets become a more crucial component during phone surveys. These should be created as a digital survey form the enumerator fills in as we can not share printed sheets on a daily basis. A tracking sheet should be provided to each enumerator to be able to track
  • ​​Respondent details
  • ​​Updated contact numbers 
  • ​​Status of survey
  • ​​Call back time in case respondent was unavailable
  • ​​Number of attempts 
  • ​​Time of each attempt
  • ​​
  • ​​Someone other than the respondent answers the phone.
  • ​​Create a protocol which directs the enumerator on this front. The enumerator can try to get other contact numbers the respondent can be reached on, take an appointment at a time the respondent will be available,  if the enumerator can survey another member if the respondent is still not available, or to replace the respondent.
  • ​​Appointments. 
  • ​​Very often respondents are not available to speak when an enumerator calls. A protocol can be put in place to allow respondents to provide times they are available to talk at and to ensure that the enumerator calls the respondent at the time of the appointment.   
  • ​​Replacement of respondent.
  • ​​Put in place a protocol for how many attempts an enumerator should make to reach a respondent before the respondent is replaced. It is also essential to think about how far apart these attempts should be from one another to be able to catch the respondent at a time they are free to speak. Example of a good protocol is 9 attempts with maximum of 3 attempts per day and each attempt spread at least 3 hours apart.
​​
  • ​​Data quality checks. 
  • ​​The data quality checks that will be run, the frequency of running the same, and how the results will be communicated to the teams should be decided on before any data collection starts.
  • ​​
  • ​​Ensuring data security.
  • ​​A data security protocol should be put in place to try and ensure no data is lost. This includes encrypting the survey form and providing guidelines to enumerators about how to share data. Create a confidentiality agreement that each enumerator signs (digitally) in an effort to ensure the respondent contact numbers do not get used for any other purpose
​​
​​DIME Analytics can support with thinking the protocols through and providing feedback on any protocols put together.
​​
​
​(Re)Train the enumerators
​​The survey team will have to be extensively trained on the protocols, the survey instrument, and ways to keep the team engaged. This is one of the most challenging tasks as there is no way to get the entire survey team together and conduct an in-person training. The training has to be done virtually, and you will need to be innovative with multiple alternative plans for training the enumerators. 
​​
​​Here are some guidelines to keep in mind when designing the training.
  • ​​Logistics
  • ​​Have smaller groups of enumerators for the training. 
  • ​​If there are many enumerators break them down into smaller manageable groups (typically 5) to make it easier to conduct the training.
  • ​​
  • ​​Use a calling platform like Skype, Zoom, Webex.  
  • ​​Depending on bandwidth, try to conduct the training using video to provide some sense of personal connection. If a live virtual training is not possible, consider alternatives such as recorded videos and phone calls. 
  • ​​You can record the training and share with enumerators for future use or in case they missed parts of the training due to connectivity issues.  Also, be sure to allocate more time in the training schedule for these additional steps, or in case things go wrong.
  • ​​ 
  • ​​Content
  • ​​Provide the team with a detailed training manual (example here). You should have a detailed script for the entire survey instrument(s) so the enumerators know how to ask the questions.
  • ​​Create recordings of test interviews. You can then use these for the training.
  • ​​Conduct more mock interviews during virtual training than you would in an in-person training
  • ​​Include content on how to probe respondents on each question 
  • ​​Train the enumerators to answer potential questions the respondents might ask. These can be questions about the project, the design, the purpose of the interview, on each question, or anything else.
  • ​​Include suggestions on how to keep respondents engaged during the interview. Some examples include
  • ​​Building on established rapport between enumerator and respondent
  • ​​Compensating respondents via mobile money or airtime if feasible
  • ​​Including the expected survey duration in the consent form.
  • ​​Attempting to answer each question of the respondent as comprehensively as possible
  • ​​Being inquisitive and kind with the respondent
  • ​​Offering to escalate to the manager in case of any issue 

​
​During collection

​​Once all the items listed above are in place, you should be ready to start data collection. While the major heavy lifting happens in the run up to the data collection, there are 2 main components to keep track of during data collection that are different in case of a phone survey.
​​
​
​Communication during data collection
​​Once data collection starts, make sure you and the rest of the team communicate frequently. To reiterate some of the communication mechanisms.
  • ​​Set up strong monitoring and communication protocols before starting data collection (test them during training or a pilot)
  • ​​Set up the hierarchy and communicate to enumerators who their first point of contact is
  • ​​Have daily check-ins with each enumerator
  • ​​Conduct survey team wide meetings once or twice a week
  • ​​Consider creating a WhatsApp group with supervisors (and enumerators if the survey team is small)
  • ​​Monitor data closely on a daily basis and provide regular feedback to enumerators
  • ​​Don’t forget to ask the enumerator how they are doing and if they have any concerns whenever communicating with them
​​
​
​Data quality checks
​​Data quality checks should be run more extensively and monitored very closely during a phone survey. It is essential to keep track of outcomes of interest as there’s a level of self selection that comes with phone surveys. This can result in samples which are not entirely representative. The following are some updates that can be made to data quality assurance plans:
  • ​​Track progress and completion rate of surveys more closely 
  • ​​As the survey is short, include consistency checks on all questions if feasible
  • ​​Include more stringent enumerator checks and ensure any learning from these are communicated to the enumerator in a timely manner
  • ​​Insert audio audits at predefined or random points of the survey. Have the supervisors review the audio audits on a daily basis and cross check the responses they hear with the submitted forms. You can create an audio audits checks form for the supervisor to fill in while reviewing audio audits. 
  • ​​If conference calls are feasible, the supervisor can do accompaniments and shadow the enumerator 
  • ​​Have the supervisor call a few respondents to confirm the enumerator did in fact call them.
  • ​​Back checks can be run on certain questions if they seem feasible in the context.
​​
​​We recommend that the data quality checks should be closely tracked throughout data collection. This is a departure from in-person data collection where the emphasis on data quality checks is higher during the initial days of data collection.
​​
​​DIME Analytics can review and provide feedback on the data quality assurance plan put in place.

​
​Other modes of data collection

​​In addition to phone surveys, there are other alternatives for remote data collection as well. Below is a brief overview.

​
​Web surveys

​​In a web survey, a link to a programmed survey is shared with respondents via text or email and the respondent answers by clicking on the link (either on a phone or a laptop). For this mode of data collection to be successful, it is essential that respondents have access to internet and that respondents are literate. 
​​
​​If the survey environment allows for it, transitioning from an in-person survey to a web-based survey can be a low-cost effort. It reduces survey costs considerably as there are no enumerators involved. Additionally, SurveyCTO programmed surveys can be used for web-based data collection. Web based surveys have the disadvantages of lower response rates because of self-selection into answering the survey. Additionally, self-selection into the response pool can bias the results.

​
​Recorded surveys

​​The survey instruments are pre-recorded and examples of recorded surveys include IVR, SMS/Text surveys, and WhatsApp surveys.  These surveys work well when the survey is very short (with 4 or 5 questions or <5 minutes on average), with zero or minimal skip patterns, and survey questions are very straightforward to understand. The downside of a recorded survey is that there is a lack of human connect and the inability of a respondent to probe further on any question. This can lead to lower response rates or potentially noisier data (if questions are subject to interpretation).
​​
​​To summarize, each mode of data collection comes with its pros and cons. Before diving into transitioning to a different mode of data collection it is essential to consider the feasibility of each method of surveying in the context you’re working in. There is no written rule that one data collection mode cannot be mixed with another. You can be innovative and combine modes of data collection to ensure higher responses rates. 

​
​List of resources 

  • ​​Blog post by Herath et al.’s on general tips on training and practicing
  • ​​Blog post by Özler and Cuevas expands on how to increase response rates
  • ​​Blog post by Goldstein and Kondylis on IEs in the time of Covid-19
  • ​​Blog post by Goldstein on using phones for repeat surveys
​​
​​
​​
​​