The Evolution of Data Collection
Using Magpi iSMS to collect data without going to the field
The first paper on mobile based data collection in the health context was written almost thirty years ago, in a 1989 paper entitled “Revolutionising health data capture: use of hand-held computers” by K.C. Lun et al.
That very early work, almost twenty years before the iPhone introduced us to modern smartphones in 2007, was based on the use of primitive electronic devices like the Psion, and later the Palm Pilot – which was the first Magpi mobile platform (under the original name “EpiSurveyor”).
Although we never worked with the Psion, for almost 20 years, Magpi mobile data collection has worked with all sorts of portable data collection device — basic phones, tablets, smartphones and more — and been a leading cloud software tool powering the transition from paper-based data collection to digital mobile data, allowing our users to utilize:
Apps for mobile data collection on Android and iOS devices
Two kinds of SMS-based mobile data collection
Chat-based data collection in any browser
All of our data collection tools, including our mobile phone data collection software app, integrate with more than 1,000 other online systems, without programming — so you can use Magpi to collect your mobile data and then easily send it to any of your existing systems. We’re talking Microsoft PowerBI, Tableau, Google Sheets, Excel, and more.
And while data collection on paper is still around, we’re happy that with a mobile phone in the hands of almost the entire world adult population (and a lot of teenagers), we are more and more able to support mobile data acquisition on electronic devices, rather than paper.
Magpi has been dedicated to helping to make this transition from paper to digital happen.
Early data collection with Magpi on feature phones
Mobile Data Collection for You
The rapid proliferation of mobile computing and communications technology in the last two decades means that big and small organizations now have more mobile data collection methods than ever for mission-critical field data collection programs — from smartphone- or tablet-based mobile data collection apps to SMS or chat-based systems, to “interactive voice response” (IVR) systems.
When it comes to choosing how mobile data capture will be performed in the field, or on the move, or even in the office or home, there are several factors to keep in mind, including:
the type of data you are collecting
who are you collecting that data from
what tech is available at the point of data collection (smartphone? laptop?)
what connectivity is available at the point of data collection
Below, we'll be discussing whether the powerful tool of mobile data collection is right for your organization. We'll review the key benefits of mobile data collection depending on the type of data being collected and the purpose it will serve, the impact that your available resources might have on the decision making process, and how to start the process of choosing the right option for your needs. And then we’ll talk a bit about when mobile data collection might not be right for you.
CONSIDERATIONS IN MOBILE DATA COLLECTION
It is important to take into account the environment your mobile data collection will take place in. There are various mobile data collection solutions available, each with different strengths and weaknesses, and maintenance requirements. It is essential to evaluate these solutions based on your program's needs and constraints, including the region you operate in, the scale of your program, and the digital literacy of your team.
One crucial factor to consider is the reliability of the mobile or Wi-Fi connection in your project region. In areas with weak or unstable connectivity, data collection can be challenging and time-consuming, leading to errors and delays. Similarly, the digital literacy of your team members can affect the ease of using a mobile data collection app. If your team is not familiar with mobile technology or has limited access to smartphones, you may need to consider simpler solutions or provide training.
It is important to note that mobile data collection tools are not a one-size-fits-all solution. A poorly-designed program will not be fixed by using a mobile app, and a basic survey tool may not be sufficient for a complex case management program. Therefore, it is crucial to assess your program's needs and choose a mobile data collection solution that aligns with your goals and requirements.
Choosing the right mobile data collection application can improve the accuracy and speed of data collection, enhance your team's productivity, and ultimately, amplify the impact of your program. However, it is crucial to keep in mind the environmental implications and consider all the factors that may affect the success of your program.
Benefits of mobile data collection
For those who are collecting mostly non-qualitative data on the move, mobile data collection offers many benefits compared with paper.
Reduced Cost 💰
Moving from paper to electronic has the potential to save big bucks (71% in one World Bank study!). Handling (and photocopying, and storing) paper is an expensive process for you. Electronic data collection usually eliminates those expenses and creates major savings. If you’re not careful, though, you could blow all those savings on programmers or tech consultants, all the savings will be lost to consulting fees, salary and other personnel-related costs. Mobile data collection systems like Magpi that allow existing staff to set up and operate the system themselves without specialized tech expertise can provide the greatest cost savings.
“In a 2011 study of the Magpi MDC system by economists at the World Bank, Magpi was found to be 71% less expensive on average than paper-based data collection - an enormous difference.”
Reduced Environmental Impact 🌲
Environmentally, it's harder to imagine a more costly approach than paper-based data collection. It's been estimated that a single survey of 5,000 households consumes twelve trees, thousands of gallons of fuel (to transport the paper), and tens or hundreds of thousands of gallons of water (to produce the paper).
But with the explosion of mobile devices, and the drop in their cost, the un-ecological use of paper is giving way to a greener approach.
Increased Speed ⚡️
If you're collecting data in an area with network coverage, your data can flow in milliseconds from the point of collection into a database online – and just as quickly can trigger a variety of actions, including messages to key personnel, instant report generation, automated backup, or forwarding of all or some of the data to another system. Even if you're collecting data offline, in the far field, you still get the benefit of near-instant analysis and response to your data – once you get to a network and upload it.
This benefit was in clear evidence during the Ebola crisis in West Africa in 2014-2015, when CDC, WHO, the Red Cross, the US military, and others turned to Magpi for a variety of purposes including funneling field reports of suspected Ebola cases to authorities – in some cases using SMS zero-reporting from the simplest of mobile phones, rather than smart phones.
And the advantages of mobile data collection have been equally evident during the worldwide COVID-19 pandemic.
Increased Data Quality 🥇
Basic Quality Measures
Any good mobile data system will allow you to impose certain quality controls at the point of data collection, thus reducing "data cleaning" later. These quality controls can take the form of multiple choice questions (which are enforced on an electronic device, unlike on paper), numeric questions with enforced range checks, and automatic logical branching ("skip patterns") that lead a user through a form based on their responses.
New Tools for Supervision
The immediacy of mobile data collection data availability allows for a higher level of supervision – hence a higher level of data quality – than was ever possible before.
Magpi’s web dashboard automatically provides supervisory meta-data analysis
Data can be examined as it is collected and uploaded, in real-time (network permitting), not days or weeks later. An often cited example of this real-time supervision involves looking at the location where each data point was collected, but it's just as useful to look at how long each interviewer is taking to fill out a complete form, and how long they are taking between interviews.
Magpi provides this kind of supervisory meta-analysis on the fly, as you upload any data into the system. Just lot into Magpi and on your dashboard you’ll automatically see:
how much data has been collected by each person submitting data
how much time do your data collectors take to complete a form or survey
a map showing where each data point was collected
None of this needs to be set up or programmed: it’s just there for you, instantly.
Automated Feedback 💬
Advanced modern mobile data collection systems like Magpi can provide instant feedback for submitted data.
This can take the form of a simple SMS acknowledgement going out to field staff who have submitted data – or the system can evaluate the Magpi data provided and ask for confirmation, or alert a supervisor for suspect values. Or the data can automatically update a report that is available to field staff, letting them know how they’re doing — as they do it.
The most advanced systems like Magpi can even go so far as to create to-do items or calendar events based on incoming data – in ways that were never possible with paper systems. Imagine if, for example, key data collected in the field indicates that certain work sites need to be visited for further review. A Magpi user with no programming or technical skill could set up the system so that a Google to-do item would be created to “Review Site A” and assigned to the correct supervisor.
Easier To Scale Up with Automatic updating 📲
The larger your data activity on paper, the more unwieldy it becomes and the more likely some of that paper will be lost. In addition, if any changes need to be made to the data collection form, paper becomes a nightmare of cost and logistics, as all previous blank versions of the form need to be tracked down and destroyed, and new versions need to be printed and transported to each and every data collection operative.
A good mobile data collection system, on the other hand, automatically "pushes" any form updates over the network to each mobile device being used for data collection. This automatic syncing of changes means that everyone is always using the right form. This can save hours or even days of configuration work – as well as the headaches that result when you discover that not all field workers are using the same version of the form.
Simple Storage and Backup ☁️
Digital records are easy to store and easy to copy, meaning that once they are in the system you have safeguards against data loss. They can also be encrypted, increasing the security of the system.
When your original data is paper, on the other hand, there is always the possibility of losing the originals – either after or even before "data entering" them into an electronic system for analysis. Paper is also easy to lose – but a digital record, once in the system, can be backed up to one or more locations.
If your mobile data collection system incorporates reliable automatic backup, this makes it harder to lose all your data, and easier and much less expensive to store for long periods of time. In Magpi, all Pro and Enterprise users can set up automatic backups to Google Sheets or Excel in just minutes with no programming, with just a few clicks. Make sure to ask if this feature is incorporated into any system you’re considering..
Nowadays there are numerous cloud-based mobile data collection systems. With Magpi and others you no longer need to worry about maintaining your own data server computers, antivirus, encryption, etc. As with other web-based systems like Gmail or Facebook, all the work of maintaining the system is taken care of by the provider, behind the scenes. Which leads to even more cost savings.
When not to use mobile data collection
There are many mobile data collection options on the market, and accessibility is quick and easy thanks to the proliferation of cell phone technology and cell networks around the globe.
So it stands to reason that mobile data collection is a good fit for most organizations, right? Not necessarily.
There are some situations in which the shiniest new tools on the market aren't necessarily the best fit.
For example, when the data you'll be collecting requires a lot of "qualitative data" – data that cannot be reduced to multiple choice or numeric questions – mobile data collection may just not be practical.
Qualitative data collection may require typing long free-form text, or recording interviews and later transcribing them from the recordings, or some other approach – and the outlook is constantly changing as new technology, like voice-to-text transcription, becomes more widely available.
Another situation in which mobile data collection might not make sense is when the respondents are … not mobile. For example, if the people who'll be providing your data are all sitting at desks in front of internet connected desktop computers, an electronic but not mobile solution may make the most sense – whether it's Survey Monkey, or simply Microsoft Excel or Google Forms.
As noted above, this last part is changing fast: all of us find ourselves doing more and more on our phones and tablets, and relatively less on our “non-mobile” computers, as the former get faster and more useful.
But one thing’s for certain: mobile data collection really shines when you're collecting numbers, multiple choice questions, dates, photos, etc. – and you're doing it on the go.
And if you’re not sure if the data you need is best collected on a phone or on a desktop, or you don’t know if you can switch your current desktop approach to a mobile device, get in touch: we’d be happy to give you the benefit of our experience, at no cost.
The Best Mobile Data Collection Platform
Just as they say the best camera is the one you have with you, the best data collection platform is the one that you’ll actually use — because it’s cost-effective, easy to set up and use, and gets the job done. Different organizations have different access to resources, and work in different settings, and this will affect which platform, and devices, you need.
Magpi’s web interface lets you easily build highly customizable forms
Below you’ll find some of the considerations in making sure you get the best possible mobile data collection platform for your particular circumstance.
Budget
Programmers Are Expensive
As mentioned above, the Magpi mobile data collection system has been shown by the World Bank and others to be dramatically less expensive than using paper, in addition to providing the other benefits of an electronic system.
But not all mobile data collection systems are created equal. While some, like Magpi, keep costs very low by eliminating the need for programmers or technology consultants, others require programmers for every aspect of setup and refinement of the system. Such systems wind up being substantially more expensive than paper, rather than less.
Does your organization have the ability to pay programmers’ salaries and benefits? Or pay tech consultants? If not, make sure that any system you consider can be set up and maintained by regular staff members.
Total Cost of Operation (TCO)
As a result of factors like the cost of programmers, when you are evaluating various mobile forms software, it is crucially important to know in advance the total cost of setup, operation, and maintenance.
These costs of mobile data collection include:
Software licensing costs
Programming costs
Server/storage costs
Setup and administration costs
Communications and data transmission costs
Travel costs (for some systems, programmers or tech consultants must be travelled to the field, which adds enormously to the total cost)
Costs per user
Costs per data volume
Power
Mobile devices need to be charged. If you will be collecting data in the far field, this may mean you'll need backup batteries and/or solar chargers and/ or car chargers.
“At Magpi, we believe the question “how to create a mobile data collection form” should be answered “easily, by anyone, without programming”.”
It’s also worth reading any online reviews of the devices you’re considering to see if they have unusually long (or short) battery life. A last consideration is to use device settings to minimize power consumption: turning off bluetooth and other transmitters when not needed, and reducing screen brightness, will add battery life. If you need to collect location data, you’ll have to keep the GPS setting on, but this will negatively impact battery.
Devices
Each mix of available devices enables a different data collection strategy. Ideally, a mobile data collection that is cross-platform and flexible in implementation is ideal. The more types of mobile devices that can be used, the more likely you can find devices that meet your specifications and your budget.
Simple Devices, SMS, and Chat
If the data collectors will be using simple “dumb” phones, this used to all-but-guarantee that SMS would be the channel over which data is collected — but modern browser-based chat systems are now available that can work just as well on a high-powered laptop, a workhorse desktop, a smartphone, OR a basic feature phone. And chat gets rid of many of the costs and problems associated with SMS data collection.
If using SMS-based data acquisition, you can use literally any mobile device with SMS capability, no matter how inexpensive. That said, before purchasing in bulk, you will want to test out the device you're considering to ensure that, for example, it allows the language or alphabet you want to use for your text data.
Smart Devices and Apps (or SMS or Chat)
If, on the other hand, you have access to smartphones or tablets, then your choices are more broad. You can use mobile apps for data collection – or you may decide that simple SMS may suit your purpose better, even on a smart device.
If using a mobile app like Magpi’s, you'll need to check the optimal requirements for compatible devices. Notice that we didn't say "minimum", but "optimal": it is a common error for teams to choose the absolutely least-expensive device available, often from little-known manufacturers.
This is a very bad idea.
If you are going to buy a car to drive across town to the grocery store once a week, an inexpensive subcompact might work very well. On the other hand, if you're planning on driving across the Sahara desert, that subcompact car would be a very bad choice.
With mobile devices, if you are planning to buy an Android phone to send a few texts and keep your electronic calendar, that's the equivalent of the grocery shopping: an inexpensive model may do very well. On the other hand, if you'll be deploying a 500 question form that includes lots of photos, complex logic, etc, you should plan on a device with more RAM memory and more storage – from a well-known and reputable manufacturer.
One tip: try Googling “best cheap Android devices” to find some well-reviewed and less expensive smart devices.
Device Setup & Management
Provisioning Tools
Nowadays it's super-easy to install an app on a smartphone, so if you won't be managing too many devices your work shouldn't be hard. If you're managing a field workforce of 25 or 50 or 100s of workers, though, it can easily become a nightmare.
Apple and Google both provide good tools to configure and manage large deployments of iOS or Android devices. In fact, Apple recently announced an expansion of their Business Essentials program — which includes device management and support — to small and medium business.
If you already have an IT policy in place for maintaining mobile hardware, integrating new software will be easier, but if this is all new to your organization, make sure there is someone responsible for this process.
Automatic Updating
It's important, regardless of platform, to use a mobile data collection system that provides automatic "push" of new forms and form changes and deletions to the devices. In the old days, every time a new version of a mobile form was created, someone would need to manually remove the old version from each device, and then download the new version.
In modern systems like Magpi, every time a form is changed the changes are instantly pushed to all devices that have access to that form. This can save hours or even days of configuration work – as well as the headaches that result when you discover that not all field workers are using the same version of the data collection instrument.
People
Who will you be relying on to do the data collection? Are they trained and experienced professionals, well-versed in the subject matter of the data collection?
Or perhaps less experienced volunteers?
Or are you planning to collect data from the general public or others that don't work for you (visitors to a facility, perhaps)?
How much time and budget is available for training?
The answers to these questions may limit the type of mobile data collection system that you can deploy. For example, while SMS-based systems are generally simple enough for everyone to use, including the public, app-based systems usually require some formal training, if only for a day or two.
Timing
Are you planning a one-time survey? Or designing an ongoing system to bring in data from mobile or field sites into a central level for years? The chronicity of your actives will affect your budget, the cost of maintenance, and other factors.
These two extremes of timing also require different types and amounts of training, even apart from the complexity of the forms that will be used, or the type of personnel.
We remember things we repeat many times, and ongoing data acquisition itself becomes a self-reinforcing training activity for the people involved. One-time activities require more training.
Training
The key to training on any mobile platform is practice. To train field workers how to use mobile data collection systems, you must allocate enough time to let them make mistakes, practice in the field, and repeat. Classroom work won't be sufficient to prepare for the field.
If your activity is for the long term, keep in mind that use of the system over time will, in itself, provide a kind of training. This is especially true because the person overseeing the system will have a real-time view of the data that is being submitted. Of course, some types of mobile data collection can be used with no training at all. Systems like Magpi's Chat (or our iSMS "interactive SMS”), use messaging back-and-forth exchanges to collect and provide information with a data collector, or a member of the public (the latter is called “crowdsourcing"). These systems often make it seem like you are chatting with another person, rather than a computer, and this can increase participation and compliance.
Data Transmission
Where will the data be collected? Is there good connectivity there? If not, can you afford to wait for your data to be uploaded from a different location? Or is the data extremely time-sensitive (e.g. in emergency settings, or with time-critical equipment checklists).
If information is very time-sensitive but will be collected in a location where no cellular data network is present, you may wish to utilize an SMS-based data collection system.
In many emergency situations there is substantial disruption of cellular and wifi connectivity, and so it may be necessary to have an mobile data collection system that connects easily with satellite communications systems – allowing urgent data to be transmitted without delay.
Security
Two main types of security concerns can impact your choices regarding mobile data collection: petty theft and data privacy. There are, of course, many other security concerns when in remote locations, including personal safety, but these don't usually impact the choice of mobile data collection system or equipment.
Petty Theft
If you plan to be collecting data either long-term or short-term in an area with substantial risk of petty theft, you will probably not want to use attention getting devices like large, expensive tablets – unless you have budgeted for security personnel. Another option, however, is to utilize a chat-based mobile data collection system that can run on very inexpensive devices that are unlikely to draw unwanted attention, or on the phones or tablets that your team already has. Magpi allows you to quickly and easily set up chat-based systems, as well as SMS-based systems, both of which have this advantage.
Data Privacy
With regard to data privacy/security, as when you are collecting valuable sensitive information that may put you at risk, you can consider using Android or iOS devices and enabling full-device encryption. Such encryption, layered on top of a password-protected mobile data collection app, provides more layers of protection than just using SMS. If using a cloud-based system, you will also need to know where your data is being stored online, what safety measures are in place, if it is automatically backed up and how often, is it monitored 24/7/365, etc. Cloud-based systems are usually monitored continuously, with robust security and privacy controls.
Just a few data visualizations available in Magpi Reports
Some mobile data collection providers choose to work with local servers to store data, but very frequently those local systems are not adequately maintained or monitored. Make sure that your confidential data is not being stored on, or backed up to, someone’s laptop computer! Keep in mind also that you may very likely need a tool like Magpi that complies with international regulations like the EU’s GDPR.
Analysis & Reporting
Some mobile data collection incorporate mapping, analysis, and reporting tools, while others do not. Some of the tools require programming or other technical assistance – which always drives up the cost.
Before finalizing on an mobile data collection, make sure you have an idea of the kind of reporting output you will need, and how much it will cost to create it and maintain it using the solutions you are considering.
Integration
As more and more applications move to the cloud (that is, they are located on servers shared over the internet, and typically accessed via a web browser or mobile app), it is becoming easier and easier to connect one with another.
The Zapier Magpi integrations page (click the image to visit)
Integration tools like Zapier allow non-technical people to connect thousands of cloud-based tools easily with no programming – including Magpi. Magpi Pro and Enterprise users can also connect their data to Google Sheets or Excel in just a few clicks — again, with no programming required.
This “no-code” approach means that it is now easy to automatically flow your data from the field to an SQL database, or a Google spreadsheet, or to DHIS2, and add automatic analysis in whatever analytic software you prefer.
It’s worth checking to see if the mobile data collection system you are considering has built-in integration with Google and Excel, and also integration with Zapier and OpenFn, because if it doesn’t you may spend your whole budget paying for programmers to custom-code your integrations.
Conclusion
Field work is an evolving process that might change rapidly during the course of a deployment.
Whether it's ongoing work for a utility company or an academic study requiring massive data collection, you need a flexible technology solution that will work seamlessly and affordably with your organization's existing processes. That's why data collection using mobile phones and tablets can be such a good fit for so many types of companies. It allows the kind of flexibility and increased efficiency that few other solutions do — if you’ve got the right software.
Still, there are many pitfalls, particularly around cost: many organizations fail to consider the ongoing costs of support and maintenance, and are then saddled with an expensive system dependent on programmers or technology consultants for each modification or adjustment.
If you are interested in learning more about how mobile data collection can benefit your organization, help you reduce costs in the field, and improve data accuracy and availability, contact Magpi today and ask to learn about our mobile data collection, checklist, survey, and SMS data collection solutions for field work.
FREQUENTLY ASKED QUESTIONS
Why is mobile data collection important?
Mobile data collection is an important tool for researchers because it enables them to collect data quickly, accurately, and in real-time using mobile devices. This method is particularly useful for research projects that require data to be more efficiently collected from multiple sources and reduces the time and costs associated with traditional paper-based methods. Mobile data collection also offers improved data quality, increased flexibility, and enhanced data security, making it a valuable tool for researchers in a variety of fields.
Who can use mobile data collection?
Mobile data collection can be used by researchers across many fields, including the social sciences, market research, and healthcare. A researcher or organization collecting data from multiple sources, locations, or respondents can benefit from using this method. This includes academic researchers, government agencies, non-profit organizations, and businesses. With the increasing availability of mobile devices and data collection tools, this type of data collection has become more accessible to researchers at all levels.
What are some challenges in data collection, and how can mobile data collection help?
Some challenges in data collection include finding a representative sample, ensuring data accuracy and completeness, protecting data privacy and security, and managing data entry and storage. Using mobile data collection can help with some of these challenges in several ways.
For example, mobile data collection tools can help ensure data accuracy and completeness by including built-in validation checks and reducing the risk of errors associated with manual data entry. Mobile data collection can also help with missing data by allowing for real-time data collection, follow-up, and reminders.
Additionally, mobile data collection tools can include features to enhance data security and privacy, such as password protection and encryption. Finally, mobile data collection can streamline logistics such as data entry and storage, reducing the time and costs associated with traditional paper-based methods.