Mobile Data Collection: How Big Data Is Collected

Magpi Plus mobile data collection app

Magpi Plus app

For 15 years, Magpi mobile data collection, or (MDC) has powered the transition from “big paper” to big data, allowing our users to utilize:

And all of our data collection tools integrate with more than 1000 other online systems, without programming — so we’ve got modern, electronic data collection covered! 🙂

But collecting data is not a new thing, and the answer to the question “what is mobile data collection” depends on what period of time you’re talking about: people have been collecting data in the field and on the move for thousands of years.

The Domesday Book

An early survey in 1086

Medieval Management Data

MDC activities have existed in various forms for thousands of years (the Egyptians were famously data-centric in managing their empire). One of the most famous was the “great survey” ordered in 1086 in England by King William the Conqueror, which sent data collectors all over the country to find out how many people and livestock there were (among other “indicators) – and which resulted in the “Domesday Book” of data.

The 20th Century: Big Paper, Souped-Up Statistics

writing data on paper form

In the early 20th century people like Jerzy Neyman helped to develop the statistics, and selection and sampling methods that formed part of the much-improved survey methods of the 20th century – and this brought about a “big paper” revolution in the amount and quality of data collected on paper and available to both governments and also businesses for planning and action.

The US Census Bureau, bound by the Constitution to collect data from every household in the US for the decennial census, was still able to utilize sampling for the first time in 1937 for a survey of unemployment during the Great Depression.

world health organization survey

Cluster surveys: a revolution in sampling

Later researchers developed new rapid survey techniques like the 30×7 cluster approach used by the Expanded Program on Immunizations, designed – like earlier sampling innovations – to enable more data to be collected more quickly and at lower cost.

Up until the 1990s, however, all these approaches were based on the use of paper forms, and so while a clipboard and pencil are technically “mobile”, nowadays when we speak of MDC we’re really talking about mobile electronic data collection using mobile phones, SMS, and other technologies that were only invented, or made widely available, in the last two decades or so. In the next section, we’ll talk about those.

The Twenty-First Century: Meet Mobile Electronic Data Collection

entering data via mobile device

Magpi on the Palm Pilot, 2007

The first paper on the use of mobile electronic data collection in the health context was written almost thirty years ago, in 1989: “Revolutionising health data capture: use of hand-held computers” by K.C. Lun et al.

That very early work, almost twenty years before the iPhone introduced us to modern smartphones, was based on the use of primitive electronic devices like the Psion, and later the Palm Pilot – which was the first Magpi mobile platform (under the original name “EpiSurveyor”).

Of course, with the incredible spread of mobile phones around the globe (an “invisible computing revolution“) MDC has also spread and changed. In “The Four Phases of Field Data“, we explored the gradual changes seen, and still to be seen, in how mobile and pervasive computing is affecting data collection:

Phase 1 (3000 B.C. – present) – collecting data on paper – oh, yes, this is still around!

Using Magpi iSMS

Using Magpi iSMS to collect data without going to the field

Phase 2 (1989 – present) – replacing paper with phones and tablets – as mobile phones were affordable by organizations: replicating paper-based approaches (i.e. sending data collectors to the field) with electronic devices

Phase 3 (~2010 – present) – eliminating data collection trips to the field – as mobile phones became affordable even by the poor: collecting data from people in “the field” (patients, staff, beneficiaries, citizens) using their mobile devices, including SMS and IVR.

Phase 4 (just starting) – the Internet of Things – as computing and connectivity spread beyond our phones to our clothing, our homes, our streets, and elsewhere: data collected by machines without human intervention

What’s Coming (and Fast): Mobile Data Meets Big Data

collection data via mobile watch

Is this a watch or a doctor? (Kardia Band for Apple Watch – PRNewsFoto/ AliveCor, Inc.)

So we expect that all kinds of new technologies will begin adding adding data that we can put to use.  As an example, when the cost of smartwatches comes down, we’ll begin to see them used first for tracking bio-data in their wearers (like heart rate, but potentially including much more), and then later for the diagnosis of illness.  And as the cost drops still further we’ll see this applied even to the poorest populations on Earth.

Does that count as MDC? Sure, but it’s also much more, as data collection – for example, of heart rhythms – and data analysis – to diagnose a problematic rhythm – and data utilization – to potentially administer a shock to correct that arrhythmia – all begin to happen seemingly at once.

For those of us interested in data, and especially health data, we’re used to a very slow process whereby health data is acquired from the community, from surveys, from medical records, etc, and then analyzed and put to use in making recommendations and, well, changing health.  This “big data” process is going to get faster and faster, and demand ever more creativity and ingenuity in making sure it’s getting accurate information, analyzing it in a helpful way, and then using it to make things better.

But the payoff will be that in healthcare, global health, and many other fields, we’ll be able to use that faster data flow to react more quickly and achieve better outcomes.

Remote Data Collection

Download Our Remote Data Collection Guide and Learn How to Use SMS and IVR to Collect More Data, for Less Money, from Remote Respondents Using any Mobile Phone, no Matter How Basic.

Download Remote Data Collection Guide