Integrating IT into Survey Data Collection

Westat continually looks for ways to bridge IT capabilities into survey research to enhance projects’ efficiencies and cost effectiveness.  When carefully crafted, this framework can be easily applied and adapted to a range of projects to provide nimble and fundamental solutions in data collection, data processing, analysis, and dissemination. We reached out to Marcelo Simas, Ph.D., a Westat Associate Director, to share examples of Westat’s expertise in this process.

Dr. Simas came to Westat to provide focused expertise in developing the architecture for a comprehensive travel survey platform, which is still in use today for national, state, and local travel use surveys. The platform seamlessly integrates applications to other databases for data collection, management, reporting, and delivery. He has since branched out, supporting other project work with a range of tools and techniques for a diverse set of data collection projects.

For the Survey of Doctorate Recipients (SDR), for example, Dr. Simas notes that Westat used container technology to automate data ingestion and processing into a research data warehouse. This longitudinal survey gathers demographic, education, and career history information from a sample of 120,000+ degree holders located across the globe. As the data come in, a process has been established, using open source components, that continuously pushes the data through a pipeline of data checks and edits. The system also includes a portal with a large set of reports that are shared with the client and that allow up-to-date reporting on data collection, coding, and editing.

The Food Acquisition and Purchase Survey (FoodAPS-2) is another good example of integrating IT solutions into data collection and providing an enhanced, unified experience. FoodAPS-2 is a nationally representative survey of nearly 5,000 households on expenditures on food, along with detailed household and community characteristics. For a pilot study of 430 households, Westat created an app that prompts participants to report their food acquisitions at detected stops and integrates that process with other data sources, including restaurant and grocery store locations, produce, and UPC databases. This improved design reduces the burden on the survey participant, accelerates the data collection process, and improves data quality for analysis. By integrating IT tools with data collection methods, the survey is able to gather and process thousands of pieces of data in real time.

Westat also considers continuous integration/continuous delivery (CI/CD) as a key component. The process helps software development teams work more efficiently and reliably. As tools pick up changes in code, the CI/CD automates the delivery of system updates, looking for mistakes, alerting developers, letting developers review and improve code on a continuous basis. “This eliminates manual staffing of the process,” notes Dr. Simas. The server builds, checks, and updates running applications in a systematic and documented way.

“There’s great value in using open source software, allowing us to do more with less,” notes Dr. Simas. Tools Westat regularly works with include, but are not limited to, the R statistical computation platform for data processing, analysis, and reporting; PostgreSQL database servers, which include spatial data storage and processing capabilities used to perform GIS functions in several projects; Git for version control along with Westat’s internal GitLab instance, which serves as a central repository to allow Teams collaboration, track issues, and create CI/CD pipelines.

These tools and processes have been beneficial as Westat continues to find ways to work better by working smarter. “The pandemic has been a real catalyst for transforming the way we work,” notes Dr. Simas. “It’s really accelerated the move to a team approach, away from the individual working in isolation, who perhaps may have been struggling with a data processing problem.” By integrating modern IT collaboration tools like GitLab and Teams in support of projects, internal silos are broken down; staff who are using the same set of tools can more efficiently collaborate and find solutions to problems quickly.

It all adds value, notes Dr. Simas, to the internal process as well as to clients. “We can do more with less by collaborating and integrating these tools,” he says. “Taking a measured and collaborative approach to these challenges provides greater returns in the long run by sparking creativity and at the same time maximizing efficiency.”