July 7, 2021

Moving from Research to Prediction with SurveyStack: A Day in the Life of a Data Point

Octavio Duarte, June 2021

Summary: A walkthrough of the Our-Sci data pipeline, from field to prediction outputs. This is part of our journey to transform our tool suite from a specialized solution for a few use cases into a generally usable continuous data merging tool, resulting in an immediately useful database. 

The journey begins when a real-world object, such as a piece of produce, is selected to be evaluated by the Bionutrient Institute. For our purposes, we might say that information exists as observable qualities in existing objects. This information comes to us from multiple sources:

Figure 1 – The object in question.
  • Data self-reported by farmers or citizen scientists (location, farming practices, harvest dates, etc.).
  • Data obtained in the lab (dimensions, scans, chemical composition).
  • Data calculated using other data, mainly indirect measurements (transforming measurements obtained in the lab into measurements in units understandable for the general public).

These data points are compiled using SurveyStack tools, and then split into two significant categories:

Measurements 

Physical dimensions

Subjective look and taste

Spectroscopic registers (XRF, UV/Visible light and NIR)

Wet Chemistry analyses.

Metadata

These are pieces of information which are relevant and distinguish points from each other, even when they are not directly measurable on the object itself:

Source location

Variety

Agricultural practices in the field.

These two forms of information are registered in multiple surveys. The past year’s database is assembled from twenty-two different surveys, and a single piece of data can involve information from as many as ten of these.

Software 

Through SurveyStack, users can design almost any kind of survey, with inputs including advanced features such as hardware integration.

Fig.2 –  Introduction screen inside the survey builder

Surveys are divided into moments of data collection within both agricultural and lab processes. These can look like any of the following:

  •  An Onboarding survey filled the moment a farmer agrees to be a part of the campaign
  • In-season surveys filled during the production stages.
  • Intake and wet chemistry surveys documenting the work carried out at the lab.

Data Control and Transparency

As a data point travels through our system, we continually check the quality of the gathered information using several dashboards that give data managers insight into the current state of the process.

Fig. 3 – Project Management Dashboard

Information gets stored in a mongoDB database and is easily accessible through several means, including an automated query generator.

Data Merging & the Data Pipeline

Once a data point has been registered in any survey, our data pipeline starts its work. It will seek out information about this particular point on all other surveys and form a single complex entry synthesizing everything we know about it. 

The goal of this stage is to transform available data into organically useful information, in a way that is transparent enough to be used for independent research. This is part of a Continuous Integration process that happens three times a day, which allows for a glimpse into the current state of data collection at any given moment. 

Organized Information Database

At the moment of merging, data is organized into clearly named columns, in tables with single-value cells. These can be opened in Excel, and are ready for more advanced data science treatments, such as those done in Python or R.

Fig.4 – Database output table

Merging Briefs  

These contain information such as the number of merged surveys, information completeness status for each point, and amount of errors (orphan entries, ambiguous naming, etc).

Fig.5 – Database brief

We are in the process of transforming this software from a specialized solution fitted to our particular needs, to a more generally usable continuous data merging tool. When paired with SurveyStack, this will allow any organization to design a complex data collection scheme and get a finished, coherent and immediately useful database as a result, just by setting parameter functions to organize how data is arranged.

Data Analysis – Briefing Scripts

Data analysis is perhaps the most complex section, and takes our data point through a host of different tasks and uses.

Cleaning

Different kinds of errors can occur in the process of collecting and analyzing data. Finding and removing these errors is critical for producing dependable insights.

The most typical error case involves outliers, which have values that are impossible to compare with the other values in a set. Even though exceptional cases exist and the values can occasionally be real, they are mostly caused by mistakes or failures, and are not useful for making predictions about the general data set. These points are marked and omitted for most processes.

Plotting

We design several kinds of visualizations, which are useful for getting a generalized, global view of the whole population, or to test hypotheses.

Researchers can get important insights by looking at well-designed plots, which enable them to focus their attention on the relevant details faster.

Fig.6 – A cluster plot showing climate region distribution for beet crops. 

Modelling

One of our main tasks is modelling, which for us involves using accessible techniques to reliably predict information that is otherwise hard or expensive to measure. Models that predict a response variable using different predictor variables are tested together and compared. We then use a range of modelling algorithms based on different mathematical grounds, which have different strengths.

Fig.7 – Prediction models of antioxidants. 

Hypothesis Testing

Hypothesis testing involves using different criteria to decide whether results observed in an experiment support the core question motivating the research at hand. 

For us, our focus so far has been comparing the effects of different farm practices on crop attributes. To get this done, we compare groups of samples in specific categories like no-till, no-spray, irrigated, organic, or greenhouse grown to those that do not fall into those categories.

Sharing – Dashboards and In-Field Predictions

Dashboards

Data dashboards allow us to share the information we’ve gathered back to the community. In turn, insights from diverse user perspectives can enrich our interpretations and questions, giving us better insights and sparking new areas of exploration. 

We’ve tried to make this as interactive and useful for independent research as possible. The main idea behind the data explorer dashboard is that it will allow a user to test their own hypotheses by visually comparing different granular selections of points.

Fig.8 – Data Explorer dashboard 

You can see it for yourself here!

In-Field Predictions

We’ve just recently reached a stage in which our models are mature enough to deploy. This is the moment when the idea of getting expensive information from cheap information crystallizes. Using the handheld UV/Vis scanner, a user can get estimates of the nutritional densities in several crops (more than ten, currently!) with an informed level of precision. This provides results in under a minute that otherwise would require delivering the samples to a lab for time-intensive testing.

Fig.9 – A polyphenol prediction readout from the in-field model

We are currently developing new models that will deliver useful predictions, from determining soil carbon to further details of a crop’s nutritional value.

At this stage of the process, we might say that the original data point is still directly involved–It’s what informs the model, allowing us to better understand new observations in the real world. The thing that’s so unique and exciting about this system lies in its integration – moving from research to prediction in the same platform, following a single data point all the way through. Thank you for coming with us on this journey! And if you have a question or idea to bring to this pipeline, please contact us. We’d love to give it a try.