Yield Calibration & Cleaning

Tutorial to run Harvesting data cleaning and calibration. Or just cleaning, or just calibration. It works great even for complex cases: multiple harvesters, no harvester id, etc.

Yield data holds immense potential for farmers, but its true power is unlocked only when its precision is perfect. The "Yield Calibration" module is designed to refine the raw Yield Dataset, aligning it with core mathematical tenets to uplift its quality. The end result is a dataset that's not only more robust but also primes it for in-depth, insightful analyses.

Harvesting data Cleaning and Calibration Video Tutorial. The difference between options explained

This calibration process is instrumental in:

1. Ensuring Data Consistency: It's not uncommon for multiple harvesters to work in tandem or across different days. This feature ensures that their data sings in harmony.

2. Homogenizing Data: Yield data can be varied; the calibration ensures it is smooth and consistent, without unwanted spikes or drops.

3. Filtering Out Noise: Like any data, yield data can have its share of 'noise' or irrelevant info. We make sure it doesn't muddy your insights.

4. Streamlining Geometries: Any turnarounds or odd geometric data patterns can skew real insights. The calibration is designed to iron these out, ensuring the data truly mirrors field realities.

5. Cropping by Field Boundary: Harvesters often operate across adjacent areas. For accurate analytical results, it's essential to consider only the data situated within the specified boundary.

The Yield Calibration interface leverages the associated GeoPard API (LINK), specifically integrating with the CALIBRATE and CLEAN operations. This functionality is accessible through the GeoPard User Interface and can be programmatically invoked via the GeoPard API.

Quick Overview

5MB

Automated_Yield_Data_Cleaning_Calibration_with_GeoPard.pdf

PDF

Open

Download Yield Cleaning PDF Brochure

Real-World Examples

In the realm of agriculture, corrupted yield datasets can pose significant challenges. Below, you can find real-world examples where such datasets were encountered. Through GeoPard's advanced calibration and cleaning algorithms, these datasets were effectively refined and optimized.

To address areas lacking logged Yield Data and achieve completeness of the Yield Map, consider utilizing the GeoPard Synthetic Yield Map approach. This method effectively restores missing data, ensuring a comprehensive yield analysis. Learn more about this technique HERE.

Multiple Harvesters Working Together

Example 1: Multiple Harvesters Working Together

When dealing with complex scenarios, a two-step calibration process is recommended for optimal accuracy. Begin by running the initial calibration using the Machine ID attribute. Following that, proceed with a second round of calibration, this time utilizing the Simulated (Synthetic) Machine Paths tickbox. This layered approach ensures a thorough and precise calibration, essential for managing intricate cases effectively.

Example 2: Multiple Harvesters Working Together

Example 3: Multiple Harvesters Working Together

J-turns, Stops, Half Equipment Width Used

Example 1: U-turns, Stops, Half Equipment Width Used

Example 2: U-turns, Stops, Half Equipment Width Used

Abnormally Large Logged Values

Example 1: Abnormally Large Logged Values

Example 2: Abnormally Large Logged Values

Example 3: Abnormally Large Logged Values

Example 4: Abnormally Large Logged Values

Example 5: Abnormally Large Logged Values

Data Beyond Field Boundary

Example: Data Beoynd Field Boundaries

Calibration Using Provided Average Yield Value

Example: Calibration Using Provided Average Yield Value (28 t/ha)

Clean Yield Attributes Ignoring Attributes with Anomalies

The Yield Dataset occasionally includes attributes with irregularities in Moisture, Speed, Elevations, or other secondary (non-yield) attributes. During the execution of Clean or Calibrate activities, it is essential to disregard these anomalies. This can be efficiently achieved using the GeoPard Yield Clean-Calibrate interface.

Example: Anomalies in the Moisture Attribute

Example: Clean Yield Data Ignoring Anomalies in Moisture

USDA Clean Protocol

Example: Clean Yield Data applying USDA Protocol

Example: Clean Yield Data applying USDA Protocol

Explanation of Calibration Logics

Pathwise Calibration

USE Pathwise Calibration when a field is harvested by multiple machines or over several days, specifically to correct systematic differences like striping or banding. It is ideal for scenarios where varying machine settings, operators, or environmental conditions cause consistent over- or under-estimation across different paths.

Crucially, the AI requires variation - such as distinct paths, machine IDs, or harvest dates - to learn and calibrate effectively.

Example: Yield WetMass and 9 Harvesters

DO NOT USE this method for single-machine harvests in one continuous session or if the yield map lacks visible spatial patterns. Additionally, avoid it if the data is sparse or if you only possess total field-level yield values without machine-level differences

Example: Statistically Correct Data Distribution

Average or Total Calibration

Average/Total Calibration IS BEST USED when you have a high level of confidence in your overall field-level yield data, such as records from a weighbridge or storage facility. Instead of adjusting individual paths, this method scales the entire dataset so that the final average or total matches your known reference value. It is often described as the simplest and safest calibration option when the overall numbers are trusted.

When to USE Average/Total Calibration:

• Known Reference Values: You should use this logic when you have official total yield records (e.g., from a weighbridge) or a highly reliable average yield for the field.

• Global Bias Correction: It is ideal if the spatial distribution in the yield map looks correct, but the values are globally shifted - meaning the yield monitor was likely uncalibrated and is reporting values that are consistently too high or too low across the entire field.

• Uniform Harvest Conditions: This method is most effective when harvesting conditions were relatively consistent throughout the operation.

• Single-Machine Consistency: It works well for harvests completed by a single machine that performed consistently across the field.

Example: Statistically Correct Data Distribution with Required Shifting using Average Yield

When NOT to USE Average/Total Calibration:

• Machine-to-Machine Bias: Do not use this method if different parts of the field were harvested by different machines or on different days that resulted in localized biases. In these cases, scaling the whole field will not fix the underlying discrepancies between machines.

• Visible Artifacts: If you see strong striping, banding, or directional artifacts in your data, this method will not resolve them; Pathwise calibration is better suited for those issues.

• Incomplete Data: Avoid this logic if only a portion of the field was harvested or if the recorded data is incomplete, as the total/average values would be misleading.

Example: Yield Data with Gaps

Conditional Calibration

Conditional Calibration serves as a safety control by ensuring yield values remain within realistic, pre-defined minimum and maximum ranges.

You SHOULD USE this logic to remove extreme outliers and sensor spikes caused by noise, machine stoppages, or turns. It is ideal for applying specific agronomic expectations - such as "yield cannot exceed X" - without performing a correction.

However, AVOID THIS METHOD if your dataset has a global bias or systematic machine differences, as it does not scale data or fix spatial patterns. Essentially, it keeps values plausible but does not resolve underlying calibration offsets.

Usage Strategy

Onepager Yield Calibration Guidance

234KB

GeoPard_Calibration_Guidance_EN_20250120.pdf

PDF

Open

Download PDF Onepager of Yield Calibration Guidance

First Step

The "Yield Calibrate and Clean" module is initiated directly from the User Interface. The primary requirement is to have an uploaded Yield Dataset. Adjacent to each Yield Dataset, you'll find a button to commence the dataset adjustments.

Start the flow

Select an option to proceed

From there, several options are available for proceeding:

1. Auto-Processing: Use the default, GeoPard-recommended settings for a one-click calibration.

2. Clean Only: Configure and execute only the CLEAN operation, including

   1. GeoPard Cleaning: Smart Cleaning of Yield dataset with AI algorithms.

   2. USDA (United States Department of Agriculture) Cleaning Protocol for yield.

   3. Conditional Cleaning: Filter data based on custom attribute thresholds.

3. Calibrate Only: Configure and execute just the CALIBRATE operation, including

   1. Pathwise: Calibrate yield for each individual machine path using AI algorithms.

   2. Average/Total: Adjust yield based on the field's known average or total yield.

   3. Conditional: Modify yield within set minimum and maximum limits to maintain expected ranges.

4. Calibrate & Clean: Choose the sequence of operations and customize the parameters.

One-Button Solution

Hint for Abnormal Values Sometimes Inherent to Yield Datasets.

If an attribute selected for calibration or cleaning predominantly contains zero values across the majority of geometries, these geometries will be excluded from the final Yield Dataset.

To ensure integrity, attributes with such anomalies should be excluded from the list of attributes to be calibrated.

Full Guidance

Choose Flow: Hint for Data Anomalies

If a user encounters anomalies in the data, such as values at or near zero, or unusually large values (for instance, an average of 10 with a maximum of 8000), the Clean & Calibration workflow is advised.

Prioritizing data Cleaning before Calibration ensures the removal of errors, missing values, or inconsistencies, thereby enhancing data quality and accuracy.

Choose Flow: Hint for Data without Initial Errors

For datasets initially free from errors, missing values, or inconsistencies, and when multiple harvesters are known to be involved, consider the Calibration & Clean workflow.

Cleaning the data post-calibration helps to refine the dataset further by potentially eliminating any artifacts introduced during calibration.

Clean Flow: Hint for Abnormal Values Sometimes Inherent to Yield Datasets.

If an attribute selected for calibration or cleaning predominantly contains zero values across a majority of geometries, these geometries will be excluded from the final Yield Dataset.

To ensure integrity, attributes with such anomalies should be excluded from the list of attributes to be cleaned (2).

Calibrate Flow: Hint for Abnormal Values Sometimes Inherent to Yield Datasets.

If an attribute selected for calibration or cleaning predominantly contains zero values across the majority of geometries, these geometries will be excluded from the final Yield Dataset.

To ensure integrity, attributes with such anomalies should be excluded from the list of attributes to be calibrated (3).

Algorithm Versions

Post-processing, the outcomes are showcased adjacent to the original dataset, distinctly marked with Calibrate" and/or "Clean" labels, as well as the algorithm version utilized.

The result of Calibrate & Clean execution (version 2)

From version 3.0 of the Clean/Calibrate algorithm onward, GeoPard introduces the Crop by Field Boundary feature. This keeps only geometries located within the Field Boundary and results in more accurate statistical data distribution.

The result of the Auto-Processing execution (version 3.0)

Starting with version 4.0, the Clean/Calibrate algorithm in GeoPard now incorporates a feature for calibration based on Average or Total Values across any attribute. A prevalent application of this enhancement is the calibration of WetMass, which can now be adjusted by the known measured Average Yield for a specific Field.

The result of the Calibration execution using Average Yield of 6 t/ha (version 4.0)

From version 5.0 of the Clean/Calibrate algorithm onward, GeoPard introduces USDA (United States Department of Agriculture) Cleaning Protocol for yield. USDA provides formal agronomic data standards that govern how yield, moisture, flow, and spatial measurements are normalized, validated, and statistically filtered to produce machine- and field-consistent agricultural datasets.

The result of the Cleaning execution using USDA Protocal (version 5.0)