Abstract
Farms run on data, but our sensors rarely agree: optical is cloud-limited, radar sees structure but not color, UAVs arrive on their own schedule, and resolutions vary by field and season. We introduce a modular multisensor fusion backbone that turns heterogeneous streams (e.g., Sentinel-2, Sentinel-1, Planet/UAV) into a single, field-centric representation designed for agronomic decision-making. The system leverages strong pretrained encoders per modality and a compact cross-modal exchange layer to integrate signals while remaining resilient to gaps (missing sensors, clouds, irregular cadence) and shifts in scale. Training relies on broadly applicable self-supervised objectives and large unlabeled archives, minimizing label demands. In preliminary studies, the fused embedding is stable, subset-robust, and deployable from plot to region with low latency. Ongoing work evaluates temporal extensions for phenology and within-season forecasting.
Potential impact in digital agriculture: early abiotic stress alerts and irrigation guidance; yield nowcasting/forecasting; nitrogen management support; emergence & gap mapping; storm/lodging and disaster impact assessment; disease/pest early warning; indicators for carbon and water accounting.
Keywords
digital agriculture, multisensor fusion, remote sensing, scale-aware representation, self-supervised learning, phenology & yield analytics
DOI
10.5703/1288284318191
MULTI-MODAL SPECTRAL FUSION FOUNDATION MODEL
Farms run on data, but our sensors rarely agree: optical is cloud-limited, radar sees structure but not color, UAVs arrive on their own schedule, and resolutions vary by field and season. We introduce a modular multisensor fusion backbone that turns heterogeneous streams (e.g., Sentinel-2, Sentinel-1, Planet/UAV) into a single, field-centric representation designed for agronomic decision-making. The system leverages strong pretrained encoders per modality and a compact cross-modal exchange layer to integrate signals while remaining resilient to gaps (missing sensors, clouds, irregular cadence) and shifts in scale. Training relies on broadly applicable self-supervised objectives and large unlabeled archives, minimizing label demands. In preliminary studies, the fused embedding is stable, subset-robust, and deployable from plot to region with low latency. Ongoing work evaluates temporal extensions for phenology and within-season forecasting.
Potential impact in digital agriculture: early abiotic stress alerts and irrigation guidance; yield nowcasting/forecasting; nitrogen management support; emergence & gap mapping; storm/lodging and disaster impact assessment; disease/pest early warning; indicators for carbon and water accounting.