Multimodal Industrial Spatiotemporal Large Model for Rail-Transit Intelligent Diagnosis
A large-model agent diagnostic framework fusing rail images, time-series waveforms, and text logbooks achieves 96% visual rail-corrugation recognition accuracy, delivering fully automated fine-grained diagnosis under complex operating conditions
Project Overview
Metro rail-corrugation inspection lacks standardization and is prone to missed or false calls: data spans image, time-series waveform, and text-logbook modalities that traditional methods struggle to fuse deeply, while anomalous samples and high-quality annotations remain severely scarce.
Research Objectives
A fast, precise annotation pipeline for corrugation anomaly samples
An efficient visual feature-extraction algorithm for corrugation
Multimodal-fused large-model agent reasoning for autonomous diagnosis
Methodology
An autocorrelation-function-based image algorithm efficiently extracts visual corrugation features; a large-model agent diagnostic framework automatically integrates time-series waveforms with logbook data through segmented parsing, identifying and distinguishing complex operating conditions such as turning and impact.
Technical Approach: How It Works
- 1
Efficient Annotation Pipeline
A rapid corrugation-anomaly annotation system was developed to address the first-order problem of "extremely scarce fault samples plus a lack of high-quality labels."
- 2
Autocorrelation-Function-Based Visual Extraction
After edge detection and binarization of rail images, the autocorrelation function (ACF) is computed to detect peaks within a 30–300mm period band, determining corrugation presence and severity — interpretable and free of large-scale training.
- 3
Large-Model Agent Multimodal Fusion Diagnosis
An agent diagnostic framework automatically combines time-series waveforms with text-logbook data through segmented parsing; beyond detecting corrugation, it identifies and distinguishes complex operating conditions such as turning and impact, with full interpretability.
Figures: Methods & Results
Key Results
Image-only rail-corrugation recognition accuracy of 96%
The agent system delivers fully automated, fine-grained diagnosis under complex conditions with end-to-end interpretability
The direction has produced 4 high-level SCI papers and 5 granted invention patents to date
Online Equipment Fault Diagnosis