2025 Advancements and challenges in earthmoving equipment automation using core technologies
Advancements and Challenges in Earthmoving Equipment Automation Using Core Technologies
Authors: Jiyu Feng, Wei Chen, Changyi Liu, Xinhui Liu, Bingwei Cao, Feixiang Xu
Journal: Automation in Construction 179 (2025) 106472
Affiliations: - School of Mechanical and Aerospace Engineering / National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China - Key Laboratory of Bionic Engineering, Ministry of Education, Jilin University, Changchun 130025, China - School of Mechanical and Vehicle Engineering, Changchun University, Changchun 130025, China - School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China
Abstract
Earthmoving equipment automation is limited by interactions with complex media, unstructured terrain and highly variable environmental conditions. Although research in this field has continued for nearly thirty years, a truly fully autonomous system remains unrealized. This paper examines automation technologies for earthmoving equipment in typical repetitive short-cycle loading operations and presents a systematic survey of core literature spanning over thirty years. Key techniques are grouped into four categories: environmental perception, path planning and tracking, autonomous loading and unloading, and safety and risk assessment. The analysis indicates that current research primarily addresses improvements to individual stages or subsystems while system-level optimization of the entire workflow is still lacking. Moreover, data-physics hybrid methods show strong potential in earthmoving equipment automation applications by compensating for errors in physical models and improving the generalization of data-driven models. Finally, the major challenges are summarized and directions for future research are proposed.
Keywords: Earthmoving equipment, Automation, Short-cycle loading, Sensing technology
1. Introduction
Earthmoving equipment is widely used in various environments, including mining operations, construction sites, railway freight yards, and agricultural estates, primarily for excavation and loading tasks. In earthworks projects, wheel loaders and excavators typically work in conjunction with dump trucks to perform short-cycle loading tasks for material transport. These tasks are inherently monotonous and repetitive, and prolonged driving operations can lead to driver distraction and physical fatigue, affecting reaction time and judgment, potentially decreasing productivity and causing safety incidents. Additionally, the dust, noise, and vibrations generated during earthmoving operations pose significant occupational health and safety risks to operators.
As a result, the automation of earthmoving equipment has become an area of significant research interest, particularly with the continuous advancements in autonomous driving technology for passenger vehicles. However, despite extensive research, commercial earthmoving machinery automation systems suitable for practical application have yet to materialize, constrained by factors such as environmental complexity, perception capabilities, autonomous decision-making and operational strategies, and safety.
Comparison with Previous Surveys
| Authors | Title | Categories | Remarks |
|---|---|---|---|
| Dadhich et al. (2016) | Key challenges in automation of earth-moving machines | Wheel loaders | Highlights the key challenges in automating and tele-operating earthmoving machines, with particular emphasis on the loading phase |
| Borngrund et al. (2022) | Deep-learning-based vision for earth-moving automation | Wheel loaders | Analyzes the relative importance of different sensor modalities, focusing on deep-learning-based vision for short-cycle loading tasks |
| Ha et al. (2019) | Robotic autonomous systems for earthmoving in military applications | Excavators; bulldozers; wheel loaders; graders and haul trucks | Platform-centric autonomous robotic systems for earthmoving in military contexts |
| Lee et al. (2019) | Challenges, tasks, and opportunities in teleoperation of excavator | Excavators | Surveys remote excavator operation, emphasizing human-machine interaction, interface design, and environmental considerations |
| Eraliev et al. (2022) | Autonomous excavation systems | Excavators | Reviews automated systems across five dimensions: sensing, perception, decision making, planning, action |
| Present study | Advancements and challenges in earthmoving equipment automation using core technologies | Earthmoving equipment | Examination of core technologies for short-cycle automated operations; pays special attention to previously overlooked technical details such as pile reconstruction and excavation difficulty assessment |
This paper fills the gap by providing a thorough examination of core technologies for short-cycle automated operations in earthmoving equipment, focusing on four enabling areas: environmental perception, path planning and tracking, autonomous loading and unloading, and safety and risk assessment.
2. Problem Definition and Decomposition
Unlike autonomous vehicles, which confront complex traffic environments, automated earthmoving equipment typically operates within enclosed and strictly controlled work sites. The driving environment is relatively simplified, thereby facilitating the implementation of automation technologies. However, the functional characteristics of earthmoving equipment dictate that, in addition to autonomous driving capabilities, they must also perform material loading and short-distance transportation — a fundamental deviation from the core tasks of autonomous vehicles.
2.1 Wheel Loader Short-Cycle Loading Workflow
The short-cycle loading operation of a wheel loader can be subdivided into six principal phases:
-
Approach and site assessment: Ascertain the location and geometric configuration of the material stockpile and the parking position of the dump truck; select the most appropriate excavation point and plan the optimal access route.
-
Excavation and loading: Select an appropriate digging trajectory based on material type and size to ensure the front bucket is fully loaded. Adjust the operating strategy in real-time to prevent tire slippage or insufficient loading.
-
Retreat and route planning: Assess the relative position of the loader and the dump truck; plan the optimal route for retreat from the stockpile to the dump truck. This is critical for reducing cycle time, lowering fuel consumption, and enhancing operational efficiency.
-
Approach and pre-assessment of dump truck: Pre-assess the distribution of material on the truck to ensure an even spread during unloading and prevent spillage. Rapidly raise the boom so that the bucket's elevation surpasses the truck body, while maintaining appropriate clearance.
-
Unloading and material adjustment: Tilt the material within the bucket as needed and make any necessary adjustments to the material on the dump truck to meet subsequent transportation requirements.
-
Return and preparation: Maneuver the loader in reverse to move away from the dump truck and lower the bucket in preparation for the subsequent short-cycle loading operation.
As achieving a fully loaded dump truck typically necessitates multiple short-cycle loading operations, the geometry of the material stockpile, the selection of excavation points, and the choice of travel paths exhibit dynamic changes as the number of scooping operations increases. This scenario necessitates that the wheel loader be equipped with an array of high-precision sensors, which, in combination with data fusion techniques, enable real-time comprehensive assessments to underpin accurate autonomous decision-making.
3. Research Methodology
This paper employs a hybrid methodology that combines bibliometric analysis and systematic review. The paper systematically searches the relevant literature from academic databases and uses keyword co-occurrence and cluster analysis for bibliometric analysis to identify key research areas and trends. Representative literature is then systematically reviewed to gain a comprehensive understanding of the evolution of the field.
3.1 Data Collection
This paper primarily focuses on earthmoving equipment such as wheel loaders, Load-Haul-Dump (LHD) loaders, excavators, and other earthmoving machinery that use a bucket as the main operational tool.
Data sources: Google Scholar, Scopus, Web of Science, and IEEE Xplore
Search keywords: - Primary: "earthmoving equipment," "wheel loader," "excavator," "automation," "construction machinery" - Extended: "environmental perception," "unstructured roads," "pile identification," "excavation difficulty," "pose recognition," "path planning," "excavation points," "excavation trajectory," "load and volume estimation," "unloading points," "worker and equipment recognition," "safety," "risk monitoring"
Selection criteria: 1. Thematic constraints: restricted to earthmoving equipment fields 2. Publication date: 1990 to present (2025) 3. Language: English only 4. Quality: peer-reviewed papers from international journals and conferences 5. Relevance filtering: manually filtered based on keywords and abstracts
Selection process (PRISMA): - 596 papers retrieved - 32 duplicates removed; 8 non-English or irrelevant records excluded - 556 articles proceeded to title/abstract screening; 298 discarded - 258 papers underwent full-text review; 144 excluded - 114 articles included; forward/backward citation tracking yielded 47 additional references - 28 of 47 additional references met inclusion standards - Total: 142 papers form the foundation for bibliometric analysis and systematic review
3.2 Bibliometric Analysis Results
3.2.1 Publication Trends by Year
| Stage | Period | Papers | Characteristics |
|---|---|---|---|
| Initial stage | 1990-2004 | 6 | Exploratory, proof-of-concept, fundamental automation principles and control models |
| Steady development | 2004-2014 | 25 | ~4x increase; computer vision, control theory, embedded systems matured; path planning research emerged |
| Rapid growth | 2014-2025 | 111 | >78% of total literature; driven by AI, deep learning, high-performance computing; advanced sensors widely applied |
3.2.2 Publications by Country/Region
| Rank | Country | Share |
|---|---|---|
| 1 | China | 32.39% (46 papers) - top institutions: Jilin University, Xiamen University, HKUST |
| 2 | USA | 13.38% |
| 3 | Japan | 9.15% |
| 4 | Canada | 9.15% |
| 5 | Korea | 7.04% |
Top 10 countries contribute 90.85% of total publications.
3.2.3 High-Frequency Publication Sources
| Rank | Count | Publication |
|---|---|---|
| 1 | 30 | Automation in Construction (2434 citations) |
| 2 | 6 | Computer-Aided Civil and Infrastructure Engineering (292 citations) |
| 3 | 4 | Journal of Field Robotics |
| 4 | 3 | IEEE Robotics and Automation Letters |
| 4 | 3 | IEEE Transactions on Vehicular Technology |
| 4 | 3 | Journal of Computing in Civil Engineering (311 citations) |
| 4 | 3 | Mechanical Systems and Signal Processing |
| 4 | 3 | Robotics and Autonomous Systems |
35 papers were presented at 33 different international conferences (~24.65% of total).
3.2.4 Research Clusters (Keyword Co-occurrence Analysis)
A threshold of 3 occurrences was applied; 39 co-occurring keywords identified, revealing four significant research clusters:
Cluster 1: Environmental Perception Keywords: "computer vision," "recognition" Environmental perception includes recognition of unstructured roads, identification of material pile locations and shapes, and recognition of dump truck positions. Information fusion-based environmental perception technologies form the foundation of autonomous earthmoving operations.
Cluster 2: Path Planning and Tracking Keywords: "path planning," "trajectory planning," "tracking" Autonomous earthmoving equipment must plan a smooth, executable trajectory based on the results of environmental perception. Path planning, particularly for the unique articulated structure of wheel loaders, has long been a focus of research.
Cluster 3: Autonomous Loading and Unloading Keywords: "machine learning," "optimization," "deep learning" Most current work is concentrated in the semi-automation domain. With the continued development of AI technologies, this field is poised to experience a new wave of research driving efficient and energy-saving autonomous operations.
Cluster 4: Safety and Risk Assessment Keywords: "proximity warning system," "blind spots," "safety" Real-time monitoring of personnel and equipment positions can promptly trigger alerts within pre-planned safety zones. Monitoring environmental factors such as landslides, collapses, and public utility pipelines during construction is critical.
4. Core Technologies for Autonomous Operations
Following the six phases of the wheel loader's short-cycle loading operation, the key technologies are organized into four parts: environmental perception (4.1), path planning and tracking (4.2), autonomous loading and unloading (4.3), and safety and risk assessment (4.4). Although these technologies are discussed separately, they are not isolated; they are closely interconnected and mutually supportive.
4.1 Environmental Perception
Environmental perception primarily involves the use of sensors such as cameras, LiDAR, and millimeter-wave radar to perceive unstructured roads, identify and reconstruct material piles, assess excavation difficulty, and recognize vehicle pose. This capability forms the foundation for the autonomous operation of earthmoving equipment.
4.1.1 Perception of Unstructured Roads
Earthmoving equipment mainly operates in unstructured environments such as construction sites and mines, where it is often impossible to perform road recognition using uniform typical features as on urban roads or highways. Factors such as unclear road boundaries, irregular shapes, deteriorated pavement appearance, variable lighting, and changing backgrounds pose great challenges.
Three categories of research:
| Category | Methods | Advantages | Limitations |
|---|---|---|---|
| Computer vision-based | Model-based, feature-based, machine learning-based (e.g., RD-Net semantic segmentation, attention modules) | Rich information content, low cost | Susceptible to lighting/weather changes; large image databases needed |
| LiDAR-based | Semantic segmentation with attention mechanism (Liu et al.) | Not affected by lighting conditions; rich point cloud data | Limited research; ideal accuracy not yet achieved |
| Multimodal fusion | Low-level and high-level feature fusion from images and point clouds (Ye et al.) | Effective complementarity; improved accuracy and real-time responsiveness | Higher system complexity |
Research on road perception for earthmoving equipment in unstructured environments is currently scarce but is a key component for enabling autonomous operations.
4.1.2 Pile Identification and Contour Reconstruction
Rapid and accurate identification of stockpiles and reconstruction of their contours are of great significance for determining the optimal excavation point, planning the operational path, evaluating operational efficiency, and achieving fully autonomous operation.
Main approaches:
- Deep learning + computer vision: Building image datasets for stockpile recognition (widely used but requires significant time/financial investment; susceptible to dust and lighting)
- Proprioceptive sensors: Using IMUs to measure acceleration data for material classification (Artan et al.); pressure sensors to distinguish between material types (Fernando et al.); energy and motion data from excavators to classify four different materials (Liu et al.); limited to single-material classification
- Pile contour reconstruction methods:
| Method | Reference | Notes |
|---|---|---|
| Stereo-vision triangulation | Koyachi et al. | Captures pile images, converts 3D data into column model |
| LiDAR column model | Multiple | Real-time capability |
| 3D point cloud profiling | Multiple | High accuracy |
| Stereo-LiDAR fusion | Chen et al. | Combines advantages of both modalities |
| Total-station measurements | Yakar et al. | Compared photogrammetry and laser scanning; both achieved 95% reliability; laser scanning higher accuracy, photogrammetry more cost-effective; real-time transmission remains a critical challenge |
4.1.3 Excavation Difficulty Assessment
Excavation difficulty assessment enables equipment and cooperating dump trucks to make more accurate autonomous decisions. This area is closely related to material identification and pile reconstruction.
4.1.4 Vehicle Pose Recognition
Accurate pose recognition of cooperating vehicles (especially dump trucks) is essential for planning unloading maneuvers and maintaining appropriate clearances.
4.2 Path Planning and Tracking
4.2.1 Path Planning
Path planning addresses route generation for Phase I (approaching the material pile), Phase III (moving from the pile to the dump truck), and Phase VI (returning from the dump truck to the staging point). It involves both point-to-point (PTP) planning and complete coverage path planning (CCPP).
Global path planning: Uses known positions of the pile, dump truck, and equipment to create a collision-free base route optimized for shortest distance and efficiency.
Local path planning: Adjusts the route in real-time based on the machine's status and surrounding conditions.
Key algorithms:
| Algorithm | Characteristics |
|---|---|
| Dijkstra | Deterministic and consistent; widely used for global planning |
| A* | Heuristic search; efficient for grid maps |
| RRT / Goal-directed RRT (improved) | Sampling-based; suitable for high-dimensional spaces with complex constraints |
| Particle Swarm Optimization (PSO) | Population-based optimization |
| Ant Colony Algorithm (improved) | Bio-inspired; suitable for combinatorial problems |
| Reinforcement Learning | Adaptive; learns from environment interaction; for path + energy optimization |
| MPC (Model Predictive Control) | Online; considers constraints and disturbances |
Industry examples: - Kajima A4CSEL: robust tracking control with heading estimation; field trials with autonomous rollers and bulldozers achieved path-tracking accuracy within +/-10 cm - Sakai Heavy Industries + JIG-SAW: autonomous roller with path-tracking deviations ~20 cm; compaction coverage ~3.5x higher than manual operation
4.2.2 Path Tracking
Path tracking ensures the vehicle follows the planned trajectory accurately and smoothly. Methods include: - Feedback linearized iterative learning (Dekker et al.) - LQR controller - PSO-NMPC control strategy - Pure tracking + fuzzy control (for unmanned electric shovels) - Nonlinear Model Predictive Control
4.3 Autonomous Loading and Unloading
4.3.1 Selection of Excavation Points
Excavation point selection involves identifying the optimal digging locations and orientations on a material pile based on its geometric shape. Reliable excavation points significantly impact the fill factor, safety, energy consumption, vehicle structural damage, travel distance, and number of excavation cycles.
Principles for reducing candidate excavation points: 1. Select points on the pile boundary 2. Choose locations adjacent to the previous excavation point to shorten travel distance 3. Choose the candidate point that minimizes the drop edge length of the pile after excavation 4. The direction of the excavation point should be perpendicular to the slope of the pile 5. Choose candidate points that exhibit a certain degree of convexity
Main methods:
| Method | Description | Reference |
|---|---|---|
| Column model + bucket trajectory | Computes moment acting on bucket to evaluate each potential excavation posture; minimizes total moment | Sarata et al. |
| Two-section bucket method | Divides bucket into left and right; evaluates convexity and inclination | Singh et al. |
| Three-section bucket method | Subdivides into left, middle, right; more accurate description of convex/concave regions; also proposes quadratic fitting model | Magnusson et al. |
| GA-based optimization with slope/distance metrics | Incorporates slope and travel distance evaluation metrics | Chen et al. |
| 3D configuration score space | Maps path length values to score space to identify optimal excavation point | Takei et al. |
| Centroid evaluation | Combines pile edge curves with height field; uses material centroid position within bucket | Lindmark et al. |
| Height-field grid method | Considers multiple metrics: distance, collision, spatial area, fill volume | Cardenas et al. |
| Reinforcement Learning | Reward function based on convexity; typically integrated with path planning | Backman et al., Kawabe et al. |
4.3.2 Excavation Trajectory Planning
Trajectory planning involves generating a time-parameterized motion profile along a predefined path. Studies show the excavation phase accounts for approximately one-third of total energy consumption in the entire operation cycle. Experienced operators can achieve fuel efficiency improvements of up to 150% and productivity increases of up to 300%.
Comparison of trajectory generation methods:
| Category | Methods | Key Characteristics | Verification |
|---|---|---|---|
| Experience-based | One-step, stepwise, digging, coordinated excavation (slicing) methods | Based on observation of skilled operators; coordinated method identified as most efficient | Real equipment |
| Geometry-based | Keypoints + arc curves, cubic polynomials, hybrid polynomials; energy minimization models | Smooth trajectories; simplified; cannot adapt to complex conditions | Simulation/real |
| Optimization-based | GA, PSO, Sparrow Search, gradient descent, SQP; minimize time, energy, jerk, moment, fill factor | Handles complex constraints; computationally intensive; force/power trajectory variants for soil uncertainty | Simulation |
| Learning-based | LSTM, RL, Inverse RL, Imitation learning | Circumvents complex modeling; adaptive to different environments; high training cost for high-dimensional problems | Platform/real |
| Hybrid | Prediction-optimization (imitation learning + SQP/GA/PSO); data generation-learning (RL, PINN) | Combines efficiency of optimization with generalization of learning; higher-quality datasets but significant time cost | Simulation/platform |
Industry examples: - Obayashi Corporation: merges operator know-how with AI to reproduce digging and loading motions with high precision - Kajima Corporation (bulldozers): developed baseline model via computer simulation + genetic algorithm; performance closely matches skilled operators
Selected method comparison (Table 5 in paper):
| Method | Vehicle | Soil-adaptive | Verification |
|---|---|---|---|
| Keypoints + arc transition (fill factor) | Excavator | Yes | Real |
| Keypoints + cubic polynomial (smoothness) | Excavator | No | Real |
| Mathematical models + energy (LHD) | LHD | Yes | Simulation |
| Surrogate model + GA (productivity, structural damage) | LHD | Yes | Simulation |
| Optimal control + nonlinear algorithm (fill factor, fuel, cycle time) | Wheel loader | Yes | Simulation |
| Keypoints + cubic spline + SQP (time, jerk) | Excavator | No | Real |
| LSTM (skilled, collision avoidance) | Excavator | No | Platform |
| RL (fill factor, time) | Excavator | Yes | Real |
| Inverse RL (skilled) | Excavator | No | Real |
| PSO + RL (time, energy, excavation rate) | Excavator | Yes | Simulation |
| Sixth-order polynomial + PSO + PINN (time, energy, fill factor) | Excavator | Yes | Platform |
4.3.3 Load Mass and Volume Estimation
Accurate estimation of bucket load and fill factor is crucial for evaluating productivity, preventing overload and underload, extending service life, and ensuring operator safety.
Methods for load estimation: 1. Kinematics and dynamics models (most common commercialized method; accurate mathematical model is challenging due to harsh environment and time-varying friction coefficients) 2. Machine learning / neural networks (inputs: boom hydraulic cylinder pressure, displacement, boom angle; supplemented by kinematics model solutions and acceleration) 3. Strain gauge installation on bucket load pin 4. LiDAR-based volume estimation 5. Computer vision (stereo cameras, depth maps, Faster R-CNN)
Quantitative comparison of payload/volume estimation methods:
| No. | Reference | Vehicle | Method | Result |
|---|---|---|---|---|
| 1 | Feng et al. | Wheel loader | Dynamic model + optimal least-squares + neural network | Payload error: 2% |
| 2 | Guan et al. | Wheel loader | Stereo camera + improved CNN + probabilistic knowledge | Volume error: 3.11% |
| 3 | Brinkschulte et al. | Wheel loader | Kinematic model | Payload error: 2% |
| 4 | Renner et al. | Excavator | Dynamic model with friction | Payload error: 3% |
| 5 | Lu et al. | Wheel loader | Stereo camera + ResNet + Faster Region-CNN | Volume error: 4.77% |
| 6 | Yung et al. | Tractor loader | Kinematic model + calibrate center of gravity | Payload error: 5% |
| 7 | Bennett et al. | Excavator | Linear interpolation | Payload error: 3% |
| 8 | Savia et al. | LHD | Kalman filter + neural network | Payload error: 3.86% |
| 9 | Kheiralla et al. | Wheel loader | Strain gauge on bucket load pin | Payload error: 1% |
| 10 | Chen et al. | Wheel loader | LiDAR + ICP registration + alpha-shape of Delaunay triangulation | Volume error: 4.64% |
| 11 | Li et al. | Excavator | Stereo vision + excavator kinematics + InceptionTime + CBAM | Volume error: 10.65% |
| 12 | Helian et al. | Excavator | 2D images + Faster Region-CNN | Volume error: 3.09% |
| 13 | Bewley et al. | Cable shovel | LiDAR + height grid | Volume error: 5% |
4.3.4 Unloading Point Selection and Material Distribution
After filling the bucket, the system must determine optimal unloading points on the dump truck bed to ensure even material distribution, prevent spillage, and meet subsequent transportation requirements.
Key methods: - 3D pose recognition of dump truck (Lee et al.): stereo cameras and point cloud processing - Load distribution estimation with attention mechanism (Wu et al.): predicts material distribution across multiple trucks
4.4 Safety and Risk Assessment
4.4.1 Construction Personnel and Equipment Identification
Sensor-based methods:
| Method | Principle | Advantages | Disadvantages |
|---|---|---|---|
| RFID | LF/HF/UHF | Rapid identification; low-cost tags; passive mode possible | Dense multipath fading; maintenance cost; restricted read/write distances |
| Bluetooth | 2.4 GHz | Low power; low cost; good compatibility with mobile devices | Limited ranging accuracy; short communication distance |
| UWB | 3.1-10.6 GHz | High positioning accuracy; strong anti-interference; supports micro-motion detection | Relatively high cost; requires cabling |
| GPS | Satellite navigation | Global coverage; high outdoor positioning accuracy | Expensive; high power consumption |
| Magnetic field sensing | Earth's/artificial magnetic field | Low power; extremely low cost in certain scenarios | Short range; highly dependent on environment; limited accuracy |
| Ultrasonic | Sound waves (>20 kHz) | Simple hardware; low cost; immune to electromagnetic interference | Short range; affected by ambient noise and temperature/humidity |
| Wi-Fi | 2.4 GHz/5 GHz (CSI/RSSI) | Utilizes existing networks; extensive coverage; motion detection | Prone to interference; limited ranging accuracy; complex algorithms |
| Radar | 24/60/77 GHz | High precision; micro-movement detection; privacy preserving | Expensive; complex algorithms; limited measurement range |
Computer vision-based methods: - Feature extraction (contours, edges, colors): intuitive but susceptible to occlusion and complex backgrounds - Deep learning instance segmentation (Xiao et al.): tracking workers in off-site construction - Hybrid vision + radio approaches (Cai et al.): 3D tracking of multiple construction workers - Local sparse transformer + LiDAR (Zhang et al.): worker detection in 3D point clouds - Night-time obstacle detection: image enhancement + multisensor fusion (Zou et al.) - Adaptive 2D tracking in complex environments (Konstantinou et al.) - Automated 2D detection using histograms of oriented gradients and colors (Memarzadeh et al.)
4.4.2 Field Construction Risk Monitoring
Landslide and collapse detection: - Terrestrial laser scanning + optical images (Ozdogan et al.) - LiDAR-based landslide inventory mapping (Bernat Gazibara et al.) - Image fusion techniques (Marcelino et al.)
Underground utility detection: - Magnetic sensors mounted on excavator (Kim et al.): low-cost wireless pipeline localization - Wireless multi-node metal pipeline localization via Bluetooth (Jiang et al.) - Deep learning + radargram inversion (Wang et al.): underground infrastructure detection
Proximity warning systems: - Bluetooth Low Energy RTLS (Huang et al.) - UWB-based proximity hazard indicator for workers near construction equipment (Teizer et al.) - Heat map generation for predictive safety planning (Golovina et al.) - Magnetic field proximity detection and alert technology (Teizer) - Wireless ad-hoc networks for proximity warning in surface mines (Sabniveesu et al.) - IoT-based autonomous system for worker safety with real-time alarming, monitoring, and positioning (Kanan et al.)
Excavator-avoidance for buried pipes: - LiDAR + GNSS-based excavator-avoidance system (Yajima et al.)
5. Major Challenges and Future Directions
5.1 Key Challenges in Practice
5.1.1 Limitations of Heterogeneous Sensor Data Fusion
To achieve comprehensive and accurate site perception, autonomous earthmoving equipment combines data from multiple sensors (stereo vision cameras, LiDAR, GPS, and more). These sources are highly complementary, but each uses different formats and stores data separately, making fusion the key to high-precision environment sensing. Key challenges during fusion: - Mismatched resolutions - Alignment errors - Uneven sensor quality - Transmission delays
Additionally, autonomy demands real-time performance, requiring raw sensor streams to be dimension-reduced, stripped of redundancy, and distilled into key features before fusion.
Future direction: A layered fusion architecture built on edge computing and cloud collaboration: - On-site edge nodes align sensor data in time and space, normalize resolutions, and perform initial dimension reduction - Only the most valuable compressed features are sent to the cloud - Hybrid fusion algorithm (blending data-driven and knowledge-driven methods) merges inputs into high-level representations
5.1.2 Insufficient Material Identification of Stockpiles and Diversified Excavation Strategies
Stockpile characteristics depend not only on inherent physical properties but also on external factors such as weather conditions (dry/loose soil vs. wet/muddy soil). Raw materials also vary widely in size, and interactions between stones of different dimensions and the bucket are complex.
Future direction: Combined digital twin and human-machine collaboration: - Digital twin simulates material properties and bucket contact mechanics, generating diverse large-scale synthetic data for training - In the field, when the system detects unfamiliar or extreme materials, a skilled operator demonstrates or fine-tunes the digging motion - Demonstrations are uploaded in real-time to edge nodes and synced to the cloud to update models and strategies
5.1.3 Lack of a Universal Safety and Risk Assessment Framework
Existing safety and risk assessment tools typically focus on personnel monitoring or static environmental sensing, but do not provide a comprehensive evaluation of major hazards such as underground utilities, landslides, or tunnel collapses.
Required framework with three core modules: 1. Personnel monitoring 2. Environmental sensing 3. Construction risk evaluation - With extensions tailored to mining, tunnel excavation, and high-slope operations - Standardized interface standards and data protocols enabling plug-and-play components - Swappable submodules (landslide prediction, collapse detection) that can be interchanged to match project needs - Validate module performance within a digital twin virtual platform
5.1.4 Lack of High-Quality, Large-Scale Datasets for Earthmoving Equipment
Existing collections fail to capture the full range of construction scenarios (various equipment types, pile geometries, unstructured terrain), limiting deep-learning models' generalization and robustness.
Future direction: - Digital twin platforms simulate diverse job-site conditions and produce precisely labeled synthetic data for cameras, LiDAR, and IMUs - Generative AI models apply domain randomization and detail augmentation to create training data resembling real-world conditions - Edge nodes collect key real-world snippets; mixed training on real and synthetic data substantially improves model stability and robustness
5.2 Future Development Directions
5.2.1 Digital Twin
Digital twin technology creates virtual models of autonomous earthmoving equipment, enabling real-time mapping of machine status and environmental changes to support operation simulation, strategy optimization, and preventive maintenance.
Industry example: DeepX Corporation (Japan) achieved automated underground excavation by combining digital twin visualization with autonomous excavators. Their system collects site data via LiDAR and GNSS, uses point-cloud recognition to visualize terrain, equipment position, and motion in 3D, synchronizing the physical world with the digital twin in real-time. The platform also computes key performance metrics and generates reports for remote monitoring.
Future focus: Enhancing model accuracy and live-update capability to enable more precise simulations and intelligent decision-making.
5.2.2 Edge Computing and Cloud Collaboration
Deploying edge computing devices on construction sites enables preprocessing, dimensionality reduction, and fusion of massive data from LiDAR, cameras, IMUs, and GPS directly on site. This delivers real-time feedback and mitigates the effects of network latency and instability. Cloud platforms use superior computing and storage resources to perform deep analysis on long-term accumulated data, refining algorithms and continuously updating models.
Industry example: Kajima's A4CSEL system (October 2021) — four operators in the centralized control center at Kajima's Tokyo headquarters managed twenty autonomous machines across three sites (Akita, Nara, and Kanagawa prefectures) around the clock, achieving notable gains in efficiency, safety, and consistency.
Supported by 5G and 6G networks, communication efficiency and reliability will further improve.
5.2.3 Standardization and Modular Design
The lack of a unified communication standard hampers efficient collaboration between machines from multiple manufacturers using different sensors, control systems, and data protocols. Required elements: - Standardized data-communication protocols: IIoT protocols, CAN bus, OPC UA - Open-source construction technology platforms
Industry example: Japan Civil Engineering Research Institute's OPERA (Open Platform for Earthwork with Robotics and Autonomy) integrates standardized control signals, hardware retrofit kits, wireless communication modules, simulators, and middleware; has successfully demonstrated autonomous excavator loading operations.
Roadmap for standardization: 1. Establish unified domestic standards for OEMs and industry applications 2. Work with international bodies (ISO, IEC) to develop compatible, interoperable global protocols 3. Build shared open-source code repositories, simulation tools, and standardized datasets 4. Form multinational, interdisciplinary alliances among research institutions, equipment manufacturers, and construction firms 5. Promote certification and mutual recognition schemes for equipment, systems, and operators
5.2.4 Human-Machine Collaboration
Future autonomous earthmoving equipment will treat human operators and intelligent systems as complementary partners rather than mere replacements. In relatively uniform, high-repetition environments (quarries, concrete batching plants), automation performs effectively; however, in complex and dynamic construction settings, operators remain essential for strategic planning, anomaly handling, and emergency decision-making.
Industry example: Kajima's A4CSEL (July 2024) fully automated every task at a mountain tunnel face: an autonomous wheel loader scoops blasted material and transfers it to a rear dump truck; when load falls below a set threshold, operators in the remote-control room deploy a teleoperated excavator to gather scattered debris before resuming autonomous scooping. This collaborative approach boosts efficiency and safety while providing flexibility for continual refinement.
Emerging commercial systems: - Built Robotics Inc.: fully autonomous pile-driving robot RPD 35 operates at 5x the speed of a human crew - SafeAI Inc.: conversion of the Caterpillar 725 electric articulated dump truck delivers 13% improvement in on-site environmental performance - ARAV Inc.: excavator automation kits (sales planned from June 2025); retrofits excavators and loaders from major OEMs (Caterpillar, John Deere, Hitachi)
6. Conclusions
This paper provided a comprehensive review of the current state of research on automation technologies for earthmoving equipment, with a particular focus on advances in environmental perception, path planning and tracking, autonomous loading and unloading, and safety and risk assessment.
Key Findings
-
Sensor fusion is fundamental: To operate autonomously in complex and uncertain environments, earthmoving equipment must rely on multisource information acquisition systems (computer vision, LiDAR, real-time dynamic positioning, pressure sensors) and utilize data fusion techniques to acquire comprehensive environmental information. Challenges including mismatches in sensor resolution, data alignment errors, inconsistent data quality, and transmission delays significantly hinder effective data fusion.
-
Individual subsystem optimization dominates: Current research primarily addresses improvements to individual stages or subsystems; system-level optimization of the entire workflow is still lacking. Researchers urgently need to undertake a holistic optimization of the entire workflow.
-
Data-physics hybrid methods show strong potential: These methods compensate for errors in physical models and improve the generalization of data-driven models.
-
Key challenges identified: - Limitations in the fusion of data from heterogeneous sensors - Insufficient sophistication in the recognition of stockpile material composition and deployment of diverse digging strategies - Absence of a universal framework for safety and risk assessment - Lack of high-quality, large-scale datasets
-
Digging strategies are critical: Adopting appropriate digging strategies tailored to the characteristics of the stockpiled material is critical for enhancing operational efficiency; these strategies are intrinsically linked to physical material properties and significantly influenced by external factors such as weather conditions.
-
Safety requires more attention: Safety, as the fundamental prerequisite for earthmoving automation, has hitherto received insufficient attention; the establishment of a universal framework for safety and risk assessment is of considerable practical significance.
State of Commercial Deployment
Significant theoretical advances have been made, but many breakthroughs remain confined to simulations or laboratory tests: - Volvo prototype: automated cooperation between an articulated tractor and a wheel loader achieved just 70% of a skilled operator's digging and unloading efficiency - ETH Zurich hydraulic excavator: autonomously built a standalone stone wall (10 m x 1.7 m x 4 m) and a permanent retaining wall (65.5 m x 1.8 m x 6 m) - University of Tokyo + Seikei University: earth-transport experiment with multiple autonomous machines; when a team's performance dropped, the lead excavator re-allocated resources by coordinating with other teams, maintaining overall productivity in real-time - Kajima A4CSEL: achieved full-process automation (bulldozers, rollers, dump trucks) at Naruse Dam and the Kamigo test tunnel, allowing a single operator to manage multiple machines
Future Outlook
- Digital twin technology will create virtual models enabling real-time mapping of machine status and environmental changes
- Edge-and-cloud collaborative architecture will balance on-site real-time performance with big-data analytics advantages; supported by 5G and 6G networks
- Standardized, modular designs will ease retrofitting of legacy machines and enable seamless interoperability
- Human-machine collaboration is likely to become the dominant mode in complex construction scenarios — autonomy does not replace operators but frees them from tedious, dangerous tasks to focus on higher-level decision-making and management
Key References
| Ref | Authors | Title / Description |
|---|---|---|
| [1] | Dadhich et al. (2016) | Key challenges in automation of earth-moving machines, Autom. Constr. 68, 212-222 |
| [7] | Borngrund et al. (2022) | Deep-learning-based vision for earth-moving automation |
| [8] | Ha et al. (2019) | Platform-centric autonomous robotic systems for earthmoving in military applications |
| [9] | Lee et al. (2019) | Challenges in teleoperation of excavator toward human-in-the-loop construction automation |
| [10] | Eraliev et al. (2022) | Autonomous excavation systems: sensing, perception, decision, planning, action |
| [12] | Chen et al. | Excavation point selection with GA-based optimization incorporating slope and distance metrics |
| [64] | Sarata et al. | Column model + bucket trajectory for excavation point evaluation |
| [89] | Singh et al. (1998) | Multi-resolution planning for earthmoving; two-section bucket method |
| [90] | Backman et al. (2021) | Continuous control of underground loader using deep RL, Machines 9(10), 216 |
| [91] | Lindmark et al. (2018) | Computational exploration of robotic rock loading, Robot. Auton. Syst. 106, 117-129 |
| [96] | Yao et al. (2023) | Bucket loading trajectory optimization for automated wheel loader, IEEE Trans. Veh. Technol. 72(6), 6948-6958 |
| [105] | Egli et al. (2022) | Soil-adaptive excavation using reinforcement learning, IEEE Robot. Automation Lett. 7(4), 9778-9785 |
| [106] | Zhang et al. (2021) | An autonomous excavator system for material loading tasks, Sci. Robot. 6(55), eabc3164 |
| [110] | Yao et al. (2023) | Real-time task-oriented continuous digging trajectory planning for excavator arms, Autom. Constr. 152, 104916 |
| [116] | Chen et al. (2025) | Estimating bucket fill factor for loaders using point cloud hole repairing, Autom. Constr. 170, 105886 |
| [117] | Li et al. (2025) | Multi-sensor data fusion and deep learning-based prediction of excavator bucket fill rates, Autom. Constr. 171, 106008 |
| [118] | Helian et al. (2024) | Computer vision-based excavator bucket fill estimation using depth map and Faster R-CNN, Autom. Constr. 166, 105592 |
| [122] | Stentz et al. (1999) | A robotic excavator for autonomous truck loading, Auton. Robot. 7(2), 175-186 |
| [125] | Huang et al. (2021) | Providing proximity alerts to workers using Bluetooth Low Energy RTLS, Autom. Constr. 132, 103928 |
| [126] | Teizer et al. (2015) | Proximity hazard indicator for workers near construction equipment, Autom. Constr. 60, 58-73 |
| [131] | Zhang et al. (2024) | Deep learning framework with local sparse transformer for construction worker detection in 3D with LiDAR, Comput. Aided Civ. Inf. Eng. 39(19), 2990-3007 |
| [139] | Zou et al. (2023) | Day-to-night obstacle detection for excavators based on image enhancement and multisensor fusion, IEEE Sensors J. 23(10), 10825-10835 |
| [144] | Yajima et al. (2021) | Development of an excavator-avoidance system for buried pipes, Adv. Robot. 35(23), 1468-1483 |
| [145] | Kim et al. (2022) | Underground metal pipeline localization using low-cost wireless magnetic sensors mounted on an excavator, IEEE Trans. Ind. Electron. 69(10), 10674-10683 |
| [147] | Wang et al. (2023) | Underground infrastructure detection using deep learning enabled radargram inversion, Autom. Constr. 154, 105004 |