2025 Towards Fully Integrated Autonomous Excavation-서울대

2025 Towards Fully Integrated Autonomous Excavation-서울대

Towards Fully Integrated Autonomous Excavation

Autonomous Excavator for Precise Earth Cutting and Onboard Landscape Inspection

By Inkyu Jang , Junha Kim , Dongjae Lee , Changhyeon Kim , Changsuk Oh , Youngbum Kim, Sangwook Woo, Heejee Sung, and H. Jin Kim

Autonomous excavator systems (AESs) can alleviate the issues caused by the shortage of skilled labor forces and increasing labor costs. For autonomous excavation, real-time landscape estimation, excavation path generation, control, and precise landscape inspection are all essential. In this article, we propose and experimentally validate an integrated AES that incorporates all these elements. Specifically, unlike previous research, we introduce a sensor arrangement capable of sufficiently covering the regions of interest (ROIs) regardless of the inclination of the target landscape, a motion planning method that satisfies geometric and physical constraints, and a precise postexcavation inspection module using onboard sensors only. The proposed methodology was experimentally validated using a real 30-ton hydraulic excavator. It successfully performed a cutting task on an upward slope with 45° inclination and achieved a centimeter-level accuracy through autonomous repetitive excavation; also, the proposed postexcavation inspection method demonstrated subcentimeter precision within seconds using onboard sensors only.

INTRODUCTION

The need for automation in construction has been consistently growing due to rising labor costs, a lack of skilled laborers, and stricter safety requirements. This has led to active research both in industry and academia, including autonomous driving for heavy equipment vehicles [1], autonomous trenching for hydraulic excavators [2], [3], [4], dry stone wall construction [5], and semiautomation in wheel loader material loading [6]. Specifically, tasks like large-scale land cuttings and precise surface grading, essential at the beginning of most construction sites, can benefit from automation as they involve repetitive actions that require precision and accuracy. These operations typically consist of two subtasks: earth cutting using excavators and postexcavation topographic inspection via external equipment. Traditionally, both are carried out by skilled human workers who must sometimes risk their safety in hazardous workspaces full of massive equipment. A fully onboard integrated AES, which reduces the need for external equipment and human effort, therefore, possesses huge potential and advantages.

The requirements for the first subtask, earth cutting, are multifaceted. First, a decision-making module that determines

Digital Object Identifier 10.1109/MRA.2024.3400772

1070-9932/24©2024IEEE

where to dig and generates the path for the arm part are needed. This module should be capable of creating a digging profile and the arm path within seconds to ensure uninterrupted workflow. Second, to facilitate the aforementioned path planning, a perception functionality that provides information about the current landscape is necessary. This perception functionality need not be overly precise considering the size and control accuracy of the hardware, and it should be capable of capturing real-time changes in the terrain made by the digging and dumping actions. Finally, to track the generated path despite disturbing forces such as bucket-soil interaction, a local trajectory planner that offers real-time tracking should be in place. This local planner should take into account the physical limitations of the excavator, such as power and arm speed constraints.

The second subtask, postexcavation landscape inspection, aims to verify whether the targeted terrain has been accurately shaped. This subtask is conducted separately from earth cutting to ensure the integrity of the assessment process and to prevent any potential influence from earth-cutting motion. Construction sites typically employ specialized surveying personnel who submit the results as evidence of completion. Although construction standards may vary by region and the type of task, a typical goal is to achieve construction accuracy within a few centimeters of the target terrain. To meet this requirement, the inspection module should be able to perceive the landscape with an accuracy level of less than 1 cm. This should be carried out within seconds to prevent delays in the construction process.

Despite several attempts both in industry and academia to implement a fully autonomous excavator, none of the existing solutions have performed both motion generation and postexcavation inspection on board [3], [4], [5]. In this research, we present an AES (see Figure 1) that combines real-time landscape estimation, onboard motion generation, and accurate postexcavation topography inspection—all achieved solely through onboard apparatus. Our system improves existing AES solutions in the following perspectives:

We demonstrate the performance of the proposed integrated AES with actual hardware in a 45° upward slope-cutting scenario. The proposed AES successfully shaped the desired landscape and conducted a postexcavation topography inspection with a mean error of less than 1 cm.

Fully autonomous excavators are receiving considerable attention from both industry and academia. In the industry,

==> picture [233 x 257] intentionally omitted <==

FIGURE 1. The 30-ton hydraulic excavator hardware considered in this research. The proposed AES is capable of performing earth cutting and postexcavation inspection works fully autonomously using onboard sensors only.

Built Robotics provides an autonomous, safety-aware trenching solution that incorporates sensors and software [2]. In academia, Zhang et al. [3] provide an integrated framework that includes sensing, planning, and control for a fully autonomous excavator, and shows applicability in multiple tasks that involve trenching, material loading, and rock removal. Similar tasks were also performed using a walking excavator [4], [5] in a fully autonomous manner. However, none of the attempts tackled integration of earth cutting and postexcavation topographic inspection. In the remainder of this section, we introduce related works in the perspectives of excavator sensor configuration, environment perception, and excavation motion generation.

Thanks to their ease of use and reliability of measurement data, 3D lidar sensors have been widely employed for autonomous excavators. By deploying them, existing works on system integration [3], [5], [7], [8], [9] installed one or two 3D lidar(s) on top of the cabin. However, in the case of slope cutting, such a configuration may have limited FOV, and the ROI for excavation may not be fully covered. In [5] and [10], a drone with a camera was used, but the additional burden of operating these equipment naturally follows. To achieve selfcontained autonomous excavation, a strategy that ensures a large FOV without external equipment is desirable.

In a typical construction site, landscape inspection should be conducted separately after the earth-cutting work because it serves as confirmation of completion of the task. Although the actual value varies depending on the type of construction work, inspection should be done at an accuracy of less than a centimeter. Although this precision is typically achieved by using land surveying solutions [11], its drawback lies in the need for separate workers and equipment. Furthermore, these solutions are expensive and can take tens of minutes, even for scanning a small area up to 10 m[2] . Existing autonomous heavy equipment [3], [5], [7], [8], [9] has utilized 3D lidar sensors to perform a perception of the surrounding landscape, but whether they are accurate and reliable enough to replace the inspection process is still unclear as there was little analysis on that.

Excavation trajectory should abide by both 1) geometric constraints, such as vehicle–ground collision avoidance, and 2) physical constraints, including power and flow rate limits, as a violation of either of these may lead to dangerous situations such as tripping over or failure to accomplish the given task. Although many studies have considered the motion constraints, they either admitted constraint violation [4], [7], [12] or partially omitted the physical constraints [13], [14]. Only a few of them, such as [15] and [16], have addressed both types of motion constraints; unfortunately, they were tested only in simulation environments. Their robustness against noisy data and real-time applicability (in terms of computation time) were therefore not validated with real-world hardware.

SYSTEM OVERVIEW

The proposed AES is capable of completing an excavation task without human intervention, where we refer a single excavation task to the whole process of achieving the desired

topography, after the base part (crawler or wheels) has settled to a designated place, and before it leaves from the location. More specifically, an excavation task in our AES consists of the following process:

During the task, the real-time landscape estimation module consistently provides real-time estimates of the current landscape given information from the onboard sensors attached to positions carefully selected for better perception capabilities. Until the real-time observed landscape sufficiently resembles the desired landscape, the motion planning module iteratively computes an excavation path and a trajectory that tracks the generated path. Then, the precise inspection software examines the completed topography after the grading process. Figure 2 provides a brief overview of how the four features collaborate to complete each step of the excavation task. In the following four sections, we delineate the details of each feature.

HARDWARE

HYDRAULIC EXCAVATOR

As shown in Figure 3(a), the kinematic model of the excavator consists of five components: the crawler, the cabin attached to the crawler with a vertical revolute joint, and the arm part, composed of three serially connected rigid bodies (boom, stick, and bucket). We call the rotation of the cabin with respect to the crawler the swing motion . The four hydraulic joints that actuate the swing motion, boom, stick, and bucket can be autonomously manipulated using signals relayed from an onboard computer.

SENSOR CONFIGURATION

To obtain kinematic information, inertial measurement units (IMUs) mounted on the cabin, boom, stick, and bucket measure the configuration of the arm part. A rotary encoder in the revolute joint that connects the crawler and the cabin measures the swing angle. Angle measurements are utilized in

==> picture [457 x 667] intentionally omitted <==

----- Start of picture text -----
205
Traversed Traj.
200
Set Points
Time (c)
Plant
MPC
195
Path Tracking Using MPC Low-Level Controller (Hydraulic Excavator)
Hydraulic Cylinder Pressure
Path
190
Pressure Sensor
Joint Angle and Angular Rate Path (b)
Reconstructed 3D Points
Excavation Path Generation
Angle Sensor
(d)
2.5D Height Map
Large FOV
Stereo Camera
Precise Landscape Inspection Using Multiview Stereo
Lid
(a)
BBoom Lidar
Lid Real-Time Landscape Estimation
Intentional Swing Motion
bi
Cabin LidarC
Lidar Map
Previous
Cylinder Lengths
----- End of picture text -----

==> picture [479 x 670] intentionally omitted <==

----- Start of picture text -----
r 1
}
{} l 0 T lb 00 { b 0 0
Calibration Gb
T
3) Lidar-to-Lidar (L2L)
θ
lb 10 (d)
T
}{ l 1 G {}
Calibration
1) Camera-to-Camera (C2C) Calibration
4) Sensors-to-Vehicle (S2V) Body Calibration
2) Lidar-to-Camera (L2C)
1
d
(b)
1
θ
Downhill
Boom Lidar
and Stereo Camera
Cabin Lidar
(c)
Bucket Stick Crawler
Uphill
Boom Lidar
and Stereo Camera
Boom (a)
(a) The hardware components of the proposed AES consist of an unmanned-controllable hydraulic excavator with onboard cameras and 3D lidar sensors. The excavator kinematics
Cabin Lidar Cabin Flat Ground
FIGURE 3. can be understood as a robotic manipulator with five joints (crawler, cabin, boom, stick, and bucket) connected in series through revolute joints. A stereo camera is attached on the boom, and two 3D lidars are attached to the cabin and the boom, respectively. The yellow labels denote the excavator parts, while the red ones represent the sensors. (b) The configuration of the employed sensors is two 3D lidars and two cameras. Sensor configuration is carefully selected to secure a uniformly and sufficiently large FOV in various arm (boom, stick, and bucket) configurations; this is for always including the ROI inside the FOV during slope cutting with arbitrary inclination, as shown in (c). (d) Sensor calibration. The numbering indicates the order of the calibration. Thanks to the carefully selected sensor configuration, sensor calibration can be performed with only one or two human worker(s) because the FOVs of the onboard sensors overlap in the ROI on the ground, which is easily accessible without additional equipment such as ladders. In addition, for the whole calibration process, a checkerboard is only required as the calibration target.
----- End of picture text -----

­excavation path generation and control, while cylinder pressures measured by sensors inside each hydraulic cylinder are employed to define a power constraint in motion planning.

As shown in Figure 3(b), two 3D lidars and a stereo camera are used. To ensure a large FOV during slope cutting, we attach one 3D lidar (i.e., a boom lidar) at the bottom of the boom where the rotating axis of the boom lidar is parallel to the rotating axis of the boom. As the boom lidar moves along with the boom, dense point cloud data in the ROI can be obtained for landscapes with any inclination, as shown in Figure 3(c). This allows the excavator to well estimate the surrounding landscape during excavation without the need for an intentional boom motion.

To additionally provide consistent observation of the ROI regardless of the motion of the boom, we attach the other 3D lidar (i.e., a cabin lidar) to the side of the cabin with a slight tilt angle. The cabin lidar compensates the boom lidar, and the landscape can be measured in the distance without severe occlusion by the arm part. By using cabin and boom lidars together, a denser point cloud at the ROI can be obtained.

To obtain precise and accurate environment reconstruction during landscape inspection, we utilize the stereo camera with the boom and cabin lidars. A stereo camera attached at the bottom of the boom alongside the boom lidar provides a sufficiently large FOV, regardless of the inclination of the landscape the excavator is facing. This is because, during inspection, the boom angle no longer needs to be manipulated for excavation, only for inspection. As we discuss in the remaining sections, the use of lidar data in environment reconstruction provides good prior knowledge of the landscape and therefore enhances computation speed.

SENSOR CALIBRATION

Note that all the sensors’ FOVs overlap near the ROI, which is easily accessible by human workers without the need for special equipment such as a ladder, as shown in Figure 3(c). This allows the calibration of the attached sensors to be done efficiently by only one or two workers, thereby reducing human effort, especially considering that the arm reach of the equipment typically ranges up to 10 m. The whole calibration process requires only a single checkerboard as the target. Using a checkerboard, we estimate the intrinsic parameters of cameras and extrinsic parameters between sensors, and between each sensor and the excavator: camera-to-camera [(C2C), i.e., within the stereo camera], lidar-to-camera (L2C), lidar-to-lidar (L2L), and sensors-to-vehicle (S2V) calibration, as shown in Figure 3(d).

C2C and L2L calibrations are performed using [17] and [18], respectively. The extrinsic parameters between the 3D lidars and the stereo camera are estimated by the automated calibration target detection algorithm proposed in [18], and by solving an optimization problem to minimize the distance between the target planes estimated by each sensor measurement in the same epoch. Then, through L2L calibration, we calculate the relative pose between the two 3D lidars for various boom angles. With the obtained relative pose, we solve for

boom angle ( i ) at each epoch and the distance ( rl ) from the boom rotation axis to the boom lidar. Finally, the S2V extrinsic calibration is performed by using poses of cabin lidar and boom lidar with respect to the frame { b 0} on the boom’s rotation axis. As a result, relative poses between all the sensors and the vehicle can be calculated for any boom angle measured by the IMU attached to the boom.

REAL-TIME LANDSCAPE ESTIMATION

For real-time landscape estimation, we maintain a grid-based 2.5D elevation map and update it in real time using 3D lidar sensor measurements similar to those of Jud et al. [7]. Such estimation inheres the following challenges:

To address the first challenge, we first filter out body-occluded points for every incoming 3D lidar point cloud data. This rejection is done by considering the kinematics and geometric model of the excavator: as shown in Figure 4(c), any 3D lidar point that falls inside any one of the box-shaped regions is rejected.

Although the proposed point rejection method sorts out most of the undesired points, there may still exist many nonrejected points. They originate mainly from the mechanical vibration of the sensor base, or the parametric error between the geometric model and the actual excavator. Also, during dumping, the falling dirt would produce undesirable 3D lidar points, which the aforementioned method cannot properly reject. Moreover, deformation of the landscape itself consistently occurs throughout the excavation task, necessitating a separate technique that is capable of capturing landscape changes in real time.

To address the concerns, we assume that the landscape follows a certain stochastic dynamics model, and run a Kalmanlike filter based on these dynamics. The whole 2.5D elevation map is considered a dynamic system, with the elevation of the cells being its state. Let us denote the elevation of cell c at time tk by xck . Then, xck + 1 (the elevation of the cell at the next time epoch tk + 1) is determined by the following update rule:

==> picture [182 x 25] intentionally omitted <==

where the update case (A) occurs if the bucket tip has swept below the ground level at elevation x[c] bucket on the cell between time tk and tk + 1, and uck , wck are white Gaussian noises with variance Qu and Qw , respectively. As the noises uck and wck represent the discrepancy between the model and actual elevation level, respectively, the values of Qu and Qw can be understood as measures indicating how unpredictable the landscape changes with respect to time. For

example, with small Qu and Qw values, the estimator believes that the landscape tends to remain still, and with larger values, it assumes that changes in the landscape (e.g., microscale landslips) occur more frequently, thereby relying more on recent measurements given from the lidar sensors. In case the actual values of noise cannot be directly measured, the values of Qu and Qw can be estimated statistically.

For filtering, each cell is assigned not only the estimated expectation of its elevation but also the estimation variance. Initially, every cell is assigned infinite variance and zero expectation, although the expectation has no significance under infinite variance. Let a nonrejected 3D lidar point be included in cell c with elevation y . As one can presume that a 3D lidar point closer to the center of the cell will better represent the true elevation at the cell (i.e., the elevation at the center of the cell), we assume that y follows the following model:

==> picture [169 x 11] intentionally omitted <==

where v 2 0 is the variance of the distance measurement from 3D lidar (given from the sensor specs), d is the horizon-

tal distance from the 3D lidar point to the center of the cell c , and C 2 0 is a parameter related to the maximum slope of the excavating landscape. We obtain a maximum a posteriori (MAP) estimate of the elevation of every cell at every update. Fusion for mean and variance is conducted using the elevation in the previous timestep tk , and the aggregation of incoming 3D lidar point clouds between tk and tk + 1. Such a computed MAP mean, x t ck + 1, represents the topographic information at time tk + 1 and is employed for collision avoidance and excavation path generation.

Empirically, we found that the performance of the proposed landscape estimator does not get significantly affected by the values of the tunable parameters, such as the size of the cell; Qu , Qw , and C ; and the time window dt = tk + 1 - tk , as long as their values remain reasonable. It can be seen in Figure 4(a) and (b) that the proposed landscape estimator succeeds in stably capturing the change of the ground shape, even while digging [see Figure 4(a)] and dumping [see Figure 4(b)] where reliable 3D lidar measurements are not available due to bucket movement and falling dirt. The snapshots of landscape

==> picture [478 x 326] intentionally omitted <==

----- Start of picture text -----
(a)
(b) (c)
(d)
----- End of picture text -----

FIGURE 4. Real-time landscape estimation. (a) and (b) The proposed landscape estimation module in operation during digging and dumping, respectively. In both situations, even though reliable sensor measurements are not available due to the bucket motion and the falling dirt, the proposed estimator successfully captures landscape changes in a stable manner. This is possible due to the kinematicsbased point cloud filtering [any point falling into one of the boxes shown in (c) is rejected], and the Kalman-like MAP estimation method assuming a stochastic dynamics (1) of the landscape. As multiple swing motions are made while excavation progresses, the 3D lidar measurements naturally reveal additional information about the surrounding landscape, as shown in (d). Throughout this figure, the color of the grid cell represents the covariance of the estimated elevation: red and white represent small (high estimation confidence) and big (low confidence) covariance, respectively. The grid cells with infinite covariance, i.e., never-observed cells, are not shown.

estimates taken during an excavation task are shown in Figure 4(d). As the excavation task progresses, the 3D lidar measurements reveal more area around the equipment.

EXCAVATION MOTION GENERATION AND TRACKING

To successfully accomplish an excavation task, generated motion has to satisfy both geometric constraints (e.g., collision avoidance while not in excavation, or bucket pose constraint while digging) and physical constraints (e.g., constraints on hydraulic cylinder speed, pump flow rate, and power limit). As geometric constraints can be formulated in configuration space, they can be imposed on a path without time information, and offline computation of such a path would suffice. On the other hand, because physical constraints require real-time state and input feedback and are defined on the entire state space, the constraints should be enforced on a trajectory , and the trajectory should be calculated online in real time. To fulfill all these constraints, similar to our previous work [15], we propose a two-staged motion generation scheme composed of path generation and path tracking motion generation.

EXCAVATION PATH GENERATION

Path generation functionality is devoted mainly to satisfying the geometric constraints. When generating an excavation path, we consider a number of geometric constraints. First, during digging,

To effectively compute a path that completes an excavation task while satisfying all the aforementioned geometric conditions, we first divide a single digging task into five phases, as illustrated in Figure 5(b):

To ensure path feasibility and continuity across phases, we first compute the path for phase 2, i.e., the digging profile, and then for the other phases.

For efficient and precise earth shaping, we establish two different types of digging profiles: normal digging and tip

grading. The normal digging profile is intended to scoop the largest possible amount of soil every repetition while not exceeding the bucket’s capacity so that the number of diggingdumping cycles is reduced. On the other hand, the tip grading profile aims to precisely shape the ground as desired while not penetrating through the target landscape. The normal digging profile is first employed, and then the tip grading profile is used after the current landscape approaches the target landscape sufficiently close after some digging repetitions.

To establish the criterion for how to switch between the two digging profiles, we first define three ground shapes as lines on the excavation plane: the current landscape obtained from real-time landscape estimation module, the prescribed target landscape, and the tip grading threshold located above the target landscape by a certain threshold, as shown in Figure 5(b). The excavator selects the tip grading profile if the current ground shape lies between the target landscape and the tip grading threshold, and the normal profile otherwise. Note that due to the piled-up dirt and microscale landslips, it is also possible to switch back from tip grading to the normal profile.

The normal digging profile is generated with the following procedure. We assume that the target landscape is

==> picture [232 x 277] intentionally omitted <==

----- Start of picture text -----
Excavation Plane
Excavation Path
Target Landscape
Current Landscape
(a)
Phase 4 Phase 1
Phase 5 Point of Attack Current
Landscape
Phase 3 Phase 2
(b)
Tip GradingTarget Landscape
Threshold
----- End of picture text -----

FIGURE 5. (a) An example of the excavation path generated during the experiment. The excavation plane is shown as a blue transparent box. It is defined as the plane spanned by the workspace of the arm, while the swing angle is being fixed with the cabin pointing at the desired digging direction. (b) The digging path consists of five phases. Phase 1: the bucket moves to the point of attack; phase 2: the bucket cuts through the ground; phase 3: the bucket is lifted above the ground; phase 4: the dirt inside the bucket is conveyed to the dumping position; and phase 5: dumping occurs.

achievable given the given excavator hardware and its crawler position, and it has no cavity. First, the excavator attempts to find the point of attack for excavation. The point of attack is decided to be an intersection of the current landscape and the tip grading threshold with the farthest distance from the cabin, as shown in Figure 5(b). To search for a feasible dig-

==> picture [232 x 529] intentionally omitted <==

----- Start of picture text -----
1) Uphill
2) Downhill
3) Trenching
4 ) Dee p Di gg in g
Landscape Before Excavation
Target Landscape
Excavation Progress (Red → Blue)
----- End of picture text -----

FIGURE 6. Normal digging profiles were generated in a simulation environment with different target landscapes. Regardless of the current and the target ground shapes, the proposed path generation module successfully generated consecutive digging profiles without encountering infeasibilities.

ging path while dealing with the path-dependent nature of the swept volume, we make use of a tree search algorithm on the 3D grid whose three axes consist of the lengths of boom, stick, and bucket cylinder lengths, respectively. Feasibility here refers to bucket capacity, the cutting angle (the angle between the tip path and the bucket), and whether the arm penetrates through the target landscape. To reduce the number of searches, we employ a depth-first search (DFS) algorithm on the 3D grid defined by the lengths of the three hydraulic cylinders that compose the excavator arm. Among the tree nodes with the same depth, the priority is determined by using a hardware-specific heuristic that takes into account both the expected excavation volume and the expected power consumption of the cylinders. By using a DFS, we are able to find a feasible path that scoops satisfactory volume at a relatively low computational cost. Figure 6 shows how the normal digging profile is generated in different landscapes. It can be seen from the figure that in each digging-dumping cycle, the amount of soil removed by the bucket is kept approximately constant, where the removed volume is close to the bucket’s capacity.

The tip grading profile, on the other hand, is generated in a manner that the bucket follows the target landscape. An example of a tip grading profile is shown in Figure 7. As shown in the figure, starting from the farthest reachable point on the target landscape, the bucket tip is pulled toward the cabin, while maintaining a constant cutting angle.

In phases 1, 3, 4, and 5, we apply an A* planner to generate a path in a 4D grid whose four axes comprise the swing motion and three cylinder lengths. The objective of the planner is to reach the goal configuration while considering the excavator kinematics and minimizing the aggregate flow in and out of each cylinder. In phase 4, to prevent the excavator from spilling the removed dirt, we impose an additional constraint in which a configuration space that has the open direction of the bucket not heading upward is considered an obstacle.

PATH TRACKING USING MPC

As the generated excavation path lacks time information, we perform time allocation before execution. We define the traversal time of each path segment to be maximal but not to

==> picture [232 x 116] intentionally omitted <==

----- Start of picture text -----
Clearance Angle
----- End of picture text -----

FIGURE 7. An example of a tip grading profile. The bucket tip follows the desired landscape, starting from its farthest reachable point on the target. The big red arrow denotes the moving direction of the bucket. The cutting angle is kept constant throughout the profile.

exceed the reference expansion/contraction velocity of each cylinder. Then, to enforce physical constraints on the timeallocated trajectory, we employ MPC in the local motion planner. MPC solves a constrained optimal control problem iteratively in a receding horizon manner and has been widely adopted in various robotics and construction applications [15], [19] because it enables computing of state and input trajectories, which satisfies motion constraints in real time.

The local planner assumes the following discrete-time first-order dynamical model:

==> picture [154 x 10] intentionally omitted <==

where q is the 4D configuration of the excavator, including the swing angle and the posture of the arm, and u is the 4D control input that comprises the swing rate and extension/ retraction rate of the hydraulic cylinders. Subscript k indicates that the corresponding variable is from the time step , tk and that fd is a function that determines the next state qk + 1 from the current state and input.

The dynamical model is derived from the first-order kinematic relation q o = J ( q ) u where J ( q ) is the Jacobian matrix. A better prediction model could be applied using a higherdimensional dynamical model that incorporates hydraulic pressure dynamics and inertial effects. However, due to the high uncertainty raised by oil temperature and hydraulic friction, model identification becomes extremely challenging, and the strong nonlinearity of the high-dimensional dynamics hinders the real-time applicability of MPC. To overcome these challenges, we opted for a relatively simple kinematics-based dynamical model in MPC, leveraging a low-level controller with sufficient tracking performance. The effectiveness of this approach was validated through experiments.

The physical constraints considered in MPC are 1) power limit, 2) cylinder displacement limit, and 3) pump flow rate limit, and they can be expressed in the following form at the time step : tk

==> picture [151 x 9] intentionally omitted <==

where fp is the measurement of hydraulic pressures inside the cylinders and is used to express the power limit constraint. Due to the difficulty of accurately estimating hydraulic pressure dynamics, we choose to assume constant pressure during the prediction horizon and instead take a fast update rate. Detailed expressions for the considered physical constraints can be found in [15].

Using (3) and (4), we formulate an MPC problem as follows:

==> picture [198 x 36] intentionally omitted <==

We take the cost function J to be quadratic with respect to the state error q - qr and input u ; this is to track the high-level

reference trajectory while regulating inputs so that excessively aggressive maneuvers are avoided. The computed local trajectory is then transformed to push/retract force input for the hydraulic cylinders by the low-level controller and relayed to the excavator.

The formulated MPC problem is solved using a constrained iterative linear quadratic regulator algorithm with the augmented Lagrangian method [20]. We validate its real-time applicability and constraint satisfaction in experiments. On the onboard computer, our MPC was able to run consistently at 10 Hz or faster, given the time horizon of 6 s, time discretization, and integration interval of 0.04 s. Furthermore, the proposed MPC ensures satisfying all the physical constraints, as visualized in Figure 10(c). An ablation study for the imposed physical constraints (the baseline results in the figure) also validates the effectiveness of our MPC formulation.

PRECISE LANDSCAPE INSPECTION USING MULTIVIEW STEREO

The proposed 3D lidar-based real-time landscape estimation module captures terrain changes promptly and reliably. However, it is not suitable for postexcavation inspection due to the following reasons. First, the structural vibration corrupts the relative pose information between the two 3D lidars. S2V alignment quality decays with respect to time due to extreme vibration and impact during the digging task. Second, the landscape estimation exhibits sensing inaccuracy of up to decimeters due to the nonnegligible grid size. One might use smaller grid cells, but the inaccuracy that originates from the poor lidar alignment still remains unresolved. Finally, the lack of 3D features on the unstructured construction site and the sparsity of 3D lidar point cloud data make point cloud alignment hard, posing difficulty in employing lidar mapping methods for inspection.

We therefore construct a separate module using a stereo camera attached to the bottom of the boom for landscape inspection. Cameras provide high-resolution information at a relatively low cost, which leads to high-quality 3D reconstruction. The proposed vision-based inspection module does not require additional external apparatus and operates with onboard sensor only s, completing the process within seconds. This inspection module shows a competent performance in precision and accuracy compared to land surveying solutions. Additionally, as the two cameras are rigidly connected to each other by short distance, the stereo camera is more reliable compared to 3D lidars in terms of robustness against severe vibration and impact.

To ensure a sufficient amount of observation data and larger horizontal coverage of the stereo camera, intentional swing and boom motions are introduced during the inspection. Because the camera is mounted under the boom facing downward, such motions will allow the camera to view the terrain from multiple viewpoints. Figure 8(a) illustrates the excavator motions that result in horizontal motion. Empirically, obtaining observations with swing motions of tens of degrees without a boom motion is sufficient to achieve sub-centimeter-level precision.

A bundle adjustment (BA) is employed, which jointly optimizes all the camera poses and 3D points using multiple image measurements. Dense representation with millions of 3D points for landscape inspection yields many pose-and-point coupling terms, which may result in burdensome computations. To that end, we propose a simple but effective two-step optimization. In the first step, all camera poses and the reduced number of sampled points are jointly optimized. In most of the cases, we found that camera poses are accurately optimized with only thousands of points. We sample several hundred features per image with the highest Features from Accelerated Segment Test [21] scores. In the second step, by fixing the optimized

We employ the Kanade–Lucas–Tomasi (KLT) algorithm for feature tracking. The original KLT algorithm assumes that the camera always observes the environment in a vertical direction, which is not our case. Thus, to achieve better performance in feature tracking, we model patch deformation with an affine pixel transformation and solve an optimization problem. A constrained affine KLT-based approach to track features in the target region is employed, where the initial guess is given from the real-time estimated topography. Through correspondences of feature points in sequential images, we reconstruct the 3D points of features and estimate camera poses at each sequence.

camera poses, a point-only BA is executed to optimize all the 3D points. By doing this, we can avoid computationally demanding calculations of pose-point coupling terms and make the proposed inspection module work within a few seconds. Note that the difference in the overall accuracy between the proposed two-step method and full BA is negligible, but the two-step method is significantly faster than the full BA. A qualitative comparison of the two methods’ computation time in different inspection scenarios is given in Table 1.

==> picture [318 x 214] intentionally omitted <==

----- Start of picture text -----
Intentional Swing Motion
(b)
(a) (c)
----- End of picture text -----

For various terrain configurations in precision earth-cutting scenarios, as shown in Figure 9, the landscape inspection module utilizing an onboard stereo camera achieves an accuracy level of 1 cm, which outperforms the two baselines using 3D lidar only or single-view stereo reconstruction, as shown in Table 2. In the table, it can also be found that the shape of the target terrain does not significantly affect accuracy. This level of accuracy surpasses the precision achievable by skilled human operators operating commonly used construction excavators of general sizes (20 tons or larger).

FIGURE 8. (a) The intentional swing motion of the excavator. It can be seen from (b) that the proposed multiview stereo method (blue region) not only provides a larger FOV but also enhances the accuracy and precision of landscape reconstruction, compared to the singleview counterpart (red region). (c) A stitched image generated using the bundle adjustment algorithm. This image is generated using 140 single-view images and −30° ~30° swing motion with the boom being fixed.

TABLE 1. The computation time comparison between a commercial solution (Trimble scanning Total Station) and our method.

==> picture [306 x 138] intentionally omitted <==

----- Start of picture text -----
COMPUTATION
COMPUTATION TIME PER UNIT
SCENARIO METHOD AREA (m [2] ) TIME (s) AREA (s/m [2] )
Upward slope Scanning Total Station 28 2542 90.8
Multiview stereo—full BA 75 833 11.11
Multiview stereo—two- 75 48 0.64
step BA (ours)
Flat ground Scanning Total Station 75 3521 46.94
Multiview stereo—full BA 101 978 9.68
Multiview stereo—two- 101 59 0.584
step BA (ours)
----- End of picture text -----

EXPERIMENTAL VALIDATION

EXPERIMENT SETUP

Now we validate the proposed AES using a real hardware in a cut slope task, in which we want the excavator to cut a 45° upward slope that is 3.5 m wide and 1.5 m high. Such an upward slope-cutting task is one of the most commonly conducted excavation tasks in the first stage of construction. Our experiment site consists of up to 10 cm-sized gravel and soil.

Bold, underlined numbers represent the fastest among the three methods, in terms of computation time per unit area. For the ground shapes, see Figure 9(a) and (d). The BA methods are capable of performing inspections within a larger FOV with less time consumption, and the proposed two-step algorithm further enhances computation speed.

==> picture [478 x 126] intentionally omitted <==

----- Start of picture text -----
(a) (b) (c) (d)
----- End of picture text -----

FIGURE 9. The four scenarios used to validate the proposed landscape inspection module. Each scenario represents a realistic construction site for excavators. We compare the results of Trimble scanning Total Station [11] (a commercial land-surveying solution) and the proposed precise landscape inspection, as shown in Table 2. For comparison, we calculated the reprojection error onto the top-down view of the scene after aligning the coordinates of the two methods. (a) A smooth slope, (b) slope with a crater, (c) uneven slope, and (d) flat ground.

The hardware is a 30-ton excavator: Hyundai HX300AL. For communication between the onboard sensors and the computer, Ethernet and controller area networks are employed. The excavator’s bucket can scoop up to 1 m[3] soil in a single excavation. On the bucket tip are steel teeth 10 cm in length, which are attached to facilitate earth cutting. The hydraulic actuators are controlled by spool valves. The positions of the spools determine the flow rate to each actuator. The spools are driven by the pilot hydraulic pressure, which is controlled by the electronic signal from the onboard computer through electrohydraulic pressure-reducing valves at 100 Hz. The 3D rotating lidar sensors used

TABLE 2. A quantitative comparison of 3D reconstruction accuracy and precision for the four validation scenarios shown in Figure 9.

==> picture [304 x 112] intentionally omitted <==

----- Start of picture text -----
SMOOTH SLOPE WITH UNEVEN FLAT
SLOPE A CRATER SLOPE GROUND
Lidar only Median 2.7 1.4 0.8 3.0
StDev. 6.0 5.9 8.8 4.2
Single-view stereo Median 1.5 1.5 2.8 9.1
StDev. 13.4 4.2 3.7 4.0
Multiview stereo (ours) Median 0.4 0.6 0.8 0.5
StDev. 4.7 2.8 7.0 2.7
----- End of picture text -----

unit: cm; StDev.: standard deviation. Bold, underlined numbers represent the best among the three methods. The measurements obtained from Trimble scanning Total Station are used as the ground truth. The lidar-only method uses lidar point clouds accumulated during the acquisition of multiple stereo images, the single-view stereo method reconstructs a 3D point cloud with only one pair of images, and the multiview stereo method is the proposed precise landscape inspection method.

A qualitative investigation of the 35 m[2] region around the excavated landscape was carried out using the proposed inspection method, as shown in Figure 10(d). For precise landscape inspection, considering the shape of the ROI, we fixed the boom angle and adopted a swing-only motion of 60°. The mean and the standard deviation of the absolute value of the elevation error were calculated as 7.4 and 5.3 cm, respectively, where the elevation error was measured within the rectangular ROI of 3.5 m in width and 3 m in depth. The results shown in Figure 10(b2) and (d) are sufficiently precise considering the excavator’s size and the bucket type; it is at a level similar to that of professional operators.

in the experiment are capable of measuring distance at 3-cmlevel accuracy. The IMUs provide the posture of the arm part with an accuracy of 0.1°, and the rotary encoder measures the swing angle at 2° accuracy.

As the amount of soil to be excavated exceeds the capacity of the bucket, the excavator needs to repeat dig cycles at different base positions. To attain the desired width of the slope, we perform excavation at three different base locations by moving the crawler in the horizontal direction to the landscape.

At every base location, excavation is performed until the bucket tip reaches the tip grading threshold, where we set the tolerance as 50 cm. Due to soil movement, tip grading sometimes results in an error in height more than the predefined tolerance. In that case, the excavator performs excavation and tip grading over again. Such a cycle repeats until the height error is uniformly smaller than the tolerance in the ROI.

Thanks to the MPC-based motion generation step, which prevents violation of the physical constraints [Figure 10(c)], stable excavation can be accomplished without wobbling the bucket tip position. We found that in the early stages of the earth-cutting task, where there is a substantial amount of soil to be excavated and significant earth-bucket interaction forces are present, the bucket tip exhibits a relatively large tracking error of up to 50 cm. However, as the operation progresses and the amount of soil to be excavated decreases, along with a reduction in the load on the excavator, we observed that the

RESULT AND DISCUSSION

The experimental results in Figure 10(a) and (b) show the excavation process, including snapshots of estimated landscape (a1)–(a4), the earth-cutting result (b1), and the landscape inspection result (b2).

excavator gradually achieves better precision in tracking the planned trajectory, resulting in the aforementioned centimeter-level earth-cutting quality. The tip position error, however, largely relies on the performance of the low-level controller, such as in [22] and [23]. This is out of the scope of this article, but we believe that excavation error can be further improved by enhancing the controller.

COMPUTATION TIME ANALYSIS

Finally, we conduct a computation time analysis. All the proposed software are implemented and run on an onboard com-

puter with a quad-core CPU at base-clock 1.8 GHz and 8-GB random-access memory. First, the landscape estimation module takes measurements of the two 3D lidars running at 10 Hz and generates a 2.5D elevation map with a grid size of 40 cm × 40 cm in 5 Hz. Considering the rate of landscape change, this update rate is sufficiently fast to reflect changes in real time. For precise landscape inspection, it takes 60 s for photo shooting (i.e., intentional swing motion), and the algorithm itself takes 25.8 s (or 0.74 s/m[2] ). Compared to the widely applied Trimble scanning Total Station [11], which spends 1,192 s (or 34.2 s/m[2] ), the proposed method consumes significantly

==> picture [478 x 453] intentionally omitted <==

----- Start of picture text -----
(1) (1)
(2)
(3)
(2)
(4)
(a) (b)
0.2
3
Baseline
2 Maximum P roposed 1
0.1
1 0
8
0 7
6 2
–1 1
0 200 400 600 800 1,000 5 0
Time (s) 4 – 2 –1 x (m) 0
(c) (d)
y
(m)
(m)
z
Power (Normalized) Absolute Error (m)
----- End of picture text -----

FIGURE 10. The experimental result. (a1)–(a4) The landscape change as excavation proceeds. Excavation starts at the state in (a1), and the target landscape is established through an iterative digging cycle. (b1) The landscape after excavation terminates. (b2) The precise landscape inspection result after excavation. (c) The power constraint, one of the physical constraints, is shown to be satisfied through the proposed MPC throughout the execution time, while the constraint-free baseline algorithm fails to adhere to the constraint. (d) A comparison between the desired landscape and the result of precise landscape inspection for the 3.5 m × 3 m ROI. Excavation is accomplished within 10-cm accuracy.

less time while providing competent performance. A digging path is computed in 0.3 s in most cases, and the path-tracking MPC that generates a local trajectory runs faster than 0.08 s. As the computation time of the digging profile generation is faster than the float time between two sequences of soil dumping and returning to the excavation plane with swing motion, ceaseless excavation can be performed. Furthermore, the physical constraints can be satisfied thanks to the fast running rate of MPC, which enables real-time response to the change of cylinder pressure and the excavator’s configuration.

CONCLUSION

SUMMARY

This research proposed an integrated AES that combines real-time landscape estimation, excavation path generation, path tracking, and precise landscape inspection. In detail, we suggested a sensor arrangement to ensure sufficient FOV— even during excavation—regardless of the target landscape, a motion planning technique that considers both geometric and physical constraints, and real-time landscape estimation and precise landscape inspection methods that rely on onboard sensors only.

These proposed techniques were validated on a real 30-ton hydraulic excavator. During the experiments, the proposed methodology successfully executed an earth-cutting task on a 45° upward slope, achieving accuracy within 10 cm through repetitive autonomous excavation tasks. The newly suggested postexcavation inspection technique completed its task within 1 min, showing an accuracy level of less than 1 cm. With its tens-of-times-faster speed than commercial solutions in terms of computation time per inspection area, our method achieved a similar level of inspection accuracy.

DISCUSSION AND FUTURE WORK

Although we successfully demonstrated the completion of a single excavation task that autonomously uses the proposed AES, there still remains the need for additional research for deployment of an AES in real-world construction sites.

In terms of perception, using cameras for landscape inspection poses drawbacks, such as the inability to work during darkness and susceptibility to weather conditions. Improvement in this aspect is needed, including employing backup sensors. Regarding excavation motion generation, our method relies on the local planner to track the global planning generated when the digging cycle begins, but the excavator may get stuck in the presence of unperceived rigid, undiggable obstacles in the soil, such as bedrock. Improvements to the local planner can be made to address this issue; for instance, creating and utilizing a 3D map of underground soil ease of excavation based on hydraulic pressure could be a viable solution.

To deploy an AES in large-scale construction sites, the movement of the excavator base should be taken into account. This includes higher-level task planning on where to begin digging, such as in [24]; cooperation with other excavators and heavy equipment, such as dump trucks; and developing addi-

tional modules to distinguish drivable and nondrivable areas using onboard sensors.

ACKNOWLEDGMENT

Inkyu Jang, Junha Kim, Dongjae Lee, and Changhyeon Kim contributed equally to this work. This research was supported by a research grant from HD Hyundai Construction Equipment Co., Ltd. This article has supplementary downloadable material available at https://doi.org/10.1109/MRA.2024.3400772, provided by the authors.

AUTHORS

Inkyu Jang , Department of Aerospace Engineering and Automation and Systems Research Institute, Seoul National University, Seoul 08826, Korea. E-mail: leplusbon@snu. ac.kr.

Junha Kim is with the Department of Mechanical and Aerospace Engineering and Automation and Systems Research Institute, Seoul National University, Seoul 08826, Korea. E-mail: wnsgk02@snu.ac.kr.

Dongjae Lee , Department of Aerospace Engineering and Automation and Systems Research Institute, Seoul National University, Seoul 08826, Korea. E-mail: ehdwo713@snu. ac.kr.

Changhyeon Kim , Samsung Research, Seoul 06765, Korea. E-mail: rlackd93@snu.ac.kr.

Changsuk Oh , Department of Aerospace Engineering and Artificial Intelligence Institute, Seoul National University, Seoul 08826, Korea. E-mail: santgo@snu.ac.kr.

Youngbum Kim , HD Hyundai XiteSolution Co., Ltd., Seongnam 13553, Korea. E-mail: youngbum.kim@hd.com.

Sangwook Woo , HD Hyundai XiteSolution Co., Ltd., Seongnam 13553, Korea. E-mail: sangwook.woo@hd.com.

Heejee Sung , HD Hyundai XiteSolution Co., Ltd., Seongnam 13553, Korea. E-mail: heejee.sung@hd.com.

H. Jin Kim , Department of Aerospace Engineering and Automation and Systems Research Institute, Seoul National University, Seoul 08826, Korea. E-mail: hjinkim@snu.ac.kr.

REFERENCES

[1] “Autonomous mining & autonomous construction solutions.” SafeAI. Accessed: Feb. 29, 2024. [Online]. Available: https://www.safeai.ai/

[2] “Robots that build the world.” Built Robotics. Accessed: Feb. 29, 2024. [Online]. Available: https://www.builtrobotics.com/

[3] L. Zhang et al., “An autonomous excavator system for material loading tasks,” Sci. Robot. , vol. 6, no. 55, 2021, Art. no. eabc3164, doi: 10.1126/scirobotics. abc3164.

[4] D. Jud et al., “Heap-the autonomous walking excavator,” Autom. Construction , vol. 129, Sep. 2021, Art. no. 103783, doi: 10.1016/j.autcon.2021.103783.

[5] R. Mascaro, M. Wermelinger, M. Hutter, and M. Chli, “Towards automating construction tasks: Large-scale object mapping, segmentation, and manipulation,” J. Field Robot. , vol. 38, no. 5, pp. 684–699, 2021, doi: 10.1002/rob.22007.

[6] B. Cao, X. Liu, W. Chen, H. Li, and X. Wang, “Intelligentization of wheel loader shoveling system based on multi-source data acquisition,” Automation Construction , vol. 147, Mar. 2023, Art. no. 104733, doi: 10.1016/j.autcon.2022.104733.

[7] D. Jud, P. Leemann, S. Kerscher, and M. Hutter, “Autonomous free-form trenching using a walking excavator,” IEEE Robot. Autom. Lett. , vol. 4, no. 4, pp. 3208–3215, Oct. 2019, doi: 10.1109/LRA.2019.2925758.

[8] R. L. Johns et al., “Autonomous dry stone: On-site planning and assembly of stone walls with a robotic excavator,” Construction Robot. , vol. 4, nos. 3–4, pp. 127–140, 2020, doi: 10.1007/s41693-020-00037-6.

[9] E. Jelavic´, D. Jud, P. Egli, and M. Hutter, “Robotic precision harvesting: Mapping, localization, planning and control for a legged tree harvester,” Field Robot. , vol. 2, no. 1, pp. 1386–1431, 2022, doi: 10.55417/fr.2022046.

[10] D. Moon, S. Chung, S. Kwon, J. Seo, and J. Shin, “Comparison and utilization of point cloud generated from photogrammetry and laser scanning: 3D world model for smart heavy equipment planning,” Autom. Construction , vol. 98, pp. 322–331, Feb. 2019, doi: 10.1016/j.autcon.2018.07.020.

[11] “Total stations.” Trimble Geospatial. Accessed: Feb. 29, 2024. [Online]. Available: https://geospatial.trimble.com/products-and-solutions/total-stations

[12] Z. Yao, S. Zhao, X. Tan, W. Wei, and Y. Wang, “Real-time task-oriented continuous digging trajectory planning for excavator arms,” Autom. Construction , vol. 152, Aug. 2023, Art. no. 104916, doi: 10.1016/j.autcon.2023.104916.

[13] Y. Yang, P. Long, X. Song, J. Pan, and L. Zhang, “Optimization-based framework for excavation trajectory generation,” IEEE Robot. Autom. Lett. , vol. 6, no. 2, pp. 1479–1486, Apr. 2021, doi: 10.1109/LRA.2021.3058071.

[14] E. Jelavic and M. Hutter, “Whole-body motion planning for walking excavators,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) , 2019, pp. 2292– 2299, doi: 10.1109/IROS40897.2019.8967631.

[15] D. Lee, I. Jang, J. Byun, H. Seo, and H. J. Kim, “Real-time motion planning of a hydraulic excavator using trajectory optimization and model predictive control,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) , 2021, pp. 2135– 2142, doi: 10.1109/IROS51168.2021.9635965.

[16] Y. B. Kim, J. Ha, H. Kang, P. Y. Kim, J. Park, and F. C. Park, “Dynamically optimal trajectories for earthmoving excavators,” Autom. Construction , vol. 35, pp. 568–578, Nov. 2013, doi: 10.1016/j.autcon.2013.01.007.

[18] J. Kim, C. Kim, Y. Han, and H. J. Kim, “Automated extrinsic calibration for 3D LiDARs with range offset correction using an arbitrary planar board,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA) , Piscataway, NJ, USA: IEEE, 2021., pp. 5082–5088, doi: 10.1109/ICRA48506.2021.9561175.

[19] F. A. Bender, S. Göltz, T. Bräunl, and O. Sawodny, “Modeling and offset-free model predictive control of a hydraulic mini excavator,” IEEE Trans. Autom. Sci. Eng. , vol. 14, no. 4, pp. 1682–1694, Oct. 2017, doi: 10.1109/TASE.2017. 2700407.

[20] B. Plancher, Z. Manchester, and S. Kuindersma, “Constrained unscented dynamic programming,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) , Piscataway, NJ, USA: IEEE, 2017, pp. 5674–5680, doi: 10.1109/ IROS.2017.8206457.

[21] E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,” in Proc. 9th Eur. Conf. Comput. Vis. (ECCV) , Graz, Austria. Berlin, Germany: Springer-Verlag, May 7–13, 2006, 2006, pp. 430–443, doi: 10.1007/11744023_34.

[22] J. Park, B. Lee, S. Kang, P. Y. Kim, and H. J. Kim, “Online learning control of hydraulic excavators based on echo-state networks,” IEEE Trans. Autom. Sci. Eng. , vol. 14, no. 1, pp. 249–259, Jan. 2017, doi: 10.1109/TASE.2016.2582213.

[23] S. Kim, J. Park, S. Kang, P. Y. Kim, and H. J. Kim, “A robust control approach for hydraulic excavators using μ -synthesis,” Int. J. Control, Autom. Syst. , vol. 16, no. 4, pp. 1615–1628, 2018, doi: 10.1007/s12555-017-0071-9.

[24] J. Kim, D-e Lee, and J. Seo, “Task planning strategy and path similarity analysis for an autonomous excavator,” Autom. Construction , vol. 112, Apr. 2020, Art. no. 103108, doi: 10.1016/j.autcon.2020.103108.

[17] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 22, no. 11, pp. 1330–1334, Nov. 2000, doi: 10.1109/34.888718.

[!info] 원본 참조 그림/표/수식이 필요하면 원본 파일을 확인하세요: 2025 Towards Fully Integrated Autonomous Excavation-서울대