TWI882582B - Camera calibration method based on vehicle positioning achievement - Google Patents

Abstract

This present invention provides a dynamic camera calibration method, the main consideration is that the vehicle is a non-ideal rigid body structure, the vibration of the vehicle during operation, uneven roads or different loads will cause changes in the extrinsics of the camera system, resulting in errors in perceived information. Therefore, the present invent integrates the sensory information such as images, inertial measurement units and speedometers, combines road and marking information provided by high-precision maps, and uses a mass spring damping model to describe the relative motion of the vehicle body and the vehicle chassis, to complete the six-degree-of-freedom position of the vehicle body and the vehicle chassis, and finally the real-time camera external parameter correction is performed by using the vehicle chassis as the reference coordinate.

Description

A camera calibration method based on vehicle positioning

The present invention relates to a camera calibration method based on vehicle positioning, and in particular to a dynamic camera calibration method using high-precision image data and positioning, which is applicable to the design of an autonomous driving system or an advanced driver assistance system (ADAS) to achieve the function of combining image positioning and dynamic camera calibration.

The operation mode of a self-driving car (automatic driving vehicle) is mainly to configure various sensors at different locations on the self-driving car according to different application requirements, so that these sensors can detect various driving information when the self-driving car is driving, so as to provide reference for the self-driving system to plan control commands, and then control the stable driving of the self-driving car.

However, self-driving cars are actually non-rigid structures composed of many parts. This means that when a self-driving car moves, its components will move relative to each other, and the relative displacement of different parts can cause different sensors on it to shake. For example, depending on the type of vehicle, the front of the articulated vehicle will shake significantly during driving; logistics vehicles will have different body heights due to different loads. In addition, uneven roads can also cause the body to shake, resulting in sensor sensing errors.

These shaking and uneven road surfaces cause the sensors to deviate from the position parameters set by the autonomous driving system, resulting in errors in the autonomous vehicle's detection of the motion state of external objects, such as the relative distance and speed between the external object and the autonomous vehicle. Errors in the estimation of the position of external objects of the autonomous vehicle can easily cause the autonomous driving system to be unable to calculate the most appropriate control command. The state of the object may vary too much, causing the autonomous vehicle to have large speed and acceleration changes, or even collisions between the autonomous vehicle and other objects.

For example, the existing autonomous driving system or ADAS usually uses a camera mounted on the vehicle body to obtain the position of external objects, and uses the image-based object detection and positioning method as the position parameter set by the autonomous driving system. However, because the vehicle body and the vehicle chassis are actually connected by a suspension system rather than a steel structure, the six-degree-of-freedom position external parameters (Extrinsics) of the camera mounted on the vehicle body relative to the vehicle chassis coordinate reference point are easily affected by the shaking of the vehicle body, different loads, and road tilt, causing the rotation angle and displacement of the camera relative to the vehicle chassis to change, resulting in errors in the distance between the image detection object and the vehicle, thereby causing the autonomous driving system or ADAS to malfunction.

In view of the lack of prior art, the inventor of this case has started to conduct research and development and improvement, and successfully completed a dynamic camera calibration method of the present invention, which is expected to effectively solve the problem of perception and positioning errors caused by changes in the rotation angle and displacement of the camera relative to the vehicle chassis due to vehicle shaking, uneven road surface or different loads.

The purpose of the present invention is to provide a camera calibration method based on vehicle positioning, which combines vehicle chassis positioning based on multiple sensor fusion and vehicle body positioning based on images, and performs dynamic model iteration of the vehicle body and vehicle chassis to complete the six-degree-of-freedom position positioning of the vehicle body and vehicle chassis, and uses the vehicle chassis as a reference coordinate to perform external parameter calibration of the camera, thereby improving the reliability and safety of the autonomous driving system or ADAS.

For image-based object detection and positioning methods, they all rely on the camera external parameters of the surround imaging system relative to the vehicle chassis, converting the detected external object or feature position from the sensor to the coordinate system required for positioning and control decisions, so as to be applied to the subsequent positioning and control decisions of the autonomous driving system. However, because the vehicle body and the vehicle chassis are connected by a suspension system rather than a steel structure, the external parameters of the camera mounted on the vehicle body relative to the vehicle chassis coordinate reference point are easily affected by the shaking of the vehicle body, different loads, and tilted and bumpy roads, resulting in errors in positioning and the distance between the detected object and the vehicle, which in turn leads to positioning failure or wrong decisions to avoid obstacles. Therefore, the six-axis information of the positioning vehicle body and vehicle floor, as well as the dynamic correction of camera external parameters, are crucial for autonomous driving systems and ADAS.

The present invention solves the problem by: A. Vehicle chassis positioning using multiple sensor fusions - using a wheel speedometer to sense the speed information of the vehicle chassis, and using an inertial measurement unit to sense the speed and acceleration of the three-dimensional displacement of the vehicle chassis and the angular velocity and angular acceleration of the three-dimensional rotation angle of the vehicle chassis, estimating the initial six-degree-of-freedom position of the vehicle chassis at present; B. Image-based vehicle body positioning - using a surround image system to obtain a surround image near the vehicle body, detecting the image object feature point set in the image and the surround image system. The vanishing points of all cameras are unified and matched with the semantic map feature point set near the vehicle to calculate the six-degree-of-freedom position of all cameras; C. Iterative calculation of the relative motion model of the vehicle body and the vehicle chassis---the three-dimensional relative displacement of the vehicle body and the vehicle chassis is estimated using the mass spring damping model, and the six-degree-of-freedom position of the vehicle chassis is iteratively fine-tuned with reference to the road surface normal vector and the rotation angle of the vehicle chassis to complete the positioning of the six-degree-of-freedom position of the vehicle body and the vehicle chassis. Finally, the external parameters of all cameras are calibrated with the vehicle chassis as the reference coordinate.

The present invention has at least the following features: a mass spring damping model is introduced into the autonomous driving system or ADAS to describe the relative displacement between the vehicle body and the vehicle chassis, and the vehicle chassis is used as the reference coordinate to perform real-time camera external parameter correction to solve the problem of perception and positioning errors caused by changes in the rotation angle and displacement of the camera relative to the vehicle chassis due to vehicle shaking, uneven road surface or different loads, thereby achieving the function of combining image positioning and dynamic camera correction.

A~A02, B~B06, C~C04: Steps

100: Vehicles

110: Car body

120: Vehicle chassis

210: Wheel speed meter

220: Inertia measurement unit

230:Mass spring damping model

300: Surround view imaging system

310: Camera

400: High-precision map information

FIG1 is a flowchart of the dynamic camera calibration method of the present invention; FIG2 is a front view of the structure of the vehicle sensor configuration of the present invention; and FIG3 is a top view of the structure of the vehicle sensor configuration of the present invention.

The following is a detailed description of the preferred embodiments of the present invention in conjunction with the attached figures, so that the advantages and features of the present invention can be more easily understood by technical personnel in this field, thereby making a clearer and more precise definition of the protection scope of the present invention.

Please refer to FIG. 1 for a camera calibration method based on vehicle positioning provided by the present invention. The system architecture of the dynamic camera calibration method is shown in FIG. 2 and FIG. 3, which includes a vehicle 100. The vehicle 100 has an upper body 110 and a lower vehicle chassis 120. The vehicle chassis 120 is provided with a wheel speed meter 210 and an inertia measurement unit 220. The vehicle body 110 is provided with a surround imaging system 300. The imaging system 300 includes a plurality of cameras 310, which are dispersed around the vehicle body 110 to obtain surrounding images. A mass spring damping model 230 is connected between the vehicle body 110 and the vehicle chassis 120. The mass spring damping model 230 uses the vehicle chassis 120 as a reference coordinate to calculate the elongation change of the mass spring damping to estimate the three-dimensional relative displacement of the vehicle body 110 and the vehicle chassis 120.

With the above structure, the dynamic camera 310 calibration method at least includes: step A, positioning the vehicle chassis 120 by fusion of multiple sensors (such as wheel speed meter 210, inertia measurement unit 220), estimating the initial six-degree-of-freedom position, velocity and acceleration of the vehicle chassis 120; step B, positioning the vehicle body 110 based on the image, applying semantic map (Semantic Map) Map) feature point set to match the detection features of the surround imaging system 300 to calculate the six-degree-of-freedom position positioning of all the cameras 310; step C, using the iterative calculation of the relative motion model of the vehicle body 110 and the vehicle chassis 120 to complete the six-degree-of-freedom position positioning of the vehicle body 110 and the vehicle chassis 120, and finally using the vehicle chassis 120 as a reference coordinate to perform external parameter calibration of all the cameras 310.

Based on the above foundation, the detailed working process of the present invention is as follows: Regarding the positioning of the vehicle chassis 120 fused with multiple sensors (such as the wheel speed meter 210 and the inertia measurement unit 220) (step A), the wheel speed meter 210 is used to sense the speed information of the vehicle chassis 120, and the inertia measurement unit 220 is used to sense the speed and acceleration of the three-dimensional displacement of the vehicle chassis 120 and the angular velocity and angular acceleration of the three-dimensional rotation angle of the vehicle chassis 120, and the current vehicle position of the vehicle 100 is estimated. The initial six-degree-of-freedom position of the chassis 120 (step A01); and for the image-based positioning of the vehicle body 110 (step B), the surrounding image system 300 is used to obtain the surrounding image of the vehicle body 110 near the vehicle body 110, such as road signs and various markings such as pedestrian crossings (step B01), and the road surface features in the image are detected as a set of image object feature points (step B02), and the vanishing points of all cameras 310 of the surrounding image system 300 are detected (step B03) , calculate the initial estimated values of the rotation angles of all cameras 310 (step B04), and match them with the semantic map feature point set near the vehicle 100 obtained from the high-precision map information 400 to calculate the six-degree-of-freedom positions of all the cameras 310 (step B05); then perform iterative calculations on the relative motion model of the vehicle body 110 and the vehicle chassis 120 (step C), use the mass spring damping model 230 to estimate the three-dimensional relative displacement of the vehicle body 110 and the vehicle chassis 120, and refer to Considering the road surface normal vector and the rotation angle of the vehicle chassis 120, the six-degree-of-freedom position of the vehicle chassis 120 is iteratively fine-tuned (step C01), the six-degree-of-freedom position of the vehicle body 110 and the surround imaging system 300 thereon is positioned (step C02), and the six-degree-of-freedom position of the vehicle chassis 120 is positioned (step C03), and finally, the vehicle chassis 120 is used as a reference coordinate to perform external parameter calibration of all the cameras 310 of the surround imaging system 300 (step C04).

In the above method, the estimation of the initial six-degree-of-freedom position of the current vehicle chassis 120 can also be performed by using the speed and acceleration information of the vehicle 100 calculated by the wheel speed meter 210 and the inertia measurement unit 220, and taking the position of the vehicle chassis 120 at the previous sampling time point as a reference point (step A03), and calculating the position of the vehicle chassis 120 at the current sampling time point. Similarly, for the six-degree-of-freedom position estimation of all the cameras 310, when the image object feature point set and the semantic map feature point set are matched, the position of the vehicle body 110 at the previous sampling time point is taken as a reference point (step B06), and the six-degree-of-freedom positions of all the cameras 310 at the current sampling time point are calculated.

Furthermore, the road surface normal vector is generated by querying the high-precision map (HD Map) information 400 based on the location of the vehicle chassis 120.

Furthermore, the semantic map feature set near the vehicle 100 is generated by querying the high-precision map information 400 based on the position of the vehicle chassis 120, including a set of road marking edges and traffic sign edges.

Furthermore, the vanishing points and the image object feature point set of all the cameras 310 of the surround image system 300 are generated by calculating the multiple image features in the surround image.

Furthermore, the six-degree-of-freedom external parameters of all the cameras 310 of the surround imaging system 300 are calculated based on the vanishing points of more than four cameras 310 in the surround imaging system 300, and the pitch and yaw angles of each camera 310 are calculated; the four cameras 310 are arranged on the vehicle body 110, and the cameras 310 and the vehicle body 110 are rigidly connected. The pitch and yaw angles of the four cameras 310 are used to determine the initial value of the three-dimensional rotation angle of the vehicle body 110; the six-degree-of-freedom position of the camera 310 and the six-degree-of-freedom position of the vehicle body 110 are calculated by matching the semantic map feature point set near the vehicle 100 and the image object feature point set detected by the camera 310.

In addition, the six-degree-of-freedom position of the vehicle body 110 and the six-degree-of-freedom position of the vehicle chassis 120 are based on the six-degree-of-freedom position initial value of the vehicle body 110, the six-degree-of-freedom position initial value of the vehicle chassis 120, the initial value of the vehicle chassis 120 speed and acceleration, and the road surface normal vector obtained by querying the high-precision map information 400 based on the position of the vehicle chassis 120, and using the mass spring damping model to describe the vehicle body 110 and the vehicle chassis The relative motion of the chassis 120 is iterated, and the dynamic values of the mass spring damping model 230, including position, velocity and acceleration, are continuously corrected until the external force applied by the vehicle body 110 and the vehicle chassis 120 approximates the dynamic change of the mass spring damping model, and the six-degree-of-freedom position of the vehicle body 110 and the six-degree-of-freedom position of the vehicle chassis 120 are generated, and the six-degree-of-freedom external parameters of the camera 310 of the surround imaging system 300 are calculated.

The above implementation methods are only for illustrating the technical concept and features of the present invention. Their purpose is to allow people familiar with this technology to understand the content of the present invention and implement it. They cannot be used to limit the scope of protection of the present invention. Any equivalent changes or modifications made according to the spirit and essence of the present invention should be included in the scope of protection of the present invention.

A~A02, B~B06, C~C04: Steps

210: Wheel speed meter

220: Inertia measurement unit

230:Mass spring damping model

400: High-precision map information

Claims (7)

A camera calibration method based on vehicle positioning includes: Obtaining the velocity and acceleration of the three-dimensional displacement of a vehicle chassis and the angular velocity and angular acceleration of the three-dimensional rotation angle of the vehicle chassis to estimate an initial six-degree-of-freedom position of the vehicle chassis at present; A surround image system is installed on the vehicle, the surround image system includes a plurality of cameras, and a surround image near the vehicle body is obtained by using the surround image system, a set of image object feature points in the surround image and a vanishing point of all the cameras of the surround image system are detected, and matched with a set of semantic map feature points near the vehicle to calculate the six-degree-of-freedom positions of all the cameras; a mass spring damping model is connected and installed between the vehicle body and the vehicle chassis, and the mass spring damping model uses the surround image system as a reference point to estimate the three-dimensional relative displacement of the vehicle body and the vehicle chassis; According to the road surface normal vector and the rotation angle of the vehicle chassis, the six-degree-of-freedom position of the vehicle chassis is iterated to complete the positioning of the six-degree-of-freedom position of the vehicle body and the vehicle chassis; and Taking the vehicle chassis as a reference coordinate, the six-degree-of-freedom position of the camera is calibrated. A camera calibration method based on vehicle positioning as described in claim 1, wherein the initial six-degree-of-freedom position of the current vehicle chassis is estimated by calculating the vehicle chassis position at the current sampling time point using the vehicle speed and acceleration information calculated by a wheel speed meter and an inertia measurement unit, with the position of the vehicle chassis at the previous sampling time point as a reference point. A camera calibration method based on vehicle positioning as described in claim 1, wherein the road surface normal vector is generated by querying high-precision map information based on the position of the vehicle chassis. A camera calibration method based on vehicle positioning as described in claim 1, wherein the semantic map feature set near the vehicle is generated by querying high-precision map information based on the position of the vehicle chassis, including a set of road marking edges and traffic sign edges. A camera calibration method based on vehicle positioning as described in claim 1, wherein the vanishing point and the image object feature point set of all the cameras in the surround image system are generated by calculating based on multiple image features in the surround image. A camera calibration method based on vehicle positioning as described in claim 1, wherein the six-degree-of-freedom position of all the cameras in the surround imaging system is calculated based on the vanishing points of more than a plurality of the cameras in the surround imaging system, and the pitch and yaw angles of each camera are calculated; the plurality of cameras are arranged on the vehicle body, and the plurality of cameras are rigidly connected to the vehicle body, and the pitch and yaw angles of the plurality of cameras are used to determine the initial value of the three-dimensional rotation angle of the vehicle body; the six-degree-of-freedom position of the camera and the six-degree-of-freedom position of the vehicle body are calculated by matching the set of semantic object feature points near the vehicle and the set of image object feature points detected by the camera. A camera calibration method based on vehicle positioning as described in claim 1, wherein the six-degree-of-freedom position of the vehicle body and the six-degree-of-freedom position of the vehicle chassis are based on the initial value of the six-degree-of-freedom position of the vehicle body, the initial value of the six-degree-of-freedom position of the vehicle chassis, the initial value of the vehicle chassis speed and acceleration, and the road surface normal vector obtained by querying the high-precision map based on the vehicle chassis position, using a mass spring The damping model describes the relative motion of the vehicle body and the vehicle chassis iteratively, and continuously corrects the dynamic values of the mass-spring-damping model, including position, velocity, and acceleration, until the external force applied by the vehicle body and the vehicle chassis approximates the dynamic change of the mass-spring-damping model, generates the six-degree-of-freedom position of the vehicle body and the six-degree-of-freedom position of the vehicle chassis, and calculates the six-degree-of-freedom position of the camera of the surround imaging system.