CN114187447B - A semantic SLAM method based on instance segmentation - Google Patents

A semantic SLAM method based on instance segmentation Download PDF

Info

Publication number
CN114187447B
CN114187447B CN202111497853.7A CN202111497853A CN114187447B CN 114187447 B CN114187447 B CN 114187447B CN 202111497853 A CN202111497853 A CN 202111497853A CN 114187447 B CN114187447 B CN 114187447B
Authority
CN
China
Prior art keywords
matching
information
semantic
instance segmentation
points
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111497853.7A
Other languages
Chinese (zh)
Other versions
CN114187447A (en
Inventor
牛毅
吴腾飞
马明明
李甫
石光明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202111497853.7A priority Critical patent/CN114187447B/en
Publication of CN114187447A publication Critical patent/CN114187447A/en
Application granted granted Critical
Publication of CN114187447B publication Critical patent/CN114187447B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The application relates to the field of ORB-SLAM2 systems and deep learning instance segmentation, and particularly provides a semantic SLAM method based on instance segmentation. The method comprises the following steps: s1, acquiring an image sequence; s2, extracting feature point information and semantic information; s3, feature point information and semantic information are fused; s4, detecting and removing dynamic objects; s5, object-level interframe matching; s6, object-level loop detection. The method can accurately identify various targets in the scene, uses the categories, bounding boxes and mask information of the targets to help remove dynamic objects, and uses a left static object to better help a system to carry out interframe matching and loop detection. Object-level matching is used for restraining inter-frame matching, and the problem that characteristic point matching errors are lost in some scenes is effectively solved. According to the method, the neural network is used for carrying out instance segmentation on the scene, so that dynamic objects can be effectively identified under the single-binocular condition, and the robustness of the SLAM system in the dynamic scene is improved.

Description

Semantic SLAM method based on instance segmentation
Technical Field
The application relates to the field of ORB-SLAM2 systems and deep learning instance segmentation, in particular to a semantic SLAM method based on instance segmentation.
Background
Along with the continuous progress of modern social science and technology level, people's demand for convenience of life is also continuously promoting. With the advent of artificial intelligence, as small as home service robots, as large as RoboTaxi, it is becoming increasingly important how to help robots build surrounding environments better and more accurately, visual SLAM being one of the best choices for simultaneous localization and mapping tasks. For example, ORB-SLAM2, RGB-D SLAM-V2 and the like have been widely used, which have the advantages of fast sensor acquisition speed, low cost, short time delay, high accuracy and the like. Taking ORB-SLAM2 as an example, the system is easy to lose in a high dynamic scene, and as the underlying visual information adopts ORB feature descriptors, only the most basic feature points can be compared, and the object level information is not utilized.
With the rapid development of deep learning technology, many visual problems have better and faster solutions, and objects of interest, particularly what types and accurate boundaries in an image can be easily identified based on a neural network. The results can combine the neural network with the visual SLAM technology, and the SLAM system is better helped to sense the world by combining the semantic information.
In recent years, many researchers have proposed a number of different semantic SLAM approaches. The ORB-SLAM2 system proposed by Raul Mur-Artal is a complete SLAM scheme of RGB-D based on monocular, binocular and feature points, and adopts a word bag (DBoW 2) model to cluster the feature points, and the feature points are matched during interframe matching and loop detection. Since the ORB feature points are 01 vectors in 32 dimensions, matching based on vector distance does not fit human visual perception, e.g., the vector distance of the feature points on the vehicle and the feature points on the ground are very similar, and the feature points are not matched from a human perspective. Also for loop detection, loop detection based on DBoW2 model only considers whether the two compared frames coincide in word vector or not when loop is performed, and no requirement is made on the spatial structure of the two frames.
The MASK-SLAM proposed by Masaya Kaneko of Tokyo university is a monocular SLAM system combined with MASK-RCNN, which can effectively divide the information of the sky, vehicles and other semantic levels, remove the characteristic points belonging to dynamic objects and convert the SLAM system in a dynamic scene into the SLAM system in a static scene. However, the method is only suitable for a monocular SLAM system, all feature points of semantic dynamics are removed, different states of various objects are not considered, related information is lost, camera pose is lost, and the method is not suitable for some scenes.
DynaSLAM proposed by Berta Bescos combines MASK-RCNN with ORB-SLAM2, improves the adaptability of a SLAM system to dynamic scenes, gives priori dynamic information to objects in monocular, and excludes the feature points in tracking; and when the two images are binocular, a multi-view geometric mode is adopted, the included angle between the corresponding points of the current frame and the reference frame is calculated, and if the included angle is larger than 30 degrees, the included angle is regarded as a dynamic point to be eliminated. The influence of the points is removed during the construction of the map, and a static map is obtained by using a background restoration method. However, this method does not consider the constraint of the object on the reference point from the object level, and there are many mismatches, and thus may also result in misjudgment of the motion state of the object.
In summary, in the prior art, the problem that the feature points are erroneously matched and the robustness is poor due to the fact that the inter-frame constraint and the object level information are not considered in the dynamic scene exists.
Disclosure of Invention
The invention aims to provide a semantic SLAM method based on instance segmentation to solve the problems of wrong feature point matching and poor robustness caused by the fact that inter-frame constraint and object level information are not considered in a dynamic scene in the prior art.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
The application provides a semantic SLAM method based on instance segmentation, which comprises the following steps: s1, acquiring an image sequence; s2, extracting feature point information and semantic information; s3, feature point information and semantic information are fused; s4, detecting and removing dynamic objects; s5, object-level interframe matching; s6, object-level loop detection.
Further, the extraction of the feature point information and the semantic information in step S2 is performed simultaneously.
Still further, the extraction of the semantic information is accomplished by a network instance segmentation module.
Still further, the network instance is partitioned into object-level partitions.
Still further, network instance partitioning incorporates MASK-RCNN networks.
Further, the MASK-RCNN network is trained in step S2.
Further, step S3 classifies the feature points inside each object in the image for fine-grained feature points.
Still further, fine-grained feature point classification is accomplished through a KD-TREE data structure.
Still further, step S5 constrains inter-frame matching using object-level matching.
Further, step S5 finds the best match using KM algorithm.
Compared with the prior art, the invention has the beneficial effects that:
firstly, the invention uses the neural network to divide the instance of the scene, can effectively identify the dynamic object under the single-binocular condition, and improves the robustness of the SLAM system under the dynamic scene;
Secondly, aiming at the defect that the DBoW model only pays attention to vector level information, the invention can not distinguish the feature point matching at a higher level, provides that object level matching is used for restraining the inter-frame matching, and effectively solves the problem that the feature point matching is lost due to error in some scenes;
thirdly, the invention provides grid division of the image, and is related to semantic information, and a three-dimensional data structure of object_ KeyFrame _dataBase is established, so that the position information of an Object rather than category information can be effectively used, and the related key frames can be efficiently searched during loop detection, so that the loop detection efficiency is higher.
Fourth, the invention uses libtorch library, which can separate the training and deployment of network, and decouple the network model and SLAM system, to modify the network model conveniently, without affecting the SLAM system to use directly, with high applicability.
Drawings
FIG. 1 is a schematic diagram of a semantic SLAM method based on instance segmentation provided by the present invention;
FIG. 2 is a schematic diagram of image meshing in step S32 of a semantic SLAM method based on instance segmentation provided by the present invention;
FIG. 3 is a schematic diagram of a three-dimensional map point transformation method under a binocular system in a semantic SLAM method based on instance segmentation provided by the invention;
FIG. 4 is a schematic diagram of a related art keyframe association scheme based on a bag of words model;
FIG. 5 is a spatial structure of a three-dimensional keyframe database KeyFrame DataBase in the present invention based on an example segmentation semantic SLAM approach provided by the present invention;
FIG. 6 is a flowchart of a semantic SLAM method based on instance segmentation provided by the present invention.
Detailed Description
In order to make the implementation of the present invention more clear, the following detailed description will be given with reference to the accompanying drawings.
Example 1:
The invention provides a semantic SLAM method based on instance segmentation, as shown in figure 1, which comprises the following steps:
s1, acquiring an image sequence;
The camera captures images to form an image sequence, wherein the image sequence can be images in a plurality of static scenes, can be images in a plurality of dynamic scenes, and can be images with larger differences or images with smaller differences; the embodiment of the invention uses an RGB camera to acquire the image sequence under the dynamic scene. The image captured by the camera released by the ROS system and the time stamp information are transmitted into the SLAM system. Since the ORB feature point extraction needs to be performed on the gray scale map, in the SLAM system, the image is first converted into a gray scale image, and then the ORB feature point extraction thread and the network instance segmentation thread are simultaneously entered. The difference between the input images of the ORB feature point extraction thread and the network instance segmentation thread is that the input images of the ORB feature point extraction thread need to be converted into a gray scale, and the input images of the network instance segmentation thread may or may not be converted into a gray scale.
S2, extracting feature point information and semantic information;
The extraction of the feature point information is completed through the ORB feature point extraction module, the extraction of the semantic information is completed through the network instance segmentation module, wherein the network instance is segmented into object-level segments, namely different objects of different same types can be distinguished, and compared with semantic segments only capable of distinguishing types, the object-level segments are closer to the actual situation and have application value. The extraction of the feature point information and the semantic information is performed simultaneously, i.e., as follows, step S21 and step S22 are performed simultaneously.
S21, extracting feature point information of the image by using an ORB feature point extraction module;
The ORB feature points consist of key points and feature descriptors. Its key point is also called "ordered FAST", which is an improved form of FAST corner. The feature descriptor is also called BRIFE. The FAST corner is known in high speed, and only one pixel point needs to be judged whether the pixel value difference between the pixel point and the surrounding pixel points is large, so that a plurality of characteristic points can be rapidly extracted, and in order to avoid the problem that the characteristic points are too concentrated, non-maximum value suppression is performed later. Because the FAST corner does not have scale invariance and rotation invariance, the ORB characteristic point extraction module adopted by the invention introduces a characteristic pyramid and gray level centroid method to solve the problems, and has the scale invariance and the rotation invariance, so that the change of the position, the scale and the direction of the characteristic point in the image brought in the sports scene can be effectively dealt with, and the method is closer to the actual scene. For the feature descriptor, BRIFE is a binary descriptor whose description vector consists of 01, which encodes two random pixels near the keypoint, if the former is large, then 1 is taken, otherwise 0 is taken. Because binary expressions are used, the calculation speed and the storage are very fast.
Because of the consideration of rotation and scaling and the rapid extraction capability thereof, the ORB feature points can meet the actual demands and are widely applied to SLAM systems.
S22, extracting object-level semantic information of the image by using a network instance segmentation module.
The network instance segmentation of the image is achieved through a MASK-RCNN network. Category information, bounding box information and MASK MASK information are obtained through a MASK-RCNN network, and feature points are screened and classified through the information, so that object-level semantic information is extracted. The steps for constructing the MASK-RCNN network are as follows:
S221, building a MASK-RCNN network;
The concrete construction steps are as follows:
Step one: the FPN network with the backbone network of ResNet as the characteristic extraction part is mainly composed of four scales, and the length and width ratio between every two is doubled.
Step two: and building an RPN network, receiving a multi-scale feature map output by the FPN network, and judging whether a target exists or not by distributing anchor frames with different sizes to each pixel position and distinguishing front backgrounds. The proposed target frame is then regressed by the ROI alignment to obtain a more accurate target frame position.
Step three: and inputting the feature map in the target frame extracted by the RPN network into a target detection branch and a mask prediction branch to finish tasks of classification, positioning and mask acquisition.
S222, training the built MASK-RCNN network;
Aiming at different tasks, different data sets are adopted to train the MASK-RCNN network, so that the accuracy of the MASK-RCNN network for completing classification tasks in different tasks can be improved, and more accurate object-level semantic information can be extracted. MASK-RCNN networks are trained using PyTorch and local datasets. The local data set includes KITTI data sets and TUM data sets; the method comprises the steps that a KITTI dataset is adopted for a local dataset during MASK-RCNN network training under an outdoor scene, and a KITTI dataset is a group of datasets which are jointly established by the university of Toyota industry and Chicago division of Toyota industry and are suitable for various computer vision tasks, wherein the datasets comprise various target information such as vehicles, guideboards and the like; the local data set used in MASK-RCNN network training under indoor scene adopts TUM data set, which is a group of continuous indoor image set collected by depth camera of Munich industrial university in Germany, including various target information such as computer, desk, chair, etc.
And judging whether training of the MASK-RCNN network is finished or not through the loss function. The network of MASK-RCNN is divided into three branches, with three outputs, so the corresponding loss function consists of three parts, namely:
L＝Lcls+Lbox+Lmask
wherein L cls is a loss function of classification, and the specific form is:
Lcls(pi，pi*)＝-log(pipi*+(1-pi)(1-pi*))
pi represents the probability of being the current category, pi represents the probability of not being the current category;
wherein L box is a loss function of the bounding box, and its specific form is:
Lbox＝smoothL1(ti-ti*)
ti is the predicted bounding box position, ti is the true value;
the loss function of the L mask mask is specifically as follows:
Lmask＝-(tlog(o)+(1-t)log(1-o))
t is a true value, and o is a predicted value.
Positive samples are bounding boxes with an overlap ratio (IOU) greater than 0.6, and negative samples are bounding boxes with an IOU less than 0.6. At training, the positive and negative sample ratio is set to 1: and 3, when the loss function approaches convergence, training is completed.
S223, calling the MASK-RCNN network trained in the step S222 in the SLAM system. And using libtorch to store the trained MASK-RCNN network as a script file, and loading in the SLAM system.
S3, feature point information and semantic information are fused; the method comprises the following specific steps:
s31, traversing all the extracted characteristic points, obtaining target information of corresponding positions on corresponding MASK MASKs through two-dimensional coordinate information p (x, y) of the extracted characteristic points, adding indexes Index of the target information into data structures of corresponding objects, wherein each object corresponds to one data structure.
S32, carrying out fine-grained feature point classification on feature points in each object by using a KD-TREE data structure, namely classifying description vector layers of the feature points into fine-grained feature point classification, wherein vector information described by the feature points in the object is fine-grained, so that a lot of unnecessary judgment can be removed when the feature points are matched, and the matching process is accelerated. And meanwhile, the current image is subjected to grid division, and the current image is divided into grid images of 28 x 32 as shown in fig. 2. According to the semantic information corresponding to the MASK, semantic categories are given to each grid, a physical-level key frame database is established for loop detection, and a specific association mode is introduced in S61.
S4, detecting and removing dynamic objects;
The invention is a monocular binocular combined system. Wherein, one image is input at a time in a monocular way, and the initialized information is less; two images are input at a time in a binocular mode, and more initialized information is obtained. The invention can effectively identify the dynamic object under the single-binocular condition, and improves the robustness of the SLAM system in the dynamic scene.
S41, monocular system;
Because the scale uncertainty of the monocular system can not obtain the distance information of the feature points through single frame information, the prior dynamic object is removed by adopting the prior semantic category screening. All objects of a class are removed by semantic class. For example, if the category of the object is a person in a room, the person is removed; when the object is a car outdoors, the car is removed from the class. This can lead to recognition accuracy of SLAM systems in some scenarios, such as parking lots.
S42, binocular system.
S421, the corresponding relation between the left view characteristic points and the right view characteristic points is obtained through a line scanning mode. And counting the characteristic points of each line of the right image, and matching in a stereo matching mode. And then searching the characteristic points pi of the ith row of the left image on the ith row of the right image to obtain the best matched points qi, wherein i represents the row number of any row, and p and q represent the characteristic points on the left side and the right side of the image respectively. And further searching with qi as a center and r as a radius, wherein the searching radius is 10pixel, and performing sub-pixel interpolation optimization on the result, so that more accurate matching points can be obtained.
S422, performing triangularization on the feature points through the corresponding feature points, obtaining the feature points corresponding to the left and right views after triangularization, and obtaining the distance by converging two rays from the optical center to the feature points in space to obtain the distance information of the feature points. By triangulation, two-dimensional points on the left and right views are converted into points in the established three-dimensional space, which are called three-dimensional map points.
S423, a frame closest to the current frame before the SLAM system operates is a reference frame, the reference frame is used as reference information, and transformation is calculated with the previous reference frame every time a new frame is input. And calculating a transformation matrix of the current frame and the reference frame through the background characteristic points, carrying out corresponding transformation on the three-dimensional map points corresponding to the reference frame, judging whether the error between the three-dimensional map points corresponding to the current frame is smaller than a threshold value, and considering the current object as a real moving object when the error of the three-dimensional map points exceeding 90% is larger than the threshold value. Specifically, objects belonging to absolute rest are found out through semantic information, initial pose calculation is carried out by using three-dimensional map points corresponding to the objects to obtain an initial pose transformation matrix Tinit, and a transformation mode from a reference frame to a current frame is estimated in a preliminary mode. Changing a three-dimensional map point corresponding to a current frame and a reference frame through Tinit, recording Plast as a characteristic point corresponding to the reference frame, pcurr as a characteristic point corresponding to the current frame, and if:
|Plast-Pcurr|>th
as shown in fig. 3, C1 and C2 respectively represent left and right cameras, dark color is the original camera position, light color is the current position, the origin represents a map point in space, when the camera moves, if the change relation of the map point accords with the transformation matrix of the camera, the map point is considered to be a static point, if the error is greater than a threshold value, the feature point is considered to be a dynamic point, th is a threshold value, and the map point is determined by prior information. If 90% or more of the feature points of an object are dynamic points, the object is considered to belong to the dynamic object, and all feature points corresponding to the dynamic object need to be removed.
S5, object-level interframe matching;
S51, solving can be performed due to the fact that the same transformation matrix is matched between static objects, and solving cannot be performed due to the fact that the same transformation matrix is not matched between dynamic objects. After the dynamic object removal of the current frame, all that remains is a static object that can be used for matching. And then adopting a KM algorithm, taking the IOU as weight information of whether the two inter-frame objects are the same object, and searching for optimal matching, so that the total weight is the largest, and the matching accuracy is higher. According to the method, the inter-frame constraint in the dynamic scene is considered, the feature point matching accuracy is high, and the robustness is improved.
Firstly, calculating weight information W of matching of two objects between two frames, namely calculating an intersection ratio value of surrounding frames of the two objects, and adding a certain offset to ensure that the matching has better translation invariance, so that the matching accuracy is higher, the matching is not easy to be mistakenly matched, and the robustness is better; then, a group of optimal matches is obtained through a KM algorithm, namely, V1 is Index information of all objects in a reference frame, V2 is Index information of all objects in a current frame, and all sides < i, j > ∈G, wherein W (i, j) represents a matching weight from an ith vertex in V1 to a jth vertex in V2. And storing graph information by using an adjacency matrix, initializing a top label by using a greedy algorithm, finding complete matching by using a Hungary algorithm, if the complete matching cannot be found, modifying the top label, adding sides, finding again, and ending when the complete matching is achieved, namely that a one-to-one correspondence is found, wherein all the top labels in a left subset in the KM algorithm have a correspondence. Thus, the matching information of objects between two frames is obtained.
S52, matching the feature point fine granularity by using a KD-TREE data structure in the matched corresponding object; and traversing word vectors of the current frame and the reference frame, and matching the feature points when the word IDs are the same. And searching the corresponding characteristic points of the characteristic points in the current word of the current frame one by one, wherein the minimum distance Dist 1 < th and Dist 1<0.8*Dist2 are required to be met during searching.
And S53, after the matching is successful, optimizing the pose information of the current frame through the three-dimensional map points corresponding to the corresponding feature points of the previous frame. And calculating a speed vector in the motion model as an initial value, and optimizing the pose of the current frame by fixing three-dimensional map points and using a graph optimization mode.
And S54, after pose optimization is successful, the three-dimensional map points associated with adjacent frames of the current frame are projected, the vector distance of the feature points corresponding to the current frame is calculated, if the distance is smaller than th, a new group of matching points are considered to be found, if the final total matching points are larger than 30, tracking is considered to be successful, otherwise, the tracking is considered to be lost, and repositioning is carried out.
S6, object-level loop detection.
S61, the key frame means that every time a frame is successfully established, and tracking is successful, the key frame has enough new information. Every time a frame of key frames is obtained, a loop detection thread needs to be entered to add current key frame information to the key frame database KeyFrame DataBase. The existing keyframe association method based on the bag-of-words model is shown in fig. 4, wherein ABCD is in four different categories, and the right side is the current keyframe, which contains four categories. The corresponding keyframe database KeyFrame DataBase also contains four categories, each category being associated with a respective number of keyframes, by which a similar keyframe can be found for loop matching. However, this correlation method does not take the position information into consideration, and there is a certain degree of mismatching. The improved method is shown in fig. 5, wherein the dark color, the gray color and the light color respectively represent 3 categories, the white color represents the background, the left side represents the information observed by the current key frame, and different colors represent different categories through meshing. The three-dimensional data structure on the right side is a key frame database KeyFrame DataBase, the length and width of the key frame database are consistent with the grid division mode of the image, the depth is the number of all categories, the background is not counted, for example, 3 categories exist, the data structure with the depth of 3 is established, therefore, different categories can be associated to different key frames at different positions, and the semantic information of all the key frames at corresponding positions is saved through the three-dimensional data structure.
S62, counting the maximum similar quantity of all related key frames, scaling the key frames by a certain coefficient, setting the scaled key frames as a threshold value, and enabling the key frames larger than the threshold value to enter the next screening step. Specifically, the number of grids matched with the key frames, commonGrids, of which the same category as the key frames of the current frame are in the corresponding position is counted. After all the key frames meeting the requirements are counted to form the associated frames, the maximum grid number (MaxCommonGrids) is counted, and 0.8 x MaxCommonGrids is taken as a threshold value, if the maximum grid number is smaller than the threshold value, the next step is not carried out, namely the key frames which do not meet the requirements are removed, so that the key frames which are closest to the current key frame in position and category can be selected, and compared with the situation that the key frames are closest to the current key frame in position and category through word vector consideration, the invention considers the position information, and has higher screening efficiency.
S63, performing non-maximum suppression, and selecting a frame with the best matching effect for the key frames with the association relation. Specifically, the common view information of all the current candidate frames is calculated, and the key frames with common view relationship are considered as a group, and only the key frame with the maximum CommonGrids is reserved in the group of key frames.
S64, carrying out continuity detection, carrying out loop detection on three continuous frames of current frames, and if the obtained key frames also have a common view relationship, considering that the current loop matching is successful, and carrying out calculation of a Sim3 matrix. That is, loop matching at the image level is successful, whether the transformation matrix between two key frames is consistent with the associated three-dimensional map points is calculated, and if so, map correction is performed.
Figure 6 shows an overall flow chart of the method of the invention.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (3)

1.一种基于实例分割的语义SLAM方法，其特征在于，所述方法包括以下步骤：S1，获取图像序列；S2，提取特征点信息和语义信息；S3，融合特征点信息和语义信息；S4，动态物体检测与去除；S5，物体级的帧间匹配；S6，物体级的回环检测；1. A semantic SLAM method based on instance segmentation, characterized in that the method comprises the following steps: S1, acquiring an image sequence; S2, extracting feature point information and semantic information; S3, fusing feature point information and semantic information; S4, dynamic object detection and removal; S5, object-level frame matching; S6, object-level loop detection; 所述步骤S2中所述特征点信息和所述语义信息的提取是同时进行的，所述语义信息的提取通过网络实例分割模块完成，所述网络实例分割为物体级的分割，所述网络实例分割结合了MASK-RCNN网络，所述步骤S2中对所述MASK-RCNN网络进行训练；The extraction of the feature point information and the semantic information in step S2 is performed simultaneously, the extraction of the semantic information is completed by a network instance segmentation module, the network instance segmentation is an object-level segmentation, the network instance segmentation is combined with a MASK-RCNN network, and the MASK-RCNN network is trained in step S2; 所述步骤S5使用物体级匹配约束帧间匹配，所述步骤S5采用KM算法寻找最优匹配；所述步骤S5包括如下步骤：The step S5 uses object-level matching to constrain inter-frame matching, and the step S5 uses the KM algorithm to find the optimal match; the step S5 includes the following steps: S51，首先，计算两帧之间两两物体匹配的权重信息W，即计算两物体包围框的交并比数值，并加上偏移量，然后，令V1为参考帧中所有物体的Index信息，V2为当前帧中所有物体的Index信息，所有的边<i,j>∈G,其中W(i,j)表示V1中的第i个顶点到V2中第j个顶点的匹配权重；用邻接矩阵存储图信息，运用贪心算法初始化顶标，使用匈牙利算法找到完备匹配，若找不到则通过修改顶标，增加边再次寻找，达到完备匹配时则结束，即一一对应关系已找到，所述KM算法中的左子集中的所有顶标都有了对应关系；S51, first, calculate the weight information W of the object matching between the two frames, that is, calculate the intersection and union ratio value of the bounding boxes of the two objects, and add the offset, then let V1 be the index information of all objects in the reference frame, V2 be the index information of all objects in the current frame, and all edges <i,j>∈G, where W(i,j) represents the matching weight from the i-th vertex in V1 to the j-th vertex in V2; use the adjacency matrix to store the graph information, use the greedy algorithm to initialize the top label, use the Hungarian algorithm to find a complete match, if not found, modify the top label, add edges and search again, and end when a complete match is reached, that is, a one-to-one correspondence has been found, and all the top labels in the left subset of the KM algorithm have a corresponding relationship; S52，在匹配好的对应物体内使用KD-TREE数据结构进行特征点细粒度的匹配；同时遍历当前帧与参考帧的词向量，当单词ID相同时，则对特征点进行匹配；S52, using the KD-TREE data structure to perform fine-grained matching of feature points in the matched corresponding objects; at the same time, traversing the word vectors of the current frame and the reference frame, when the word IDs are the same, matching the feature points; S53，匹配成功后，通过上一帧对应特征点对应的三维地图点优化当前帧的位姿信息；S53, after the matching is successful, optimizing the pose information of the current frame through the three-dimensional map points corresponding to the corresponding feature points of the previous frame; S54，对位姿优化成功后，通过将在当前帧的相邻帧关联的三维地图点进行投影，并计算与当前帧对应处特征点的向量距离，若距离小于th，则认为找到了新的一组匹配点，若最终总的匹配点大于30，则认为追踪成功，否则认为丢失，进行重定位；S54, after the pose optimization is successful, the three-dimensional map points associated with the adjacent frames of the current frame are projected, and the vector distance with the feature points corresponding to the current frame is calculated. If the distance is less than th, it is considered that a new set of matching points is found. If the final total number of matching points is greater than 30, the tracking is considered successful, otherwise it is considered lost and repositioning is performed; 其中，所述th为阈值，所述阈值的表达式为|Plast-Pcurr| > th，其中，所述Plast为参考帧对应的特征点，所述Pcurr为当前帧对应的特征点。Wherein, th is a threshold value, and the expression of the threshold value is |Plast-Pcurr| > th, wherein Plast is a feature point corresponding to a reference frame, and Pcurr is a feature point corresponding to a current frame. 2.根据权利要求1所述的基于实例分割的语义SLAM方法，其特征在于，所述步骤S3对图像中每个物体内部的特征点进行细粒度特征点分类。2. The semantic SLAM method based on instance segmentation according to claim 1, wherein step S3 classifies the feature points inside each object in the image into fine-grained feature points. 3.根据权利要求2所述的基于实例分割的语义SLAM方法，其特征在于，所述细粒度特征点分类通过KD-TREE数据结构完成。3. the semantic SLAM method based on instance segmentation according to claim 2, is characterized in that the classification of the fine-grained feature points is completed by KD-TREE data structure.
CN202111497853.7A 2021-12-09 2021-12-09 A semantic SLAM method based on instance segmentation Active CN114187447B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111497853.7A CN114187447B (en) 2021-12-09 2021-12-09 A semantic SLAM method based on instance segmentation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111497853.7A CN114187447B (en) 2021-12-09 2021-12-09 A semantic SLAM method based on instance segmentation

Publications (2)

Publication Number Publication Date
CN114187447A CN114187447A (en) 2022-03-15
CN114187447B true CN114187447B (en) 2024-11-19

Family

ID=80603995

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111497853.7A Active CN114187447B (en) 2021-12-09 2021-12-09 A semantic SLAM method based on instance segmentation

Country Status (1)

Country Link
CN (1) CN114187447B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116758148A (en) * 2023-05-08 2023-09-15 苏州科技大学 SLAM method and system in dynamic environment
CN116883586B (en) * 2023-06-14 2024-08-23 重庆大学 Terrain semantic map construction method, system and product based on binocular camera
CN116592897B (en) * 2023-07-17 2023-09-22 河海大学 Improved ORB-SLAM2 positioning method based on pose uncertainty

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110738673A (en) * 2019-10-21 2020-01-31 哈尔滨理工大学 Visual SLAM method based on example segmentation

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109753940B (en) * 2019-01-11 2022-02-22 京东方科技集团股份有限公司 Image processing method and device
CN112381841A (en) * 2020-11-27 2021-02-19 广东电网有限责任公司肇庆供电局 Semantic SLAM method based on GMS feature matching in dynamic scene
CN112465858A (en) * 2020-12-10 2021-03-09 武汉工程大学 Semantic vision SLAM method based on probability grid filtering

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110738673A (en) * 2019-10-21 2020-01-31 哈尔滨理工大学 Visual SLAM method based on example segmentation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于深度学习的动态场景语义 SLAM;房立金等;《华 中 科 技 大 学 学 报（自 然 科 学 版）》;20200131;第48卷(第1期);第121-125页 *

Also Published As

Publication number Publication date
CN114187447A (en) 2022-03-15

Similar Documents

Publication Publication Date Title
CN110956651B (en) Terrain semantic perception method based on fusion of vision and vibrotactile sense
CN111126304B (en) Augmented reality navigation method based on indoor natural scene image deep learning
Zhang et al. Semantic segmentation of urban scenes using dense depth maps
Xiao et al. Multiple view semantic segmentation for street view images
Zhu et al. A review of 6d object pose estimation
CN111563442A (en) Slam method and system for fusing point cloud and camera image data based on laser radar
GB2612029A (en) Lifted semantic graph embedding for omnidirectional place recognition
Zhao et al. Light-SLAM: A robust deep-learning visual SLAM system based on LightGlue under challenging lighting conditions
Košecka Detecting changes in images of street scenes
CN111767854B (en) SLAM loop detection method combined with scene text semantic information
CN114187447A (en) Semantic SLAM method based on instance segmentation
CN116485894B (en) Video scene mapping and localization methods, devices, electronic equipment and storage media
CN111274964B (en) Detection method for analyzing water surface pollutants based on visual saliency of unmanned aerial vehicle
CN108151728A (en) A kind of half dense cognitive map creation method for binocular SLAM
Delmerico et al. Building facade detection, segmentation, and parameter estimation for mobile robot localization and guidance
CN118887353A (en) A SLAM mapping method integrating points, lines and visual labels
CN116311223A (en) Method and system for target 3D pose estimation based on RGBD
Debaque et al. Thermal and visible image registration using deep homography
CN120220044A (en) A method for re-identification of occluded pedestrians based on posture guidance and feature fusion
CN117576363A (en) A positioning technology method based on monocular vision/inertia in complex indoor environments
CN116912404A (en) LiDAR point cloud mapping method for scanning distribution lines in dynamic environments
CN121095480A (en) Robust visual SLAM method oriented to complex dynamic environment
Zhang et al. Joint segmentation of images and scanned point cloud in large-scale street scenes with low-annotation cost
CN121033166A (en) A robust visual SLAM method for dynamic scenes based on multi-feature collaborative optimization
CN120765870A (en) 3D reconstruction method and system for indoor dynamic environment based on deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant