2020 International Conference on Unmanned Aircraft Systems (ICUAS)

Athens, Greece. September 1-4, 2020

Image-Based Visual Servo Control for Ground Target Tracking Using a

Fixed-Wing UAV with Pan-Tilt Camera
Lingjie Yang1 , Zhihong Liu∗1 , Guanzheng Wang1 , Xiangke Wang1

Abstract— This paper proposes a control framework to it only allows an optimal trajectory in 3D Cartesian space
achieve the tracking of the moving target by a fixed-wing to be followed theoretically, instead of in the image space.
unmanned aerial vehicle (UAV) with a monocular pan-tilt As a result, even small errors in the image measurements
camera. This control framework is based on the image-based
visual servoing (IBVS) method, which takes the target feature can lead to errors in the pose [9], which may affect the
point on the captured image as the input and outputs the control tracking performance. Compared to PBVS, IBVS directly
signals directly with the aid of image Jacobian matrix. However, defines the difference between the current and desired image
the image is affected by the attitude of both the UAV and the characteristics as the control error, thereby eliminating the
pan-tilt, and the attitude of the pan-tilt is coupled with that process of 3D modeling, and insensitive to the calibration
of the UAV simultaneously. To solve this problem, we present
an “Ideal State” as the reference state, and make sure the errors of the sensors, cameras and robots. Therefore, this
coordinates of the feature point in the state are only affected kind of methods can better ensure that the target is in the
by the change of the yaw angle of the UAV. In this way, we can camera field of view [14].
integrate the attitude control of the UAV and the pan-tilt. By Recently, applying the IBVS technique to tackle the vision
using this control framework, the fixed-wing UAV can track the based control problem in UAVs is more and more common.
ground target continuously on the one hand, and the target will
tend to locate at image center on the other hand. This prevents Serra et al. [15], [16] use a quadrotor to track a moving plat-
the target from moving toward to the edge of the image or even form and then land on it with the aid of IBVS. Srivastava et
disappearing. Besides, we prove the controller is exponentially al. [17] control the quadrotor to keep following a target drone
convergent by the Lyapunov method. In order to evaluate the while maintaining a fixed distance from it in the absence of
performance of our controller, we build a hardware-in-the-loop GPS. Lyu et al. [18] propose a framework to realize vision-
(HIL) simulation platform and a prototype platform. Based
on these platforms, extensive experiments including simulations based multi-UAV cooperative mapping and control based on
and real flight tests are conducted. The results show that our IBVS. Although a number of approaches based on IBVS for
controller can achieve continuous and robust tracking of the UAVs emerge, most of the existing work is proposed for
target with a speed of 20km/h when the speed of the UAV is rotor UAVs [19], [20], [21], [22]. Compared to rotor UAVs,
16m/s. the fixed-wing UAVs have much more dynamical constraints,
I. INTRODUCTION such as it cannot move omni-direction on the one hand, and
its minimum speed is limited by the stalling speed on the
The past decade has witnessed a rapid development of other hand. Therefore, applying IBVS in target tracking for
UAVs. They can execute dull, dirty or dangerous tasks fixed-wing UAVs is more challenging.
instead of humans and have been widely used in both the To this end, Florent et al. [23] design a controller for the
civilian and military fields. Due to the excellent maneuver- fixed-wing UAV with a fixed camera to track a ground target.
ability of the UAVs, applying them in ground target tracking Their work enforces the trajectory of the UAV to converge to
has tremendous benefits [1], [2], [3], [4]. Considering that the a cone and a plane at the same time, and only the tracking of
fixed-wing UAVs have longer navigation time, larger payload stationary target is studied. Pietro et al. [24] achieve the target
and faster flight speed than rotor UAVs, the fixed-wing UAVs tracking for the fixed-wing UAV equipped with a pan-tilt
are more suitable for the long endurance tracking tasks. camera. They require the pan-tilt to tend to be perpendicular
Visual servo control is a commonly used tracking method to the body, which causes the yaw capability of the pan-
[5], [6], [7]. There are mainly two categories [8], [9], [10]: tilt cannot be fully utilized. As a result, the target may be
position-based visual servoing (PBVS) and image-based vi- out of sight when it moves fast. Wang et al. [25] present a
sual servoing (IBVS). PBVS requires the controller to be framework for tracking a mobile ground target using a fixed-
designed in 3D Cartesian space, thus the camera needs to wing UAV. Yet the design of guidance law in this approach
be calibrated for the transformation of coordinate frame. is based on target localization, which is easily affected by
Basically, most of the existing work of target tracking for the calibration of the camera. In our previous work [26], we
the UAVs adopt PBVS [11], [12], [13]. In this approaches, have studied the tracking of ground target by fixed-wing UAV
with fixed camera. Nevertheless, due to the limited sight of
This work was funded by the National Natural Science Foundation of
China (61906209) and (61973309) view and the fixed attitude of the camera, it is easy to lost
1 authors are with the College of Intelligence Science and the target when the target moves fast.
Technology, National University of Defense Technology, Changsha, In this work, we propose a control framework based on
China. ljyang13@163.com, zhliu@nudt.edu.cn,
guanzhengw@163.com, xkwang@nudt.edu.cn IBVS for tracking a moving target by a fixed-wing UAV
* corresponding author with a pan-tilt camera. This approach integrates the attitude

978-1-7281-4277-7/20/$31.00 ©2020 IEEE 354

Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
Fig. 1: The ground target tracking problem for a fixed-wing Fig. 2: The relationships among four coordinate frames.
UAV with a pan-tilt camera.

the controller for the fixed-wing UAVs is more complicated

control of the pan-tilt camera and the UAV. Hence, the than the rotor UAVs.
UAV can loiter over the moving target while maintaining To illustrate the relationships among the UAV, the pan-
the target at the image center. In this case, the target will tilt and the camera, four right-hand Cartesian frames are
not be out of sight even if it moves fast. Besides, we build a constructed, shown in Fig. 2.
HIL simulation platform and a prototype platform. Based on (a) The body coordinate frame (FB )
these platforms, extensive experiments including simulations The origin of FB is located at the center of mass of
and real flight tests are conducted to evaluate our controller. the fixed-wing UAV. Besides, xb -axis is along the heading
The results show that our controller can achieve continuous direction of the UAV, and yb -axis points to the right side of
robust tracking of the target, and the speed of the target can the UAV.
be as fast as 20km/h when that of the UAV is 16m/s. The (b) The pan-tilt coordinate frame (FG )
contributions of our work are: The top of the pan-tilt is fixed to the body, and we denote
• Proposing a framework to integrate the attitude control the center of the connecting plane as the origin of FG .
of the pan-tilt camera and the fixed-wing UAV, which Besides, yg -axis is along the optic axis and xg -axis points
ensures that the UAV can loiter over the moving target to the right side of yg -axis.
while maintaining the target at the image center. (c) The camera coordinate frame (FC )
• Conducting extensive experiments including HIL sim- The optical center of the camera is defined as the origin of
ulations and field experiments. To the best of our Fc , and the optic axis is denoted by zc -axis. Besides, xc -axis
knowledge, the field experiments for the moving target is opposite to xg -axis.
tracking by the fixed-wing UAV based on IBVS are (d) The image coordinate frame (FI )
barely performed in the existing work. This highlights The origin of FI lies in the center of the image, xi -axis
the feasibility and superiority of our controller. and yi -axis point to the right and bottom of the image,
The paper is organized as follows. Section II describes respectively.
the problem of target tracking for the fixed-wing UAV with Assume that the UAV autopilot has a speed controller that
a pan-tilt camera. Section III introduces the main work of the holds the speed constant (or nearly so) and has a height
paper, including the design of the controller and proof of the holding controller for constant UAV altitude. Actually, the
algorithm. Section IV provides the results and analysis for assumption is commonly used in related tasks performed by
the HIL simulations and field experiments. Section V gives the fixed-wing UAVs [27]. Therefore, a simplified kinematic
the conclusion of the paper. model in two dimensional plane is considered. Besides, the
paper adopts a pan-tilt with two limited degrees of freedom
II. PROBLEM STATEMENT
for the reason of low cost and weight, and its yaw and pitch
In this section, we provide the description of the problem rate are denoted by θ˙p and θ̇t , respectively. Furthermore, we
to be addressed and the overall framework of the proposed denote current speed of the UAV (denoted by Vt ) along the
controller. inertial frame by {Vx , Vy }, and yaw angle of the UAV by ψ .
A. Problem Description Therefore, the “UAV & pan-tilt” model can be represented
by Eq. (1).
We consider the ground target tracking problem for a 
miniature fixed-wing UAV equipped with a pan-tilt camera 
 Vx = Vt · cosψ


in this work. As Fig. 1 shows, the target lies in the image 
 Vy = Vt · sinψ

center with the aid of the pan-tilt when the UAV is tracking
ψ̇ = uψ , (1)
it. It is known that the fixed-wing UAVs have a minimum 


 θ̇ p = u p , θ p ∈ (γ p1 , γ p2 )
stall speed limit, and can only track the target by loitering, 


rather than hovering of rotor UAVs. Therefore, the design of θ̇t = ut , θt ∈ (γt1 , 0)

355
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3: The range of the pan-tilt attitude. Fig. 4: The framework of the controller.

where uψ , u p and ut are control inputs. The range of the detection is not the focus of the paper, we adopt existing
pan-tilt attitude is shown as Fig. 3, γ p1 (< 0), γ p2 (> 0) and method named YOLOv3 [29] to solve the problem directly.
γt1 (< 0) are the thresholds of the pan-tilt attitude. 2) Servo Control: Image-based visual servoing is able to
Different from the conventional position-based tracking directly map the desired velocity of feature point into the
method, we intend to propose an IBVS control law for the motion velocity of the UAV. With the aid of it, there is no
UAV and the pan-tilt together by using the image information need to calculate the relative position of the target to the
directly, without target positioning processes in FC . However, UAV. As a result, the generated error in this process can be
there exist two challenges in the work as follows. eliminated.
We denote the desired velocity of the centroid coordinates
• A small jitter of the UAV attitude will cause a dramatical
by Ṡ(u̇, v̇) in FC , and it is related with the altitude of the
jump for the position of the target in the image, and
UAV and the pan-tilt. Simultaneously, the linear velocity of
the attitude of the fixed-wing UAV is dynamically
the UAV is denoted by T = (Vx ,Vy ,Vz )T , and angular velocity
changed all the time during the mission. Therefore, how
of it is denoted by Ω = (ωx , ωy , ωz )T . There exists image
to implement the tracking under this circumstance is
Jacobian matrix Jv that associates Ṡ with {T, Ω}, which is
challenging.
shown as Eq. (2).
• Both the attitude of the pan-tilt and that of the UAV are Å ã Å ã
coupled, how to integrate the attitude control of them is u̇ T
= Jv · , (2)
challenging as well. v̇ Ω
where Jv ∈ R2×6 [9], and we can obtain the control output
B. Overall Framework
directly from Ṡ through the inverse of the equation. Simul-
In this subsection, we design the overall framework of taneously, we need to adjust θ p and θt to make (u1 , v1 ) tend
the visual control scheme (Fig .4). The pitch, roll and yaw to (0, 0).
angle of the UAV in current state are denoted by {θ , ϕ , ψc }
in order, and the desired yaw angle of the UAV is denoted III. MAIN WORK
by ψe . Besides, H represents the flight height, α represents A. Control Law Design
a desired pitch angle of the pan-tilt, and kψ is a coefficient. During the target tracking process, the attitude of the UAV
Firstly, an image detection module is used to obtain the {ψ , θ , ϕ } is coupled with that of the pan-tilt {θ p , θt }. As a
target information, which is the target position (u1 , v1 ) in FI . result, the centroid coordinates of the target are affected by
And then, the servo control module is performed with the the attitude of both the UAV and the pan-tilt. Furthermore,
aid of the (u1 , v1 ) information and the UAV’s state, to obtain the coordinates of the desired feature point changes at
the desired uψ , u p and ut . runtime before the UAV reaches a stable loitering state,
1) Image Detection: UAV-based image detection is in which causes the unknown of Ṡ.
charge of processing image and detecting target. It takes In order to make Ṡ unaffected by the change of
the image captured by the airborne camera as the input, by {θ , ϕ , θ p , θt }, we propose a reference state named “Ideal
locating the position of the target on the image, and outputs State” (Definition 1). Through this method, we map the
the coordinates of the target feature point (denoted by (u1 , v1 ) current system state into an reference state, and Ṡ in the latter
in FI ). It should be noted that high requirement of accuracy can be determined uniquely. Moreover, Ṡ is only affected by
and real-time performance is necessary in field experiments. the change of ψ , then uψ can be obtained on the basis.
To the best of our knowledge, there exist two kinds of image Definition 1 (Ideal State): The “UAV & pan-tilt” system
detection methods [28], which can be divided into traditional that the model corresponding to must simultaneously satisfy
method and neural network method. Compared to the former, the following four conditions: (a) The pitch angle of the UAV
the latter can obtain (u1 , v1 ) directly from the image, which is is 0◦ ; (b) The roll angle of the UAV is 0◦ ; (c) The pitch angle
more efficient. Besides, considering that this part of image of the pan-tilt is α ; (d) The yaw angle of the pan-tilt is 90◦ .

356
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
Fig. 5: Flow chart of the transformation for the camera
coordinate frame.

When the fixed-wing UAV loiters clockwise over the

Fig. 6: Relationships between the position of the pan-tilt
target, according to the signals of the angles in Fig. 3, the
camera and coordinates of the feature point.
ideal state for the pan-tilt can be represented as: θ p = −90◦ ,
θt = α (α < 0). In order to obtain Ṡ, we need to transform
current state of the system into the reference state. The by adjusting θ p and θt . The reason is that it makes poor
transformation process is shown as Fig. 5, where Ri (i ∈ detection at the edge of the image, which causes the target
{1, 2, 3, 4, 5}) represents the rotation matrix that changes the to be lost easily. As Fig. 6 shows, O1 O2 is the optic axis, O2
current state into the state inside the corresponding rectangle, is the center of the image, and (|O2 A|, −|O2 B|) represents
with the other state unchanged. Besides, all matrices are the coordinates of the target in FI . With the pan-tilt rotating,
based on the rotation of FC . the changes of θ p and θt will affect the abscissa and ordinate
The coordinates of the target in current FC are denoted of the target, respectively. Since ∆θ p < 0 when the pan-tilt
by (X1 ,Y1 , Z1 ), then the corresponding coordinates under the deflects to the right and ∆θt < 0 when it rotates downward,
reference state can be expressed as: the relationship between them can be expressed as:
Ñ é Ñ é ß
X2 X1 tan(∆θ p ) = −|O2 A|/|O1 O2 |
(6)
Y2 = R5 · R4 · R3 · R2 · R1 Y1 (3) tan(∆θt ) = |O2 B|/|O1 O2 |
Z2 Z1 Since the feature point needs to tend to locate at the image
We define R = R5 · R4 · R3 · R2 · R1 . According to the princi- center, u̇1 and v̇1 can be expressed as:
ple of pinhole imaging, the relationship between (X1 ,Y1 , Z1 ) Å ã Å ãÅ ã Å ã
u̇1 λu u1 u1
and (u1 , v1 ) satisfies the triangle similarity. The correspond- =− = −λuv , (7)
v̇1 λv v1 v1
ing similarity coefficient is defined as k. We denote the focal
length of the camera by f , which can be obtained directly where {λu , λv } ∈ R+ .
from the adopted camera parameters. After letting Z2 = − f , We denote the depth from the target to the camera by z,
(i, j)
we can get the corresponding coordinates (u2 , v2 ) under the the element in i-th row and j-th column of Jv by Jv . Based
ideal state. on the previous work (see detail in Theorem 3 in [26]), we
Å ã Å ã have Å ã Ç å
u2 u1 M1 u˙2 − zf Vt
= k·R (1−2,·)
, (4) ψ̇ = , (8)
v2 v1 M2 v˙2
where R(1−2,·) represents first two rows of R, and the coef- where
(1,5) (1,6)
ficient k can be calculated by the third line of Eq. (3). M1 = −Jv · sinα + Jv · cosα ,
In the “Ideal State”, the desired feature point is the center (2,5) (2,6)
M2 = −Jv · sinα + Jv · cosα .
of the image, then the desired rate of the feature point can
be obtained: By using the least square method, we are able to obtain
Å ã Å ãÅ ã Å ã uψ . Furthermore, combining Eqs. (6) (7), the control law of
u̇2 λ1 u2 u2
=− = −λ , (5) the “UAV & pan-tilt” can be represented as
v̇2 λ2 v2 v2
 f
 M1 (u˙2 − z Vt )+M2 v˙2
where {λ1 , λ2 } ∈ R+ . 
 uψ = M12 +M22
During the control, we design uψ based on Eq. (5) to make
 u p = arctan( −λfu u1 ) , (9)
the feature point tend to locate at the image center, thereby 
 −λv v1
achieving the tracking. However, the target is moving and the ut = arctan( f )
speed is unknown, then the actual (u̇2 , v̇2 ) are different, and Furthermore, we analyze the effect of the parameter z on
the feature point may even deviate from the image center. uψ . Compared with uψ when z is measured in real time,
Therefore, we adopt the pan-tilt camera to adjust the angle it will only cause a small deviation of each yaw amplitude
of view, so that the target can be kept near the image center when z is a constant, without affecting the trend of the UAV
during the tracking. circling and tracking the target.
When analyzing the control of the pan-tilt, we hope the The implementation of the algorithm is shown as Algo-
target can be as close as possible to the center of the image rithm 1.

357
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
Algorithm 1 Using coordinates of the feature point to obtain
the control of the UAV and the pan-tilt
Require: the image captured by camera
Ensure: the control of UAV and pan-tilt
1: while discover the target do
2: detect the centroid coordinates (u1 , v1 );
3: calculate the rotation matrices R1 , R2 , R3 , R4 , R5 ;
4: obtain the transformed centroid coordinates:
Ñ é
Å ã u1
u2
= k · (R5 · R4 · R3 · R2 · R1 )(1−2,·) v1
v2 Fig. 7: The HIL simulation environment.
−f
5: obtain the velocity of the transformed target:
Å ã Å ãÅ ã Set a Lyapunov function
u˙2 λ1 u2
=−
v˙2 λ2 v2 1
V = eT · e,
6: calculate M1 and M2 : 2
(1,5) (1,6)
it is obvious that V > 0, and the time derivative of V can be
M1 = −Jv · sinα + Jv · cosα represented as
(2,5) (2,6)
M2 = −Jv · sinα + Jv · cosα V̇ = eT · e = eT · Pe · e, (10)

7: obtain the yaw rate of the UAV: Since Pe is negative, and we choose

M1 (u˙2 − zf Vt ) + M2 v˙2 η = ||Pe ||∞ ,

uψ =
M12 + M22 then Eq. (10) can be changed into
8: obtain the velocity of the target V̇ < η · eT · e < 0, (11)
Å ã Å ãÅ ã
u˙1 λu u1 The result shows that e is exponentially convergent. There-
=−
v˙1 λv v1 fore, the algorithm is exponentially asymptotically stable.
9: obtain the deflection rate of the pan-tilt: Q.E.D.
®
u p = arctan( u˙f1 ) IV. E XPERIMENTS A ND R ESULTS
ut = arctan( v˙f1 ) In this section, we build a HIL simulation platform based
on Gazebo and complete the preliminary validations of the
10: end while controller. Furthermore, we conduct field experiments to
evaluate the performance of our controller.

B. Stability Analysis A. HIL Simulations

We denote the error between (u1 , v1 ) and image center by 1) Simulation Setup: The HIL simulation platform is
e1 , and the error between (u2 , v2 ) and image center by e2 . composed of three parts: Gazebo, Pixhawk and QGround-
The convergence of e1 to 0 indicates that the feature point Control (QGC), as Fig. 7 shows.
tends to locate at the image center with the aid of the pan- Gazebo is a simulator that offers the ability to accurately
tilt. Simultaneously, the UAV will circle around the stationary and efficiently simulate populations of robots in complex
target when e2 converges to 0. Furthermore, we denote the environments. The weight of the fixed-wing UAV is 2.65kg,
error state by e = (e1 , e2 )T . the wing surface and wingspan are 0.47m2 and 2.59m,
Theorem 1: With the controller designed as Eq. (9), when respectively. With a 60◦ field of view, the camera captures
we define images of 1280 × 720 in pixels. Besides, the weight of the
car is 771kg, and its front wheels can be deflected to change
ė = Pe · e, Pe = −diag{λu , λv , λ1 , λ2 } the motion direction.
the UAV is able to asymptotically track the stationary target Pixhawk is a high performance autopilot that is widely
while maintaining it at the image center. used in small UAVs. With the aid of MAVLink protocol,
Proof: According to Eqs. (6) (8), we can construct the we implement the communication between Pixhawk and the
following equations to obtain the control output: UAV in Gazebo.
Ü ê Ü ê QGC is a ground control station that is adapted to the
f Ñ é 0 Pixhawk autopilot. Similarily, it communicates with the
tan(u p )
f 0 autopilot via the MAVLink protocol. We use it to observe
ė = tan(ut ) + fVt
M1 z the flight state (flight speed, altitude, trajectory, etc.) of the
uψ
M2 0 UAV in real time.

358
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
 120 50

u (pixel)
110

100 0
90

Distance (m)
80 -50
0 20 40 60 80 100
70
Time (s)
60
50
50

v (pixel)
40
0
30

20
-50
10 0 20 40 60 80 100
0 20 40 60 80 100 120

Time (s) Time (s)

(a) (b) (c)

Fig. 8: The fixed-wing UAV tracks the target of uniform linear motion in the HIL simulation. The speed of the target is
4m/s, the flight height and speed of the UAV are 100m and 16m/s, respectively. (a) Tracking trajectories; (b) Horizontal
distance between the target and the fixed-wing UAV; (c) Image stabilizing precision.

2) Simulation Results and Analysis: To evaluate the per-

formance of our controller, we record the position infor-
mation of the UAV and the target in Gazebo. After that,
the results are presented and analyzed through plotting the
trajectories in MATLAB.
In this experiment, the fixed-wing UAV tracks the tar-
get with uniform linear motion at a constant speed. The
velocity and flight height of the fixed-wing UAV is 16m/s
and 100m, respectively. Simultaneously, the target keeps
moving forward at a constant speed of 4m/s. The values of
{λ1 , λ2 , λu , λv } in our work are all chosen as 0.5.
Fig. 8a shows the tracking trajectory under this circum-
stance. Note that we conduct the experiments under this
circumstance several times and similar results are found. It
is obvious that the fixed-wing UAV can circle and track the
moving target.
Besides, we depict the horizontal distance between the
UAV and the target during the experiment in Fig. 8b. When Fig. 9: The comparison of tracking performance between
the target moves, the distance will fluctuate within a small the fixed-wing UAV with a pan-tilt camera (figure a) and
range. The two black dashed lines represent the maximum that with a fixed camera (figure b). Green line represents
range of the distance is approximately [32, 96]. the distance between the feature point and image center in
Fig. 8c shows the image stabilizing precision for the the image coordinate. Red line and blue line represent the
experiment. When the target makes a uniform linear motion, change of θ p and θt , respectively.
both u and v will fluctuate regularly around 0. It can be easily
found that the maximum of the variation will not exceed 50
pixels. pixels when the target is stationary, what accounts for it not
Furthermore, in order to embody the superiority of the near 0 is that the attitude of the camera is fixed. When the
pan-tilt camera in target tracking, we compare the tracking target begins to move after 25s, the distance changes and
performance of the fixed-wing UAV with a pan-tilt camera starts to increase sharply after 50s. As a result, the target is
and that with a fixed camera. We set the flight speed and out of sight at about 60s, and we denote the distance after
height of the UAV as 16m/s and 100m, respectively. Besides, that by (0, 0) to make the UAV circle around where the target
the motion speed of the target is 4m/s. The result is shown in disappears.
Fig. 9 indicates that with the aid of the adjustment of θ p and
θt , the distance between the feature point and image center B. Field Experiments
in the image coordinate will not exceed 50 pixels. 1) Field Experimental Setup: The results of the HIL sim-
However, since the camera is fixed as Fig. 9b shows (θ p = ulation verify the effectiveness of our controller in virtual en-
−90◦ , θt = −45◦ ), the distance between the feature point and vironment. However, there still exist disturbances (e.g. wind
the image center in the image coordinate will change greatly. disturbance and inaccurate airspeed measurement) during the
As the green line shows, the distance remains at about 230 real flight tests. Therefore, we set up the field experiment to

359
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
 stationary, the angles of the pan-tilt cannot keep constant for
the unstable state of the fixed-wing UAV in field experiments.
Besides, the points inside blue circles deviate significantly,
which is caused by misdetection. And there are also some
points missing inside blue rectangles. The reasons accounting
for this can be divided into two kinds, one is that the
target is too small in the image edge to be detected, the
other is the angle of the pan-tilt reaches threshold, which
Fig. 10: The fixed-wing UAV and the car (as the ground makes the target out of sight briefly. Under the circumstance,
target) we use in the field experiments. we use the uψ calculated from (u2 , v2 ) at the last moment
before the target is lost for control. After that, the UAV will
yaw to the direction where the target disappears from the
image. As a result, the UAV can still track the target despite
these situations, which further verifies the feasibility of our
controller in real flight tests.

V. CONCLUSION
This paper has presented a control framework for a fixed-
wing UAV with a monocular pan-tilt camera to track a
moving target. More specifically, this control framework
uses the target detection algorithm based on YOLOv3 to
obtain the centroid coordinates of the target firstly. Then,
this framework uses a reference state called “Ideal State” to
make the coordinates of the feature point only affected by the
change of the yaw angle of the UAV. After that, the controller
is designed based on the image Jacobian matrix. In order to
Fig. 11: The scenes of the real flight tests. The red circle verify the feasibility of the controller, we have conducted the
represents the flying fixed-wing UAV, and the blue box shows HIL simulation experiments based on Gazebo, and then have
the target, which is a white car. carried out field experiments based on a prototype system.
The results show that our controller can achieve continuous
and robust tracking of the target by the UAV.
further evaluate our controller, and it is developed based on Future work will focus on target tracking by multiple
the existing architecture of our team [30], [31]. fixed-wing UAVs. We believe that by using the images
of diverse perspectives and surrounded trajectories from
The fixed-wing UAV and the car we use in the field
different UAVs cooperatively, the challenge of tracking the
experiments are shown in Fig. 10. The UAV has a wingspan
high speed maneuvering ground target can be met.
of 1.8m and a weight of 5kg. Besides, the length and height
of the UAV are 1.23m and 0.35m, respectively. For the pan- R EFERENCES
tilt camera, it is the same as we use in the HIL simulation
platform. [1] Wu J, Wang H, Li N, et al. Distributed trajectory optimization for
multiple solar-powered UAVs target tracking in urban environment by
Fig. 11 displays the scenes of our field experiments. As Adaptive Grasshopper Optimization Algorithm[J]. Aerospace Science
the figure shows, our experimental site is a meadow, and the and Technology, 2017, 70: 497-510.
target we use is a white car (the blue box). The four pictures [2] Zhang M, Liu H H T. Cooperative tracking a moving target using mul-
tiple fixed-wing UAVs[J]. Journal of Intelligent & Robotic Systems,
represent a sequence of scenes from the same experiment. 2016, 81(3-4): 505-529.
2) Experimental Results and Analysis: We make the [3] Vanegas F, Campbell D, Roy N, et al. UAV tracking and following
fixed-wing UAV track target with uniform linear motion in a ground target under motion and localisation uncertainty[C]//2017
IEEE Aerospace Conference. IEEE, 2017: 1-10.
the experiment. The flight airspeed and height are 16m/s [4] Vanegas F, Gonzalez F. Uncertainty based online planning for UAV
and 100m, respectively. Besides, the target is moving at a target finding in cluttered and GPS-denied environments[C]//2016
constant speed of 20km/h. IEEE Aerospace Conference. IEEE, 2016: 1-9.
[5] Larouche B P, Zhu Z H. Autonomous robotic capture of non-
The tracking trajectories are shown as Fig. 12a, and the cooperative target using visual servoing and motion predictive con-
horizontal distance between the UAV and the target is shown trol[J]. Autonomous Robots, 2014, 37(2): 157-167.
as Fig. 12b. We can find that the horizontal distance keeps [6] Pestana J, Sanchez-Lopez J L, Campoy P, et al. Vision based gps-
denied object tracking and following for unmanned aerial vehi-
at about 95m when the target is stationary, and it will vary cles[C]//2013 IEEE International Symposium on Safety, Security, and
within the range of [23, 210] (denoted by two black dashed rescue Robotics (SSRR). IEEE, 2013: 1-6.
lines). [7] Olivares Mendez M A, Mondragon I, Campoy P, et al. Aerial object
following using visual fuzzy servoing[C]//First Workshop on Research,
Fig. 12c shows the image stabilizing precision for the Development and Education on Unmanned Aerial Systems (RED-UAS
experiment. The green part indicates that even the target is 2011). 2011.

360
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.
 250
step 1
step 2
200

Distance (m)
150

100

50

0
0 20 40 60 80 100 120
Time (s)
(a) (b) (c)

Fig. 12: The fixed-wing UAV tracks the target of uniform linear motion in the field experiment. The speed of the target is
20km/h, the flight height and speed of the UAV are 100m and 16m/s, respectively. Besides, the target is stationary in step
1 and moving in step 2. (a) Tracking trajectories; (b) Horizontal distance between the target and the fixed-wing UAV; (c)
Image stabilizing precision.

[8] Hutchinson S, Hager G D, Corke P I. A tutorial on visual servo inspection of a ground target using a multirotor UAV system[J].
control[J]. IEEE Transactions on Robotics and Automation, 1996, Sensors, 2017, 17(12): 2929.
12(5): 651-670. [22] Wei Y, Lin Z. Vision-based Tracking by a Quadrotor on ROS[J]. IFAC-
[9] Chaumette F, Hutchinson S. Visual servo control. I. Basic ap- PapersOnLine, 2017, 50(1): 11447-11452.
proaches[J]. IEEE Robotics & Automation Magazine, 2006, 13(4): [23] Le Bras F, Hamel T, Mahony R. Image-based visual servo control for
82-90. circular trajectories for a fixed-wing aircraft[C]//Proceedings of the
[10] Chaumette F, Hutchinson S. Visual servo control. II. Advanced ap- 48h IEEE Conference on Decision and Control (CDC) held jointly
proaches [Tutorial][J]. IEEE Robotics & Automation Magazine, 2007, with 2009 28th Chinese Control Conference. IEEE, 2009: 3430-3435.
14(1): 109-118. [24] Peliti P, Rosa L, Oriolo G, et al. Vision-based loitering over a target
[11] Teulire C, Eck L, Marchand E, et al. 3D model-based tracking for for a fixed-wing UAV[J]. IFAC Proceedings Volumes, 2012, 45(22):
UAV position control[C]//2010 IEEE/RSJ International Conference on 51-57.
Intelligent Robots and Systems. IEEE, 2010: 1084-1089. [25] Wang X, Zhu H, Zhang D, et al. Vision-based detection and tracking
of a mobile ground target using a fixed-wing UAV[J]. International
[12] Wise R, Rysdyk R. UAV coordination for autonomous target track-
Journal of Advanced Robotic Systems, 2014, 11(9): 156.
ing[C]//AIAA Guidance, Navigation, and Control Conference and
[26] Yang L, Liu Z, Wang X, et al. An optimized image-based visual servo
Exhibit. 2006: 6453.
control for fixed-wing unmanned aerial vehicle target tracking with
[13] Chen J, Dawson D M. UAV tracking with a monocular cam- fixed camera[J]. IEEE Access, 2019, 7: 68455-68468.
era[C]//Proceedings of the 45th IEEE Conference on Decision and [27] Zhao S, Wang X, Lin Z, et al. Integrating Vector Field Approach and
Control. IEEE, 2006: 3873-3878. Input-to-State Stability Curved Path Following for Unmanned Aerial
[14] Yongchun F. A survey of robot visual servoing [J]. CAAI Transactions Vehicles[J]. IEEE Transactions on Systems, Man, and Cybernetics:
on Intelligent Systems, 2008, 3(2): 109-114. Systems, 2018:1-8.
[15] Serra P , Cunha R , Hamel T , et al. Landing of a Quadrotor on a [28] Zhiqiang W, Jun L. A review of object detection based on convo-
Moving Target Using Dynamic Image-Based Visual Servo Control[J]. lutional neural network[C]//2017 36th Chinese Control Conference
IEEE Transactions on Robotics, 2017, 32(6):1524-1535. (CCC). IEEE, 2017: 11104-11109.
[16] Araar O , Aouf N , Vitanov I . Vision Based Autonomous Landing [29] Redmon J, Farhadi A. Yolov3: An incremental improvement[J]. arXiv
of Multirotor UAV on Moving Platform[J]. Journal of Intelligent & preprint arXiv:1804.02767, 2018.
Robotic Systems, 2017, 85(2):369-384. [30] Wang X, Shen L, Liu Z, et al. Coordinated flight control of miniature
[17] Srivastava R, Lima R, Das K, et al. Least Square Policy Iteration for fixed-wing UAV swarms: methods and experiments[J]. Science China
IBVS based Dynamic Target Tracking[C]//2019 International Confer- Information Sciences, 2019, 62(11): 212204.
ence on Unmanned Aircraft Systems (ICUAS). IEEE, 2019: 1089- [31] Liu Z, Wang X, Shen L, et al. Mission Oriented Miniature Fixed-wing
1098. UAV Swarms: A Multi-layered and Distributed Architecture[J]. arXiv
[18] Lyu Y, Pan Q, Zhang Y, et al. Simultaneously multi-UAV mapping preprint arXiv:1912.06285, 2019.
and control with visual servoing[C]//2015 International Conference
on Unmanned Aircraft Systems (ICUAS). IEEE, 2015: 125-131.
[19] Popova M G. Visual Servoing for a Quadrotor UAV in Target Tracking
Applications[M]. University of Toronto, 2015.
[20] Zarudzki M, Shin H S, Lee C H. An image based visual servoing ap-
proach for multi-target tracking using an quad-tilt rotor UAV[C]//2017
International Conference on Unmanned Aircraft Systems (ICUAS).
IEEE, 2017: 781-790.
[21] Hinas A, Roberts J M, Gonzalez F. Vision-based target finding and

361
Authorized licensed use limited to: DELHI TECHNICAL UNIV. Downloaded on March 12,2025 at 18:45:41 UTC from IEEE Xplore. Restrictions apply.