2009 Brazilian Symposium on Aerospace Eng.

& Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

VISIO-BASED COTROL OF FIXED-WIG UMAED AERIAL

VEHICLE FOR AUTOOMOUS ROAD-FOLLOWIG

João Paulo de Souza, jpsouza@ita.br

Elder Moreira Hemerly, hemerly@ita.br
Instituto Tecnológico de Aeronáutica, ITA – Pça. Mal. Eduardo Gomes, 50 – Vila das Acácias – São José dos Campos - SP

Abstract. avigation and control techniques using the GPS signals have problems in areas where the terrain or other
obstacles can jam or block the satellite signal. avigation systems based on vision emit no external signs, are passive
and low cost. This work aims to enable the autonomous navigation of a fixed-wing unmanned aerial vehicle (UAV) for
road tracking using only image processing and inertial sensors. Image processing algorithms were implemented to
generate waypoints and projection of the image reference frame to the world reference frame, thereby enabling the
design of strategies for lateral and altitude control. Performance evaluation is carried out with a realistic UAV model
and virtual scenario using the FlightGear Simulator. This simulator provides visualization of three-dimensional
models, which run in accordance with the information of position and attitude received from the control algorithms
implemented in MatLab.

Keywords: Vision, UAV, Tracking Trajectory, avigation, Image Processing

1. ITRODUCTIO

Unmanned aerial vehicles (UAV’s) are remotely piloted or autonomous aircraft that can carry cameras, sensors,
communication equipments, among others. The UAV’s have very important in applications that include traffic and
environmental monitoring, pest control and fire, search and rescue, border patrol, aerial surveillance, crop and flock
inspection, police surveillance of urban areas, inspection of power transmission lines and oil products, assistance in
disaster and military applications.
The versatility and low cost of UAV’s are the main factors for the increase in its use both in public and in private
sectors. Some of the disadvantages of the UAV’s include payload capacity, and limited supply of energy. However,
there are several advantages associated with the use of UAV’s that override these limitations. Because they are more
maneuverable and are not operated by pilot, their missions are in general more complex, such as very low altitude
flights or flying through hostile environment. The main advantage of UAV is the operational cost, which is up to three
times cheaper than a traditional aircraft. Besides having a lower cost than manned aircraft, they are more easily
transported and can be launched even on uneven terrain.
Standard techniques that rely heavily on GPS for navigation and control will have difficulties in urban environments
or hostile environments in which the satellite signals can be easily blocked, jammed or have their accuracy degraded.
The computer vision is an important sensor for UAV’s operating in natural environments. A sequence of video images
contains a large amount of information that can be used for navigation and control, detection and identification of
objects, obstacle avoidance and many other tasks. Unlike systems based on radar or laser, the computer vision is passive
and emits no external signals. As a result, systems based on vision can be small in size, reducing the load of the UAV
platform.
The contribution of this work is the development of image processing and attitude and position control for easy
implementation, aiming at UAV applications. A comparison of performance between two techniques for lateral control
is realized, keeping the parameters of the autopilot unaltered. The discussion focuses on the characteristics and
capabilities of the control based on vision for civil and military use. The main difference with respect to Niculescu
(2001) and Frew (2004) are: a) Niculescu (2001) consider only lateral control, without addressing the image processing
task; b) no performance comparison between the 2 strategies is carried out in Frew (2004), and c) neither employs
FlightGear for visualization, which is efficient and freely available.

2. UAV MODEL

The general equations of motion presented in this section were developed for a rigid aircraft and are divided into
three sets: force equations, moment equations and kinematics equations (Roskan, 2001).

( )
m U& − VR + WQ = − mg sin (Θ ) + FAx + FTx
m(V& + UR − WP ) = mg sin (Φ ) cos(Θ ) + FA + FT
y y
(1)
m(W& − UQ + VP ) = mg cos(Φ ) cos(Θ ) + FA + FT
Z Z
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

( )
I xx P& − I xz R& − I xz PQ + I zz − I yy RQ = L A + LT
( )
I xx Q& + (I xx − I zz )PR + I xz P − R 2 = M A + M T
2
(2)
( )
I zz R& − I xz P& + I yy − I xx PQ + I xz QR =  A +  T

& −Ψ
P=Φ & sin (Θ )
& cos(Φ ) + Ψ
Q=Θ & cos(Θ )sin (Φ ) (3)
& cos(Θ ) cos(Φ ) − Θ
R=Ψ & sin (Φ )

Where m is the mass of the UAV; U, V, W are the UAV linear velocities components about XYZ; P, Q, R are the
UAV angular velocities components about XYZ; g is the acceleration of gravity; Φ, Θ, Ψ are the Euler angles of the
UAV; FAx, FAy, FAz are the aerodynamic force components along XYZ; FTx, FTy, FTz are the thrust force components
along XYZ; Ixx, Iyy, Izz are the UAV moments of inertia about XYZ; Ixy, Iyz, Ixz are the UAV products of inertia about XYZ;
LA, MA, A are the aerodynamic moment components about XYZ; LT, MT, T are the thrust moment components about
XYZ.
The Equations (1)-(3) are necessary for the understanding and use of mathematical model for the aerodynamic
forces and moments that act on the UAV. These describe the aerodynamic behavior of an UAV, which were used in the
analysis and design of the control system.
In this work we used the typical model of the UAV and autopilot design presented and developed in Hemerly et al.
(2006). Figure 1 shows the block diagram of the autopilot.

Figure 1. Autopilot block diagram.

In Figure 1, V and Vref are the measured and reference linear velocity, respectively; ay and ayref are the measured and
reference lateral acceleration, respectively; θ and θref are the measured and reference pitch angle, respectively; ψ and ψref
are the measured and reference yaw angle, respectively; π, δr, δe and δa are the throttle, rudder, elevator and aileron
commands.
In section 5, simulation results for a realistic UAV model using the control techniques implemented in this work are
presented and discussion.

3. COMPUTER VISIO

There are many applications of computer vision techniques. Traditionally, most of the computer vision systems have
been designed for industrial and military applications. Common military applications include target recognition, visual
guidance for autonomous vehicles, recognition and interpretation of images. Common industrial applications include
visual inspection of parts and automated control systems.
In this work, the interaction with the user is only required to classify the pixels as positive (track) or negative (non-
track) for a particular training sample (Figures 2a and 2b). The statistics for a sampled region is computed through the
mean and standard deviations of the interest pixels.
We use a connected components analysis algorithm (Fig. 2c) available in the image processing toolbox of the
MatLab, which is a method of identifying objects, regions and features to convert the previous segmentation in adjacent
regions. This algorithm marks regions with areas smaller than a give threshold (window). This window is placed on
each pixel of the background, and the algorithm assigns it to the class that is the closest to its neighborhood. In this
work were selected windows with up to 4 pixels square area.
Finally, using the location information of regions selected by the user in classification step, we determine the class
to be defined as the track (Fig. 2d).
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

A disadvantage of these techniques is that only the local information is used. One way to enter the global
information is through models. The techniques of the Hough transform (Fig. 2e) are well known techniques of this type
that can be applied to straight lines, circles and any other kind of curves that can be expressed by a small set of
parameters. In this work, we consider only the problem of detecting straight lines in images with the application of
Hough transform.

Figure 2. Road detection algorithm. a) Original Image. b) Pixels Classification. c) Connected components analysis.
d) Holes removal. e) Hough transform. f) Robust line fitting.

Representation in polar coordinates of a straight line is used in processing, to avoid infinitely large slope
coefficients for straight lines parallel to y-axis.
A robust alignment (Fig. 2f) of the line using the least squares method (LSM) is applied to analyze the position and
orientation of the center of the runway. The data points used as input for the algorithm that implements the LSM are
obtained from an algorithm for detecting edges. These edges are taken as the edges of the track, which after being
subjected to LSM return the center of the runway.

4. AVIGATIO AD COTROL

4.1. Projection equations

The set of points in the image that represents the centerline of the track are the waypoints to be flown by UAV. The
waypoints are defined, in general, by latitude, longitude, altitude, direction, aircraft speed or speed relative to the
ground, flying time, among others. However, until the moment, waypoints are points (pixels) defined in the reference
plane of the image (Xi, Yi). The Figure 3 shows the layout of the camera relative to the UAV and image reference
system as seen by the camera.
As can be seen in Figure 3, the center of rotation of the gimbal (C) at which the camera is fixed is not on the center
of gravity (A) of UAV. The same as sensing element of the camera is not in the center of rotation of the gimbal.
To obtain the equations of projection is necessary first to apply the model of the pinhole camera, and then align the
coordinate systems by means of rotations and translation.
In the pinhole camera model, the location of a point in the coordinate system of the camera (XC, YC, ZC) is related to
a point in the image by the equation:

xc y
xi = f ⋅ , yi = f ⋅ c (4)
zc zc
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

where f is the focal length of the camera in pixel dimension; xc, yc, zc are the coordinates of a point in the coordinate
system of the camera; xi, yi are the coordinates of a point in the coordinate system of the image; XAYAZA is the coordinate
system of the UAV; and XWYWZW is the coordinate system of the world.

Figure 3. Coordinate systems of image, camera, aircraft and runway.

Equation (5) shows the matrices of rotation (Rx, Ry, Rz) around the axis x, y and z, respectively, counter-clockwise:

1 0 0   cos(β ) 0 sin (β ) cos(γ ) − sin (γ ) 0

  
R x (α ) = 0 cos(α ) − sin (α ) , R y (β ) =  0 1 0  , R z (γ ) =  sin (γ ) cos(γ ) 0
 (5)
0 sin (α ) cos(α )  − sin (β ) 0 cos(β )  0 0 1

Finally, the translation matrix is defined by:

t x 
T = t y  (6)
t z 

where tx, ty, tz is the components of the translation matrix.

For the alignment of the camera two rotations are required and one translation as in Eq. (7). This is because the
camera has only movements of pan and tilt.

 xc  xA 
 y  = R (τ ) ⋅ R (ρ ) ⋅  y  + T (7)
 c y z  A
 z c   z A 

According to Fig. 3, this alignment requires first a rotation of angle τ (tilt angle of the camera) around the axis YC in
counter-clockwise and a second rotation, of angle ρ (pan angle of the camera) around the axis ZC in counter-clockwise,
as in Eq. (7).

4.2. Arctan controller

This nonlinear lateral control strategy developed in Frew (2004) supposes that the speed of the aircraft is constant.
The goal is to bring the aircraft from point Wp1 to point Wp2 located on the runway, as shown in Fig. 4.
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

Figure 4. Arctan Controller.

Given the aircraft's current position (Wp1) and the next waypoint (Wp2), the heading reference ψref can be calculated
from equation:

ψ ref = arctan(− y track , x track ) (8)

where ytrack and xtrack are obtained from the geometry between the roll angle, altitude measured of aircraft and projection
equations (subsection 4.1 and fig. 3).
The measured heading of the aircraft ψm is calculated from the components of its velocity, which obtained from the
UAV model developed in section 2:

ψ m = arctan( y& track , x& track ) (9)

The signal error of the heading is defined by the difference between the reference and measured headings, as in Eq.
(10).

e = ∆ψ = ψ m −ψ ref (10)

This signal is to be brought to zero by the proportional integral control law with the saturation limit of ± 0.2 rad/s.
4.3. Velocity ratio controller

This strategy developed in Niculescu (2001) is also a nonlinear lateral control that assumes the speed of the aircraft
as being constant. As in the previous strategy, the goal is to bring the aircraft to its current position (point A) to the next
waypoint (point Wp2), as shown in Fig. 5.
From the geometry of similarity between the triangles ABC and ADE, this control strategy is based on the
establishment of the position and speed of the vehicle according to the relationship:

x& track y&

= track (11)
k ⋅ xtrack y track

E = k ⋅ xtrack ⋅ y& track − y track ⋅ x& track = 0 (12)

ω CMD = K R ⋅ E = K R (k ⋅ xtrack ⋅ y& track − y track ⋅ x& track ) (13)

The parameter k determines the speed with which the aircraft converges to the reference trajectory and the parameter
KR is determined interactively by simulation until a good tracking trajectory is found without overshoot. For safety, it is
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

imposed a limit of saturation of ± 0.2 rad/s in the angular speed at the reference ωCMD. This variable is integrated for
supply the yaw reference. The values of k = 0.000025 and KR = 0.24 and were found satisfactory for this lateral control
law.

Figure 5. Velocity ratio controller.

5. RESULTS

This section presents the results of simulation for the UAV model developed in section 2 applying the laws of lateral
control presented in subsections 4.2 and 4.3. The UAV first is in equilibrium condition, when the signals are then
applied as reference. The variables to be controlled by the autopilot are speed, lateral acceleration, yaw angle and pitch
angle (inner loop).

5.1. Arctan controller

The Figure 6 shows the signs of reference for the MIMO controller with feedback and feedforward signals and the
measured output in the UAV model shown in section 2 (see Fig. 1) for a simulation horizon of 8000 steps. The UAV is
in the condition of equilibrium up to 1000 steps of simulation, when the lateral and altitude controls (Fig. 7) are then
applied. The reference linear velocity is 12 m/s, and lateral acceleration reference is 0 m/s2.

12.5 0.15

12

0.1
11.5
Lateral Acceleration (m/s 2)

11
0.05
Velocity (m/s)

10.5

0
10

9.5
-0.05
9 V(k) ay (k)
Vref (k) ayref (k)
8.5 -0.1
0 1000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000
Multiples of the sampling period T=0.033 Multiples of the sampling period T=0.033

a. b.
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

1.4 0.6
θ(k) ψ(k)
1.2 θref (k) ψref (k)
0.5

1
0.4
0.8
Pitch angle (rad)

Yaw angle (rad)

0.6 0.3

0.4 0.2

0.2
0.1
0

0
-0.2

-0.4 -0.1
0 1000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000
Multiples of the sampling period T=0.033 Multiples of the sampling period T=0.033

c. d.

Figure 6. Variables controlled by the autopilot and its references with the arctan controller. a) Velocity measured and
reference. b) Lateral acceleration measured and reference. c) Pitch angle measured and reference. d) Roll angle
measured and reference.

The reference for the pitch and yaw angles are generated by the external loop of the proportional integral (PI)
guidance type (see Tab. 1). The pitch angle in equilibrium condition is about 1.36 rad. The yaw angle reference is
limited to ± 0.2 rad. This explains why the reference signal keeps the value of 0.2 rad to about 2000 steps of
simulation.

90 220

85 200

Aircraft Trajectory
80 180 Reference Trajectory
Relative Position Y (m)

Actual Altitude
Altitude (m)

75 Reference Altitude 160

70 140

65 120

60 100

55 80
0 1000 2000 3000 4000 5000 6000 7000 8000 0 500 1000 1500 2000 2500 3000
Multiples of the sampling period T=0.033 Relative Position X (m)

a. b.

Figure 7. Altitude and relative position of the aircraft with the arctan controller. a) Altitude measured and reference. b)
Relative trajectory measured and reference.

As can be seen, the error converge to zero. We obtained a good tracking for the pitch angle, and the other variables
reached the reference in less than 3000 steps of simulation after the application of control signals.

Table 1. Gains for the PI controllers with the Arctan controller.

Linear Lateral Pitch Yaw Altitude

Velocity Acceleration Angle Angle Hold
kp 1.6 0.119 2.5 0.119 0.068
ki 0.075 0.00009 0.8 0.001 0.0056

The control signals are defined by: throttle (π (k)) limited between the values from 0 to 1 (dimensionless value), the
rudder deflection (δr (k)) limited to ± 1 rad, the elevator deflection (δe ( k)) limited to ± 0.4 rad and the aileron
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

deflection (δa (k)) limited to ± 1 rad. The parameters kp and ki are the proportional and integral gains of the PI
controllers.

5.2. Velocity ratio controller

Similar to the previous section, the Fig. 8 shows the signs of reference for the MIMO controller with feedback and
feedforward and measured signals in the output of the UAV model for the velocity ratio controller.

12.5 0.03

12 0.02

11.5 0.01

Lateral Acceleration (m/s 2)

11 0
Velocity (m/s)

10.5 -0.01

10 -0.02

9.5 -0.03

9 -0.04

V(k) ay (k)
8.5 -0.05
Vref (k) ayref (k)
8 -0.06
0 1000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000
Multiples of the sampling period T=0.033 Multiples of the sampling period T=0.033

a. b.
1.4 0.35
θ(k) ψ(k)
1.2 θref (k) 0.3 ψref (k)

1
0.25

0.8
0.2
Pitch angle (rad)

Yaw angle (rad)

0.6
0.15
0.4
0.1
0.2

0.05
0

-0.2 0

-0.4 -0.05
0 1000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000
Multiples of the sampling period T=0.033 Multiples of the sampling period T=0.033

c. d.

Figure 8. Variables controlled by the autopilot and its references with the velocity ratio controller. a) Velocity measured
and reference. b) Lateral acceleration measured and reference. c) Pitch angle measured and reference. d) Roll angle
measured and reference.

As can be seen, the error converge to zero. We obtained a good tracking for the pitch angle, and the other variables
reached the reference in less than 3000 steps of simulation after the control signals application. The Figure 9 shows the
altitude obtained for the pitch angle shown in the Fig. 7 and the trajectory of the aircraft due to the technique of lateral
control, respectively.

Table 2. Gains for the P and PI controllers with the velocity ratio controller.

Linear Lateral Pitch Yaw Altitude Angular

Velocity Acceleration Angle Angle Hold Velocity
kp 1.6 0.119 2.5 0.119 0.068 0.000025
ki 0.075 0.00009 0.8 0.001 0.0056 -

Different reference values were used to evaluate the performance of controllers. It is important to express the
performance achieved is not optimal, so it is possible to obtain better results.
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

90 220

85 200

Actual Trajectory
80 180 Reference Trajectory

Relative Position Y (m)

Actual Altitude
Altitude (m)

75 Reference Altitude 160

70 140

65 120

60 100

55 80
0 1000 2000 3000 4000 5000 6000 7000 8000 0 500 1000 1500 2000 2500 3000
Multiples of the sampling period T=0.033 Relative Position X (m)

a. b.

Figure 9 Altitude and position relative of the aircraft with the velocity ratio controller. a) Altitude measured and
reference. b) Relative trajectory measured and reference.

The two techniques for lateral control were implemented in MatLab software in 2 different ways: the package
Simulink blocks (simple and s-functions) and the programming code on file “.m”. Data of time, latitude, longitude,
altitude, roll, pitch and yaw angles are sent to FlightGear via IP address. This simulator has the advantage of having
known scenarios, with several airports available with their surroundings.
Figure 10 shows simulations results for the lateral control of the velocity ratio type. The starting point for the
simulations carried out in Simulink was a position located on top of the entrance of the Electronics Division of
Technological Institute of Aeronautics, with known latitude and longitude. The reference trajectory is given by the
center of the airport runway at São José dos Campos.

Figure 10. Virtual scenario of the aircraft with the velocity ratio controller flying over São José dos Campos airport.

The algorithms were implemented by a “.m” file. Thus, a “.dll” file that captures the screen image of the simulator
is called as a subroutine by MatLab. This dll simulates the camera taking a frame for processing by the vision
algorithm.

5.3. Robustness to Image oise

Due to image noise, the waypoints are not precisely known. The effects of this noise are evaluated by simulations.
For a variance of 5 m in the waypoint positions (3D reference trajectory), the obtained altitude (Fig. 11a) and relative
position (Fig. 11b), for the arctan controller, indicate no substantial difference with respect to the noiseless case.
The same applies to the altitude (Fig. 11c) and relative position (Fig. 11d), for the velocity ratio controller with
noise.
2009 Brazilian Symposium on Aerospace Eng. & Applications 3rd CTA-DLR Workshop on Data Analysis & Flight Control
Copyright © 2009 by AAB September 14-16, 2009, S. J. Campos, SP, Brazil

90 220

85 200

Aircraft Trajectory
80 180 Reference Trajectory

Relative Position Y (m)

Altitude (m)

75 160

70 Actual Altitude 140

Reference Altitude

65 120

60 100

55 80
0 1000 2000 3000 4000 5000 6000 7000 8000 0 500 1000 1500 2000 2500 3000
Multiples of the sampling period T=0.033 Relative Position X (m)

a. b.
90 220

85 200

Actual Trajectory
80 180 Reference Trajectory
Relative Position Y (m)
Altitude (m)

75 160

70 Actual Altitude 140

Reference Altitude

65 120

60 100

55 80
0 1000 2000 3000 4000 5000 6000 7000 8000 0 500 1000 1500 2000 2500 3000
Multiples of the sampling period T=0.033 Relative Position X (m)

c. d.

Figure 11. Altitude and relative position of the UAV with image noise. a) Altitude of the UAV with the arctan
controller. b) Relative position of the UAV with the arctan controller. c) Altitude of the UAV with the velocity ratio
controller. d) Relative position of the UAV with the velocity ratio controller.

6. COCLUSIOS

Techniques of image processing for detection and localization of the center of a track were implemented,and used
for comparing two techniques for lateral control existent in the literature. Therefore, a system for navigation and control
of UAVs based on computational vision was presented and implemented in a virtual scenario, by employing a free
software for flight visualization. The performances of the control strategies are directly related to the PI parameters of
the yaw controller for the arctan controller; and to the parameters k and KR , for the velocity ratio controller.
Future works will address applications for night vision and development of electronics onboard.

7. REFERECES

Frew, E., Mcgee, T., Kim, Z., Xiao, X., Jackson, S., Morimoto, M., Rathinam, S., Padial, J., Sengupta, R., 2004,
“Vision-Based Road-Following Using a Small Autonomous Aircraft”, IEEE Aerospace Conference.
Hemerly, E. M., Neto, N. S. B., Maciel, B. C. O., Goes, L. C. S., 2006, “Identificação e Controle de Veículos
Autônomos Não Tripulados com Asa Fixa”, XVI Congresso Brasileiro de Automática, Salvador-BA, pp. 1231-1236.
Niculescu, M., 2001, “Lateral Track Control Law for Aerosonde UAV”, Proceedings of the 39TH AIAA Aerospace
Sciences Meeting and Exhibit.
Roskan, J., 2001, “Airplane Flight Dynamics and Automatic Flight Controls. Part 1”, Lawrence, KS, DARcorp.

8. RESPOSIBILITY OTICE

The author(s) is (are) the only responsible for the printed material included in this paper.