J Intell Robot Syst(2012)65:295–308
DOI10.1007/s10846-011-9588-y
Vision Based UAV Attitude Estimation: Progress and Insights
Abd El Rahman Shabayek·
Cédric Demonceaux·Olivier Morel·David Fofi
Received:15February2011/Accepted:18April2011/Published online:17August2011©Springer Science+Business Media B.V.2011
Abstract Unmanned aerial vehicles(UAVs)are increasingly replacing manned systems in situa-tions that are dangerous,remote,or difficult for manned aircraft to access.Its control tasks are empowered by computer vision technology.Vi-sual sensors are robustly used for stabilization as primary or at least secondary sensors.Hence, UAV stabilization by attitude estimation from visual sensors is a very active research area.Vision based techniques are proving their effectiveness and robustness in handling this problem.In this work a comprehensive review of UAV vision based attitude estimation approaches is covered, starting from horizon based methods and passing by vanishing points,optical
flow,and stereoscopic based techniques.A novel segmentation approach for UAV attitude estimation based on polariza-A.E.R.Shabayek(B)·C.Demonceaux·
O.Morel·D.Fofi
Le2i-UMR CNRS5158,IUT Le Creusot,
Universitéde Bourgogne,Dijon,France
e-mail:a.shabayek@gmail
C.Demonceaux
e-mail:Cedric.Demonceaux@u-bourgogne.fr
O.Morel
e-mail:Olivier.Morel@u-bourgogne.fr
D.Fofi
e-mail:David.Fofi@u-bourgogne.fr tion is proposed.Our future insightes for attitude estimation from uncalibrated catadioptric sensors are also discussed.
Keywords UAV·Attitude estimation·Vision·Polarization·Catadioptric·Horizon·Segmentation·Vanishing points·Optical flow·Stereoscopic
1Introduction
In order to determine the pose of the vehicle accu-rately and rapidly,the regular approach is to use inertial sensors with other sensors and applying sensor fusion.Some sensors used for this purpose are the Global positioning sensor(GPS),inertial navigation sensor(INS),as well as other sensors such as altitude sensors(ALS)and speedometers. These sensors have some limitations.GPS sensor for example,is not available at some locations or readings subject to error.INS has the disadvan-tage of accumulation of errors.To overcome these limitations,vision-based navigation approaches have been developed.These approaches can be used where GPS or INS systems are not available or can be used with other sensors to obtain better estimations.UAV attitude estimation has been deeply studied in terms of data fusion of multiple low cost sensors in a Kalman filter(KF)frame-work to have the vehicle full state of position and
orientation.But in pure vision based methods,if a horizontal world reference is horizon) the camera attitude can be obtained.
In order to control a flying vehicle at least six parameters(pose of the vehicle)should be known; Euler angles representing the orientation of the vehicle and a vector of coordinates,representing the position of the vehicle.Pose estimation ba-sically depends on viewing a world unchanging physical landmarks on the ground) for accurate estimation.Our main concern in this work is to review the work that focuses on atti-tude(roll,pitch,and yaw angles shown in Fig.1) estimation rather than pose estimation.
In a typical flight,the demand for yaw angle will be largely constant and hence disturbances tend to have a relatively small effect on yaw.Further, small steady state errors are normally acceptable since(unlike roll and pitch)any errors will have no further effect on the UAV motion.Therefor, for the sake of UAV stabilization,the most impor-tant angles to be estimated are the pitch and roll angles as most of the work in literature propose.In this work,the focus will be on attitude estimation from perspective and omnidirectional cameras.It is intended to give a complete review with some views to enhance current work and propose novel ideas under investigation and development by our research
group.
Fig.1An illustrative sketch of the attitude(roll,pitch,and yaw angles)1.1Vision Sensors for Attitude Estimation Vision based methods were first introduced by [1].They proposed to equip a Micro Air Vehi-cle(MAV)with a perspective camera to have a vision-guided flight stability and autonomy sys-tem.Omnidirectional sensors for attitude estima-tion were first introduced by[2].The omnidirec-tional sensors(fisheye and catadioptric cameras shown in Fig.2)were used in different scenarios. Catadioptric sensors are commercially available for reasonable prices.A catadioptric sensor has two
main parts,the mirror and the lens.The lens could be telecentric or perspective.The sensor is in general assembled as shown in Fig.2c.
Omnidirectional sensors were used alone or in stereo configurations.Omnidirectional vision presents several advantages:(a)a complete sur-rounding of the UAV can be captured and the horizon is totally visible,(b)possible occlusions will have lower impact on the estimation of the final results,(c)whatever the attitude of the UAV, the horizon is always present in the image,even partially,and the angles can always be computed,
(a) Perspective(b) Fisheye
(c) Catadioptric
Fig.2Perspective and omnidirectional(fisheye and cata-dioptric)cameras
Fig.3Horizon in a a perspective image,b a non-central catadioptric image
(a) Perspective
(b) Non-central catadioptric
and (d)it is also possible to compute the roll and pitch angles without any prior hypothesis,con-trary to the applications using a perspective cam-era.Yet,catadioptric vision also presents some drawbacks.For example,(a)a catadioptric im-age contains significant deformations due to the geometry of the mirror and to the sampling of the camera,and (b)catadioptric cameras should be redesigned to a lower scale to be attached to a micro air vehicle (MAV).
1.2The Main Techniques for Attitude
Estimation
In literature,the first group of methods tries to detect a horizontal reference frame in the world to estimate the up direction and hence the at-titude of the vehicle.The horizon,if visible,is the best natural horizontal reference to be used [1].However,in urban environments the horizon might not be visible.Hence,the second group tries to find the vanishing points from parallel vertical and horizontal l
ines which are basic features of man made structure (e.g.[3]).The third group was biologically inspired from insects,it employs the UAV motion (optical flow)for the sake of required estimation [4].Stereo vision based tech-niques came to the play to open the door for more accurate estimation specially if combined with optical flow (e.g.[5]).All these techniques will be discussed in the following sections.
Most of the employed techniques in literature use the Kalman filter (KF)or one of its varia-tions in order to obtain an accurate and reliable estimation specially if more than one sensor is
reference group
used and their measurements are fused.For a general parameter estimation issue,the extended Kalman filter (EKF)technique is widely adopted.Due to the processing of EKF in a linear manner,it may lead to sub-optimal estimation and even filter divergence.Nevertheless,state estimation using EKF assumes that both state recursion and covariance propagation are Gaussian.Unscented Kalman filter (UKF)resolves the nonlinear para-meter estimation and machine learning problems.It can outperform the EKF especially for those highly nonlinear system dynamics/measurement processes.None of the Jacobean or derivatives of any functions are taken under the UKF processing [6].For example in [7],using an EFK,the can-didate horizon lines are propagated and tracked through successive image frames,with statistically unlikely horizon candidates eliminated.In [8],they followed
the EKF framework to combine inertial and visual sensor for real time attitude estimation.They have designed a KF for image line measurements.1.3Paper Organization
The paper will be organized as follows:Sections 2,3and 4will review the general techniques for at-titude estimation from visual sensors (perspective and omnidirectional only)in detail.In Section 2,horizon detection algorithms will be briefly ex-plained and reviewed.Vanishing points based techniques are reviewed in Section 3.The classical and hybrid approaches using stereo-vision and optical flow are reviewed in Section 4.Finally we conclude in Section 5.
2Horizon Detection
The visual sensor is not only a self-contained and passive like an INS but also interactive with its en-vironment.An absolute attitude can be provided by detecting a reliable world reference frame. Attitude computation by vision is based on the detection of the horizon,which appears as a line in perspective images or a curve in omnidirectional images as shown in Fig.3,and on the estimation of the angle between the horizon and a horizontal reference.
Due to the difficulty in obtaining ground-truth for aircraft attitude,most of the work in literature do not provide a quantitative measure of error in their estimates of roll and pitch.In[9],they provided a compl
exity and performance compar-ison between their method and other methods in literature.They have included a comparison table of execution times for various published studies on visual attitude estimation.
In the following subsections,we will cover in detail the different segmentation approaches for horizon detection in Section2.1,a proposal to seg-ment using polarization in Section2.2,and both the perspective and omnidirectional scenarios will be reviewed.Section2.3will briefly discuss hori-zon estimation and attitude computation in the perspective case.Section2.4will briefly discuss the same in the omnidirectional case specially in the catadioptric scenario which is frequently used.
2.1Sky/Ground Segmentation
As the segmentation of sky and ground is a cru-cial step toward extracting the horizon line/curve, which is used for attitude estimation,these seg-mentation methods will be discussed here.
Using perspective vision,algorithms employ-ing Gaussian assumptions for sky/ground seg-mentation fails in scenarios where the underlying Gaussian assumption for the sky and ground ap-pearances is not appropriate[1].These assump-tions might be enhanced by a statistical image modeling framework by building prior models of the sky and ground then trained.Since the appear-ances of the
sky and ground vary enormously,no single feature is sufficient for accurate modeling;as such,these algorithms rely both on color and texture as critical features.They may use hue and intensity for color representation,and the com-plex wavelet transform for texture representation. Then they may use Hidden Markov Tree models as underlying statistical models over the feature space[10].In[7],the algorithm is based on de-tecting lines in an image which may correspond to the horizon,followed by testing the optical flow against the measurements expected by the motion filter.
Using omnidirectional vision,some algorithms use Markovian formulation of sky/ground seg-mentation based on color information[2],or the sky/ground partitioning is done in the spher-ical image thanks to the optimization of the Mahalanobis distance between these regions.The search for points in either regions takes place in the RGB space[11].In order to isolate the sky from the ground[12,13],an approach based on the method employed by[14]weights the RGB components of each pixel using the function f(RGB)=3B2/(R+G+B).
In[9],they propose an algorithm which can be incorporated into any vision ar-row angle,wide angle or panoramic),irrespective of the way in which the environment is imaged (e.g.through lenses or mirrors).The proposed horizon detection method consists of four stages: (a)enhancing sky/ground contrast,(b)determin-ing optimum threshold for sky and ground seg-mentati
on,(c)converting horizon points to vec-tors in the view sphere,and(d)fitting3D plane to horizon vectors to estimate the attitude.
In[15]they proposed segmentation using tem-perature from thermopile sensors in the thermal infrared band.However,in this work,the focus will be on attitude estimation from perspective and omnidirectional sensors only.
The previous segmentation solutions are either complex and/or time consuming.A method based on polarization for segmentation in Section2.2 is proposed.We believe it will have significant enhancements in both complexity and time due to its simplicity.We propose a novel non-central catadioptric sensor where the mirror is a free-form shape and the camera is FD-1665P Polarization Camera[16])to be used for attitude estimation.
2.2Polarization Based Segmentation
Instead of using color information or edge de-tection algorithms for segmentation which may require different complex models and offline processing as shown,we propose to use polar-ization information which exists in the surround-ing nature.Polarization information are directly computed from three intensity images taken at three different angles of a linear polarization filter (0,45,and 90◦)
or at one shot using a polarimetric camera.
Using polarization for segmentation is not new.It was used for rough surface segmentation [17],material classification [18],water hazards detec-tion for autonomous off-road navigation [19],and similar applications.However,to the best of our knowledge,it is the first time to propose using po-larization for sky/ground segmentation for UAV attitude estimation.
The most important polarization information are phase (angle)and degree.According to [18],the phase of polarization is computed as follows:
θ=0.5∗tan −1 I 0+I 90−2I 45
I 900
+90(1)
if I 90<I 0
if I 45<I 0θ=θ+90else
θ=θ−90
and the degree of polarization is:φ=
I 90−I 0
(I 90+I 0)∗cos (2θ)
(2)
where I 0,I 45,and I 90are intensity images taken at 0,45,and 90◦of the rotating polarizer respec-tively (or at one shot from a polarimetric camera).Figure 4shows the segmentation results for non-central catadioptric images with the horizon detected by simply detecting the transition area.This technique is very simple and can be opti-mized by kind of binary search in the image having very rapid and robust results for the detected horizon in the image.Only few regions of the image are needed to be inspected for their degree or angle of polarization to decide for the search di-rection.Unlike conventional segmentation meth-ods,thanks to polarization,we do not face the
illumination problem caused by the sun being in the image.
In future work,we will provide detailed algo-rithms with complexity and run time comparison with other methods found in literature.2.3Using Perspective Sensors
The horizon is projected as a line in the perspec-tive image.Intuitively,it is required to extract that line.Most methods first segment the image into sky/ground areas,then take the separating points as the horizon line.The attitude is dependant on the gradient of that horizon line on the image plane.In literature,the general approach is to find the normal to the plane of the horizon in order to estimate the roll and pitch angles.The normal vector has direct mathematical relation with the attitude as expressed in different methods.The work done by [20,21]are examples of successful autonomous control of a MAV based on attitude estimation from the horizon detected.
In literature,horizon detection problem has been addressed by segmentation and edge detec-tion.In [1,22]they proposed to equip a MAV with a perspective camera to have a vision-guided flight stability and autonomy system.They de-tected the horizon by extracting the straight line that separates the sky from the ground using the context difference of the two regions.In [10]they treated the horizon detection problem as a subset of image segmentation and object recognition,and used a percentage of the sky seen as an error signal to a flight stability controller on a MAV.The resulting system was stable enough to be safely flown by an untrained operator in real time.In contrast,[20]uses a direct edge-detection technique,followed by automatic threshold and a Hough-like algorithm to generate a “projection statistic”for the horizon.It claims a 99%success rate over several hours of video.Important
ly,it deals only with detection,not estimation of at-titude.In [7]they propose an algorithm slightly similar to [20]in that it uses an edge detection technique followed by a Hough transform.How-ever,they propose different image pre-filtering.In [14,23–25]they use the centroids of sky and ground to extract the horizon and derive the different angles.They try to simplify their work by
(a) 0 degree(b) 45 degree(c) 90 degree
(d) Segmentation based on the  degree of polarization (e) Segmentation based on the
angle of polarization
(f) Extracted horizon curve
Fig.4Sky/ground segmentation and horizon extraction based on polarization from non-central catadioptric images
using a circular mask to reduce image asymmetry and to simplify the calculations.
2.4Using Omnidirectional Sensors
The use of a single perspective camera generates several drawbacks.Firstly,a partial view of the environment and important occlusions in the hori-zon can have a serious influence on the final result. Secondly,the horizon is visible only in a particular interval of roll and pitch values.If the UAV gets out of this interval,the final image is exclusively made of sky or earth and the horizon cannot be detected.Thirdly,it is only possible to compute the roll angle while the pitch is only approximated thanks to a hypothesis on the altitude of the UAV. All that pushed the need toward employing om-nidirectional sensors to capture the horizon in al-most all scenarios.The horizon appears as a curve in the omnidirectional image.It is common to use both fisheye and central catadioptric sensors. As both are treated by the equivalence sphere theory proposed by[26].The particular geometric characteristics of the catadioptric sensor will be briefly explained in the next section.Once the horizon is detected,these characteristics are used to compute the attitude of the UAV.
2.4.1Central Catadioptric Projection
of the Horizon
As demonstrated in[26],a3D sphere projects on the equivalence sphere in a small circle,and then on the catadioptric image plane in an ellipse(see Fig.5).Consequently,the attitude computation is based
on searching for an ellipse in the omnidi-rectional image or a small circle on the equivalent sphere which corresponds to the horizon.The geometrical properties of the equivalent sphere allow to deduce the roll and pitch angles.Indeed, the normal of the projected horizon on the sphere, which is also confounded with the line passing through the center of the sphere of equivalence and through the center of the earth represents in fact the attitude of the UAV depending on the position of the optical axis.Then,the computation of the coordinates of the optical axis is sufficient in order to deduce the roll and pitch angles.

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。