A survey on motion prediction and risk assessment for intelligent vehicles

Abstract

With the objective to improve road safety, the automotive industry is moving toward more “intelligent” vehicles. One of the major challenges is to detect dangerous situations and react accordingly in order to avoid or mitigate accidents. This requires predicting the likely evolution of the current traffic situation, and assessing how dangerous that future situation might be. This paper is a survey of existing methods for motion prediction and risk assessment for intelligent vehicles. The proposed classification is based on the semantics used to define motion and risk. We point out the tradeoff between model completeness and real-time constraints, and the fact that the choice of a risk assessment method is influenced by the selected motion model.

Introduction

Safety is a cornerstone of both Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles. A closely related concept is that of risk, which can be intuitively understood as the likelihood and severity of the damage that a vehicle of interest may suffer in the future. From this definition it is clear that, in order to assess the risk associated with a particular situation, it is necessary to have mathematical models which allow us to predict how this situation will evolve in the future. This paper surveys such models and their relation with risk assessment.

We have chosen to organize motion modeling and prediction approaches according to the kind of hypotheses they make about the modeled entities. As depicted in Figure 1, we propose a classification in three levels with an increasing degree of abstraction:

• 1. Physics-based motion models are the simplest models, they consider that the motion of vehicles only depends on the laws of physics. They are described in Section ‘Physics-based motion models’.

Figure 1

Motion modeling overview.

• 2. Maneuver-based motion models are more advanced as they consider that the future motion of a vehicle also depends on the maneuver that the driver intends to perform. These models are presented in Section ‘Maneuver-based motion models’.

• 3. Interaction-aware motion models take into account the inter-dependencies between vehicles’ maneuvers. Only a few examples of such advanced representations may be found in the literature. They are reviewed in Section ‘Interaction-aware motion models’.

The differences between the three families of motion models are illustrated by Figure 2. In these examples, the Physics-based motion model assumes a constant speed and orientation for the cars, the Maneuver-based motion model assumes that the black car goes straight and the blue car turns left, the Interaction-aware motion model assumes that the black car goes straight, that the blue car turns left and that the joint motion of the cars is constrained by the traffic rules.

Figure 2

Examples of motion prediction with the different types of motion models.

Section ‘Risk assessment’ deals with a different aspect of the problem: how to apply these motion models for risk estimation. Since the actual meaning of the word “risk” is not the same for all these approaches, we propose to classify existing approaches into two broad families. The first one considers only the risk associated with physical collisions between entities. The second family of approaches introduces the idea that risk is related to vehicles behaving differently from what is expected of them given the context (e.g. according to traffic rules).

Physics-based motion models

Physics-based motion models represent vehicles as dynamic entities governed by the laws of physics. Future motion is predicted using dynamic and kinematic models linking some control inputs (e.g. steering, acceleration), car properties (e.g. weight) and external conditions (e.g. friction coefficient of the road surface) to the evolution of the state of the vehicle (e.g. position, heading, speed).

Extensive work has been done on such Physics-based motion models for vehicles, and they remain the most commonly used motion models for trajectory prediction and collision risk estimation in the context of road safety. The models are more or less complex depending on how fine the representation of the dynamics and kinematics of a vehicle is, how uncertainties are handled, whether or not the geometry of the road is taken into account, etc.

This section is divided into three parts. In Section ‘Evolution models’, the most standard evolution models are described. In Section ‘Trajectory prediction’, a review is provided of the different methods for predicting trajectories using these evolution models. Finally, the limitations of Physics-based motion models are addressed in Section ‘Limitations’.

Evolution models

Dynamic models

Dynamic models describe motion based on Lagrange’s equations, taking into account the different forces that affect the motion of a vehicle, such as the longitudinal and lateral tire forces, or the road banking angle [[1]]. Car-like vehicles are governed by complex physics (effect of driver actions on the engine, transmission, wheels etc.), therefore dynamic models can get extremely large and involve many internal parameters of the vehicle. Such complex models are relevant for control-oriented applications, but for applications such as trajectory prediction simpler models are preferred. They are often based on a “bicycle” representation, which represents a car as a two-wheeled vehicle with front-wheel drive moving on a 2-D plane. Examples of such simple dynamic models are found in several works [[2]–[7]].

Kinematic models

Kinematic models describe a vehicle’s motion based on the mathematical relationship between the parameters of the movement (e.g. position, velocity, acceleration), without considering the forces that affect the motion. The friction force is neglected, and it is assumed that the velocity at each wheel is in the direction of the wheel [[1]]. Kinematic models are far more popular than dynamic models for trajectory prediction because they are much simpler and usually sufficient for this type of applications (i.e. applications which are not vehicle control). In addition the internal parameters of a vehicle needed by dynamic models are not observable by exteroceptive sensors, which rules out the use of dynamical models for ITS applications involving other vehicles than the ego-vehicle. A survey of kinematic models for car-like vehicles was done by Schubert et al. [[8]]. The simplest of these are the Constant Velocity (CV) and Constant Acceleration (CA) models, which both assume straight motion for vehicles [[9]–[13]]. The Constant Turn Rate and Velocity (CTRV) and Constant Turn Rate and Acceleration (CTRA) models take into account the variation around the z-axis by introducing the yaw angle and yaw rate variables in the vehicle state vector [[10],[12],[14]–[17]]. The complexity remains low as the velocity and yaw rate are decoupled. By considering the steering angle instead of the yaw rate in the state variables, one obtains a “bicycle” representation, which takes into account the correlation between the velocity and the yaw rate. From this representation, the Constant Steering Angle and Velocity (CSAV) and the Constant Steering Angle and Acceleration (CSAA) can be derived.

Trajectory prediction

The evolution models described above can be used for trajectory prediction in various ways, the main difference being in the handling of uncertainties.

Single trajectory simulation

A straightforward manner to predict the future trajectory of a vehicle is to apply an evolution model (see Section ‘Evolution models)’ to the current state of a vehicle, assuming that the current state is perfectly known and that the evolution model is a perfect representation of the motion of the vehicle. This strategy can be used with dynamic models [[2]] or kinematic models [[11],[13],[17]], and is illustrated in Figure 3. The advantage of this single forward simulation is its computational efficiency, which makes it suitable for applications with strong real-time constraints. However the predictions do not take into account the uncertainties on the current state nor the shortcomings of the evolution model, and as a result the predicted trajectories are not reliable for long term prediction (more than one second).

Figure 3

Trajectory prediction with a constant velocity motion model (based on [[13]]).

Gaussian noise simulation

Uncertainty on the current vehicle state and on its evolution can be modeled by a normal distribution [[9],[10],[12],[16]]. The popularity of the “Gaussian noise” representation of uncertainty is due to its use in the Kalman Filter (KF). Kalman filtering is a standard technique for recursively estimating a vehicle’s state from noisy sensor measurements. It is a special case of Bayesian filtering where the evolution model and the sensor model are linear^a, and uncertainty is represented using a normal distribution. In a first step (prediction step) the estimated state at time t is fed to the evolution model, resulting in a predicted state for time t+1 which takes the form of a Gaussian distribution. In a second step (update step) the sensor measurements at time t+1 are combined with the predicted state into an estimated state for time t+1, which is also a Gaussian distribution. Looping on the prediction and update steps each time a new measurement is available is called filtering^b.

By looping on the prediction step, one can obtain for each future timestep a mean and covariance matrix for the state of the vehicle, which can be transformed into a mean trajectory with associated uncertainty (normal distribution at each timestep) [[9],[15],[16]]. This process is illustrated in Figure 4.

Figure 4

Trajectory prediction with a constant velocity motion model and Gaussian noise simulation. Ellipses represent the uncertainty on the predicted positions (based on [[9]]).

Compared to the “single trajectory simulation” approaches, these techniques have the advantage that they represent the uncertainty on the predicted trajectory. However they suffer form a similar limitation in the sense that modeling uncertainties using a unimodal normal distribution is insufficient to represent the different possible maneuvers. A solution to this problem is to represent uncertainty using mixtures of Gaussians. Switching Kalman Filters (SKF) [[18]] can be used for this purpose. They rely on a bank of Kalman Filters to represent the possible evolution models of a vehicle and switch between them [[10],[19],[20]]. An alternative to the SKF is to use heuristics to switch between the different kinematic models depending on the situation [[12]].

Monte Carlo simulation

In the general case, i.e. when no assumption is made on the linearity of the models or on the Gaussianity of the uncertainties, the analytical expression for the distribution on the predicted states is usually not known. Monte Carlo methods provide tools to approximate this distribution. The idea is to randomly sample from the input variables of the evolution model in order to generate potential future trajectories. In order to take into account the road topology, weights can be applied to the generated trajectories to penalize the ones which do not respect the constraints of the road layout. The evolution models described in Section ‘Evolution models’ can be used for Monte Carlo simulation by sampling on the inputs instead of considering them to be constant. Typical inputs to be sampled from are the acceleration and steering angle or lateral deviation. In order to take into account the feasibility of a maneuver, one can either remove the generated trajectory samples with a higher lateral acceleration than what is physically allowed [[21]], or take into account the physical limitations of a vehicle in the evolution model so that the inputs are distributed in a more realistic manner and the post-processing step for removing unfeasible trajectories is not needed [[6],[22]]. Example predicted trajectories are displayed in Figure 5. Monte Carlo simulation can be used to predict a vehicle’s trajectory either from a perfectly known current state or from an uncertain current state estimated by a filtering algorithm.

Figure 5

Trajectory prediction (yellow arrows) with Monte Carlo simulation (e.g. [[21]]).

Limitations

Since they only rely on the low level properties of motion (dynamic and kinematic properties), Physics-based motion models are limited to short-term (less than a second) motion prediction. Typically, they are unable to anticipate any change in the motion of the car caused by the execution of a particular maneuver (e.g. slow down, turn at constant speed, then accelerate to make a turn at an intersection), or changes caused by external factors (e.g. slowing down because of a vehicle in front).

Maneuver-based motion models

Maneuver-based motion models represent vehicles as independent maneuvering entities, i.e. they assume that the motion of a vehicle on the road network corresponds to a series of maneuvers executed independently from the other vehicles. Here, a maneuver is defined as “a physical movement or series of moves requiring skill and care” [[23]]. The word “behavior” is sometimes used in the literature for the same purpose [[24]–[28]], but for the sake of clarity the word “maneuver” will be used throughout this paper. Trajectory prediction with Maneuver-based motion models is based on the early recognition of the maneuvers that drivers intend to perform. If one can identify the maneuver intention of a driver, one can assume that the future motion of the vehicle will match that maneuver. Thanks to this a priori, trajectories derived from this scheme are more relevant and reliable in the long term than the ones derived from Physics-based motion models. Maneuver-based motion models are either based on prototype trajectories or based on maneuver intention estimation. Section ‘Prototype trajectories’ introduces prototype trajectories and how they can be used for trajectory prediction. Section ‘Maneuver intention estimation and maneuver execution’ covers approaches based on maneuver intention estimation. Finally, the limitations of Maneuver-based motion models are analyzed in Section ‘Limitations’.

Prototype trajectories

The idea is that the trajectories of vehicles on the road network can be grouped into a finite set of clusters, each cluster corresponding to a typical motion pattern. Example clusters are displayed in Figure 6.

Figure 6

Clustered trajectories: each cluster is represented by a color and corresponds to a typical motion pattern (e.g. [[29]]).

Motion patterns are represented using prototype trajectories which are learned from data during a training phase. Subsequently prediction can be performed online given a partial trajectory by finding the most likely motion pattern(s) and using the prototype trajectories as a model for future motion.

Representation

Motion patterns are represented using prototype trajectories which are learned from sample (previously observed) trajectories. Because the road network is a structured environment, it is generally assumed that the motion patterns can be identified in advance (they can for example be extracted from a digital map, by identifying all the possible maneuvers at a given location). In this case no clustering process is needed, i.e. each trajectory in the training dataset is already assigned to a cluster^c.

Starting from there, several possibilities exist for representing a motion pattern based on the sample trajectories. One solution is to compute a unique prototype trajectory for each motion pattern, by agglomerating the previously observed trajectories. For example, a stochastic representation of a motion pattern can be derived by computing the mean and standard deviation of the sample trajectories [[30]]. Another way to account for the variations in the execution of a motion pattern is to have several prototypes for each class, e.g. a subset of the training samples [[31]]. A different approach was proposed in [[32]], where the different behaviors were not represented by individual prototypes, but merged in a single graph structure, learned online using a Topology Learning Network.

More recently, several works showed that Gaussian Processes (GPs) are well-suited for representing motion patterns in the context of road traffic [[25],[33]–[35]]. GPs can be seen as a generalization of Gaussian probability distributions. They model a process as a Gaussian distribution over a function. When applied in the context of vehicle trajectories, the assumption is that the trajectories in the learning dataset are sample functions from a Gaussian Process. Therefore the learning consists in fitting a Gaussian distribution over these functions. The main advantages of GPs are their robustness to noise in the observed trajectories (compared to the approaches presented above) and their ability to represent the variations in the execution of a motion pattern in a consistent and probabilistic manner. For example, a GP featuring the function f(t)=(x,y), with t the time and (x,y) the 2D coordinates of the vehicle will be able to partially account for variations in the speed of execution of a maneuver [[25]]. An alternative is to use the function$f(x,y)=(\frac{\mathrm{\Delta x}}{\mathrm{\Delta t}},\frac{\mathrm{\Delta y}}{\mathrm{\Delta t}})$, which maps locations to velocities and therefore removes any time-dependent aspect from the model [[33],[34]]. The variability in the velocity due to varying traffic conditions is captured in the covariance function of the GP. However this ability to represent the inherent variations of a pattern comes at a price; a naive implementation of a GP has a complexity of O(n³), where n is the number of training sample points.

Trajectory prediction

Starting from the partial trajectory executed by a vehicle so far, prediction can be performed by comparing it with the learned motion patterns, selecting the most likely one(s) and using the prototype trajectories as a model for future motion. First, metrics need to be defined to measure the distance of a partial trajectory to a motion pattern. When motion patterns are represented by Gaussian Processes, the distance is computed as the probability that the partial trajectory corresponds to the GP, by integrating over the possible futures of the trajectory [[25],[33],[34]]. When motion patterns are represented by a finite set of prototype trajectories, the distance of a partial trajectory to a motion pattern is measured by its similarity with the prototype trajectories. A number of metrics have been defined to measure the similarity between two trajectories, including the average Euclidian distance between points of the trajectories [[36]], the modified Hausdorff [[29]], the Longest Common Subsequence (LCS) [[37]] and its translation-and-rotation invariant version the Quaternion-based Rotationally Invariant LCS [[31]].

The simplest solution to predict future motion after the distance to each motion pattern has been computed is to select the most likely motion pattern and to use it as a unique model [[36]]. Alternatively it is possible to consider a mixture of motion patterns: a probability distribution over the different motion patterns is computed, and then either the different motion models are combined (weighted) into one [[33]], or a set of potential future trajectories are generated with associated weights [[31],[34]]. Another solution proposed recently is to use a Hierarchical Mixture of Experts (HME) [[38]], which has the advantage that categorical variables can be handled (e.g. turn signal, class of road layout).

Limitations

For a long time, the main limitation of prototype trajectories was their strictly deterministic representation of time. Indeed, when motion patterns are represented using a finite set of trajectories it would take a very large number of prototypes to model the large variation in the execution of a motion pattern. In order to recognize maneuvers involving a waiting period at a stop line for example, one has to resort to hard thresholds to identify waiting intervals and ignore them when computing the distance between two trajectories. Handling more subtle variations in velocity like the ones caused by heavy traffic is still an issue for such models. To a certain extent, the introduction of Gaussian Processes solved this problem by allowing a time-independent representation of motion patterns [[33],[34]]. However GPs suffer from other limitations. In addition to their heavy computational burden, they lack the ability to take into account the physical limitations of a vehicle and therefore may generate unrealistic sample trajectories. To the best of our knowledge the only solution to this problem proposed in the literature is the one introduced by Aoude et al. [[34]], which uses a Rapidly-exploring Random Tree (RRT) algorithm to randomly sample points toward dynamically feasible trajectories, using as inputs the current state of the vehicle and the sample trajectories generated by the GPs.

Another difficulty when using prototype trajectories is their adaptation to different road layouts, in particular when applied to road intersections. Because each motion model is trained for a specific intersection geometry and topology, they only can be reused at intersections with a similar layout.

Maneuver intention estimation and maneuver execution

An alternative to trajectory prototypes is to first estimate the maneuver intention of the driver (e.g. waiting at the stop line, following another vehicle, executing a left turn) and then predict the successive physical states so that they correspond to a possible execution of the identified maneuver. A major advantage over trajectory prototypes is that there is no need to match the partial trajectory with a previously observed trajectory. Instead, higher-level characteristics are extracted and used to recognize maneuvers, which makes it easier to generalize the learnt model to arbitrary layouts.

Maneuver intention estimation

Many cues can be used to estimate the maneuver intention of a driver, for example the physical state of the vehicle (position, speed, heading, acceleration, yaw rate, turn signal, etc.), information about the road network (geometry and topology of the road, speed limit, traffic rules, etc.), driver behavior (head movement, driving style, etc.). Maneuver intention estimation has been investigated by many works. This survey focuses on maneuver intention estimation at road intersections, but most of the approaches presented here can be applied to other traffic situations.

Context and heuristics can be used to determine what maneuvers are likely to be performed in the near future in a deterministic manner [[39]]. For classifying maneuvers in more complex scenarios, discriminative learning algorithms are very popular, such as Multi-Layer Perceptrons (MLP) [[28]] Logistic regression [[40]], Relevance Vector Machines (RVM) [[41]], or Support Vector Machines (SVM) [[42]–[44]]. An equally popular alternative is to break down each maneuver into a chain of consecutive events and to represent this sequence of events using a Hidden Markov Model (HMM). The transition probabilities between the different events can be learned from data, as well as the observation model (i.e. the relationship between the non-observable events and the available observations). For a new sequence of observations, the maneuver intention is estimated by comparing the likelihood of the observations for each HMM [[24],[25],[45]–[47]].

A comparative review of works on maneuver intention estimation is provided in Table 1.

Table 1 Comparative review of works on maneuver intention estimation at road intersections

Maneuver execution

Trajectories are predicted so that they match the identified maneuver(s). This can be done in a deterministic manner, by deriving the input controls corresponding to the recognized maneuver and then generating a single trajectory from a kinematic motion model [[48]]. One issue with this strictly deterministic approach is that it cannot take into account uncertainties on the current vehicle state, on the maneuver which is being performed or on the execution of this maneuver. In order to explore the space of the potential executions of a maneuver in a probabilistic manner, GPs or RRTs can be used. A GP can be learned for each maneuver from training data and used in a generative manner to create sample trajectories for each maneuver [[25],[49]]. Alternatively, a RRT tree can be grown by sampling points in the input space of the vehicle’s evolution model, applying a bias in the sampling according to the estimated maneuver intentions [[42]]. This method has the advantage that it always generates dynamically feasible trajectories for a maneuver. As an alternative to trajectories, reachable sets can be used to represent the future motion of vehicles. This representation can be stochastic [[50]] or geometric [[39]] (see Figure 7).

Figure 7

Maneuver execution using three different approaches. (a) RRT (adapted from [[42]]) (b) GP (adapted from [[49]]) (c) Stochastic reachable sets (adapted from [[50]]).

Limitations

In practice, the assumption that vehicles move independently from each other does not hold. Vehicles share the road with other vehicles, and the maneuvers performed by one vehicle will necessarily influence the maneuvers of the other vehicles. Inter-vehicle dependencies are particularly strong at road intersections, where priority rules force vehicles to take into account the maneuvers performed by the other vehicles. Disregarding these dependencies can lead to erroneous interpretations of the situations, and affects the evaluation of the risk.

Interaction-aware motion models

Interaction-aware motion models represent vehicles as maneuvering entities which interact with each other, i.e. the motion of a vehicle is assumed to be influenced by the motion of the other vehicles in the scene. Taking into account the dependencies between the vehicles leads to a better interpretation of their motion compared to the Maneuver-based motion models described in the previous section. As a result, it contributes to a better understanding of the situation and a more reliable evaluation of the risk. Despite this, there are few Interaction-aware motion models in the literature. They are based either on prototype trajectories or on Dynamic Bayesian Networks. These solutions are presented in Sections ‘Models based on trajectory prototypes’ and ‘Models based on Dynamic Bayesian Networks’, followed by the limitations of Interaction-aware motion models.

Models based on trajectory prototypes

For methods relying on trajectory prototypes, inter-vehicle influences cannot be taken into account during the learning phase because the resulting number of motion patterns would quickly become intractable. However, it is possible to take into account the mutual influences during the matching phase by assuming that drivers have a strong tendency to avoid collisions when they can [[51],[52]]. Pairs of trajectories which lead to an unavoidable collision are penalized in the matching process, and as a result safe trajectories are always considered to be more likely than hazardous ones. This approach is an elegant workaround for taking into account inter-dependencies when using trajectory prototypes. However the issue of modeling other types of influences remains, since the influence of one vehicle on the trajectory of another cannot be modeled directly.

Models based on Dynamic Bayesian Networks

Most Interaction-aware motion models are based on Dynamic Bayesian Networks (DBN).

Pairwise dependencies between multiple moving entities can be modeled with Coupled HMMs (CHMMs) [[53]]. However, since the number of possible pairwise dependencies grows quadratically with the number of entities, the complexity is not manageable in the context of complex traffic situations. A solution to simplify the model is to make CHMMs asymmetric by assuming that the surrounding traffic affects the vehicle of interest, but not vice versa [[54]]. The assumption of asymmetric dependencies greatly reduces the computational complexity of the problem. It has since then been used in a number of works, in particular when dealing with lane change and overtaking maneuvers [[25]] or car following [[55]].

The fact that vehicles interactions are regulated by traffic rules is exploited by Agamennoni et al. [[56],[57]]. Multi-agent influences are decomposed as log-linear combinations of pairwise dependencies, with pairwise dependencies of the type “vehicles on a smaller road yield to vehicles on the main road”. This is illustrated in Figure 8.

Figure 8

Trajectory prediction with an Interaction-aware motion model (adapted from [[56]]). The model is able to predict that the yellow agent should yield to the blue one because of the traffic rules (yield line on the floor).

Finally, a few works have proposed general probabilistic frameworks for tracking vehicles and predicting their future motion. Instead of modeling pairwise dependencies, the model proposed by Gindele et al. [[26]] accounts for mutual influences by using factored states. The causal dependencies between the vehicles are modeled as a function of the local situational context, which reduces greatly the computational complexity. The approach was validated on simulated highway scenarios, but could in theory be applied to other traffic situations. A similar approach was used by Lefevre et al. [[58]–[60]] to model the joint motion of vehicles at road intersections. However, instead of directly modeling the dependencies between the local situational context and the intention of a driver, they introduced an intermediate variable called “Expected maneuver”. In the proposed model, the situational context influences what the driver is expected to do, which in turn influences what the driver intends to do. This framework was tested at road intersections in simulation and with real passenger vehicles. The results showed the benefits of taking into account interactions between vehicles when reasoning about traffic situations and risk at intersections. A generalization to general traffic situations was also developed [[60]].

Limitations

The Interaction-aware motion models are the most comprehensive models proposed so far in the literature. They allow longer-term predictions compared to Physics-based motion models, and are more reliable than Maneuver-based motion models since they account for the dependencies between the vehicles. However, this exhaustiveness has some drawbacks: computing all the potential trajectories of the vehicles with these models is computationally expensive and not compatible with real-time risk assessment. For this reason, some risk assessment techniques have been proposed recently which do not rely on trajectory prediction. These methods will be described in Section ‘Risk based on unexpected behavior’.

Risk assessment

The previous sections dealt with motion models which can be used to predict the future motion of vehicles. How can these predictions be used to actually evaluate the risk of a situation?

The very notion of risk is not always a clearly defined concept. In the context of intelligent vehicles, it is generally associated with the idea that a situation may be dangerous for the driver, i.e. may result in harm or injury. Following this, it is natural to consider collisions as the main source of risk, and to base the assessment of risk solely on collision prediction. This approach, which has been adopted by many works in the ITS community, is the focus of Section ‘Risk based on colliding future trajectories’. Recently a number of works have suggested that predicting collisions is only part of the road safety problem. These works propose a more general interpretation for the notion of risk, saying that dangerous situations arise from drivers performing unexpected maneuvers. These approaches are described in Section ‘Risk based on unexpected behavior’.

Risk based on colliding future trajectories

This section provides a review of approaches to risk assessment based on collision prediction. Typically they are composed of two steps:

• 1. Predict the potential future trajectories for all the moving entities in the scene.

• 2. Detect collisions between each possible pair of trajectories, and derive a risk estimate based on the overall chance of collision.

Techniques to perform trajectory prediction (step 1) were presented in Sections ‘Physics-based motion models’, ‘Maneuver-based motion models’, ‘Interaction-aware motion models’. Given the future trajectories of two vehicles, many tools exist to extract information regarding the occurrence of a collision (step 2). While the most basic methods only provide basic information such as whether, where and when a collision will occur, the more advanced ones can compute its probability, type, severity, etc.

Binary collision prediction

The computation of the collision risk can be binary. In the special case of linear Physics-based motion models, the analytical solution for the state of the vehicles at a specific time can easily be derived by solving the linear differential equations of the motion model. It follows that the intersection point between two trajectories can be computed in an efficient manner [[11],[13]]. However, in the general case the motion equations are too complex for a closed-form solution to be derived. One solution is to approximate each trajectory by a piecewise-straight line trajectory [[17]]. A more common approach is to discretize the trajectories and to check iteratively for a collision at each discrete timestep. Following this reasoning, collisions can be detected in a simple manner by defining a threshold on the distance between two points (from two trajectories at the same timestep) [[42]]. In order to take into account the shape of the vehicles, this threshold can be replaced by a condition on the “overlap between the shapes of the two vehicles”. Although the exact shape is not always mentioned, vehicles are often represented as polygons [[2],[6],[7],[21],[25]] or can be represented by loom test points [[61]]. If information is available about the uncertainty on the state of the vehicles, and if this uncertainty is Gaussian, an ellipse can be used instead of a polygon by applying a threshold on the standard deviations [[9],[16]]. In order to simplify the calculation of the intersection area, ellipses can be approximated by a set of circles [[9]] or by a set of points [[16]].

Some driver assistance systems focus on detecting unavoidable collisions. This computation is a special case of binary collision prediction where risk is assigned the value 0 or 1 depending on whether there exists a collision-free maneuver that the driver can perform. Determining whether such a maneuver exists can be done in two ways. The first one consists in computing escape maneuvers (i.e. how the vehicle should steer, brake or accelerate to avoid the collision) and check whether these maneuvers are feasible (with “feasible” meaning that the steering, braking or accelerating does not exceed the physical limitations of the vehicle) [[2]]. The second one is to consider the entire space of combined steering, braking and accelerating maneuvers, and to perform an optimized search for collision-free trajectories [[7]]. This concept is closely related to the notion of Inevitable Collision States (ICS) [[62]] used in robotics.

Probabilistic collision prediction

The collision risk can be computed in a probabilistic manner, taking into account the uncertainty on the future motion of vehicles.

In the case of stochastic reachable sets, the collision probability can be computed on a discretized position space by calculating the probability that the center of both vehicles is in the same cell, for all the possible combinations of cells [[22]]. With the geometric version of reachable sets, the collision probability can be measured as the percentage of overlap between the geometric shapes representing the future motion of vehicles [[39]].

For a normally distributed uncertainty on the current state, a solution based on stochastic linearization via the unscented transformation has been proposed [[63]].

When the future motion of a vehicle is represented by a probability distribution on sample trajectories (which is typically the case with approaches relying on Monte Carlo simulations or Gaussian Processes), it is possible to compute risk as the “probability of a collision in the future” by integrating over all the possible future trajectories and detecting collisions between each possible pair. This approach provides a lot of flexibility in the handling of uncertainties. For example, for a Maneuver-based motion model the calculation can either sum over both the maneuvers and their executions, or assume that the maneuvers are known and sum on the possible executions only [[25]]. Furthermore, depending on the final application one can compute the risk of colliding with a specific vehicle or sum over all the vehicles and obtain a global collision risk [[25],[52]].

Other risk indicators

Several risk indicators exist which can complement the collision risk.

By further analyzing the predicted trajectories and their intersecting points, it is possible to derive some indicators which give more information about the potential collision. Popular indicators of the criticality of a potential collision are the velocity of the vehicles [[7],[64]], the amount of overlap between the shapes representing the vehicles [[9]], the probability of simultaneous occupancy of the conflict area by both vehicles [[65]], and the configuration of the collision [[9]]. The information provided by these indicators can be used to determine the best way to mitigate or avoid the potential collision.

Other popular risk indicators are based on a measure of the “Time-To-X” (or TTX) where X corresponds to a relevant event in the course toward the collision.

• Time-To-Collision (TTC): The most standard indicator is the Time-To-Collision, which corresponds to the time remaining before the collision occurs^d. It can be used as an indication of what action (if any) should be taken [[2],[6],[9],[15],[66]]. For example, when the TTC is still large, it might be preferable to inform or warn the driver rather than to apply the brakes. For autonomous emergency braking applications, the TTC can be compared with the time required for the vehicle to come to a full stop in order to decide when to apply the brakes [[67]]. For driver warning applications, the driver reaction time needs to be added to the time to stop the vehicle [[13]]. The TTC can also be used as a tool to identify the least dangerous maneuver for an autonomous vehicle, by assuming that the risk of executing a specific trajectory for the autonomous vehicle is inversely proportional to the earliest TTC (the TTC is calculated for all the possible trajectories of the other vehicles in the scene) [[42]].

• Time-To-React (TTR): A closely related indicator is the Time-To-React, which corresponds to the time available for the driver to act before the collision is inevitable. The idea is to simulate different driver actions (such as braking, accelerating, steering) and to identify the latest moment at which one of these maneuvers is able to avoid the collision [[11],[48]].

Risk based on unexpected behavior

While the approaches presented in Section ‘Risk based on colliding future trajectories’ estimate the risk of a situation by predicting the future trajectories of vehicles and looking for collisions, the approaches presented in this section detect behaviors which deviate from the nominal behavior expected on the road. This formulation extends the concept of risk beyond collisions, by taking into account the emotional strain caused by drivers performing unexpected maneuvers. For example, a vehicle proceeding in an intersection when only a very short gap is available will not necessarily result in a collision, but most drivers will consider it to be dangerous since they expected the vehicle to wait for a longer gap.

Two types of approaches are presented below: detecting unusual events and detecting conflicting maneuvers.

Detecting unusual events

The risk of a situation can be estimated by defining the nominal behavior of vehicles on the road and detecting events which do not match that nominal behavior.

An intuitive solution is to define a set of rules which characterize the nominal behavior of a vehicle depending on the context, and to consider any deviation from that nominal behavior as a danger. The rules can be simple heuristics on acceptable speeds when approaching an intersection [[68]], or can include more advanced concepts such as the semantics of the location, weather conditions or the level of fatigue of the driver [[69]]. However an established limitation of rule-based systems is their inability to account for uncertainties (both on the data and on the model).

As an alternative to manually defining what is a nominal behavior, one can use real data to learn the typical behavior of road users. In [[70]], models are learned for nominal behaviors in the form of Gaussian Mixture Models. The likelihood of the current observations characterizes how “usual” the current situation is. Unusual activities such as running a red light or performing a forbidden turn are detected using a simple threshold on that likelihood.

Detecting conflicting maneuvers

A number of works propose to assess the risk of a situation by estimating the maneuver intentions of the drivers and detecting potential conflicts between them, or conflicts with the traffic laws. Since these approaches rely on estimated maneuver intentions, and since the concept of maneuver does not exist in Physics-based motion models, vehicle motion is usually represented using Maneuver-based or Interaction-aware motion models.

If enough data is available from vehicles violating traffic laws, it is possible to learn models for specific dangerous events in addition to the models for the nominal behavior. Each model is labeled as a “dangerous behavior” or a “nominal behavior”. The problem of detecting dangerous situations then becomes a classification problem, where the likelihood of observations is used to select the model which best explains the current observations. This approach was implemented in [[24]], using tens of thousands of real trajectories recorded by radars and cameras at a road intersection. The presence of a significant number of traffic light violations in the data allowed them to learn a model of that behavior. In parallel, a nominal model was learned from vehicles which complied with the traffic rules. Two classification algorithms were implemented and successfully validated on the dataset: the first one combined Support Vector Machines with a Bayesian Filter, the second one relied on Hidden Markov Models.

For traffic situations involving more than one vehicle, one can use traffic rules to label pairs of maneuvers as “conflicting” or “not conflicting”. This method has been implemented in the context of road intersections, by manually labeling pairs of maneuvers based on the priority rules (e.g. stop, yield) [[71],[72]]. This concept of conflicting maneuvers was recently generalized to automatically extract what maneuvers are expected from drivers given the traffic rules and the current situational context [[59]]. The expected maneuvers can then be directly compared with the driver’s intended maneuver in order to compute the risk value.

Conclusion

This survey addressed motion prediction and risk assessment techniques in the context of intelligent vehicles.

Motion prediction approaches were categorized based on their degree of abstraction. Physics-based motion models operate at a low level, they consider that the motion of vehicles only depends on the laws of physics. They allow for an efficient computation of the risk, but are limited to short-term collision prediction. Maneuver-based motion models operate at a higher level, as they consider that the future motion of a vehicle also depends on the maneuver that the driver intends to perform. They provide a more reliable estimation of long-term motion and risk, but are not always reliable since they ignore the dependencies between the vehicles in the scene. Finally, Interaction-aware motion models add another level of abstraction by taking into account the inter-dependencies between vehicles’ maneuvers. However, their computational complexity mean they are not always compatible with real-time risk assessment.

Overall, the main difficulty faced by these approaches is that in order to reliably estimate the risk of a traffic situation it is necessary to reason at a high level about a set of interacting maneuvering entities, taking into account uncertainties associated to the data and the models. This high-level reasoning is computationally expensive, and not always compatible with real-time risk estimation. For this reason, a lot of effort has been put recently into designing novel, more efficient risk estimation algorithms which do not need to predict all the possible future trajectories of all the vehicles in the scene and check for collisions. Instead, algorithms have been proposed which focus on the most relevant trajectories to speed up the computation, or to use alternative risk indicators such as conflicts between maneuver intentions.

This survey also highlighted the fact that the choice of a risk assessment method is tightly coupled with the choice of a motion model. Therefore, we believe that major improvements in this field will be brought by approaches which address jointly vehicle motion modeling and risk estimation.

Endnotes

^a Extensions of the Kalman Filter exist which deal with non-linear models. A comparison of their performances for trajectory prediction is done in [[73]].

^b More details about probabilistic filtering can be found in [[74]].

^c For readers interested in a state of the art of clustering, a comparative survey of the most popular methods was presented by Morris and Trivedi [[75]]. In particular, road traffic situations were investigated by Buzan et al. [[37]], Hu et al. [[36]], and Atev et al. [[29]].

^d The term Time-To-Collision is often used to refer specifically to collision detection methods which assume a constant velocity for the vehicles. Here the alternative definition of the Time-To-Collision is used, where it corresponds to the time remaining before the collision occurs.