Computer Vision for Intelligent Vehicles


Intelligent vehicles - they get more and more attention: in research institutes, in industry as well as in media. The development is driven by a desire to reduce traffic accidents, reduce environmental pollution and increase traveling comfort, and companies such as Tesla, and Mercedes show impressive demonstrations. These vehicles have Advanced Driver Assist Systems (ADAS) that help the driver with different tasks (lane-departure warnings, steering autonomously, etc.). As you can imagine, a key component is the sensing system that perceives and interprets the environment of the vehicle.
Our group has a car available with many cameras (360º stereo vision coverage and more), a MobilEye radar module, vehicle sensors, a GPS sensor, communication modules and computing power (such as an NVIDIA DrivePX2 platform). This offers a lot of possibilities for research projects that can be verified with real-world data or even, if you dare, live in a real car.
Note that in this ADAS context, a sophisticated trade-off between robustness (for traffic safety) and efficiency (it should run in real-time in a car) is paramount and offers interesting challenges to resolve – both in terms of algorithm design and for the evaluation.


We have several research tracks running in parallel, all of which offer interesting and relevant research questions that can be addressed in a graduation project.

Deep Learning for Semantic Scene Parsing
Semantic Scene parsing is about classifying each pixel in an input image as one of many classes, such as vehicle, pedestrian, road, building, etc. Currently, Deep Convolutional Networks are dominating this task (Illustration 1). However: they are far from perfect. For example, they require a lot of memory and computational power; and they still are easily fooled by tricky conditions such as reflective pools on the road, classes that rarely occur, etc. Potential research projects can look into incorporating world models in the currently 'dumb' nets, exploiting time consistency in video, extending the number of classes, robustness against bad weather and many more.

Segmentation, detection and tracking of obstacles for collision prediction
Understanding the dynamics of your surroundings in traffic is one of the key issues for autonomous or assisted driving. You do not only need to know where which obstacles are, but you need to know where they will be in a few seconds. This requires robust and efficient obstacle path prediction algorithms. We work with the 'Stixel World' framework, which represents the scene efficiently in rectangles with a 3D position, using stereo vision. One of our ongoing extensions is tracking these stixels and calculate where they will go next to avoid collisions (Illustration 2). This needs to be reliable, or at least, should have a reliable confidence metric to know how sure the system is of its assessment. Even though stixels are an state-of-the-art method, all input signals degrade heavily under bad weather conditions -unavoidable in the Netherlands- or at night time, so there is plenty of room for clever improvements.

On top of these subjects, we also have researchers working on ego lane detection, sensor fusion for lane-accurate localization (such as GPS, IMU and vehicle odometry) which can be used for creating and updating digital maps automatically and several other topics.

In general, we work in C++ with libraries such as OpenCV and CUDA modules to use GPUs. Our Deep Learning projects rely on Tensorflow and Python. To test software in the car, the code needs to be integrated in RTMaps, the framework that connects all hardware in the car.

Most projects can be executed internally at the TU/e. Working together with (or at) industrial partners such as TomTom, NXP and Mapscape can be discussed.
Image/Video analysis, Sensor Fusion, Machine learning, Deep Learning, Object Tracking, 3D video processing, Efficient processing algorithms
Contact person: