Technologies that will be key to autonomous driving

Latest developments in sensors, actuators, digital maps, and telematics



 We recently outlined Mercedes Benz's self-driving experiments and several OEM's roadmaps for telematics in our report Autonomous Driving Technology: Telematics Japan 2014. This report introduces various types of technology for autonomous driving: sensors, actuators that control acceleration/deceleration and steering, 3D digital maps, and telematics including cloud communication. Autonomous driving system firstly obtained information from sensors, and correlates between these information and high-precision 3D digital maps. The system recognizes location of the vehicle secondly. Thirdly it detects road conditions, pedestrian, other vehicles on the road. In consideration of all factors, the system judge trajectory planning and chooses travel routes. Then the system can give instructions of vehicles' driving operations to actuators and enable the vehicles to drive ahead autonomously. Once the vehicle starts traveling, it will be able to instantly sense its new positions to where they advance ahead and sense any changes in their surroundings by receiving data through sensors. The data will enable the self-driving car to recognize its next situations and determine next routes.

autonomous driving

Related reports:

Telematics Japan 2014
  Autonomous Driving Technology: Telematics Japan 2014 (Nov. 2014)
  Latest Developments on Telematics in China and Thailand: Telematics Japan 2014 (Nov. 2014)

ITS World Congress: Detroit
  Outline of CTO Symposium: GM, Ford, Toyota, Honda, and parts suppliers deliberating the future of ITS (Sep. 2014)
  Further advanced automated driving and driver assist systems (Oct. 2014)

Sensor technology

Millimeter wave radar excels in measuring long distances at high speed:
Millimeter wave radar, laser radar, laser scanners, cameras, and ultrasonic sonar are used in autonomous driving experiments conducted by various OEMs. The functions and performance of camera and millimeter wave radar sensors have advanced, thanks to its repeated use in driver assistance systems. In order to measure the distance with a vehicle ahead, driver assistance systems make use of millimeter wave radars, laser sensors, and stereo cameras. However, when OEMs consider the need for high precision, better- performing millimeter wave radar will be the key to accurately measure distances at high speed and in adverse weather conditions.
Cameras with greatly enhanced performance of digital image processing:
The image processing technology used in cameras is most rapidly advanced. Cameras that can recognize signs and pedestrian based on digitalization technology are indispensable to autonomous driving. In autonomous driving, the most critical issue is to recognize their surroundings by receiving 3D data and collating it with 3D map data. Therefore, suppliers are focusing on improving the level of precision of 3-dimentional recognition based on stereo-camera image processing.
3-dimensional recognition will rely on either laser scanners or stereo cameras:
Currently, laser scanners provide a far more superior level of precision in terms of measuring space based on 3D digital data. They have been tested many times by various OEMs. However, because laser scanners are currently extremely expensive, suppliers are making efforts to reduce their costs. Laser scanners and stereo cameras are both being considered as 3-dimensional recognition sensors, provided that the costs of scanners can be reduced and the level of performance of image recognition and image processing by cameras can be improved.
Multiple sensors and cameras: each sensor or camera with designated detection range
Mercedes Benz conducted its autonomous-driving experiment on public roads using many sensors for millimeter wave radar and cameras as follows. Each sensor has designated detection range to detect designated objects.

  - 2 stereo cameras for front view
  - 1 monocular camera for traffic signal
  - 1 monocular camera for rear view
  - 4 long-range millimeter wave radars for front view ( 77GHz )
  - 1 short-range millimeter wave radars for front view ( 24GHz )
  - 2 long-range millimeter wave radars for rear view ( 77GHz )
  - 2 short-range millimeter wave radars for rear view ( 24GHz )
  - 2 short-range millimeter wave radars for side view ( 24GHz )
  - 4 millimeter wave radars at corners of vehicle for the nearer surroundings( 24GHz )
 The company reported that one of the problems identified in the experiment is that autonomous vehicles need to accurately determine where they should stop at intersections while checking vehicles at intersections, stating that precision of vehicle positioning needs to be improved. Also, another issue was the difficulty that cameras had in recognizing the colors of traffic signals.


Comparison of sensors

Millimeter wave radar Laser radar Scanradar Camera Ultrasonic sonar
77, 79 GHz 24 GHz Stereo Monocular
Object detection (100-250m)
Obstacles/fallen objects
Object detection(at close distance) ○within 5m
Vehicles ○within 5m
Motorcycles ○within 5m
Pedestrians ○within 5m
Obstacles/fallen objects ○within 5m
Relative velocity detection
Performance in bad weather
Night vision detection
Lane detection
Signal/sign distinction
Vehicle surroundings map

Performance: ◎ Excellent; ○ designed to handle; △ capable of being used


Millimeter wave radar

Millimeter wave radar
Millimeter wave radar ( Bosch )
Photo: MarkLines
Millimeter wave radar transmits high-frequency radio waves (77,79GHz or 24GHz) in front of the vehicle and receives back radio waves reflected off of cars or obstacles ahead. Depending on time lapses or changes in the wave frequencies received, the radar can calculate distances and relative velocity. The spectrum of high frequency radio waves that are known as millimeter waves, travel in straight waves like light. Their advantage is that they are affected less by rain, snow, fog, etc., because their wave lengths are longer than those of light. There are two types of millimeter wave radar devices. One is capable of measuring long distances to a maximum of 250m and the other is capable of measuring short distances, which expands the illuminating angle. Recently, new millimeter wave radars, which can recognize pedestrians, are under development.


Laser radar

Laser radar
Laser radar ( Denso )
Photo : MarkLines
This laser radar device is comprised of a laser diode that emits laser light, a polygon mirror used for scanning laser light, a photodiode that detects light that is reflected from obstacles, and a CPU circuit. It emits a laser light and measures the amount of time it takes for reflectors on cars ahead to return the reflected light, calculating the distance and relative velocity. Laser light is scanned up and down and left and right by changing the tilt angles of all the surfaces of the polygon mirror that spins the motor. Even though Denso manufactures a device that can measure up to 100 meters, it doesn't attain the performance level of a millimeter radar device. In addition, its performance drops in inclement weather such as rain, snow, fog, etc.


Monocular camera

Monocular camera
Monocular camera ( Delphi )
The microcomputer processes images of camera, detecting and recognizing objects the camera photographed as digital images. Many use a CMOS image sensor. It generates an electrical charge according to the amount of light received by the photodiode. It transmits to the accumulating diode through a forwarding transmitter, and then reads out the voltage signal. Its performance is being improved by enhancing resolution through increasing the number of pixels, by enhancing the level of sensitivity based on back-side illumination technology, and by enhancing the dynamic range. The advantage of the camera is its ability to recognize pedestrian, road sign, etc. by recognizing patterns created through image processing. This is an ability that radar devices are incapable of achieving.


Stereo camera

Stereo camera
Stereo camera ( Continental )
The system is horizontally placed apart from each other to recognize 3 dimensional spaces by comparing the images that the two cameras take simultaneously. While stereo camera is capable of measuring distances too, the level of performance is inferior millimeter wave radar devices in terms of measuring long distances at high speed. In addition, the level of performance is reduced in inclement weather such as rain, fog and snow. Stereo camera can also include monocular camera functions. There are some examples of stereo cameras and monocular cameras being used together by changing their angles and direction, according to objects they photograph. Like in the experimental car used by Mercedes Benz, millimeter wave radar devices, monocular cameras, and stereo cameras are used in combination, depending on the detection object and range.


Laser scanner

Laser scanner
Laser scanner ( Velodyne )
The omnidirectional Laser Imaging Detection And Ranging (LIDAR) scanner is equipped with multipoint sensors for sending and receiving lasers. The sensor unit is placed on vehicle roof, rotates, measures, and scans 360 degrees at high speed, in order to obtain 3D images in real time. The scanner creates high-precision 3D data, and by comparing it with previously created data, accurately detect its current positions and precisely determine travelable routes.The laser scanner costs very high at JPY 8 million, and used on Google's driverless car and other experimental vehicles by universities and research institutions. Another restriction of the scanner is that the cameras have to be mounted on a relatively high position on the roof.


Ultrasonic sonar

Ultrasonic sonar
Ultrasonic sonar ( Bosch )
The ultrasonic sonar sensors built into the front and rear bumpers transmit ultrasonic waves, detecting sound waves that are reflected off obstacles. The sonar units can calculate distances to objects, based on the round-trip travel time of the emitted sound waves. They are best suited for detecting close-range sounds waves within 5m or less. They are being equipped as sensors on almost all premium vehicles, functioning to determine distances to walls and other vehicles when the vehicles are being parked.



Actuators for acceleration/deceleration and steering

Actuators used in existing driver-assist systems can be used on autonomous driving cars:
Self-driving vehicles have to automatically control all vehicle operations that are currently being performed by drivers, such as accelerating, braking, and steering. Currently, some actuators are already being used in existing driver assistance systems. These include intelligent cruise-control systems and emergency-braking support systems that control actuators. The electronically-controled autuators on the systems control acceleration and deceleration to enable vehicles to maintain a constant distance between vehicles ahead. In addition, these systems also apply the brakes when detecting obstacles. These acceleration and braking actuators used in existing driver assistance systems can basically be used in their current form in self-driving cars.
Electric power-steering system becomes a breakthrough for autonomous driving:
Hydraulically controlled power-steering systems are increasingly being replaced by electronically controlled power-steering systems in vehicles that undergo redesigns recently. Compared to hydraulically controlled power-steering systems, the latest electronically controlled power-steering systems can control all sorts of actuator operations, becoming breakthroughs to achieve autonomous driving. The driver assistance systems equipped on the latest Nissan Skyline and Mercedes Benz S, E, C class vehicles have cameras capable of recognizing driving lanes. These deriver assistance systems, via the motor, assist with steering and keep vehicles in their own lanes. If the driver assistance system can assess driving condition and control operations more accurately, these actuators in the systems can be used in autonomous driving systems as they are.


Steering actuator: electronically controlled steering system

The direct adaptive steering and the active lane-control driver assistance systems on the Nissan Skyline are able to recognize driving lanes based on input from cameras. As a result, they can control steering and keep the vehicle in its proper lane. Also, by assessing driving conditions, these systems control the amount of steering and reaction to drive the vehicle in line with the driver's expectation. The new technology called "steer-by-wire" is able to sense driver steering input and driving conditions, functioning to control steering electronically. If the steer-by-wire system can fully control driving operations, it could be used in autonomous driving. Direct adaptive steering
Nissan Skyline/Infiniti Q50 Direct adaptive steering


Braking actuator: ESP

ESP units are used in braking actuators that control brake hydraulics. They follow instructions to ensure that the exact, required amount of braking is applied. A majority of premium vehicles are already equipped with driver assistance systems such as intelligent cruise control systems, which maintain a constant separation distance wih a vehicle traveling ahead. ESP
ESP ( Bosch )


Accelerator actuator: electronically controlled throttle valve

Electrically controlled throttle valves on engines, which are used in acceleration actuators, are currently being equipped on a large number of mass-marketed vehicles. Depending on driver-based acceleration and road conditions, the throttle valves are controlled by the motor and opened to an appropriate degree. All controlling is done comprehensively, with engine control-units controlling the appropriate degree of opening on the electronically controlled throttle valves and selecting the appropriate transmission gears. Driver assistance systems such as intelligent cruise control systems, which maintain a constant distance between vehicles traveling ahead, are being installed on a wide range of vehicles as driver assistance systems. Electronically controlled throttle valve
Electronically controlled throttle valve( Hitachi automotive )



3D maps used for autonomous driving

3D maps requiring a high degree of precision and accuracy:
3D maps
Source: Daimler

Compared to existing car-navigation systems that rely solely on GPS to determine vehicle locations, autonomous driving requires an extremely higher level of precision and accuracy in order for self-driving vehicles to accurately identify their positions on the road. Mercedes Benz gave a presentation on the results of its autonomous driving experiment, reporting that the level of precision needed for vehicles to correctly recognize their correct driving lanes and precise stopping points is +/- 5-10cm. This is because vehicles need to determine whether they can safely pass through the space between themselves and obstacles on the road or parked vehicles.

Greater need for information about signals, stop-lines, traffic signs, etc., currently not found on existing maps:
Information, which was never needed before, will be required to make autonomous driving possible. For example, in addition to information on the locations of roads, the map for autonomous driving vehicles require a lot of information on stop lines, driving lanes, guard rails, signals, and number and directions of traffic signs. In addition, in order for vehicles to confirm their current locations, they will also need information about the locations of buildings in their vicinity. The major map-maker Zenrin is actively creating 3D digital maps needed for automated driving. These maps will not only require a high degree of precision but also will need to display the most up-to-date road conditions such as detours created because of road construction.
future digital map
Image of future digital map needed for automated driving
Source: Zenrin
Need for standardizing map data:
OEMs (such as Mercedes Benz) and research institutes that are experimenting with autonomous driving are creating and using their own special maps, which show only the sectors on where they are conducting experiments. However, when considering the overall spectrum of high-precision maps needed, the vast volume of data are required to make these kinds of maps. There is so much data that it can't all be stored in conventional car navigation systems. One way to solve this situation is to store the needed information in mega cloud servers and use it by communicating through telematics. However, the huge volume of map data being stored in cloud servers would need to be made available to every OEM. The U.S. and European countries are starting to deliberate pressing issues such as how such map data should be standardized. Who is going to pay for creating it, and which companies are going to take the initiative? These topics attract attentions of people.



Telematics (accessing/ transmitting data stored in cloud servers)

Telematics are vital to automated driving:
The volume of high-precision map data that will be needed is going to be huge. If it is stored on cloud servers, vehicles could access it via telematics. The information provided to self-driving vehicles from cloud servers would not only be map data but also the most up-to-date, real-time information on road conditions, accidents, obstacles in the road, etc. If the number of vehicles equipped with telematics devices increases  on the road, they could transmit the latest traffic information for storing on cloud servers, such as information about road construction, accidents, etc. These information will eventually be transmitted to other vehicles traveling in the same vicinity. There is no question that telematics are vital to achieving autonomous driving, which requires an enormous amount of the most up-to-date information.
Source: MarkLines
Two ways being considered for transmitting telematics-based information:
Currently, two ways are being considered for transmitting telematics-based information. One way is for OEMs to preinstall designated telephone units in vehicles. The other way is for consumers to use their own smartphones. As for the first proposal, some OEMs such as Lexus have already created and activated their own service programs. In addition, in the case of the all-electric Nissan Leaf, designated device on vehicle which are factory installed, use mobile phone line and send the vehicle's driving data to a central control center, so that Nissan can use the data in its future R&D activities. As for the second proposal calling for the use of consumers' smartphones, insurance companies in the U.S.A. already track drivers' driving habits and distances driven via connected-car platforms, and use that data to adjust insurance premiums. While both of these two ways have potential, it is still necessary to standardize telematics, specifically the actual map data itself, the format for the data, and the format for transmitting the data.
Direction of overall business model still remains uncertain:
These cloud services will grow exponentially in the future, requiring a huge amount of investment. However, several questions still need to be answered about the actual business model itself, such as who is going to pay for all of this and how can profits be made. The business model of smartphone-based Google Map service is based on advertisement income, since end-users don't pay to use the service. This business model could be considered as one possibility for the telematics business model; however, it is still uncertain. Therefore, the key to all of this will depend on what kind of business model is actually adopted and how it will be developed.
Takahiko Yoshida, "Technology Trends of Sensors Contributing to the Safty Systems," Journal of Society of Automotive Engneers of Japan, Vol.68, Apr. 2014.
Takayuki Inaba, Tetsuo Kirimoto, "Automotive Milimeter Wave Radars," Journal of Society of Automotive Engeers of Japan, Vol.64, Feb. 2010

<Automotive Industry Portal MarkLines>