CEATEC JAPAN 2017: Driver Monitoring Technologies

Drowsiness prevention features and biometric sensing for autonomous driving



At the CEATEC JAPAN 2017 exposition held in the Makuhari, Chiba convention center on October 3-6, various advanced driver-assistance system technologies were displayed by electronics manufacturers.

This report focuses on exhibits relating to driver state monitoring technologies that are essential for the safe and smooth transfer of control over vehicle functions involved in Level 3 autonomous driving systems (conditional automation).

Omron showed off a cockpit demonstration of its "Onboard Driver Monitoring Sensor" that utilizes its proprietary facial-recognition technology. Panasonic demonstrated its camera imaging and AI processing technology for regulating driver drowsiness, which was announced in July 2017. Denso conducted a VR imaging demonstration using a wearable biometric sensor to detect data including the driver's pulse and blood pressure. At the NTT booth, the NTT group demonstrated its wearable biometric sensing "hitoe" wear technology and introduced its driver drowsiness detection system that utilizes a dedicated smartphone application.


Related Reports:
Toyota's autonomous driving (2): Striving for "trillion-mile reliability" (March 2017)
JSAE Exposition 2016: Latest autonomous systems and ADAS technologies (June 2016)

Omron: Advances in driver state monitoring sensors and facial-imaging technology

Omron exhibited their "Onboard Driver Monitoring Sensor" technology using a cockpit demonstration. Omron's new sensor reflects further advancements made in their onboard sensor equipped with "driver concentration sensing technology" that was announced in June 2016.

As discussions continue on the legislation surrounding autonomous driving, Omron will continue to focus on the development of improved onboard sensor technology, extensive field testing, and the development of features with high functionality to analyze integrated face and posture data in real time.

Omron aims to have its "Onboard Driver Monitoring Sensor" adopted in vehicles to be launched by 2020.

Omron's cockpit
Omron's cockpit demonstration
Sensor mounted in the instrument panel, above the steering wheel
Instrument panel / Display Panel

Technological advancement in driver condition monitoring sensors

1. Simultaneous judgment of driver condition according to three categories
  1. Whether the driver is focused on the vehicle's operation? (Eyes ON/OFF)
    The Eyes ON/OFF function judges between two alternatives based on the manner in which the eyes are blinking and the direction they are facing to determine whether the driver is focused (ON/OFF) on the direction of the car and its periphery.
  2. How quickly will the driver be able to resume driving? (Readiness High/Mid/Low)
    The Readiness High/Mid/Low function judges between three alternatives as to the driver's capacity to resume driving based on what they are doing, determining whether they are ready to drive immediately (High), able to resume driving momentarily (Mid) or will require some time before resuming driving (Low).
  3. Is the driver in the driver's seat? (Seating ON/OFF)
    The Seating ON/OFF function judges between two alternatives to determine whether the driver is sitting in the driver's seat.
    With advances in autonomous driving technology, it will also be possible to assume whether the driver has moved from the driver's seat to some other location in the passenger compartment.

Conventional driver state monitoring technology is only capable of determining whether "the driver is in a condition to drive safely". But, with advances in deep learning technology, it is now possible to simultaneously monitor and judge the three driver conditions described above.

2. Advances in "OKAO Vision" facial-recognition technology

Omron has significantly improved the facial recognition accuracy of their "OKAO Vision" technology by applying time-series deep learning technology to the extent that it is now possible to determine the condition of the driver even if they are wearing a surgical mask or sunglasses.

3. Improved real-world driving data enriches database

By incorporating real-world driving data and enriching its database, Omron has accumulated a wealth of information on driver conditions that it can apply in the development of its technologies.

Omron has announced their Onboard Driver Monitoring Sensor as a 'world's first' technology able to simultaneously monitor and judge three kinds of driver conditions. Among them, determining "how quickly the driver can resume driving" is made possible with technology utilizing algorithms reflecting changes in the real-world driving data accumulated by the company. In addition, Omron is considered a leader in the field of facial-recognition technology, with over 20 years research experience as its facial image sensing technology has been extensively used in digital cameras.

In 2017, and in collaboration with Nagoya University's Institutes of Innovation for Future Society and the HMHS*, Omron has been conducting field tests on a wide variety of vehicles installed with its new Onboard Driver Monitoring Sensor as well as biosensors to measure heart activity and pulse rates to collect additional data for future technology research and development.

* Human Machine Harmonization System Consortium

Panasonic: Technology for detecting, predicting, and controlling driver drowsiness

Panasonic Automotive & Industrial Systems (AIS) Company announced in July 2017 that it has developed drowsiness control technology to detect and predict a driver's state of drowsiness, including features to allow the driver to stay comfortably alert. Panasonic will start accepting orders for samples from October. At the CEATEC JAPAN 2017 exhibition, their technology was introduced as "Emotion & Physical Condition Sensing" technology.

In an actual demonstration, a cockpit would be installed with cameras, environmental sensors, their Grid-EYE infrared array sensor, and other equipment (this was demonstrated by video simulation only, and there was no cockpit at Panasonic's booth)

  1. The technology uses AI processing of multiple sources of biometric data obtained through camera images such as blinks and facial expressions to detect the onset of driver drowsiness in a highly accurate and contactless manner. The level of a driver's drowsiness is classified on a scale of 1 to 5 and displayed on the instrument panel.
  2. Using Panasonic's Grid-EYE sensor technology, the amount of heat loss from a driver is measured in a contactless manner, and taking into account the surrounding brightness measured with an environmental sensor, the level of drowsiness that will occur in 15 minutes will be predicted and displayed on the instrument panel.
  3. While monitoring the thermal sensation of the driver with Grid-EYE, the technology can control the HVAC system to adjust the cabin's air temperature and volume according to the level of predicted drowsiness. Other options to optimize conditions to ensure the driver remains comfortably alert include increasing the volume of the car's audio system. When a driver's current drowsiness level is detected as high, a sound alarm or a command to pull over for a rest break is issued.
Panasonic Emotion &
Physical Condition Sensing
Panasonic Emotion & Physical Condition Sensing
Panasonic display to detect, predict, and control driver drowsiness

In collaboration with the Ohara Memorial Institute of Labor Science, Panasonic has developed AI processing technology to estimate a driver's drowsiness level based on the results of the analysis of drowsy facial expressions. The scale for determining the level of the driver's drowsiness from facial expressions is classified into 5 levels (1. Not drowsy at all, 2. Slightly drowsy, 3. Drowsy, 4. Very drowsy, 5. Seriously drowsy).

In joint research conducted with Chiba University, Panasonic has revealed that the amount of heat loss from a person's body is correlated with drowsiness after a predetermined time elapses. Research results also make it possible to predict how a driver's current state of drowsiness can change by means of heat loss and the surrounding brightness.

Furthermore, Panasonic has applied their knowledge of thermal environments and physiology accumulated through R&D of indoor air conditioners and other products, and in collaboration with Nara Women's University, developed contactless technology to measure the thermal sensation (whether hot or cold) in a vehicle's interior, where the effects of airflow and others factors are significant.

Conventional drowsiness detection systems have been unable to detect drowsiness at the level of subtleness required to predict future drowsiness states. Also, these systems disturb the driver's comfort level by the use of features such as alert sounds and vibrations to stimulate the driver to stay awake. Panasonic's drowsiness-control technology is able to detect and predict the driver's level of drowsiness, while allowing them to stay comfortably awake. The system helps prevent drowsy driving by using measurement data from the vehicle's interior to stimulate a drowsy driver in an unobtrusive manner by adjusting the vehicles air conditioning or audio system accordingly.

Denso: Development of driver-assistance technology using biometric signals

Denso's research and development is focused on driver-assistance technologies that sense biological signals. Denso is aiming to establish driver-assist technologies by clarifying the relationship between situations specified by the driver in various driving scenarios and biometric feedback data.

At this year's exhibition, the Denso booth had a demonstration unit used to develop gaming applications that projected VR images allowing the visitor to drive cars around in outer space for the purpose of sensing changes in a driver's emotions encountered in various driving environments. The wearable sensor attached to the driver's arm detects and collects biometric data such as pulse and blood pressure.

Biometric sensing
Biometric sensing demonstration
Wearable sensor attached to a person's arm
Display Panel / Wearable sensor

About 10% of all traffic accident deaths result from impairment of physical conditions. To reduce traffic accidents caused by sudden deterioration in the physical condition of a driver, Denso has been developing biometric sensors for automotive use to monitor and provide real-time feedback on vital signs.

For the early detection of a driver's impaired physical condition, initially Denso's biometric sensor development focused on vital sign sensors for heart rate and pulse, measured by imbedding electrocardiogram sensors into the steering wheel, which when gripped by the driver could be used to derive electrocardiogram waveform data and other types of biometric information on the driver such as heart rate, autonomous nerve activity, and blood pressure.

However, the problem with steering wheel biometric sensing systems is that they require the driver to grip the steering wheel with both hands, so that the palms cover the sensors, which makes it difficult to collect reliable biometric data when the driver steers with one hand or turns the steering wheel often.

Denso is working to achieve contactless and unattached sensing technologies that can be used to advance the research and development of practical applications of advanced driver-assistance products. (Quoted from Denso Technical Review Vol. 21, 2016)

Denso collaborated with a number of partners to create this year's exhibition theme, utilizing technologies and concepts from various fields such as Virtual Reality (VR). Denso used the "VR-CAR", a small (single-occupant) EV, and VR imaging to demonstrate the ability to cooperatively control the movement of a moving object to travel around to visit tourist spots in flying cars.

The company plans to accelerate the development of practical applications of ADAS products by promoting collaboration with partners in various fields from the early planning stage. Denso is aiming to apply the same collaborative process for the research and development of advanced driver-assistance system technologies using biometric signal data.

Denso's VR-CAR
Denso's VR-CAR
Display Panel

NTT Group: Drowsiness detection system for drivers wearing "hitoe" wear

At the NTT booth, the NTT Group demonstrated a driver drowsiness detection system with a driver wearing "hitoe" clothing and gave presentations on other biosensing devices and services. Simply by wearing hitoe wear, highly accurate electrocardiogram data of a driver can be recorded in a comfortable and natural manner as it does not restrict body movements while driving, and because the hitoe material is embedded in the material near the heart.

Heartbeat variations and electrocardiogram signals detected while wearing the hitoe clothing are transmitted wirelessly by the hitoe Transmitter 01 located on the chest area of the hitoe wear to a smartphone. The dedicated smartphone application then initiates a drowsiness detection algorithm for the transmitted biometric data, making it possible to detect the drowsiness state of a driver during nighttime driving or long drives. This technology can also help prevent drowsy state driving beforehand by signaling the driver with an alarm or by notifying the system administrator of an alert by e-mail.

"hitoe" is a functional material to measure biometric signals, jointly developed by NTT and textile manufacturer Toray Industries. NTT provided their "Conductive Fiber Technology" to conduct electricity through fiber material, while Toray provided their cutting-edge "nanofiber technology" and "biometric weaving technology". Such technology can be used for new functional clothing, instead of electrocardiograph and pulse rate electrode wearables, and is expected to lead to the creation of new services in fields such as sports training, safety management, and medical care support.

NTT Group's drowsiness
detection system
NTT Group's drowsiness detection system
Biometric data detection on the back side (gray area) of hitoe wear

NTT Docomo is developing the hitoe transmitter 01 while NTT Data MSE works on dedicated apps for smartphones.

The drowsiness detection algorithm used in this technology was jointly developed by researchers at Kyoto University and Kumamoto University to predict epileptic seizures with a high degree of accuracy by correlating indices of heart rate variability.

In May 2016, NTT started field testing of its driver drowsiness detection system with a delivery service company with drivers using the hitoe wear. This testing is aimed at improving the timing and accuracy of notification methods to delivery drivers and system administrators by conducting evaluations of real-world driving situations encountered to optimize the technology for commercial use.

In the future, NTT plans to develop services to further support drivers and system administrators by developing algorithms to analyze electrocardiogram and other biometric data transmitted and stored in a cloud to compare trends in past heartbeat data acquired through the onboard biometric sensors, including white lane deviation, and vehicle swerving.

Presentation of drowsiness
detection system
Presentation of drowsiness detection system

CEATEC, autonomous driving, driver monitoring, biometric sensing, Omron, Panasonic, Denso, NTT

<Automotive Industry Portal MarkLines>