Toyota's autonomous driving (1): More sophisticated ADAS and fully autonomous driving

The OEM unveils the Toyota Concept-i



TOYOTA Concept-i
TOYOTA Concept-i exhibited at 2017 International CES

  The Automotive World 2017 Conference was held at Tokyo Big Sight from January 18, 2017, to January 20. This report will describe the direction of Toyota's long-term goal for autonomous driving. It will focus on a lecture given by Yoshiaki Matsuo, chief examiner of the Advanced Safety System Research and Development Division at Toyota Motor Corporation titled, "Toyota's Direction of Automatic Driving and Future Issues - Cooperation between Intelligent Vehicles and People, and the Realization of a Mobility Society;" and also examine Toyota's efforts for autonomous driving.

  Under its "Mobility Teammate Concept," Toyota is aiming for autonomous driving that enables cooperation between humans and automobiles. While many OEMs are skipping implementation of level 3 autonomous driving, where both the system and driver perform operation, and aim for higher levels of autonomous driving, Toyota will realize level 3 autonomous driving with advanced artificial intelligence (AI) and a human machine interface (HMI), and will strive to achieve level 4 and 5 autonomous driving through advancements in AI technology.

  The Toyota Research Institute (TRI), which researches AI, is currently working on 2 approaches to autonomous driving: "Level 4 and level 5 fully autonomous driving (referred to as a 'chauffeur')," and "autonomous driving through advanced driving support systems (referred to as a 'guardian')." The driving support (guardian) system will also feature AI technology equal to the Concept-i (details below), and gradually increase in intelligence. Toyota believes that enhancing this aspect to create a smart AI is the most effective path to achieving fully autonomous driving.

  Additionally, at the 2017 International CES, Toyota unveiled the Toyota Concept-i, the embodiment of the OEM's vision for future mobility. The latest advances in AI have been applied to the concept to establish a composite technology for understanding humans, such as recognizing emotions and learning driver preferences. The concept will provide a novel experience based on new ideas, under the umbrella of safety and security for humans and the Fun to Drive concept. The AI will also support human drivers in making decisions to switch between autonomous and manual driving. The Concept-I is Toyota's long-term goal, and test vehicles are scheduled to start driving in Japanese cities in the next few years.

Related Report: Toyota's ADAS technology: Autonomous Vehicle and ADAS Japan 2016 (1) (August 2016)

Expectations for autonomous driving and the role of an independent system

  Issues surrounding automobiles include safety, the desire for free mobility (including for the elderly), and environmental issues such as traffic and fuel economy. Toyota aims to achieve the realization of a "society where everyone can travel safely, smoothly, and freely."

  With 96% of fatal accidents caused by driver error (the remaining 4% being the fault of pedestrians, etc.), there are high social expectations for autonomous driving, which if realized, can lead to the reduction of traffic accidents, support for the elderly, and solutions to issues such as the shortage of bus and truck drivers.

  Moreover, it has been said until now that autonomous driving can be realized through the integration of independent and cooperative systems with onboard technology. Based on a recent technological revolution in sensors, computer processing power, and software, the role of independent systems has risen in importance, and Toyota has renewed its goal to develop such a system.

Technological advancements in recent years

Sensor performance Cameras: high resolution, high sensitivity Lasers: 2D→3D image scanners
Hardware processing power Increase in performance through GPU and FPGA, etc.
Software Improvements in recognition logic performance through fundamental autonomous driving algorithms and machine learning
(Note) GPU is an acronym for Graphics Processing Unit. FPGA stands for Field-programmable Gate Array, and refers to an integrated circuit that can be arranged by consumers or designers after production.

Toyota's concept of autonomous driving: MOBILITY TEAMMATE CONCEPT

Source: Toyota
(Source: Toyota)

  Toyota's Mobility Teammate Concept gives insight into the OEM's way of concept for autonomous driving, wherein humans and automobiles cooperate with one another. The logo of the concept shows a driver and autonomous driving system holding a steering wheel side-by-side. The concept is based on the idea of not impeding the driver's 'Fun to Drive,' allowing them to take control when they want to, and letting the system take over when they don't want to or cannot operate the vehicle.

  Toyota's autonomous driving will focus on safety, environment, mobility (freedom of travel), and Fun to Drive.

Development approach for autonomous driving technology: Achieve level 4 and 5 autonomous driving after realizing level 3

(Source: Toyota)

  Toyota's policy, as noted above, can be seen in its work on level 3 of the 5-level SAE standard. The SAE defines level 3 as meaning that "autonomous driving systems are required to constantly monitor for situations where the driver must take control of the vehicle, and revert control to the driver in such situations." However, it is believed that it is extremely difficult for drivers to take control in such a short period of time after having allowed the system to take control, leading to some OEMs to skip level 3 autonomous driving, and instead aiming straight for levels 4 and 5.

  Toyota aims to realize level 3 autonomous driving by establishing a cooperative and trusting relationship between humans and automobiles, and will only move to level 4 after that has been established.

  According to Gill Pratt, CEO of the TRI, "It will take several decades for American streets to be filled with primarily level 4 and 5 autonomous vehicles. During that time, we must pursue autonomous driving technology that can save as many lives as possible. Toyota will work on level 2 through level 5 fully autonomous driving, and will continue to make automobiles that can further contribute to safety."


Developing an advanced driving support system and fully autonomous driving

  Under the guidance of Pratt, Toyota is developing the "guardian" driving support system and "chauffeur" level 4 and 5 fully autonomous driving systems. The following section will summarize Toyota's policy for developing autonomous driving technology based on 2 lectures given by Pratt (from CES in January 2017 and the GPU Technology Conference (GTC) held in April 2016).

<Comments regarding the SAE standard levels 2, 3, 4, and 5>

  Pratt has repeatedly emphasized the difficulty in realizing level 5 (unconditional, fully autonomous driving). According to Pratt, "Society can forgive many of the mistakes made by drivers (humans), but demands extremely high safety performance from autonomous driving (machines). People tend to not forgive any injuries or deaths mistakenly caused by machines. In order to realize the degree of safety required from level 5, there needs to be an immense amount of machine learning, simulations, and real driving tests."

  Compared to level 5, there is a higher possibility that level 4 can be achieved earlier than planned.

  This also can be said for level 3, but Pratt continued to explain by using numbers.

  With the fatal accident caused by a level 2 Tesla vehicle, autonomous driving is currently receiving an enormous amount of attention. Pratt also pointed out the difficulty for drivers to constantly monitor their vehicles for extended periods while relinquishing all control to autonomous driving systems. TRI is currently researching ways to maintain driver attention through new HMI and other measures.

Comments from Gill Pratt regarding the SAE standard levels 5, 4, 3, and 2

Level 5   In order to achieve the level of completion required for level 5 (unconditional, fully autonomous driving), there needs to be several years' worth of machine learning, as well as tens of thousands of miles in simulations and real driving tests. We are still only halfway.
  However, there are people who are unable to drive, so level 5 is necessary, and Toyota is continuing its research.
Level 4   Level 4 (autonomous driving in limited regions, speeds, times, and weather conditions) can be realized much more quickly. In the next ten years, it is possible that many automakers will achieve level 4. It is particularly appealing to companies that provide mobility services such as ridesharing.
Level 3   With level 3, the system must constantly be able to detect situations where it must revert control back to the driver, and the driver must also be prepared to take control once this happens. For example, if the required time for a driver to take back control is 15 seconds, an automobile driving at 65 mph (104 km/h) travels roughly 1,500 feet (approximately 460 meters) in 15 seconds, so the system must be able to detect trouble 460 meters ahead. It may be just as difficult to achieve level 3 as level 4.
Level 2   Level 2, which is already realized, can autonomously drive on highways, but the driver is required to constantly monitor driving conditions. Additionally, if the system cannot detect hazards on its own, the driver must be able to intervene. However, drivers can either underestimate or overly trust the system. The more autonomous driving advances, the more drivers tend to overly trust the system.
Source: Toyota's press releases dated January 5, 2017/April 7, 2016

<Two approaches: Guardian and Chauffeur>

Hybrid Autonomy
Hybrid Autonomy (Series Autonomy and Parallel Autonomy) (although the image says Guardian Angel, this is equivalent to "Guardian") (source: Gill Pratt's slides from his lecture, posted on the GTC 2016 website)

  Guardian is a driving support system, along the lines of ABS, VSC, and AEB (automatic emergency brake). The system integrates the automobile and driver's situational awareness, where the driver constantly monitors road conditions, and the system acts as a safety feature that operates only when necessary. Guardian will have AI similar to "Yui," which is featured in the Concept-i, and will collect data from the automobile and driver, and then share it via the cloud to grow smarter.

  Gill Pratt refers to this as "parallel autonomy." A similar concept would involve a parent teaching their child how to play golf, where initially both the parent and child hold the golf club. However, as the child improves, eventually they would be able to hit the ball on their own, and gradually increase the distance they can hit the ball. This imagery coincides with Toyota's way of thinking regarding autonomous driving, the "Mobility Teammate Concept."

  Chauffeur is a level 4 and 5 fully autonomous driving system. Although the peripheral recognition and control technology being developed are fundamentally the same as Guardian, the difference is that Guardian only activates when necessary, and Chauffeur is constantly online while driving autonomously. The image is of a commanding officer giving orders to subordinates, and who then carry out the tasks. This is referred to as Series Autonomy.

  Pratt has referred to the two types of autonomy as Hybrid Autonomy, as shown in the image to the right, and Toyota will pursue the development of these autonomous driving technologies. Pratt also emphasized that the parallel axis in the image to the right has immeasurable value in strengthening the series autonomy (vertical axis), which the automobile industry is striving for.

Two approaches: Guardian and Chauffeur

Autonomy Name Overview Related driving support systems
Parallel Autonomy Guardian Driving support system that integrates automobile and driver situational awareness ABS, VSC, and AEB
Series Autonomy Chauffeur Level 4 and 5 fully autonomous driving system Adaptive Cruise Control, Automatic Highway Passing, etc
Source: Toyota's press releases dated January 5, 2017/April 7, 2016

Enhancing AI research framework in the U.S.

(Source: Toyota)

  In January 2016, Toyota established TRI (CEO: Gill Pratt), a new company to research AI. By January 2017, the company had already hired roughly 100 staff members, and 50 employees from the Japanese research department have also joined. In the next year, the company plans to hire an additional 100 staff members.

    Currently, the company has three facilities in the U.S., which carry out the following initiatives:
  • Ann Arbor, Michigan: Primarily focused on fully autonomous driving research
  • Palo Alto, California: Focused on technologies (Guardian) where drivers are the primary focus and the automobiles provide support when necessary.
  • Cambridge, Massachusetts: Focused on areas such as simulations and deep learning

  Each facility has partnerships with institutions close to their respective premises (the University of Michigan, Stanford University, and the Massachusetts Institute of Technology).

  Additionally, TRI plans to set up highly advanced, large simulators and begin operating them soon.

Toyota: Enhancing research framework for AI in the U.S.

September 2015   Toyota has signed an agreement with the Massachusetts Institute of Technology's (MIT) Computer Science and Artificial Intelligence Laboratory, as well as Stanford University's Artificial Intelligence Laboratory regarding the research of AI, and established collaborative research centers with both universities. Toyota announced that it will invest a total of USD 50 million over the next 5 years in order to advance research.
  Toyota announced it would hire Gill Pratt. Pratt was the program manager of the disaster relief robot competition held by the U.S. Department of Defense's Defense Advanced Research Projects Bureau. Partnerships with MIT and Stanford will advance based on Pratt's guidance.
January 2016   TRI, a new AI research company was established in Ann Arbor, Michigan, with Dr. Gill Pratt as the CEO. Toyota plans to invest roughly USD 1 billion over five years for AI research and the development of indoor robots utilizing mobility technology.
August 2016   TRI announced it would collaborate with the University of Michigan on AI research. Toyota plans to invest USD 22 million over the next four years. The Toyota Technical Center, Toyota's North American technology development base, has a long history of conducting joint research on safety technology with the university.
Source: Toyota's press releases dated September 4, 2016/ January 5, 2016/ August 11, 2016

Toyota unveils the "Toyota Concept-i," featuring AI technology capable of understanding humans, at the 2017 CES

  At the 2017 International CES, held in Las Vegas, Nevada, from January 5 to January 8 2017, Toyota exhibited its Toyota Concept-i concept car. The vehicle embodies Toyota's ideas for a future mobility society where humans and automobiles become partners through AI technology.

  Toyota has created a new relationship for drivers and automobiles as partners that grow together thanks to AI that can understand humans. By combining technology that can "understand humans," through means such as emotional recognition and learning driver preferences, with autonomous driving technology, Toyota will provide a new experience to drivers that fulfills needs for safety, security, and the pleasure of traveling.

  The Concept-i features AI technology called Yui. Yui deepens its understanding of the driver as it grows along with them. It will eventually grow beyond simply setting routes, but begin to also predict the driver's expectations, becoming more than an interface between driver and machine, and will deepen the relationship between the two. Yui is a teammate that works with the person operating the vehicle.

  The Concept-i features fully autonomous and manual driving modes, and Yui provides support in making the decision to switch between them. In all released photos, the concept features a steering wheel and pedals.

  The Concept-i is scheduled for public road verification testing in the next few years. An experimental vehicle with some of the features from the exhibition model will drive in Japanese cities.

Toyota Concept-I's
exterior Toyota Concept-I's
exterior Toyota Concept-I's
Toyota Concept-I's exterior (1) (Source: Toyota) Toyota Concept-I's exterior (2) (Source: Toyota) Toyota Concept-I's interior (Source: Toyota)

Toyota Concept-i exhibited at CES 2017

The Concept-i is based on Toyota's philosophy of automobiles as an industrial product that drivers become emotionally attached to, and provides a new experience by having vehicles work for people.
Technology that understands people (Learn)   Using the latest AI technology, Toyota has established a composite technology that can understand humans, through means such as emotional recognition and learning driver preferences. By converting driver expressions, actions, and alertness into data; and monitoring social media posts, actions, and conversations; the automobile predicts the driver's preferences.
Safety and security (Protect)   By combining technology "that understands people" with autonomous driving technology, Toyota aims to direct drivers to a more safe and secure experience. Depending on the driver's emotions, fatigue, and awareness, the system will stimulate their autonomic nerves through visual and tactile prompts that guide them to drive safer. For example, in order to increase driver awareness, the vehicle will stimulate the driver's sympathetic nerves by providing visual feedback through blue light, as well as tactile feedback through stretch sheets. The system will also prioritize parasympathetic nerves by providing warm light, a lavender aroma, and slow music to relax drivers.
  Moreover, in addition to the surroundings of the vehicle, the system will also monitor the driver's emotions and degree of trust in autonomous driving, and switch to autonomous mode if there is danger (example: when there is a risk of deviating from the road), and guide occupants to safety. The vehicle is an embodiment of Toyota's "Mobility Teammate Concept" philosophy for autonomous driving, where the system "watches over the driver, and also helps."
New Fun to Drive (Inspire)   By combining technology "that understands people" with technology that can communicate, the vehicle is able to provide suggestions that anticipate the driver's emotions, giving them a new experience.
Source: Toyota's press release dated January 5, 2017

Key words

Toyota, autonomous driving, chauffeur, guardian

<Automotive Industry Portal MarkLines>