Toyota's autonomous driving (2): Striving for "trillion-mile reliability"

Driving Intelligence, Interactive Intelligence, and Connected Intelligence



  The Automotive World 2017 conference was held from January 18 to 20 2017 at Tokyo Big Sight. This report will focus on a lecture given by Yoshiaki Matsuo, chief examiner of the Advanced Safety System Research and Development Division at Toyota Motor Corporation, titled, "Toyota's Direction of Automatic Driving and Future Issues - Cooperation between Intelligent Vehicles and People, and the Realization of a Mobility Society." It will also examine Toyota's efforts for autonomous driving and future challenges.

  The previous looked at Toyota's long-term direction for autonomous driving (developing both a more sophisticated ADAS and a fully autonomous driving system). This report will focus on specific plans and items currently under consideration pertaining to the three types of intelligence required for autonomous driving: driving intelligence, interactive intelligence and connected intelligence.

  The Toyota Research Institute (TRI), which will handle core development for driving intelligence, established its third U.S. facility in Ann Arbor, Michigan, and will utilize the University of Michigan's drive testing facility "Mcity" as it conducts its research. TRI is also developing a greatly improved simulator. According to Gill Pratt, CEO of TRI, globally Toyota vehicles drive 1 trillion miles annually (assuming 100 million Toyota vehicles drive an average of 10,000 miles each year). In order to realize fully autonomous driving, TRI needs to conduct 1 trillion miles worth of driving tests. However, as this will be difficult to realize, simulations are one effective method of testing. By combining real driving tests with simulations of extremely difficult conditions, TRI will attempt to achieve the same level of reliability as if it had conducted 1 trillion miles worth of tests, which in itself will be a difficult task to accomplish.

  With regards to interactive intelligence between drivers and vehicles, Toyota will conduct tests with its driving simulator at the company's Higashifuji Technical Center while also developing an HMI for autonomous driving.

  As for connected intelligence, Toyota will establish a framework where autonomous vehicles connect to a cloud server and exchange traffic information. Additionally, in situations where on-board systems alone would have difficulty controlling the vehicle like at intersections with poor visibility, Toyota vehicles will utilize a cooperative safety system that uses communication technology.

Related reports:
Toyota's autonomous driving (1): More sophisticated ADAS and fully autonomous driving (February 2017)
Toyota's ADAS technology: Autonomous Vehicle and ADAS Japan 2016 (1) (August 2016)
Toyota accelerates efforts for "connected car technology" (April 2016)

Driving intelligence

  Toyota is currently strengthening its research and development efforts for artificial intelligence. The OEM established TRI in January 2016, and announced it would invest USD 1 billion in it over five years. Moreover, TRI will conduct research and development in cooperation with the Massachusetts Institution of Technology, Stanford University, and the University of Michigan.

  With artificial intelligence, automobiles will be able to record knowledge gained while driving as experiences, instantly compare those experiences with real-time information to make decisions, and drive while formulating a safe driving plan.

  Toyota and Preferred Networks, a company in which Toyota has invested JPY 1 billion, exhibited robo-cars at the 2016 International CES. In the demonstration, several robo-cars, with a length of 43 cm and a width of 20 cm, drove around a 3 m×3 m stage. Initially, the vehicles often crashed, and could not drive at all, but thanks to machine learning, they gained enough experience to be able to drive smoothly without crashing into one another.

  During Matsuo's lecture at the 2017 Automotive World, a video of an autonomous vehicle passing through a roundabout (a circular intersection with no traffic lights) was shown. Initially, the vehicle was unable to enter the roundabout due to congestion, but thanks to machine learning, it was able to smoothly pass through.

TRI's third facility established in Ann Arbor, Michigan

The University of Michigan's autonomous driving testing facility "Mcity," in which Toyota has invested (Photo: The University of Michigan)

  In June 2016, Toyota established TRI's third facility in Ann Arbor, Michigan. The facility will primarily focus on researching fully autonomous driving. The new facility is within walking distance of the University of Michigan, and TRI will closely coordinate with the university in its research.

  The University of Michigan's Mcity is the only U.S. testing facility for autonomous vehicles and connected cars, and Toyota has invested in the University of Michigan Mobility Transformation Center, which operates it. Incidentally, among Japanese OEMs, Toyota, Nissan, Honda, and Denso are the members granted priority use of the facility.

  Moreover, an American Center for Mobility (see note below), one of 10 driving verification facilities the United States Department of Transportation (USDOT) is taking the lead on establishing around the country, is scheduled to open in the vicinity of Mcity, and this will make Ann Arbor an ideal location for TRI to move ahead with its autonomous driving research.

  Ryan Eustice and Edwin Olson, 2 professors at the University of Michigan, will also work with TRI. The professors remarked, "The development of sensors and algorithm technology is remarkable. TRI will further advance its efforts in these fields," and that "TRI will cooperate with the University of Michigan and will conduct tests in various conditions, including conditions that would occur only in the rarest of scenarios."

  TRI's CEO Gill Pratt commented that, "Including Toyota, in recent years, the automobile industry has made large advancements in the field of autonomous driving. However, much of what we have collectively accomplished has been easy, because most driving is easy. It is actually in difficult conditions where autonomous driving will be most beneficial, and TRI will undertake this difficult challenge."

  (Note) The American Center for Mobility is a drive testing facility similar to Mcity that is to be established in Willow Run, which is near Ann Arbor and in the vicinity of the Michigan state capitol building and University of Michigan. The facility will be the world's largest and most advanced drive testing facility. It will be one of the first ten USDOT designated national level driving test facilities.

The University of Michigan's "Mcity" driving test facility

  Mcity was opened in July 2015 as a testing facility for testing autonomous vehicles and connected cars on real roads and with actual driving conditions. It is located within the University of Michigan's campus, operated by the university's Mobility Transformation Center, and sponsored by various automakers.
  Mcity's 32 acres (roughly 130,000 square meters) reproduce various driving environments such as straight roads, city roads, tunnels, and railroad crossings. Traffic facilities including buildings, streetlights, and road signs can also be placed according to the needs of a given test. Additionally, Mcity features research infrastructure for traffic control systems, vehicle-to-infrastructure communication systems, high precision digital maps, and traffic simulations.
  Michigan is known for the vicissitudes of its weather, which includes rain, snow, and even tornadoes, and this provides an environment that allows for drive tests to be conducted in bad conditions. Mcity replicates real-world conditions such as poorly maintained roads, road signs that have been covered in graffiti, and faded lane markers. The facility is designed to repeatedly test new technology under tough conditions before testing on public roads.
Source: Toyota's press release dated April 7, 2016/University of Michigan's press release dated July 20, 2015

TRI develops a new, significantly more advanced simulator

  In his lecture at the 2016 GPU Technology Conference (held in April 2016) Gill Pratt announced that TRI is currently developing a new and significantly more advanced simulator. Although the details remain under wraps, Pratt said the facility is the size of a football stadium, and features a test vehicle with a GPU made by NVIDIA operating inside a dome that both moves in all directions and tilts. The surrounding scenery is shown as a graphics display, allowing testers to feel as if they were driving a real car.

Toyota aims for "trillion-mile reliability"

  According to Pratt, globally, Toyota vehicles drive 1 trillion miles annually (assuming 100 million Toyota vehicles drive an average of 10,000 miles annually). If this many vehicles drive this many miles, however unlikely, extremely dangerous situations are bound to happen. For example, if an autonomous vehicle comes across a car being driven violently (most likely by a human), even if the other vehicle is at fault, the autonomously driving vehicle must act in a manner consistent with laws and regulations.

  Therefore, in order to realize fully autonomous vehicles, they need to be tested for 1 trillion miles. However, this is difficult to accomplish in reality, making simulations an effective alternative. By combining actual driving tests with simulations of extremely dangerous conditions (due to the high possibility of crashes, there are limits to what can be accomplished through driving tests), Toyota will take on the challenge of achieving reliability equivalent to 1 trillion miles, although it has also noted that this will be no easy feat.

  Additionally, Gill Pratt has stated that the Guardian driver support system, where vehicles cooperate with drivers, does not require the level of perfection that is demanded from fully autonomous vehicles, but that there need to be simulations to see how drivers and vehicles cooperate with one another.

Investigating causes of accidents involving autonomous vehicles and preventing their recurrence

  Pratt discussed the need to investigate the causes of accidents involving autonomous vehicles in order to prevent their recurrence.

  Although driving records are expected to be attainable in the event of an accident, a complete investigation of the causes is difficult. Despite the astounding breakthroughs achieved with machine learning, particularly deep learning, autonomous systems cannot explain how conclusions were arrived at. There is also no guarantee that deep learning will choose to take the correct actions in response to various inputs. Because the range of necessary countermeasures is so broad, it is difficult to determine if they are sufficient or not. As a result, checking with test runs and simulations is of the highest priority.

Interactive intelligence

  Toyota aims to develop autonomous driving wherein drivers and vehicles cooperate with one another. The automaker will establish a system to form a relationship between them in levels 2 and 3 of the five-stage SAE standard, and use artificial intelligence that has learned from these stages to advance into level 4 and 5 fully autonomous driving (bottom left image).

  The driver takes prominence in level 2 driving, and must be ready to take control at any given situation during autonomous driving. In level 3, the system and driver share equal responsibility. When the system determines it is at the limit of its capabilities, it will swiftly request that the driver take control (center image below).

  In either situation, in order to make the transition from autonomous to manual driving as smooth as possible, the system needs to constantly communicate the system conditions to the driver, along with ascertaining the driver's condition so it can hand over the controls. Additionally, in level 3, controls will be entrusted to the system, thereby requiring more time to revert control back to the driver than in level 2 (bottom right image).

Development of an HMI for autonomous driving with a driving simulator

  Toyota has identified the balance point between the system and driver based on the need to secure safety and free drivers from the task of driving, and will develop a HMI that allows the system to help the driver maintain that balance. The center image below diagrams this process. Advanced technology is implemented, and, for example, when the sensors detect that there are few moving cars around the vehicle, the driver will be allowed to remain relaxed. However, when traffic becomes congested, and the system determines that autonomous driving is complicated, it will encourage the driver to stay focused. The driver is required to be above or to the right of the balance line as depicted in the image (within the shaded acceptable range in the image). If the driver falls below this line, the system will stimulate the driver to wake them up.

  Toyota is utilizing driving simulators to conduct research on optimal driver monitoring, driver information displays (HUDs and Multi Information Displays (MIDs), and etc.), light indicators, and sounds to help drivers maintain cognition.

  Driving simulators are machines that use video footage and acceleration/deceleration generators to simulate driving, and analyze an individual's fitness for driving by monitoring declines in awareness (falling asleep, dazing off) and inattentiveness to danger (not looking at the road, not checking for safety). Driving simulators have helped develop preventive safety technologies that reduce accidents effectively. The Higashifuji Technical Center has a driving simulator that had the highest level of functionality in the world at the time it was established in 2007.

Examples of HMI developed by Toyota (upper right)

HUD   The HUD in the photo above and to the right not only displays vehicle speed, but also the steering wheel and driver's hands in order to encourage the driver to drive on their own.
MID   The display communicates information about the vehicle in an easy to understand manner. For example, the MID shows information such as surrounding vehicles, predetermined maximum speed, current speed, and as announcements including when the vehicle is about to change lanes.
Approaches to driver cognition   Toyota is developing a system that will help maintain driver cognition through light indicators and sounds.
Source: Toyota's press release dated September 5, 2014/January 5, 2017

Connected intelligence

  In April 2016 Toyota partnered with Microsoft to establish Toyota Connected (TC) in the U.S. for the purpose of consolidating and utilizing big data collected by automobiles. TC will apply big data to product development, as well as a broad field of applications. The company will also work closely with TRI.

  In the field of autonomous driving, all vehicles will be connected to a cloud server and send and receive various types of traffic information. In the future, 3D maps of highways and car-exclusive roads developed primarily by Dynamic Map Planning Co., Ltd., as well as T probe traffic information collected by Toyota's telematics vehicles will be distributed to autonomous vehicles via large-capacity communication.

  Additionally, in situations where the onboard system alone will have difficulty responding to conditions like at intersections with poor visibility, Toyota will utilize cooperative safety systems (vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication). As of February 2017, these safety systems can be equipped in the Toyota Prius, Majesta, Crown (Royal/Athlete), and the Lexus RX as an option.

  In April 2016, Toyota USA announced it would begin joint V2V/V2I verification experiments with the University of Michigan in southeast Michigan and downtown Ann Arbor using dedicated short range communication. Previously, V2V/V2I verification experiments had to deal with a lack of vehicles equipped with communication equipment. However, Toyota asked its employees and their families to participate in the experiment, with an aim to have 5,000 vehicles participate. Toyota has followed USDOT's policy of deploying vehicles equipped with V2V/V2I across the U.S.

  (Note) V2V: Vehicle-to-vehicle, V2I: Vehicle-to-infrastructure

Key words

Toyota, autonomous driving, Toyota Research Institute, trillion-mile reliability

<Automotive Industry Portal MarkLines>