NVIDIA's GPU: Toyota widely adopting for the development of autonomous vehicles

Monozukuri: Musashi Seimitsu Industry commercializes AI systems for automated visual inspection



Toyota and NVIDIA fully cooperate in the development of autonomous vehicles
(Source: NVIDIA)

This report covers the recently announced partnership between NVIDIA Corporation (hereinafter referred to as NVIDIA) and Toyota Motor Corporation (hereinafter referred to as Toyota) and Musashi Seimitsu Industry Co., Ltd. (hereinafter referred to as Musashi Seimitsu Industry).

Toyota has partnered with NVIDIA since 2017. In March 2019, at the GPU Technology Conference (GTC) held in San Jose, California, NVIDIA announced that Toyota will adopt NVIDIA's DRIVE Constellation cloud-based autonomous vehicle simulation platform. The collaboration between the two companies will cover a wide range of activities for the end-to-end development to commercialization of autonomous vehicles. Toyota will be NVIDIA’s first automaker customer to adopt its Constellation platform.

On the Toyota side, Toyota, the Toyota Research Institute (TRI), and the Toyota Research Institute - Advanced Development (TRI-AD) will collaborate with NVIDIA to develop, train and validate self-driving vehicles. In March 2018, Toyota established TRI-AD in Tokyo in collaboration with Denso and Aisin Seiki to jointly conduct research and develop software for autonomous driving technologies.

Musashi Seimitsu Industry is a tier 1 supplier of high-precision auto parts for automobiles and motorcycles. The company is seeking to improve productivity by incorporating NVIDIA's Jetson AGX Xavier embedded system-on-module (SoM) developer kits for the automated visual inspection process in manufacturing. In addition, the company is developing "Self-Driving Vehicles (SDV)" for applications both within and outside of manufacturing plant environments. The company plans to start productivity improvement projects for both of these new business areas.

At the 3rd annual Artificial Intelligence · AI EXPO held in Tokyo April 3-5, 2019, Musashi Seimitsu Industry announced that it is establishing Musashi AI Ltd. as a joint venture with Israel-based SixEye Interactive Ltd. SixEye Interactive specializes in providing SaaS (Software as a Service) with a particular focus on the AV, lighting, and entertainment environment, and will jointly develop AI algorithms for Musashi Seimitsu Industry’s automated visual inspection machine and self-driving vehicle (SDV) control systems.

☆ NVIDIA offers SoC (System-on-a-chip) processors, known as the “DRIVE AGX Xavier" for autonomous vehicles (AV) and the "Jetson AGX Xavier" for robots.

Related reports:
Toyota’s CASE Strategy: Seminars from Automotive World and CES 2019 (February 2019)
NVIDIA’s DRIVE AGX AI platform for autonomous driving (October 2018)
NVIDIA: The rapid evolution of Deep Learning and its applications (August 2018)

Configuration and role of NVIDIA DRIVE Constellation

NVIDIA’s DRIVE Constellation platform was announced in March 2018. Capable of quickly simulating billions of miles of test driving in a realistic virtual world, TRI and TRI-AD are adopting the Constellation platform as part of their simulation tools for software validation and testing that are critical for the development of safe autonomous driving technologies.

DRIVE Constellation consists of two different kinds of cloud-based servers. The first is the Constellation Simulator server, which powers the NVIDIA DRIVE Sim software platform (described later) to generate output data from simulated sensors such as cameras, radars and LiDAR from a virtual car driving in a realistic virtual world. The second is the Constellation Vehicle server that contains a NVIDIA DRIVE AGX in-vehicle Pegasus chip, which processes the simulated sensor data from the Constellation Simulator as if it were data collected by sensors on real-world cars being driven on actual roads, making decisions to control virtually the accelerator, brake pedals and steering wheel the way a self-driving car control system would.

Commands from the Drive Pegasus are fed back to the simulator to verify whether the executed algorithms and software are correctly operating the simulation vehicle. Such a feedback loop is called a “hardware-in-the-loop” test loop (a loop having hardware, an ECU, that issues commands in the bit-accurate digital feedback loop).

According to NVIDIA, Toyota is the world's first automaker to adopt Constellation and the first automaker to adopt the full breadth of NVIDIA's self-driving systems for the construction of an AI-powered system to develop autonomous vehicles, from AI deep learning and the training of deep neural networks to simulation and to in-vehicle computers based on NVIDIA’s computing platforms. The two companies intend to closely collaborate going forward.

DRIVE Constellation DRIVE Constellation
The NVIDIA DRIVE Constellation simulation system provides a wide range of testing environments including images, traffic scenarios, weather conditions, and unique vehicle and sensor models

Source: NVIDIA


NVIDIA Constellation is an open platform that accumulates a wide range of information

The DRIVE Sim software generates streams of photo-realistic (photograph-like) data, and a vast array of test scenarios where accidents seem likely to occur. Constellation observes and processes real-world examples usually reflecting rare or hazardous driving scenarios such as sudden braking by the preceding vehicle, the rapid approach of a vehicle in the adjacent lane, jaywalking pedestrians, and children or animals darting out into the street ahead. Concerning "hazardous situations", it seems that Constellation also creates additional scenarios based on accident history data compiled by auto insurance companies. In addition, digital simulation specialists such as Israel-based Cognata and the automotive simulation company IPG Automotive are cooperating to accumulate simulation data.

The Constellation Simulation server also generates various driving scenarios, simulating a variety of changing conditions such as the weather, brightness, and road surfaces. For example, it can be used to mimic abnormal weather conditions such as rainstorms and snowstorms, the blinding glare of sunlight at different times of the day and limited vision at night, along with different road surfaces and terrain. The local traffic patterns of the driving environment to be tested must also be faithfully reproduced. Constellation also accumulates the model-specific performance of vehicles and sensors.

In hazardous situations such as above, the ability of an autonomous vehicle to respond can be tested using simulation without compromising the human driver as in actual test driving. The accumulation of actual driving miles and data is also important, but there is a physical limit to accumulating sufficient mileage. There are also many cases during actual test driving where nothing unusual happens. A simulation run that can reproduce a rare but very hazardous situation is considered to be far more valuable than only accumulating driving distance data.

However, because it is difficult to cover all possible hazardous situations, NVIDIA has opened its DRIVE Constellation platform to its simulation partners. The open platform makes it possible for participating simulation ecosystem partners to exchange and collaborate on simulation datasets to generate comprehensive, diverse and complex testing environments. Partner companies can access the Constellation cloud-based platform from any location.

Toyota: Widely adopting NVIDIA's GPU for the development to commercialization of autonomous vehicles

At NVIDIA’s GTC 2019 held in March 2019, NVIDIA announced that Toyota, TRI, and TRI-AD would adopt the NVIDIA DRIVE Constellation cloud-based autonomous vehicle simulation platform. Toyota is already engaged in the development of autonomous driving technologies using the NVIDIA DRIVE AGX Xavier AV computer, but going forward the partnership between the two companies will include advancements in the following three areas.

  • Building an AI computing infrastructure that leverages NVIDIA GPUs
  • Simulation using the NVIDIA DRIVE Constellation platform
  • Development of in-vehicle AV computers based on DRIVE AGX Xavier or DRIVE AGX Pegasus

☆ “Xavier” is the world's first processor developed for autonomous machines, and the most complex SoC (system-on-a-chip) ever produced. The Xavier processor was the result of 8,000 man-years of engineering effort to develop the technology. NVIDIA has been fully engaged in taking measures to ensure functional safety such as using the Dual Execution (redundancy) concept.

トヨタとNVIDIAが、自動運転車開発で全面的に協力 新型実験車両「TRI-P4」
Toyota and NVIDIA cooperate broadly in the development of autonomous vehicles
(Source: NVIDIA)
The new TRI-P4 test vehicle announced at CES 2019, based on the 5th generation Lexus LS, is equipped with the NVIDIA DRIVE AGX Xavier processor
(Source: Toyota)


Toyota Group: Established TRI-AD and J-QuAD DYNAMICS to commercialize autonomous vehicles

Toyota's MaaS-dedicated vehicle line-up, which will be extended to Toyota's autonomous mobility initiative (Toyota’s financial results for Q3 FY2018)

In March 2018 Toyota established a new company named "Toyota Research Institute - Advanced Development (TRI-AD)" in Tokyo in collaboration with Denso and Aisin Seiki to jointly develop software to further accelerate its efforts in advanced development for autonomous driving. Dr. James Kuffner, Chief Technology Officer at TRI, has been named CEO of the new company, and will coordinate closely with Toyota Research Institute (TRI) to efficiently link research results to advanced development and commercialization. TMC plans to invest more than JPY 300 billion in TRI-AD. Established with a staff of 300 employees, TRI-AD is targeting to build a staff of approximately 1,000 employees, including the external recruitment of top-level engineers globally.

In December 2018, Denso, Aisin Seiki, ADVICS and JTEKT announced that they agreed to establish a new company, J-QuAD DYNAMICS, to develop integrated control software that will be used for autonomous driving, vehicle motion control, and other related functions. The company will develop an integrated vehicle control system required to coordinate sensors, brakes, and steering systems in autonomous vehicles at a higher level of sophistication, with plans to implement the system on volume production vehicles. J-QuAD DYNAMICS will be launched in April 2019 after receiving the approval of antitrust authorities.

Many OEMs such as GM and Daimler plan to introduce autonomous driving first in mobility service vehicles.  Toyota plans to sequentially introduce the "e-Palette" for the 2020 Tokyo Olympics, the "MaaS Sienna" in 2021, and a "MaaS EV" (refer to the upper right diagram). Toyota has also indicated that these vehicles be extended to Toyota's future autonomous mobility initiative.

Musashi Seimitsu Industry: Aiming to automate visual inspection and logistics using AI

Musashi Seimitsu Industry is an automotive supplier of high-precision parts for automobiles and motorcycles. AI is being incorporated into the visual inspection process to improve productivity. In addition, the company is developing a fully-autonomous self-driving plant logistics vehicle.

Note: Musashi Seimitsu Industry is a listed company on the Tokyo Stock Exchange and the Nagoya Stock Exchange. Sales in the fiscal year ended March 31, 2018 were JPY 237.9 billion (53% for Honda, and approximately 88% for overseas customers).

If you classify the manufacturing pipeline in a plant (see lower left figure), 60% of the flow involves setup and machining (manufacturing activities) to generate added value, 20% is for the inspection process, and 20% involves the transfer of parts across the manufacturing floor. So, it turns out that 40% can be considered so-called ‘incidental work’ that the company has targeted to streamline. Musashi Seimitsu Industry implements AI on manufacturing sites under the philosophy of "more human-friendly work" and is also working on manufacturing innovation.

Visual Inspection Process:

With regards to visual inspection, even slight imperfections may lead to the malfunction and breakage of parts, and for many years visual inspection has been conducted manually. However, manual visual inspection requires a certain degree of experience, and capability variation exists among humans, depending on the person involved and how they might be feeling on any given day. There have been many attempts to automate the visual inspection process, but all have failed to match the capability of experienced veteran workers. However, the company believes the time has come to be able to improve the productivity of the manufacturing process by the deployment of AI.

Currently, the company is working on the automated visual inspection of differential bevel gears and welded gears. Imperfections in bevel gears cause noise and vibration in a vehicle. Utilizing NVIDIA’s Jetson platform, the company has succeeded in detecting minute defects on the surfaces of metal parts. Currently, as part of the trial phase of the prototype system at the company's factories, both AI and humans are double checking product quality visually. 

With regards to welded gears, since spatters (metal particles) that can occur from joining metal with a welded part may come off after the gears are installed on an actual vehicle, there is the risk that these spatters may cause a serious accident, so at present all of these gears are being visually inspected by humans. Musashi Seimitsu Industry is working on automating the visual inspection process for this spatter.

Development of SDV (Self-Driving Logistics Vehicles):

For the physical distribution of parts inside and outside the factory, Musashi Seimitsu Industry studied the utilization of an existing automated guided vehicle (AGV) but the experiment did not go well, so the company is currently developing a fully-autonomous self-driving plant logistics vehicle (SDV) that guides itself autonomously using AI. The SDV is equipped with LiDAR and a stereo camera, and AI determines the course to autonomously guide the vehicle. The SDV in the center of the lower right photo is the indoor use SDV, and the red SDV at the upper right area of the photo is the SDV for outdoor use. The SDV for indoor use can haul over 1 ton of weight and the outdoor use SDV can haul up to 770kg of weight. The company is also jointly developing a system with SixEye to centrally control multiple SDVs.

AIシステムを開発中 ベベルギヤの検査工程 SDVのデモ走行
Developing an AI system that creates new value at manufacturing sites (Source: Musashi Seimitsu Industry) Bevel gear inspection process  (Musashi Seimitsu Industry exhibit at the 3rd Artificial Intelligence • AI  EXPO 2019) Demo run of SDV (Self Driving Vehicle) (Musashi Seimitsu Industry exhibit at the AI EXPO 2019)

Established Musashi AI Ltd. to offer an optimized manufacturing process service using AI

At the 3rd annual Artificial Intelligence AI EXPO" held in April 2019, Musashi Seimitsu Industry announced that it will establish Musashi AI Ltd., a joint venture with SixEye Interactive Ltd. of Israel. The company has launched a service that realizes collaboration between humans and machines, and provides customers with new value created by optimizing the manufacturing process. The company aims to expand its AI manufacturing business to manufacturing companies, which are facing similar issues, and to  other industries.

Musashi Seimitsu Industry has been developing AI technology in collaboration with Mr. Ran Poliakine, who represents SixEye. SixEye Interactive provides SaaS (Software as a Service) for control systems, with particular focus on the AV, lighting, and entertainment environment. SixEye is jointly developing the AI algorithms for the company’s automated visual inspection system and its fully-autonomous self-driving transport vehicle (SDV).

Musashi Seimitsu Industry has already received an order from one company for PoC (Proof of Concept field testing for system introduction) services for an AI-powered visual inspection system for yoke joints. The purpose of the system is to inspect for any imperfections that may have been generated during the manufacturing process by classifying images using AI.

Starting the sales of Neural Cube:

Musashi Seimitsu Industry has developed the "Neural Cube", a tool that dramatically reduces the time it takes for client companies to introduce an AI system into their manufacturing operations. An NVIDIA Jetson TX2 embedded AI computing device can be installed with one touch to the control panel that controls the actual manufacturing work. After the installation of software such as NVIDIA's CUDA app, the Neural Cube provides all of the hardware and software required to deploy AI into the manufacturing process. Size: H178 x W116 x D139mm.


Concept model (Mockup) of a general purpose inspection machine:

The company exhibited a concept model (mockup) of a "general-purpose visual inspection machine" jointly developed with SixEye. The manufactured parts to be inspected are continuously input from the production line into the prototype general-purpose visual inspection machine shown in the lower right photo, where six parts are inspected in lots, with the inspected parts automatically sorted as good or defective products.

One-by-one inspection takes too much time even if the inspection process is converted to an AI-powered system because one-by-one inspection cannot keep pace with the speed of the entire production line.

合弁会社設立を発表 Neural Cube、NVIDIAのJetson TX2を内蔵 「全自動外観検査システム」の開発
Mr. Otsuka, president of Musashi Seimitsu Industry, and Mr. Ran Poliakine announcing the establishment of their joint venture at the 3rd Artificial Intelligence • AI EXPO 2019 Neural Cube, with the NVIDIA Jetson TX2 embedded AI computing device (There were four Neural Cubes on display) Musashi Seimitsu Industry is developing a general-purpose "Fully-automated visual inspection system" for the future


Musashi Seimitsu Industry: Exhibited at NVIDIA's GTC 2019

Musashi Seimitsu Industry exhibited at NVIDIA's GTC held in San Jose, California in March 2019, and its AI project member, Mr. Keisuke Fujita, gave a presentation with the theme of “AI Deployment in Manufacturing: Deep Learning Visual Inspection to Improve Productivity”.

In addition, a NVIDIA blog article describes the background, purpose and future prospects of this plan as follows.

In addition to being a measure to deal with labor shortages that are forecasted to result due to Japan’s declining population, Musashi Seimitsu Industry is promoting the implementation of AI technology in manufacturing sites as a major revolution in the manufacturing industry. The rationalization of the manufacturing flow using the company’s fully-automated visual inspection machine powered by NVIDIA’s Jetson platform and its fully-autonomous self-driving plant logistics vehicle can be applied to a wide range of manufacturing industries, and represents just the start of industrial transformation.

Musashi Seimitsu Industry is taking a role typically reserved for software startups, but that the company’s CEO says is the rightful provenance of a massive operation with vast data. Musashi Seimitsu Industry intends to become a strong leader in the utilization of AI-powered systems in the manufacturing industry. Also, Musashi Seimitsu Industry is working on its next-generation Neural Cube, with the idea of building more complex neural networks to identify the smallest of imperfections, which is part of the company’s bigger vision to take what it has developed from its massive manufacturing datasets to deploy its systems to other industries beyond the manufacturing industry. So to speak, the company has indicated that its innovative concept should be called "AI manufacturing as a service".

Toyota, Toyota Research Institute, Toyota Research Institute Advanced Development, NVIDIA Constellation, Musashi Seimitsu Industry, Musashi AI, NVIDIA, Jetson AGX Xavier, Self Driving Vehicle

<Automobile Industry Portal MarkLines>