Recipient Organization Regents of the University of Minnesota 200 Oak Street SE; Suite 450 Minneapolis, MN 55455-2070 Recipient Identifying Number CON# 042705 DUNS and EIN Numbers DUNS: 55-591-7996 EIN: 41-6007513 Federal Grant Number DTRT13-G-UTC35 Development of a Workshop on Automated Vehicle Technologies Brian Davis Reed Johnson Department of Mechanical Engineering University of Minnesota Final Report Recipient Organization Regents of the University of Minnesota 200 Oak Street SE; Suite 450 Minneapolis, MN 55455-2070 Recipient Identifying Number CON# 042705 DUNS and EIN Numbers DUNS: 55-591-7996 EIN: 41-6007513 Federal Grant Number DTRT13-G-UTC35 Recipient Organiz ti n Regents of the University of Minnesota 200 Oak Street SE; Suite 450 Minneapolis, MN 55455-2070 Recipient Identifying Number CON# 042705 DUNS and EIN umbers DUNS: 55-591-7996 EIN: 41-6007513 Federal Grant Number DTRT13-G-UTC35 CTS 19-23 Human-centered solutions to advanced roadway safety Technical Report Documentation Page 1. Report No. 2. 3. Recipients Accession No. CTS 19-23 4. Title and Subtitle 5. Report Date Development of a Workshop on Automated Vehicle Technologies July 2019 6. 7. Author(s) 8. Performing Organization Report No. Brian Davis, Reed Johnson 9. Performing Organization Name and Address 10. Project/Task/Work Unit No. Department of Mechanical Engineering University of Minnesota 111 Church St. SE Minneapolis, MN 55455 CTS #2018052 11. Contract (C) or Grant (G) No. DTRT13-G-UTC35 12. Sponsoring Organization Name and Address 13. Type of Report and Period Covered Roadway Safety Institute Center for Transportation Studies University of Minnesota University Office Plaza, Suite 440 2221 University Ave SE Minneapolis, MN 55414 Final Report 14. Sponsoring Agency Code 15. Supplementary Notes http://www.roadwaysafety.umn.edu/publications/ 16. Abstract (Limit: 250 words) Connectivity and automation in vehicles have the potential to change nearly every aspect of our transportation system. Vehicles with these capabilities are already being tested on public roads and are beginning to enter the national fleet. These vehicles have the potential to bring numerous benefits to the public, but these benefits come with tradeoffs that must be carefully considered by transportation professionals. This project focuses on the development and delivery of a two-day workshop on connected and automated vehicles (CAVs) for an audience of public transportation professionals. This workshop was designed to not only provide an overview of CAV technologies but to also to take an in-depth look at Global Navigation Satellite Systems, LIDAR, and the software and algorithms that use these sensors for localization and navigation. The goal of the workshop was to help develop technical literacy about these technologies among workshop participants to provide context when encountering CAV technologies. A workshop was developed that incorporates classroom lectures and demonstrations using a robotic unmanned ground vehicle. A pilot offering of the workshop was delivered in May 2019, which was attended by eight public transportation professionals representing different agencies at different levels of government. Participant feedback was collected to evaluate the efficacy of the workshop. Responses were almost exclusively positive, highlighting participants’ interest in both classroom sessions and demonstrations. Using constructive feedback, improvements have been identified for future offerings. Workshop participants found the workshop to be interesting and reported learning about the topics covered, motivating future offerings. 17. Document Analysis/Descriptors 18. Availability Statement Connected vehicles, Intelligent vehicles, Education and training, Workshops, Continuing education No restrictions. Document available from: National Technical Information Services, Alexandria, Virginia 22312 19. Security Class (this report) 20. Security Class (this page) 21. No. of Pages 22. Price Unclassified Unclassified 27 Development of a Workshop on Automated Vehicle Technologies FINAL REPORT Prepared by: Brian Davis Reed Johnson Department of Mechanical Engineering University of Minnesota July 2019 Published by: Roadway Safety Institute Center for Transportation Studies University of Minnesota University Office Plaza, Suite 440 2221 University Ave SE Minneapolis, MN 55414 The contents of this report reflect the views of the authors, who are responsible for the facts and the accuracy of the information presented herein. The contents do not necessarily represent the views or policies of the United States Department of Transportation (USDOT) or the University of Minnesota. This document is disseminated under the sponsorship of the USDOT’s University Transportation Centers Program, in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof. The authors, the USDOT, and the University of Minnesota do not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to this report. ACKNOWLEDGMENTS The funding for this project was provided by the United States Department of Transportation’s Office of the Assistant Secretary for Research and Technology for the Roadway Safety Institute, the University Transportation Center for USDOT Region 5 under the Moving Ahead for Progress in the 21st Century Act (MAP-21) federal transportation bill passed in 2012. The author would also like to acknowledge the following people for their assistance with this project: Rodney Dockter at Boon Logic, Inc. for his work preparing and delivering workshop lecture material. Max Donath, of the University of Minnesota for his assistance preparing workshop demonstrations and lecture material. The participants of the pilot workshop for offering their attendance, engagement, and feedback. TABLE OF CONTENTS CHAPTER 1: Introduction ....................................................................................................................1 1.1 Project Overview ................................................................................................................................ 1 1.2 Workshop Format and Topics Covered .............................................................................................. 1 1.3 Report Organization ........................................................................................................................... 2 CHAPTER 2: Summary of Classroom Sessions ......................................................................................3 2.1 Introduction to Connected and Automated Vehicles ......................................................................... 3 2.2 Introduction to the Clearpath Husky and Robot Operating System................................................... 5 2.3 Introduction to GNSS: Part 1 .............................................................................................................. 6 2.4 Introduction to GNSS: Part 2 .............................................................................................................. 7 2.5 Introduction to LIDAR ......................................................................................................................... 7 2.6 Machine Learning ............................................................................................................................... 8 2.7 Software and Algorithms for Localization and Navigation ................................................................. 9 CHAPTER 3: Workshop Demonstrations ............................................................................................ 11 3.1 Demonstration Platform ................................................................................................................... 11 3.2 Summary of Demonstration Sessions ............................................................................................... 12 3.2.1 GNSS Demonstration ................................................................................................................. 12 3.2.2 LIDAR Demonstration ................................................................................................................ 13 3.2.3 Sensor Fusion Demonstration ................................................................................................... 13 CHAPTER 4: Conclusions ................................................................................................................... 14 4.1 Participant Feedback ........................................................................................................................ 14 4.2 Future Work ...................................................................................................................................... 15 4.3 Project Findings ................................................................................................................................ 15 REFERENCES .................................................................................................................................... 16 APPENDIX A Final Workshop Schedule LIST OF FIGURES Figure 2.1 Simplified CAV architecture. ........................................................................................................ 4 Figure 2.2 Generalized neural network model. ............................................................................................ 9 Figure 3.1 Clearpath Husky. ........................................................................................................................ 11 Figure 3.2 Workshop demonstration area. ................................................................................................. 12 LIST OF ABBREVIATIONS AHRS Attitude and heading reference system CAV Connected and automated vehicle DSRC Dedicated short range communication GNSS Global navigation satellite system GPS Global Positioning System IMU Inertial measurement unit LIDAR Light detection and ranging RADAR Radio detection and ranging ROS Robot Operating System UGV Unmanned ground vehicle 1 CHAPTER 1: INTRODUCTION 1.1 PROJECT OVERVIEW Connectivity and automation in vehicles have the potential to change nearly every aspect of our transportation system. Vehicles with limited automation capabilities are already entering the national fleet. Companies developing fully self-driving systems are currently testing their vehicles in a number of states. These vehicles have the potential to increase safety among road users, improve accessibility, especially for those with mobility impairments, and optimize vehicle utilization and cost sharing. However, managing this emerging technology requires careful planning among public transportation professionals who are tasked with ensuring public safety and maximizing the benefits of this technology. Transportation professionals at the state, county, and local levels of government are increasingly becoming responsible for understanding the technology enabling, infrastructure supporting, and policies regulating connected and automated vehicles (CAVs). This can be challenging for professionals who have not traditionally required highly technical knowledge in these areas. One way to support these professionals is by providing education that will promote a deeper understanding of the underlying technologies enabling connected and automated vehicles. This project focuses on the development and delivery of a workshop on connected and automated vehicle technologies designed for an audience of public transportation professionals. The goal of the workshop was to not only provide an overview of CAVs but also in-depth knowledge about select enabling technologies that would allow participants to demonstrate technical literacy about these components and sub-systems. This was accomplished over a two-day workshop that included both classroom lecture material and live demonstrations. The course material was designed to be interesting, informative, and applicable, seeking to provide information that would be useful for professionals preparing for CAVs. 1.2 WORKSHOP FORMAT AND TOPICS COVERED A pilot offering of the developed workshop was delivered as a part of this project. The goal of this offering was to solicit participant feedback to assess the suitability of the workshop for the intended audience of public transportation professionals. The workshop was offered over two days on the University of Minnesota Campus. The format of the workshop included both classroom-based lecture sessions as well as outdoor demonstrations featuring sensors, algorithms, and software discussed in the lectures. The classroom sessions included an overview of CAVs as well as in-depth sessions on Global Navigation Satellite Systems (GNSS)/Global Positioning System (GPS), LIDAR, machine learning, and navigation algorithms and software. The topics discussed in-depth were identified based on interest from project stakeholders and applicability to CAVs operating in northern climates. 2 The demonstrations utilized a small unmanned ground vehicle (UGV) equipped with sensors and software commonly used in CAVs. This robot was chosen as a platform for the demonstrations because it would be much safer than a passenger vehicle operating in an autonomous or partially autonomous mode, it was more cost effective than procuring or developing an automated vehicle, and it would be much easier for a group of workshop participants to simultaneously interact with an unmanned ground vehicle than take turns riding in a passenger vehicle. The demonstrations developed for the workshop included examples of GNSS performance and navigation, LIDAR-based navigation, and sensor fusion. 1.3 REPORT ORGANIZATION This report documents the work performed as a part of this project including the development of the workshop’s lectures and demonstrations as well as the analysis of the workshop feedback. Chapter 1 provides an overview of the project and a summary of the workshop. Chapter 2 summarizes the classroom-based lecture sessions at a high level. The workshop included seven classroom sessions each lasting approximately 45 to 60 minutes. While Chapter 2 focuses on describing the content presented in each session, it is not intended to be stand-alone reference on the topics. It is noted that this report does not necessarily include all the material presented in the lectures. It also does not discuss the topics in the same depth. Chapter 3 discusses the demonstrations developed for the workshop as well as the unmanned ground vehicle used as a platform for these demonstrations. Chapter 4 summarizes the feedback collected from workshop participants, possible workshop changes to improve future offerings, and the project’s findings. Appendix A provides the workshops final schedule for reference. 3 CHAPTER 2: SUMMARY OF CLASSROOM SESSIONS The workshop included seven classroom-based lecture sessions each lasting approximately 45 to 60 minutes. The primary focus of these sessions was to deliver lecture content, but an emphasis was also put on soliciting questions and discussion from the workshop participants. The topics included a high-level overview of CAV technology as well as in-depth sessions on a single sensor type or category of algorithms. The content was developed in this way to first establish a baseline level of knowledge and define a common vocabulary with which to discuss further topics. In-depth sessions on GNSS, LIDAR, navigation algorithms, and machine learning were designed to provide information about some of the technologies enabling CAVs. The goal of these sessions was to provide foundational information about the technologies, highlight some applications, and to discuss their advantages and disadvantages when used in CAVs. 2.1 INTRODUCTION TO CONNECTED AND AUTOMATED VEHICLES Connectivity and automation in vehicles have the potential to change nearly every aspect of our transportation system. Vehicles with limited automation capabilities are already entering the national fleet and sophisticated, self-driving vehicles are currently being tested. According to the U.S. DOT, automation is the “use of electronic or mechanical devices to operate one or more functions of a vehicle without direct human input [1].” This is related but distinct from connectivity in vehicles which is the use of wireless communications enabling a vehicle to exchange information with other vehicles, infrastructure, and road users. Collectively, vehicles with advanced features are referred to as connected and automated vehicles or CAVs. It is noted that the title “automated vehicle” encapsulates varying levels of automation ranging from a vehicle with adaptive cruise control to a fully self-driving car. To better describe the capabilities of automated vehicle systems, SAE International has published the SAE J3016 [3] standard to describe different levels of automation. These range from Level 0, a vehicle with features “limited to providing warnings and momentary assistance” to Level 5, a vehicle that can drive itself under all conditions. Connected and automated vehicles are complex systems that require sensors to observe their environment and software to implement algorithms to interpret and act on that information. At a basic level, these vehicles must first take in raw information about their position, dynamics, and environment using sensors such as GPS, LIDAR, vision, etc. Then that information can be processed to form a basic perception of the environment such as nearby obstacles, the vehicles position, or information from road signs. That basic information must then be interpreted in order to create a plan for where the vehicle will go and how it will accomplish that goal. Lastly, the vehicle must act on that plan by actuating vehicle controls such as steering, throttle, and braking. A representation of this architecture is shown in Figure 2.1. 4 Figure 2.1 Simplified CAV architecture. There are many different types of sensors used in CAVs that allow the system to determine its location, the vehicle’s dynamics, and the position of other vehicles, road users, obstacles, and road geometry. Global Navigation Satellite Systems (GNSS) are one type of sensor that allow the system to determine its location on the globe by triangulating its position based on signals sent from satellites. The most common GNSS is the Global Positioning System (GPS) which is the GNSS operated by the U.S. Department of Defense. GNSS receivers can determine their location to the centimeter level or better depending on the quality of the signals received and the availability of local corrections data. Another common sensor is LIDAR (light detection and ranging) which emits laser pulses and times their return to determine the geometry of the environment around the sensor. This technique generates a point cloud, a 3D representation of the environment around the sensor made up of individual x, y, z points. This data is useful when determining the position of nearby objects and determining their trajectory. It can also be used to estimate the vehicle’s motion by observing how stationary environmental objects move with respect to the sensor. RADAR (radio detection and ranging) is a sensor that uses radio waves to capture information about individual objects or targets around the vehicle. Unlike LIDAR, RADAR does not return a complete 3D representation of the environment around the sensor but rather discrete positions of individual targets. Because of this, RADAR is best suited for obstacle detection tasks as opposed to navigation tasks. It is generally less accurate than LIDAR but is also less expensive which is why it is a frequently used sensor. Attitude and heading reference systems (AHRS) are a class of sensor that detect changes in the vehicle’s motion such as acceleration, braking, turns, etc. using inertial measurements and magnetometers (i.e. a compass). These measurements when combines provide an estimate of the vehicle’s orientation in space (i.e. its roll, pitch, and yaw). An AHRS sensor doesn’t directly provide location information but by integrating the inertial measurements a position estimate can be calculated. It is noted however, that this estimate is susceptible to dead reckoning errors which is to say that small errors in the inertial 5 measurements will add up over time causing the position estimate to become less accurate. However, these sensors are frequently paired with GNSS receivers which provide accurate position but generally are not capable of generating accurate orientation information. Cameras and computer vision when paired together are capable of sensing visual information about the environment around the vehicle. This is done by using a camera (or multiple cameras together) to capture video and then using computer vision algorithms to process the video to extract information such as the relative position of lane markings and other road users. They are also capable of capturing and interpreting the meaning of objects such as road signs or whether or not an emergency vehicle’s lights are active. A major advantage of video-based sensors is that cameras are relatively inexpensive as compared to other sensors although the tradeoff is that they require complex processing necessitating more powerful computers to interpret the raw video feed. Connectivity is another powerful tool for automated vehicles. This can refer to communications with other vehicles, the infrastructure, or other road users. By exchanging information with others, connected vehicles can collect information beyond what they can sense themselves. For example, a vehicle could broadcast a notification that it is about to perform an emergency braking maneuver. This would allow a following vehicle to take action on this information before it would have been able to detect the braking maneuver using its own sensors. Communications can also enable a number of other applications increasing safety, network efficiency, and driver convenience. 2.2 INTRODUCTION TO THE CLEARPATH HUSKY AND ROBOT OPERATING SYSTEM The Husky is a rugged, outdoor-ready unmanned ground vehicle made by Clearpath Robotics [4]. To support the workshop, this robot was procured to be used as a platform for demonstrations. It is equipped with a number of CAV technologies including GNSS/GPS, LIDAR, AHRS/IMU, cellular communications, and a computer. These sensors enable the robot to autonomously complete certain navigation tasks such as waypoint following and obstacle detection and avoidance. It is noted that additional information about the robot is presented in Chapter 3. To enable these automation applications, the robot’s computer must at a minimum, interface with the sensors and process their raw output, combine and interpret the information collected from the different sensors, make decisions about how to complete its task(s), and issue control commands to the robot’s wheels. To facilitate these tasks, the robot runs a collection of software called the Robot Operating System (ROS) [5]. ROS is an open-source architecture for distributed inter-machine communication and configuration. Functionally, it is a middleware framework designed to provide a standardized interface for roboticists working with different sensors, algorithms, and robots. ROS is also accompanied by a strong community of developers who create open-source software to support ROS. The ROS ecosystem includes a number of software tools for interfacing with a robot such as RViz [6]. RViz is a 3D visualizer for displaying sensor data and state information from ROS. This may include visualizing data such as LIDAR point clouds or video camera imagery. It allows for an intuitive understanding of the processes going on behind the scene which is particularly useful for debugging 6 issues. Another useful tool is Gazebo [7], a 3D dynamic system simulator. It can be used to create a virtual 3D environment in which a simulated robot is placed. The system can then generate simulated data from LIDAR, RADAR, cameras, etc. to feed back into ROS. By default, ROS includes a fully functional navigation stack (i.e. a collection of related packages when combined handle navigation behaviors). This stack is capable of map generation, path planning, obstacle avoidance, and combining these sub-components in order to issue movement commands to the robot. 2.3 INTRODUCTION TO GNSS: PART 1 A global navigation satellite system (GNSS) is the general term to describe a satellite-based position and timing system. The Global Positioning System (GPS) [8] is the most ubiquitous example and is the GNSS operated by the U.S. Department of Defense. Other systems operated by other entities include Russia’s GLONASS, China’s BeiDou, and the European Union’s Galileo. Other regional (non-global) systems are operated by countries such as India, Japan, and France. The Global Positioning System is made up of three segments. The first is the space segment which is comprised of the 31 satellites in orbit around the Earth. The system is designed such that no matter where a receiver is on Earth, 6 satellites are visible (although not accounting for local terrain or man- made objects). The space segment is monitored and managed by the ground or control segment. This consists of a master control station in Colorado which is supported by command and control antennas as well as additional monitoring sites around the Earth. This system monitors satellite health and accurately tracks the position of the satellites to account for any orbital drift. Lastly, the user segment is made up of the individual GPS receivers using the system to determine positioning and timing information. Receivers receive timing information from the satellites that can be used to determine the distance between the receiver and the satellites to which it’s listening. Because the satellites broadcast their own position (among other information), the receiver can then triangulate its own position. This is the basis for the standard positioning service, the most basic type of position solution that is accurate on the order of 10 meters. System accuracy is affected by a number of error sources. One source of error is the atmospheric effects which affect the time it takes for the signal to travel from the satellite to the receiver. Other errors can stem from satellite position or timing errors which affects the receiver’s range measurement. A significant source of error can be multipath which is when the satellite signals reflect off of terrain or buildings such that the receiver hears multiple echoes of the same message. This can cause errors on the order of meters depending on the local geometry. System errors can be reduced by using corrections systems. One such example is the Wide Area Augmentation System (WAAS) which uses ground stations at known positions to estimate system errors. These corrections can then be re-broadcast back to corrections satellites which then in turn, broadcast the information to receivers. Using this method, accuracies can improve to roughly 1 meter. 7 Another common method for improving system accuracy is to use real-time kinematic positioning. This technique involves using nearby reference stations or reference station networks to provide corrections data. Unlike WAAS, this technique generally requires higher end receivers capable of fully incorporating the information from the correction station. One method for receiving corrections is through the use of a statewide continually operating reference station (CORS) network such as the MnCORS network [9]. When pairing MnCORS corrections with a high-end receiver, the horizontal accuracy of the position can be better than 1 cm. 2.4 INTRODUCTION TO GNSS: PART 2 When using GNSS for a given application, it’s important to consider the performance of the system. This includes not only the accuracy of the position solution but also other metrics such as the initialization time or time to first fix (i.e., the time it takes to first calculate a valid position solution) as well as the continuity and availability of the solution (i.e., how frequently a high quality position fix is available and how the receiver recovers when it isn’t). Another important factor when using GNSS is how the position is being reported and how to transform that into a useful coordinate system. There are two main types of coordinate systems for Earth-based navigation. The first is geodedic datums which model the earth as a non-flat ellipsoid. The most common system is the World Geodetic System 1984 (WGS84) model which is the system used by GPS. Another common system in the U.S. is the North American Datum 1983 (NAD83) which is a local approximation of Earth’s shape in North America. The other main type of coordinate system is the 2D projection. Projections model a portion of Earth as a flat plane. These models are more accurate over smaller areas. Common projections for navigation in the U.S. include the Universal Transverse Mercator (UTM) system, the state plane coordinate systems, and other more local systems such as county- or city-specific systems. Although GNSS gives the user a relatively precise estimate of their location, this information by itself is not useful without context about the environment in which a user is navigating. Maps are an important tool to provide context to the position information given by GNSS. This information can be coarse network-level information like the rough position of roads and points of interest or highly accurate maps providing information about individual lane markings at the decimeter- or centimeter-level. Overall, GNSS is a powerful tool enabling navigation-based tasks in automated vehicles. It provides a highly accurate measure of a vehicle’s absolute position. However, it requires line of sight to the sky and additional context in the form of maps to allow for a meaningful interpretation of the position. 2.5 INTRODUCTION TO LIDAR Light Detection and Ranging (LIDAR) is a type of sensor that uses laser pulses to detect the physical geometry of the environment around the sensor. The sensor emits a laser pulse and then times how long it takes for the pulse to return in order to calculate the distance to the nearest object in a given 8 direction. This process is repeated as many as million of times per second in many directions to generate a 3D point cloud of individual x, y, z points representing the geometry of the environment. LIDAR hardware varies in a few key metrics. The first is the accuracy of the point cloud created which for many automotive LIDAR units is roughly 1 to 5 cm. Range and field of view are also very important factors which together describe the volume where the sensor can detect objects. Range is a measure of the furthest object from the sensor that can be detected. The field of view represents in what directions a sensor can collect information. Lastly, point density describes the spacing of the emitted laser pulses. This is particularly important because it affects how many points in a point cloud represent an object of a given size at a given distance. Detecting smaller objects accurately generally requires a higher point density as compared to large objects. Processing the data collected by LIDAR is a challenging task. It requires a high bandwidth connection between the computer and the sensor in order to receive all the data in real-time. For example, a Velodyne HDL-64E LIDAR [8] scanner can create 2.2 million points per second. This data rate creates a about 2.5 to 3 GB of data in 10 minutes. Processing the data is a computationally expensive task and can require powerful processing power to do so in real-time. For CAVs this frequently involves segmenting the point cloud into objects and classifying them (e.g., the ground, cars, pedestrians, etc.). LIDAR data can also be used for local navigation by observing how stationary objects move relative to the sensor. LIDAR sensing provides dense and relatively accurate (roughly 1 to 5cm) data about the geometry of the environment around the sensor. This is particularly useful for obstacle detection and avoidance as well as local navigation tasks. However, LIDAR sensors are relatively expensive when compared to other CAV sensor types and processing the data they create is challenging and computationally expensive. 2.6 MACHINE LEARNING It is noted that this lecture was prepared and delivered by Rodney Dockter of Boon Logic Inc. and an adjunct professor in the Department of Mechanical Engineering at the University of Minnesota. Vision-based systems enable CAV applications such as object detection and classification, lane marking detection, and semantic segmentation. Across these varied applications, computer vision is commonly performed using neural networks. Neural networks are a machine learning computational model that at its most basic level receives input which is processed through several layers of operations which in turn generates some output. The general form of a neural network is represented in Figure 2.2 which shows the input on the left, the processing layer in the middle, and the output on the right. 9 Figure 2.2 Generalized neural network model. In a neural network, each layer is made of a number of nodes (also called neurons). Each node performs a linear combination (i.e., weighted sums) of all the values from the previous layer. By combining the values in this way, the “knowledge” of the network is stored in the weights. Although this is a simple mathematical operation, the complexity of the model is created by adding additional layers and additional nodes per layer. As the model grows, so does its ability to generate useful output. Neural networks are an example of a supervised machine learning which means they require annotated data with which the model is trained. The training data is a set of inputs paired with known-correct outputs. This dataset can then be used to train the model, which is to say set the weights for the individual nodes. The training process can be very challenging often requiring a great deal of training data and computational power to determine an appropriate set of weights. Another challenging aspect to using neural networks is choosing an appropriate model architecture. One type of neural network that is frequently used for computer vision is a convolutional neural network (CNN). CNNs make the assumption that the input is an image so nearby pixels can be grouped together without losing information. Neural networks are powerful mathematical models that take inputs such as video in order to perform a task such as segmenting the image, detecting lane lines, or identifying and classifying obstacles in the image. However, this technique is computationally expensive often requiring powerful processors to run in real-time. Additionally, constructing and training these models is even more computationally expensive and requires a great deal of training data. 2.7 SOFTWARE AND ALGORITHMS FOR LOCALIZATION AND NAVIGATION Localization and navigation are processes by which a vehicle processes the information available from sensors and determines where it is and how to get to its goal position. GNSS is commonly used for localization as it gives absolute position data which can then be tied to information from a map. 10 However, in GNSS denied areas such as urban canyons, other techniques must be used to augment the position solution. One such method is simultaneous localization and mapping (SLAM) which is a technique that uses LIDAR to create a map of the environment around the vehicle and also determine its own position within that map. This is particularly useful in areas the vehicle may travel at a later time since the created map can be reused to aid in the navigation task. Another method is Adaptive Monte Carlo Localization (AMCL) which uses a similar technique with a pre-existing map. Here, the map describes the expected LIDAR point cloud which can then be matched to the actual point cloud generated from the environment. When combined, the vehicle can determine its position within the map. Another method for LIDAR-based localization is LIDAR odometry. This is a technique that does not directly determine the position of the vehicle but rather calculates the difference in position between sequential data frames. This results in a set of deltas that can be used to construct an estimate of the vehicle’s position but is susceptible to dead reckoning errors. This is done by extracting stationary landmark features from the environment and in subsequent data frames, searching for those landmarks. Then based on their position relative to the sensor, the vehicle’s change in position can be determined. LIDAR can also be used for obstacle detection and avoidance. Here, the point cloud is segmented to filter out points corresponding to the ground and other non-obstacles. The remaining points are then assumed to belong to obstacles which can then be avoided. The manner in which drivable and non- drivable areas are represented is a local cost map. This is a map of the area directly around the vehicle and the locations of obstacles that must be avoided. This is in contrast to the global cost map which is the map of the entire area the vehicle knows about including its position as well as any waypoint or goal positions. Together the global and local cost maps determine how the vehicle should travel through the environment. 11 CHAPTER 3: WORKSHOP DEMONSTRATIONS The workshop was designed with a focus on providing real-world examples of the technologies discussed in the classroom sessions. This was accomplished through demonstrations using the Husky, an unmanned ground vehicle (UGV) made by Clearpath Robotics. This chapter describes the Husky UGV and the technologies it uses to support the demonstrations. 3.1 DEMONSTRATION PLATFORM To facilitate workshop demonstrations, a Clearpath Husky [4] unmanned ground vehicle was procured along with sensors commonly used in connected and automated vehicle. The robot and its sensors serve as a platform capable of demonstrating a number of different concepts. The Husky itself is a rugged mobile robot with four wheels that is roughly 3 feet long and 2 feet wide. It is designed to accommodate many different sensor and payload configurations for use in different applications. The configuration used in this workshop is shown in Figure 3.1. Figure 3.1 Clearpath Husky. Additional equipment not shown in the image includes wheel encoders for wheel odometry, an internal computer to run the system software, and a WI-FI router to enable wireless connectivity to the robot. The primary sensors used for the demonstrations included the GNSS, LIDAR and AHRS/IMU. The GNSS receiver on the Husky is a Trimble BX992 [11] paired with a single Trimble Zephyr 3 antenna. This combination is capable of operating in RTK mode when using the cell modem to receive corrections data from the MnCORS network. In this mode, the receiver is capable of a position solution accurate to 12 the 1cm level. The LIDAR sensor is a Velodyne VLP-16 [12] which is capable of scanning in 360 degrees around the robot with a 30-degree vertical field of view producing 300,000 points per second. It has an accuracy of 3cm which is suitable for both obstacle detection and LIDAR-based localization techniques. Lastly, the AHRS/IMU is a PhidgetsSpatial Precision 3/3/3 High Resolution, model 1044_0B [13]. This unit has a 3-axis accelerometer, gyroscope, and compass. The UGV uses the Robot Operating System (ROS) to integrate with the sensors and perform the localization and navigation tasks. It runs on the internal single-board computer which has an Intel i7 processor. In addition to the computer, the inside compartment of the robot also houses the power supply, batteries, and a WI-FI router used to wirelessly connect to the computer. 3.2 SUMMARY OF DEMONSTRATION SESSIONS The demonstration sessions took place near the Civil Engineering and Shepherd Laboratories buildings on the University of Minnesota campus. This area provided paved open spaces, grassy areas with hills, sidewalk paths, and varied obstacles including buildings, walls, bike racks, and pedestrians. A map marking the demonstration area is shown in Figure 3.2. Figure 3.2 Workshop demonstration area. 3.2.1 GNSS Demonstration The GNSS demonstration was designed to show the relative accuracy with which a GNSS guided vehicle can follow user-defined waypoints. A series of waypoints was created that would take the robot through the demonstration area. Using GNSS for position information and the AHRS only for heading, the system traversed the waypoints, driving through the area on the pre-determined path. 13 In addition to the waypoint path, the robot was also given a simpler set of waypoints creating a square with 5m sides in order to show the accuracy with which the robot could return to its starting position when using GNSS-provided position. Lastly, the GNSS-provided position was visualized superimposed on a map of the area as the robot was teleoperated by a human. The robot was taken on a path between two relatively tall buildings, creating an urban canyon. This caused drastic errors in the GNSS position. This was visualized by noting where the robot should have been as compared to the reported position, which at its worst, was over a city block away. This demonstrates that although GNSS can be very accurate, it is still susceptible to issues caused by sky occlusion and multipath. 3.2.2 LIDAR Demonstration The LIDAR demonstration was designed to show two examples of LIDAR-based localization. The first was LIDAR odometry using an algorithm called LeGO-LOAM (lightweight and ground optimized LIDAR odometry and mapping) [14]. This was demonstrated using the same square shape from the GNSS demonstration. Again, the robot was tasked with driving the perimeter of a square with 5m sides. This time, only lidar odometry was used to determine the vehicle’s position and heading. Even with dead reckoning errors typical of odometry-based localization, the robot was able to return to the spot where it started as accurately as when using GNSS. This demonstration also included a RViz visualization of simultaneous localization and mapping (SLAM) as implemented in the OpenSlam Gmapping package [15]. Here, the system started fresh with no concept of its location within the environment. It was teleoperated by hand and as it moved through the environment, it built up a map of the surrounding area and accurately placed itself within the map. The goal of this demonstration was to show that as the robot moves through an unknown area, it can build a model of what areas are traversable and which are non-traversable (e.g., walls, obstacles, etc.) as well as determine its own position relative to the environment. 3.2.3 Sensor Fusion Demonstration The sensor fusion demonstration was designed to show how a system can transition between two sensors in order to mitigate any single sensor’s weaknesses. Here, the system used GNSS-based positioning unless it detected the quality of the position solution (as reported by the receiver) had deteriorated. In this case, the system would automatically switch to using LIDAR odometry until a high quality GNSS position fix was re-acquired. Once re-acquired, the GNSS would update the robot’s position to account for any drift in the position due to dead reckoning errors. It is noted that at least an initial GNSS position fix was required in order to provide the robot’s initial position within the environment. To demonstrate this, the UGV was given a set of waypoints similar to the GNSS waypoint following demonstration. The difference here was that the waypoints would now take the robot under a skyway and into an urban canyon between two buildings. Using this technique, the robot was capable of driving the path defined by the waypoints, even when traveling under the skyway and between buildings. 14 CHAPTER 4: CONCLUSIONS The workshop developed in this project was delivered in a pilot offering on May 16 and 17, 2019, on the University of Minnesota Campus. The goal of this offering was not only to provide value to participants but also to assess the success of the workshop and determine improvements for future offerings. The pilot offering was attended by eight public transportation professionals representing state, county and local agencies as well as law enforcement. 4.1 PARTICIPANT FEEDBACK Feedback was collected in a survey sent to participants after the workshop. The survey was broken into five sections focusing on the classroom sessions, demonstrations, the topics covered, workshop logistics, and overall workshop feedback. There were eight completed survey responses. The primary goal of the survey was to assess how well the workshop met its goals and to determine what areas could be improved. When asked about the classroom sessions and their content, most participants thought the sessions were interesting and applicable to their CAV work. Some sessions such as the Introduction to the Husky and ROS were noted as interesting but not particularly applicable to their jobs. Most participants had heard the information provided in the Introduction to CAVs lecture but appreciated establishing a foundation and some did learn new information. When asked explicitly about what sessions they liked best and disliked most, responses were varied and included a relatively even distribution of all sessions. All participants reported enjoying the demonstration sessions, with most noting that they helped solidify the concepts presented in the classroom. Most survey responses included suggestions for improving the demonstrations, generally focusing on showing more situations, different sensors, or having the robot perform additional or more challenging tasks. Participants were asked about the topics covered in the workshop and whether the time spent on each topic was appropriate. Most responses centered on the idea that the relative time spent on each topic was appropriate as opposed to covering more topics in less detail. When explicitly asked about additional topics that could have been covered or that would be good for a follow-on workshop, most respondents did not identify any topics. The few who did generally mentioned connected vehicle applications and other CAV infrastructure considerations. Most participants thought the two-day format of the workshop was “about right,” although two respondents thought it was “a little too long.” Regarding the pacing of each individual day, participants thought the approximately 6-7-hour day was an appropriate length and the pacing of the day (i.e., the ratio of sessions to breaks) was “about right.” All participants reported agreeing or strongly agreeing with the statement, “I found the workshop interesting.” When asked about whether they thought the material was applicable or would be useful to their jobs, responses ranged from strongly agree to somewhat agree. Half (4) of the responses indicated 15 strong agreement with the statement, “If the workshop were offered again, I’d recommend it to a co- worker,” with three additional responses in the agree or somewhat agree categories. 4.2 FUTURE WORK Participant feedback was almost exclusively positive noting that the topics covered and the technical level at which they were discussed were both interesting and understandable. Respondents were less emphatic regarding the content’s direct applicability to their job. Future work could involve additional information gathering to determine what, if any, knowledge gaps exist among transportation professionals and how best to address them in a workshop format. This could include adding sessions about new topics or adjusting existing sessions. One example of a topic area not covered in the workshop was connected vehicle technology and applications. This would be an area particularly interesting for public transportation professionals, especially those who are or will be responsible for evaluating and deploying vehicle-to-infrastructure communications technology. Other topics might include the state of commercially available OEM connected and automated vehicles. Demonstrations were well liked and thought to be useful in solidifying concepts presented in the classroom sessions. Future work could enhance the demonstrations by adding additional or more complicated scenarios. In addition, future work could seek to improve the visualization tools used to represent the system’s internal state. The tools used now are designed for use by developers generally performing debugging tasks. Some aspects of the visualizations could be made easier to understand for participants viewing these tools for the first time and without prior knowledge about the system’s architecture or internal state model. This project focused on the development of the workshop material and included a pilot offering primarily for evaluation purposes. Future efforts should seek to offer this, or similar workshops open to a larger pool of participants. 4.3 PROJECT FINDINGS Workshop participants provided feedback that was almost exclusively positive. Although noting some areas for improvement, attendees reported that the material was interesting and understandable and that they took away new information they learned by participating in the workshop. Future iterations of the workshop could be further improved based on participant feedback, although even in its current version, the workshop was valuable to participants. Overall, these findings showed that the workshop was beneficial to public transportation professionals seeking to learn information about connected and automated vehicle technologies. 16 REFERENCES [1] U.S. Department of Transportation. (2018). Preparing for the future of transportation: Automated vehicle 3.0. Washington D.C.: U.S. Department of Transportation. Retrieved from https://www.transportation.gov/av/3 [2] U.S. Department of Transportation Intelligent Transportation Systems Joint Program Office. (n.d.). Connected Vehicle Basics. Retrieved May 2019, from https://www.its.dot.gov/cv_basics/cv_basics_20qs.htm [3] SAE International. (2018). SAE J3016_201806 taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE International. Retrieved from https://www.sae.org/standards/content/j3016_201806/ [4] Clearpath Robotics. (n.d.). Husky unmanned ground vehicle. Retrieved May 2019, from https://www.clearpathrobotics.com/husky-unmanned-ground-vehicle-robot/ [5] Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A. Y. (2009). ROS: An open-source robot operating system. In IEEE International Conference on Robotics and Automation (IRCA): Workshop on Open Source Software, Kobe, Japan. Retrieved from http://www.willowgarage.com/sites/default/files/icraoss09-ROS.pdf [6] Open Source Robotics Foundation. (n.d.). rviz. Retrieved May 2019, from http://wiki.ros.org/rviz [7] Open Source Robotics Foundation. (n.d.). Gazebo. Retrieved May 2019, from http://gazebosim.org/ [8] National Coordination Office for Space-Based Positioning, Navigation, and Timing. (n.d.). GPS: The global positioning system. Retrieved May 2019, from https://www.gps.gov/ [9] Minnesota Department of Transportation. (n.d.). MnCORS GNSS network. Retrieved May 2019, from http://www.dot.state.mn.us/surveying/cors/ [10] Velodyne Lidar. (n.d.). HDL-64E. Retrieved May 2019, from https://velodynelidar.com/hdl- 64e.html [11] Trimble. (n.d.). Trimble BX992 enclosure. Retrieved May 2019, from https://www.trimble.com/Precision-GNSS/BX992-Enclosure.aspx [12] Velodyne Lidar. (n.d.). Puck. Retrieved May 2019, from https://velodynelidar.com/vlp-16.html [13] Phidgets Inc. (n.d.). PhidgetSpatial precision 3/3/3 high resolution. Retrieved May 2019, from https://www.phidgets.com/?tier=3&catid=10&pcid=8&prodid=1038 17 [14] Shan, T. & Englot, B. (2018) LeGO-LOAM: Lightweight and ground optimized lidar odometry and mapping on variable terrain. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 4758-4765. https://doi.org/10.1109/IROS.2018.8594299 [15] Grisetti, G., Stachniss, C., & Burgard, W. GMapping. (n.d.). Retrieved May 2019, from https://openslam-org.github.io/gmapping.html APPENDIX A FINAL WORKSHOP SCHEDULE A-1 Day 1 (May 16, 2019) 9:00 – 9:30 Registration and refreshments 9:30 – 10:00 Introductions, workshop goals, schedule, and logistics 10:00 – 10:45 Introduction to Connected and Automated Vehicles Common definitions and levels of automation Introduction to CAV sensors Global Satellite Navigation Systems (GNSS)/GPS LIDAR, RADAR Inertial Navigation Systems (INS)/Inertial Measurement Units (IMU) Cameras and computer vision Communications and Dedicated Short Range Communications (DSRC) State of public CAV technology and testbeds 10:45 – 11:00 Break 11:00 – 12:00 Introduction to the Clearpath Husky and Robot Operating System Hardware and sensors onboard the Husky Robot Operating System (ROS) overview ROS navigation stack In-room Husky demonstration – sensor visualization 12:00 – 1:00 Lunch/Break (Lunch Provided) 1:00 – 1:45 Introduction to GNSS: Part 1 GNSS operating principles GNSS corrections – base stations, base station networks Real-time Kinematic solutions (RTK), Precise Point Positioning (PPP) Overview of available technology Evaluation techniques 1:45 - 2:00 Break 2:00 – 2:30 GNSS Lab Evaluation of GPS sensors Husky GPS waypoint following 2:30 – 3:30 Introduction to GNSS: Part 2 Discussion of GNSS demo Applications, strengths, weaknesses Mapping and maps 3:30 – 4:00 Discussion and Q & A A-2 Day 2 (May 17, 2019) 9:00 – 9:15 Refreshments 9:15 – 10:15 Introduction to LIDAR Lecture LIDAR operating principles Overview of available technology Visualization of 64 channel LIDAR on-road data LIDAR localization techniques 11:15 – 12:00 LIDAR Lab Simultaneous Localization and Mapping (SLAM) LIDAR odometry 10:45 - 11:00 Break 11:00 - 1:00 Machine Learning Lecture (Lunch Provided) Neural networks Convolutional Neural Networks Applications: Image classification, object detection, and tracking 1:00 - 1:15 Break 1:15 – 2:00 Software and Algorithms for Localization and Navigation Lecture Localization and navigation techniques (SLAM, AMCL) Obstacle detection Global and local route planning Sensor fusion overview 2:00 – 2:45 Sensor Fusion Demo LIDAR and GNSS combined navigation 2:45 – 3:00 Discussion, Q & A, and wrap-up