Browsing by Subject "reinforcement learning"

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    Data-Driven Framework for Energy Management in Extended Range Electric Vehicles Used in Package Delivery Applications
    (2020-08) Wang, Pengyue
    Plug-in Hybrid Electric Vehicles (PHEVs) have potential to achieve high fuel efficiency and reduce on-road emissions compared to engine-powered vehicles when using well-designed Energy Management Strategies (EMSs). The EMS of PHEVs has been a research focus for many years and optimal or near optimal performance has been achieved using control-oriented approaches like Dynamic Programming (DP) and Model Predictive Control (MPC). These approaches either require accurate predictive models for the trip information during driving cycles or detailed velocity profiles in advance. However, such detailed information is not feasible to obtain in some real-world applications like the delivery vehicle application studied in this work. Here, data-driven approaches were developed and tested over real-world trips with the help of two-way Vehicle-to-Cloud (V2C) connectivity. First, the EMS problem was formulated as a probability density estimation problem and solved by Bayesian inference. The Bayesian algorithm deals with the condition where only small amounts of data are available and sequential parameter estimation problem elegantly, which matches the characteristics of the data generated by delivery vehicles. The predicted value of the parameter for the next trip is determined by the carefully designed prior information and all the available data of the vehicle so far. The parameter is updated before the delivery tasks using the latest trip information and stays static during the trip. This method was demonstrated on 13 vehicles with 155 real-world delivery trips in total and achieved an average of 8.9% energy efficiency improvement with respect to MPGe (miles per gallon equivalent). For vehicles with sufficient data that can represent the characteristics of future delivery trips, the EMS problem was formulated as a sequential decision-making problem under uncertainty and solved by deep reinforcement learning (DRL) algorithms. An intelligent agent was trained by interacting with the simulated environment built based on the vehicle model and historical trips. After training and validation, optimized parameter in the EMS was updated by the trained intelligent agent during the trip. This method was demonstrated on 3 vehicles with 36 real-world delivery trips in total and achieved an average of 20.8% energy efficiency improvement in MPGe. Finally, I investigated three problems that could be encountered when the developed DRL algorithms are deployed in real-world applications: model uncertainty, environment uncertainty and adversarial attacks. For model uncertainty, an uncertainty-aware DRL agent was developed, enabled by the technique of Bayesian ensemble. Given a state, the agent quantifies the uncertainty about the output action, which means although actions will be calculated for all input states, the high uncertainty associated with unfamiliar or novel states is captured. For environment uncertainty, a risk-aware DRL agent was built based on distributional RL algorithms. Instead of making decisions based on expected returns as standard RL algorithms, actions were chosen with respect to conditional value at risk, which gives more flexibility to the user and can be adapted according to different application scenarios. Lastly, the influence of adversarial attacks on the developed neural network based DRL agents was quantified. My work shows that to apply DRL agents on real-world transportation systems, adversarial examples in the form of cyber-attack should be considered carefully.
  • Thumbnail Image
    Item
    Learning to cooperate using deep reinforcement learning in a multi-agent system
    (2020-12) Khan, Nabil
    In this thesis we address the problem of emergence of cooperation between agents that operate in a simulated environment, where they need to accomplish a complex task that is decomposed into sub-tasks and that can be completed only with cooperation. A deep reinforcement learning approach using a multi-layered neural network is used by the agents within a reinforcement learning algorithm to learn how to accomplish the task of moving heavy loads that require cooperation between agents. The goal of this work is to empirically show that cooperation can emerge without explicit instructions, whereby agents learn to cooperate to perform complex tasks, and to analyze the correlation between task complexity and training time. The series of experiments we conducted helps establish that cooperation can emerge but becomes unlikely in partially observable environments as the environment size scales up. Another series of experiments shows that communication makes the cooperative behavior more likely, even as environments scale up, when the environment is only partially observable. However, communication is not a necessary condition for cooperation to emerge, since agents with knowledge of the environment can also learn to cooperate, as demonstrated in the fully observable environment.
  • Thumbnail Image
    Item
    Sex-correlated variability in exploration strategy in uncertain environments
    (2022-07) Chen, Sijin 'Cathy'
    Every organism must balance between two goals: exploiting rewarding options when they are available and exploring more new information about potential better alternatives. Adaptively transition between exploration and exploitation is essential when navigating an uncertain world. Exploration is dysregulated in numerous neuropsychiatric disorders, many of which are sex-biased in risk, presentation, and prognosis. This raises the possibility that sex-linked mechanisms could modulate exploration differently and contribute to sex-linked individual variability in the vulnerability or resilience to these conditions. Understanding how individuals explore uncertain environments can give us in sight into how brains implement divergent exploration strategy. In this dissertation, I present three studies investigating 1. exploration strategy in a complex novel environment, 2. exploration strategy in a changing environment, 3. neuromodulatory systems underlying exploration strategy. In experiment 1 and 2, I observed a spectrum of strategies that individuals adopted to navigate the environment and sex captured a major source of variability in the strategies adopted. Both sexes did not differ in the ability to learn the task but they differed in the preferred strategy employed to explore an uncertain environment. Females preferred a more energy-conserving, systematic and exploitative approach across both tasks, where as males predominantly used more variable and exploratory approach. In experiment 3, I modulated tonic dopamine and norepinephrine level and examined the modulatory effect on exploration. The results suggested novel role of dopamine in mediating exploration and highlighted the sex-differentiated modulatory effect of norepinephrine on exploration. This dissertation took advantage of computational tools and revealed sex-correlated variability in strategies employed when interacting with an uncertain environment, rather than any difference in ability. This highlighted sex as source of individual variability and implicated potential sex-modulated circuits and systems that could contribute to vulnerability or risk for neuropsychiatric disorders.