Oct 22 2020
Robots have been replacing humans in hazardous conditions like search and rescue missions, but they must have the potential for fast assessment and decision-making—to respond and adapt similar to a human being.
Photo taken in 2019 of (left to right) Neale Van Stralen, Ayberk Yaraneri, and Huy Tran testing a drone on the U of I campus. Image Credit: University of Illinois at Urbana-Champaign.
At the University of Illinois at Urbana-Champaign, a research team used a model based on the Capture the Flag game to create a new perspective on deep reinforcement learning that assists robots to assess their next move.
The researchers opted for Capture the Flag as it is played with two teams, where each team has several teammates and the opposing team also makes decisions.
Robots can learn how to react in an environment like a competitive game by using a kind of trial and error process, called reinforcement learning. They learn what actions to take in a given situation by playing the game. The challenge is to figure out how to create agents that can also adapt to unexpected situations.
Huy Tran, Researcher, Department of Aerospace Engineering, University of Illinois at Urbana-Champaign
Tran added that his group understood that the robots required assistance to prioritize tasks.
“Given the overall task of capturing the flag, there are actually sub tasks to accomplish the along the way, which we model in a hierarchical structure. What we wanted to explore was whether or not this type of hierarchy would help with the ability to adapt,” he explained.
According to Tran, in hierarchical deep reinforcement learning, tasks are split up—to capture the flag or to tag a member of the opposite team to remove them—which enables the model to handle more complex problems.
By breaking the task into sub tasks, we can improve adaptation. We trained a high-level decision maker who assigns a sub task for each agent to focus on.
Huy Tran, Researcher, Department of Aerospace Engineering, University of Illinois at Urbana-Champaign
He added that the hierarchical structure is useful as it simplifies any updates to the model. Rather than updating each of the agents, it is enough if only the hierarchical controller is updated.
This approach has the potential to solve interesting and challenging problems, but there are a lot of issues that we still need to address before we can deploy these systems in real-world situations. For example, we learned that this framework can help with adaptation, but we recognize that in this study we decided what the sub tasks should be based on our own intuition of how the game works.
Huy Tran, Researcher, Department of Aerospace Engineering, University of Illinois at Urbana-Champaign
“That is not ideal because it has our own biases. What we’re doing now is looking at new techniques to allow agents to figure out what those sub goals should be on their own,” Tran added.
The paper titled “Evaluating Adaptation Performance of Hierarchical Deep Reinforcement Learning,” was authored by Neale Van Stralen, Seung Hyun Kim, Huy T. Tran, and CSL’s Girish Chowdhary.
The study was financially supported by the Defense Advanced Research Projects Agency and was presented at the 2020 IEEE International Conference on Robotics and Automation (ICRA).
Journal Reference:
Van Stralen, N., et al. (2020) Evaluating Adaptation Performance of Hierarchical Deep Reinforcement Learning. 2020 IEEE International Conference on Robotics and Automation. doi.org/10.1109/ICRA40945.2020.9197052.