Policy shaping in domains with multiple optimal policies

Himanshu Sahni; Brent Harrison; Kaushik Subramanian; Thomas Cederborg; Charles Isbell; Andrea Thomaz

Policy shaping in domains with multiple optimal policies

Himanshu Sahni, Brent Harrison, Kaushik Subramanian, Thomas Cederborg, Charles Isbell, Andrea Thomaz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

In many domains, there exist multiple ways for an agent to achieve optimal performance. Feedback may be provided along one or more of them to aid learning. In this work, we investigate whether humans have a preference towards providing feedback along one optimal policy over the other in two gridworld domains. We find that for the domain with significant risk to exploration, 60% of our participants prefer to discourage the agent's exploration along the risky portion of the state space, while 40% state that they have no preference. We also use the interactive reinforcement learning algorithm Policy Shaping to evaluate the performance of simulated oracles with a number of feedback strategies. We find that certain domain traits, such as risk during exploration and number of optimal policies play an important role in determining the best performing feedback strategy.

Original language	English
Title of host publication	AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems
Pages	1455-1456
Number of pages	2
ISBN (Electronic)	9781450342391
State	Published - 2016
Event	15th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2016 - Singapore, Singapore Duration: May 9 2016 → May 13 2016

Publication series

Name	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
ISSN (Print)	1548-8403
ISSN (Electronic)	1558-2914

Conference

Conference	15th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2016
Country/Territory	Singapore
City	Singapore
Period	5/9/16 → 5/13/16

Bibliographical note

Publisher Copyright:
Copyright © 2016, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.

Keywords

Interactive machine learning
Learning from critique
Policy shaping
Reinforcement learning

ASJC Scopus subject areas

Artificial Intelligence
Software
Control and Systems Engineering

Cite this

Sahni, H., Harrison, B., Subramanian, K., Cederborg, T., Isbell, C., & Thomaz, A. (2016). Policy shaping in domains with multiple optimal policies. In AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems (pp. 1455-1456). (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

@inproceedings{f855645ee0204b5dbc062042b1f0f133,

title = "Policy shaping in domains with multiple optimal policies",

abstract = "In many domains, there exist multiple ways for an agent to achieve optimal performance. Feedback may be provided along one or more of them to aid learning. In this work, we investigate whether humans have a preference towards providing feedback along one optimal policy over the other in two gridworld domains. We find that for the domain with significant risk to exploration, 60% of our participants prefer to discourage the agent's exploration along the risky portion of the state space, while 40% state that they have no preference. We also use the interactive reinforcement learning algorithm Policy Shaping to evaluate the performance of simulated oracles with a number of feedback strategies. We find that certain domain traits, such as risk during exploration and number of optimal policies play an important role in determining the best performing feedback strategy.",

keywords = "Interactive machine learning, Learning from critique, Policy shaping, Reinforcement learning",

author = "Himanshu Sahni and Brent Harrison and Kaushik Subramanian and Thomas Cederborg and Charles Isbell and Andrea Thomaz",

note = "Publisher Copyright: Copyright {\textcopyright} 2016, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.; 15th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2016 ; Conference date: 09-05-2016 Through 13-05-2016",

year = "2016",

language = "English",

series = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

pages = "1455--1456",

booktitle = "AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems",

}

Sahni, H, Harrison, B, Subramanian, K, Cederborg, T, Isbell, C & Thomaz, A 2016, Policy shaping in domains with multiple optimal policies. in AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, pp. 1455-1456, 15th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2016, Singapore, Singapore, 5/9/16.

Policy shaping in domains with multiple optimal policies. / Sahni, Himanshu; Harrison, Brent; Subramanian, Kaushik et al.
AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems. 2016. p. 1455-1456 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Policy shaping in domains with multiple optimal policies

AU - Sahni, Himanshu

AU - Harrison, Brent

AU - Subramanian, Kaushik

AU - Cederborg, Thomas

AU - Isbell, Charles

AU - Thomaz, Andrea

PY - 2016

Y1 - 2016

N2 - In many domains, there exist multiple ways for an agent to achieve optimal performance. Feedback may be provided along one or more of them to aid learning. In this work, we investigate whether humans have a preference towards providing feedback along one optimal policy over the other in two gridworld domains. We find that for the domain with significant risk to exploration, 60% of our participants prefer to discourage the agent's exploration along the risky portion of the state space, while 40% state that they have no preference. We also use the interactive reinforcement learning algorithm Policy Shaping to evaluate the performance of simulated oracles with a number of feedback strategies. We find that certain domain traits, such as risk during exploration and number of optimal policies play an important role in determining the best performing feedback strategy.

AB - In many domains, there exist multiple ways for an agent to achieve optimal performance. Feedback may be provided along one or more of them to aid learning. In this work, we investigate whether humans have a preference towards providing feedback along one optimal policy over the other in two gridworld domains. We find that for the domain with significant risk to exploration, 60% of our participants prefer to discourage the agent's exploration along the risky portion of the state space, while 40% state that they have no preference. We also use the interactive reinforcement learning algorithm Policy Shaping to evaluate the performance of simulated oracles with a number of feedback strategies. We find that certain domain traits, such as risk during exploration and number of optimal policies play an important role in determining the best performing feedback strategy.

KW - Interactive machine learning

KW - Learning from critique

KW - Policy shaping

KW - Reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85014277099&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85014277099&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85014277099

T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

SP - 1455

EP - 1456

BT - AAMAS 2016 - Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems

T2 - 15th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2016

Y2 - 9 May 2016 through 13 May 2016

ER -

Policy shaping in domains with multiple optimal policies

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this