Scribd Logo
Upload

Welcome to Scribd!

  • 26832-Article Text-30895-1-2-20230626

    Uploaded by

    Võ Minh Trí
    4 views1 page
    The document discusses the challenge of policy evaluation in reinforcement learning. Policy evaluation aims to assess how good a reinforcement learning agent is through numeric metrics. While evaluating agents in simulated environments is straightforward using Monte Carlo methods, evaluating agents in real-world scenarios is much more challenging due to the high cost of executing agents in the real world. Recent offline evaluation methods aim to perform policy evaluation using existing logged offline data instead of new online data, but often suffer from biases. The document argues that the next important challenge is to reduce the use of online data in Monte Carlo methods via information extracted from offline data, while maintaining the unbiasedness of Monte Carlo methods.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses the challenge of policy evaluation in reinforcement learning. Policy evaluation aims to assess how good a reinforcement learning agent is through numeric metrics. While evaluating agents in simulated environments is straightforward using Monte Carlo methods, evaluating agents in real-world scenarios is much more challenging due to the high cost of executing agents in the real world. Recent offline evaluation methods aim to perform policy evaluation using existing logged offline data instead of new online data, but often suffer from biases. The document argues that the next important challenge is to reduce the use of online data in Monte Carlo methods via information extracted from offline data, while maintaining the unbiasedness of Monte Carlo methods.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    4 views1 page

    26832-Article Text-30895-1-2-20230626

    Uploaded by

    Võ Minh Trí
    The document discusses the challenge of policy evaluation in reinforcement learning. Policy evaluation aims to assess how good a reinforcement learning agent is through numeric metrics. While evaluating agents in simulated environments is straightforward using Monte Carlo methods, evaluating agents in real-world scenarios is much more challenging due to the high cost of executing agents in the real world. Recent offline evaluation methods aim to perform policy evaluation using existing logged offline data instead of new online data, but often suffer from biases. The document argues that the next important challenge is to reduce the use of online data in Monte Carlo methods via information extracted from offline data, while maintaining the unbiasedness of Monte Carlo methods.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)

    A New Challenge in Policy Evaluation

    Shangtong Zhang
    University of Virginia
    85 Engineer’s Way, Charlottesville, VA, 22903, USA
    shangtong@virginia.edu

    Introduction The Challenge


    Given the dominance of Monte Carlo methods, I argue that
    Reinforcement Learning (RL) builds agents that make deci- it might be too ambitious at the moment to remove online
    sions in multiple consecutive stages to achieve some goal. data from our policy evaluation pipeline entirely. Reducing,
    Enormous success of RL has been witnessed in simulated instead of removing, the use of online data in Monte Carlo
    environments, e.g., Go and StarCraft II. Our ultimate goal is, methods seems to be a more practical objective in the near
    however, to deploy RL agents in real-world scenarios, e.g., future. Notably, the improved Monte Carlo methods should
    autonomous vehicles. Before such deployment, we have to still be unbiased, or at least consistent, because it is the unbi-
    know how good an RL agent is, ideally through some nu- asedness that makes Monte Carlo methods the golden stan-
    meric metrics. This is the policy evaluation problem. Eval- dard in policy evaluation.
    uating an agent in a simulated environment is straightfor- Efforts have been made to improve the data efficiency
    ward. One can run the agent multiple times and take the av- of Monte Carlo methods in RL, including Zinkevich et al.
    erage outcome. This is the widely used Monte Carlo method. (2006); Hanna et al. (2017); Mukherjee, Hanna, and Nowak
    Evaluating an agent in real-world scenarios is, however, (2022); Zhong et al. (2022). Offline data is, however, not
    much more challenging. This is because actually executing fully exploited to improve Monte Carlo methods yet, despite
    an agent in the real world can be very expensive; but build- that offline data contains rich information about the environ-
    ing a high-fidelity simulator is often more challenging than ment. I, therefore, argue that the next important challenge
    solving the problem itself. in policy evaluation is to reduce Monte Carlo methods’
    Recent offline evaluation methods take this challenge use of online data via information extracted from offline
    via performing policy evaluation with existing previously data while maintaining the unbiasedness of Monte Carlo
    logged offline data, instead of new online data from agent- methods. Liu and Zhang (2023) shed light on this challenge
    environment interaction. Offline methods typically rely on via learning a provably variance-reducing behavior policy
    learning some other quantities, e.g., density ratio and action from offline data.
    value function. However, those learning processes unavoid-
    ably suffer from various biases, resulting from, e.g., the mis- References
    specification of function class, the difficulty in optimizing Hanna, J. P.; Thomas, P. S.; Stone, P.; and Niekum, S.
    nonconvex function, etc. As a result, offline methods rarely 2017. Data-efficient policy evaluation through behavior pol-
    have evaluation accuracy guarantees without restrictive as- icy search. In Proceedings of the International Conference
    sumptions. Even worse, many offline methods still require a on Machine Learning.
    large amount of online data for model selection and thus do
    not really fulfill the promise to eliminate the use of online Liu, S.; and Zhang, S. 2023. Improving Online Monte Carlo
    data. Offline model selection methods are being developed Evaluation with Offline Data. arXiv preprint.
    but rarely have model selection accuracy guarantees without Mukherjee, S.; Hanna, J. P.; and Nowak, R. D. 2022. ReVar:
    restrictive assumptions. As a result, the Monte Carlo method Strengthening policy evaluation via reduced variance sam-
    using massive online data is still the most widely used policy pling. In Proceedings of the Conference in Uncertainty in
    evaluation method. The development and deployment of ev- Artificial Intelligence.
    ery RL algorithm implicitly or explicitly depend on Monte Zhong, R.; Hanna, J. P.; Schäfer, L.; and Albrecht, S. V.
    Carlo methods more or less. For example, when an RL re- 2022. Robust On-Policy Data Collection for Data-Efficient
    searcher wants to plot the performance of an agent against Policy Evaluation. In Advances in Neural Information Pro-
    training steps, the Monte Carlo method is almost always the cessing Systems.
    first choice. Zinkevich, M.; Bowling, M.; Bard, N.; Kan, M.; and
    Billings, D. 2006. Optimal unbiased estimators for evalu-
    Copyright c 2023, Association for the Advancement of Artificial ating agent performance. In Proceedings of the AAAI Con-
    Intelligence (www.aaai.org). All rights reserved. ference on Artificial Intelligence.

    15465

    You might also like