[en] This paper introduces an algorithm for direct search of control policies in continuous-state, discrete-action Markov decision processes. The algorithm looks for the best closed-loop policy that can be represented using a given number of basis functions (BFs), where a discrete action is assigned to each BF. The type of the BFs and their number are specified in advance and determine the complexity of the representation. Considerable flexibility is achieved by optimizing the locations and shapes of the BFs, together with the action assignments. The optimization is carried out with the cross-entropy method and evaluates the policies by their empirical return from a representative set of initial states. The return for each representative state is estimated using Monte Carlo simulations. The resulting algorithm for crossentropy policy search with adaptive BFs is extensively evaluated in problems with two to six state variables, for which it reliably obtains good policies with only a small number of BFs. In these experiments, cross-entropy policy search requires vastly fewer BFs than value-function techniques with equidistant BFs, and outperforms policy search with a competing optimization
algorithm called DIRECT.
Disciplines :
Computer science
Author, co-author :
Busoniu, Lucian
Ernst, Damien ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Systèmes et modélisation
Babuska, Robert
De Schutter, Bart
Language :
English
Title :
Cross-entropy optimization of control policies with adaptive basis functions
Publication date :
February 2011
Journal title :
IEEE Transactions on Systems, Man and Cybernetics. Part B, Cybernetics
ISSN :
1083-4419
eISSN :
1941-0492
Publisher :
Institute of Electrical and Electronics Engineers, New-York, United States - New York
D. P. Bertsekas, Dynamic Programming and Optimal Control. 3rd ed. Belmont, MA: Athena Scientific, 2007.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
J. N. Tsitsiklis and B. Van Roy, "Feature-based methods for large scale dynamic programming," Mach. Learn., vol. 22, no. 1-3, pp. 59-94, Jan.-Mar. 1996.
R. Munos and A. Moore, "Variable-resolution discretization in optimal control," Mach. Learn., vol. 49, no. 2/3, pp. 291-323, Nov. 2002.
M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, 2003.
L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuka, "Continuous-state reinforcement learning with fuzzy approximation," in Adaptive Agents and Multi-Agent Systems III, K. Tuyls, A. Nowé, Z. Guessoum, and D. Kudenko, Eds. New York: Springer-Verlag, 2008, ser. Lecture Notes in Computer Science, pp. 27-43.
D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," J. Mach. Learn. Res., vol. 6, pp. 503-556, 2005.
S. Mahadevan and M. Maggioni, "Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes," J. Mach. Learn. Res., vol. 8, pp. 2169-2231, 2007.
P. Marbach and J. N. Tsitsiklis, "Approximate gradient methods in policy-space optimization of Markov reward processes," Discrete Event Dyn. Syst.: Theory Appl., vol. 13, no. 1/2, pp. 111-148, Jan. 2003.
S. Mannor, R. Y. Rubinstein, and Y. Gat, "The cross-entropy method for fast policy search," in Proc. 20th ICML, Washington, DC, Aug. 21-24, 2003, pp. 512-519.
R. Munos, "Policy gradient in continuous time," J. Mach. Learn. Res., vol. 7, pp. 771-791, 2006.
H. S. Chang, M. C. Fu, J. Hu, and S. I. Marcus, Simulation-Based Algorithms for Markov Decision Processes. New York: Springer-Verlag, 2007.
M. Riedmiller, J. Peters, and S. Schaal, "Evaluation of policy gradient methods and variants on the cart-pole benchmark," in Proc. IEEE Symp. ADPRL, Honolulu, HI, Apr. 1-5, 2007, pp. 254-261.
R. Y. Rubinstein and D. P. Kroese, The Cross Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation, and Machine Learning. New York: Springer-Verlag, 2004.
D. R. Jones, "DIRECT global optimization algorithm," in Encyclopedia of Optimization, C. A. Floudas and P. M. Pardalos, Eds. New York: Springer-Verlag, 2009, pp. 725-735.
A. Y. Ng and M. I. Jordan, "PEGASUS: A policy search method for large MDPs and POMDPs," in Proc. 16th Conf. UAI, Palo Alto, CA, Jun. 30-Jul. 3, 2000, pp. 406-415.
R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Advances in Neural Information Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R. Müller, Eds. Cambridge, MA: MIT Press, 2000, pp. 1057-1063.
V. R. Konda and J. N. Tsitsiklis, "On actor-critic algorithms," SIAM J. Control Optim., vol. 42, no. 4, pp. 1143-1166, 2003.
J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, no. 7-9, pp. 1180-1190, Mar. 2008.
D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
H. H. Chin and A. A. Jafari, "Genetic algorithm methods for solving the best stationary policy of finite Markov decision processes," in Proc. 30th Southeastern Symp. Syst. Theory, Morgantown, WV, Mar. 8-10, 1998, pp. 538-543.
D. Barash, "A genetic search in policy space for solving Markov decision processes," in Proc. AAAI Spring Symp. Search Techn. Probl. Solving Under Uncertainty Incomplete Inf., Palo Alto, CA, Mar. 22-24, 1999.
S.-M. Tse, Y. Liang, K.-S. Leung, K.-H. Lee, and T. S.-K. Mok, "A memetic algorithm for multiple-drug cancer chemotherapy schedule optimization," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 1, pp. 84-91, Feb. 2007.
I. Menache, S. Mannor, and N. Shimkin, "Basis function adaptation in temporal difference reinforcement learning," Ann. Oper. Res., vol. 134, no. 1, pp. 215-238, Feb. 2005.
C. G. Atkeson and B. J. Stephens, "Random sampling of states in dynamic programming," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 924-929, Aug. 2008.
M. G. Lagoudakis and R. Parr, "Reinforcement learning as classification: Leveraging modern classifiers," in Proc. 20th ICML, Washington, DC, Aug. 21-24, 2003, pp. 424-431.
D. P. Bertsekas, "Dynamic programming and suboptimal control: A survey from ADP to MPC," Eur. J. Control-Special Issue for the CDC-ECC-05 in Seville, Spain, vol. 11, no. 4/5, pp. 310-334, 2005.
C. Dimitrakakis and M. Lagoudakis, "Rollout sampling approximate policy iteration," Mach. Learn., vol. 72, no. 3, pp. 157-171, Sep. 2008.
L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuka, "Policy search with cross-entropy optimization of basis functions," in Proc. IEEE Int. Symp. ADPRL, Nashville, TN, Mar. 30-Apr., 2, 2009, pp. 153-160.
J. Rust, "Numerical dynamic programming in economics," in Handbook of Computational Economics, H. M. Amman, D. A. Kendrick, and J. Rust, Eds. Amsterdam, The Netherlands: Elsevier, 1996, ch. 14, pp. 619-729.
L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, 1996.
D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Mach. Learn., vol. 49, no. 2/3, pp. 161-178, Nov. 2002.
A. Costa, O. D. Jones, and D. Kroese, "Convergence properties of the cross-entropy method for discrete optimization," Oper. Res. Lett., vol. 35, no. 5, pp. 573-580, Sep. 2007. (Pubitemid 47198343)
J. Randløv and P. Alstrøm, "Learning to drive a bicycle using reinforcement learning and shaping," in Proc. 15th ICML, Madison, WI, Jul. 24-27, 1998, pp. 463-471.
B. Adams, H. Banks, H.-D. Kwon, and H. Tran, "Dynamic multidrug therapies for HIV: Optimal and STI control approaches," Math. Biosci. Eng., vol. 1, no. 2, pp. 223-241, 2004.
D. Ernst, G.-B. Stan, J. Gonçalves, and L. Wehenkel, "Clinical data based optimal STI strategies for HIV: A reinforcement learning approach," in Proc. 45th IEEE Conf. Decision Control, San Diego, CA, Dec. 13-15, 2006, pp. 667-672
