Brainstormers 2D Research

Reinforcement Learning describes the situation of a machine learning system, where the only training signal provided by the environment is that of success or failure of the agent, after the system has acted over a sequence of decision cycles. This learning problem can be formulated as a Markov Decision Process (MDP) within the framework of Dynamic Programming. The main motivation behind the Brainstormers' effort in the soccer domain is to investigate Reinforcement Learning (RL) methods in complex domains and to develop new variants and practical algprithmus. We consider it important that we not only demonstrate the principal feasibility of RL, but actually do apply learned behavior in our competition team. Our long term goal is a team of learning agent, where we only plug in 'Win the match' - and our agents learn to generate the appropriate behavior.

Below you'll find a mini tutorial, made up of five lections, on the use of Reinforcement Learning methods in the context of Robotic Soccer.
[Note that Shockwave Flash is required for the tutorial's animations.]

For the purpose of illustration this tutorial is kept quite straightforward. The first page includes an explanation of Reinforcement Learning and wants to equip you with the mandatory notion used throughout the tutorial.
During the subsequent lections you will get to know how a soccer-playing agent can manage to autonomously acquire some specific capabilities, such as intercepting a ball or dribbling. Finally, a multi agent setting is considered, where two players must cooperatively learn to shoot a goal.

• T. Gabel and M. Riedmiller. On Progress in RoboCup: The Simulation League Showcase. In E. Chown, A. Matsumoto, P. Plöger, J.R. del Solar (editors): RoboCup 2010: Robot Soccer World Cup XIV, LNCS. Springer, Singapore, 2010. pdf
• T. Gabel, M. Riedmiller. Brainstormers 2D — Team Description 2010. unpublished, 2010 (CD supplement). pdf

• M. Riedmiller, T. Gabel, R. Hafner and S. Lange. Reinforcement Learning for Robot Soccer. Autonomous Robots, 27(1):55–74, Springer, 2009. pdf
• T. Gabel, M. Riedmiller. Brainstormers 2D — Team Description 2009 In J. Baltes, M. G. Lagoudakis, T. Naruse, S. Shiry (Eds.), RoboCup 2009: Robot Soccer World Cup XIII. Lecture Notes in Computer Science, Graz, Austria, Springer, 2009 (CD supplement). pdf

• T. Gabel, M. Riedmiller and F. Trost. A Case Study on Improving Defense Behavior in Soccer Simulation 2D: The NeuroHassle Approach. RoboCup 2008: Robot Soccer World Cup XII, LNCS. Springer, Shouzou, China, 2008. pdf
• M. Riedmiller, T. Gabel, F. Trost, T. Schwegmann. Brainstormers 2D - Team Description 2008 In L. Iocchi, H. Matsubara, A. Weitzenfeld, C. Zhou, editors, RoboCup 2008: Robot Soccer World Cup XII, LNCS. Springer, 2008. (CD Supplement) pdf

• T. Gabel and M. Riedmiller. On Experiences in a Complex and Competitive Gaming Domain: Reinforcement Learning Meets RoboCup. Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG 2007). Honolulu, USA, 2007. pdf
• M. Riedmiller and T. Gabel. Brainstormers 2D - Team Description 2007. In U. Visser, F. Ribeiro, T. Ohashi, F. Dellaert, editors, RoboCup 2007: Robot Soccer World Cup XI, LNCS. Springer, 2007. (CD Supplement) pdf

• T. Gabel, R. Hafner, S. Lange, M. Lauer and M. Riedmiller. Bridging the Gap: Learning in the RoboCup Simulation and Midsize League. Proceedings of the 7th Portuguese Conference on Automatic Control (Controlo 2006). Portuguese Society of Automatic Control, Porto, Portugal, 2006. pdf
• T. Gabel and M. Riedmiller. Learning a Partial Behavior for a competitive Soccer Agent. Künstliche Intelligenz, 2:18-23, Springer, 2006. pdf
• M. Riedmiller, T. Gabel, R. Hafner, S. Lange and M. Lauer. Die Brainstormers: Entwurfsprinzipien lernfähiger autonomer Roboter. Informatik-Spektrum, 20(3):175-190, Springer, 2006.
• M. Riedmiller and T. Gabel. Brainstormers 2D - Team Description 2007. In G. Lakemeyer, E. Sklar, D. G. Sorrenti, and T. Takahashi, editors, RoboCup 2006: Robot Soccer World Cup X, LNCS. Springer, 2006. (CD Supplement) pdf

• T. Gabel and M. Riedmiller. CBR for State Value Function Approximation in Reinforcement Learning. Proceedings of the International Conference on Case Based Reasoning 2005 (ICCBR 2005). Springer, Chicago, USA, August 2005. pdf
• A. Sung, A. Merke and M. Riedmiller. Reinforcement Learning using a Grid-Based Function Approximator. Biomimetic Neural Learning for Intelligent Robots. Lecture Notes in Artificial Intelligence, Vol. 3575 -volume-, 2005.
• D. Withopf and M. Riedmiller. Comparing Different Methods to Speed-Up Reinforcement Learning in a Complex Domain. Proc. of the Int. Conference on Systems, Man and Cybernetics, 2005. Big Island, USA, October 2005.
• D. Withopf and M. Riedmiller. Effective Methods for Reinforcement Learning in Large Multi-Agent Domains. it - Information Technology Journal, 5(47):241-249, Oldenbourg, 2005.
• M. Riedmiller, T. Gabel, J. Knabe, H. Strasdat. Brainstormers 2D - Team Description 2005. In A. Bredenfeld, A. Jacoff, I. Noda and Y. Takahashi, editors, RoboCup 2005: Robot Soccer World Cup IX, LNCS. Springer, 2005. pdf

• M. Riedmiller and A. Merke. Using machine learning techniques in complex multi-agent domains. In I. Stamatescu and W. Menzel and M. Richter and U. Ratsch, editor, Adaptivity and Learning, Springer, 2003.
• M. Riedmiller, A. Merke, M. Nickschas, W. Nowak, and D. Withopf. Brainstormers 2003 - Team Description. In D. Polani, B. Browning, A. Bonarini, K. Yoshida, editors, RoboCup 2003: Robot Soccer World Cup VII, LNCS. Springer, Padova, Italy, 2003. (CD supplement) pdf
• M. Arbatzat, S. Freitag, M. Fricke, R. Hafner, C. Heermann, K. Hegelich, A. Krause, J. Krüger, M. Lauer, M. Lewandowski, A. Merke, H. Müller, M. Riedmiller, J. Schanko, M. Schulte-Hobein, M. Theile, S. Welker, D. Withopf: Creating a Robot Soccer Team from Scratch: the Brainstormers Tribots. Proceedings of Robocup 2003, Padua, Italy, 2003, Springer.

• Martin Riedmiller, Artur Merke, Andreas Hoffmann, Daniel Withopf, Manuel Nickschas, Franziska Zacharias. Brainstormers 2002 - Team Description. RoboCup-202: Robot Soccer World Cup VI, LNCS, Springer. pdf

• T. Gabel, M. Veloso. Selecting Heterogeneous Team Players by Case-Based Reasoning: A Case Study in Robotic Soccer Simulation. Technical Report CMU-CS-01-165, Carnegie Mellon University, Pittsburgh, USA. 2001. pdf
• M. Riedmiller, A. Merke: Karlsruhe Brainstormers - A Reinforcement Learning Approach to Robotic Soccer II. RoboCup-2001: Robot Soccer World Cup V, LNCS, Springer. pdf

• M. Riedmiller, A. Merke, D. Meier, A. Hoffmann, A. Sinner, O. Thate, R. Ehrmann. Karlsruhe Brainstormers - A Reinforcement Learning Approach to Robotic Soccer. RoboCup-00: Robot Soccer World Cup IV, LNCS, Springer. pdf

• M. Riedmiller, S. Buck, A. Merke, R. Ehrmann, O. Thate, S. Dilger, A. Sinner, A. Hoffmann, L. Frommberger. Karlsruhe Brainstormers - Design Principles. RoboCup-99: Robot Soccer World Cup III, LNCS, Springer. pdf