Competitive Benchmarking: Lessons Learned from the Trading Agent Competition

Many important developments in artificial intelligence have been stimulated by organized competitions that tackle interesting, difficult challenge problems, such as chess, robot soccer, poker, robot navigation, stock trading, and others. Economics and artificial intelligence share a strong focus on rational behavior. Yet the real-time demands of many domains do not lend hemselves to traditional assumptions of rationality. This is the case in many trading environments, where self-interested entities need to operate subject to limited time and information. With the web mediating an ever broader range of transactions and opening the door for participants to concurrently trade across multiple markets, there is a growing need for technologies that empower participants to rapidly evaluate very large numbers of alternatives in the face of constantly changing market conditions. AI and machine-learning techniques, including neural networks and genetic algorithms, are already routinely used in support of automated trading scenarios. Yet, the deployment of these technologies remains limited, and their proprietary nature precludes the type of open benchmarking that is critical for further scientific progress.

The Trading Agent Competition was conceived to provide a platform for study of agent behavior in competitive economic environments. Research teams from around the world develop agents for these environments. During annual competitions, they are tested against each other in simulated market environments. Results can be mined for information on agent behaviors, and their effects on agent performance, market conditions, and the performance and behavior of competing agents. After each competition, competing agents are made available for offline research. We will discuss results from various competitions (Travel, Supply-Chain Management, Market Design, Sponsored Search, and Power Markets).