2012: Results

The results of the seventh Annual Computer Poker Competition were announced July 23rd at the Twenty-Sixth Conference on Artificial Intelligence in Toronto during the first Computer Poker Symposium.  There was a good field of participants.

Three variations of Texas Hold'em poker were played: heads-up limit, heads-up no-limit, and three-player limit. Two different winner determination rules were used to decide the winners in each variant: bankroll instant run-off and total bankroll.

About the Results

As with previous years, the rules for this year's competition called for matches of duplicate poker to help mitigate the effects of "luck" and more accurately determine this year's winner. In total, about 70 million hands of poker were played.

Each competition has a PDF (and LaTeX source) of numerical results.  The top table in the PDF is a crosstable including all combinations of matches between players in the given variant of poker.  The second table presents the results for the total bankroll competition and the third table shows results for the bankroll instant run-off competition.  In each table, cells are shaded bright green/red if the row player won/lost by a statistically significant margin.  Cells are shaded light green/red if the row player won/lost by a statistically insignificant margin.  If a cell is grey no data is available for that combination of players (e.g., matches of an agent against itself or an agent entered exclusively under a different winner determination rule).

95% confidence intervals were not possible for all results.  Instead of using 95% confidence intervals, we decided to use weaker confidence intervals corresponding to one standard deviation.  In cases of narrower margins, we declared ties.  Please see the section discussing the statistical analysis used to generate the results for additional details.


 

Heads-up Limit Texas Hold'em

Download results and logs

Total Bankroll

gold_medal Slumbot (Eric Jackson, USA)
silver_medal Little Rock (Rod Byrnes, Australia)

Zbot (Ilkka Rajala, Finland)

Bankroll Instant Run-off

gold_medal Slumbot (Eric Jackson, USA)
silver_medal Hyperborean (University of Alberta, Canada)
bronze_medal Zbot (Ilkka Rajala, Finland)

Heads-up No-Limit Texas Hold'em

Download results and logs.

Total Bankroll

gold_medal Little Rock (Rod Byrnes, Australia)
silver_medal Hyperborean (University of Alberta, Canada)
bronze_medal Tartanian5 (Carnegie Mellon University, USA)

Bankroll Instant Run-off

gold_medal Hyperborean (University of Alberta, Canada)
silver_medal Tartanian5 (Carnegie Mellon University, USA)
bronze_medal Neo Poker Bot (Alexander Lee, Spain)

3-player Limit Texas Hold'em

Download results and logs

Total Bankroll

gold_medal Hyperborean (University of Alberta, Canada)
silver_medal Little Rock (Rod Byrnes, Australia)
bronze_medal

Neo Poker Bot (Alexander Lee, Spain)

Sartre (University of Auckland, New Zealand)

Bankroll Instant Run-off

gold_medal Hyperborean (University of Alberta, Canada)
silver_medal Little Rock (Rod Byrnes, Australia)
bronze_medal

Neo Poker Bot (Alexander Lee, Spain)

Sartre (University of Auckland, New Zealand)


Statistical Analysis

There are many ways to analyse the data we get from running matches between agents and some have better statistical properties than others.  One thing we need to remain mindful of is what statistic we are actually trying to compute.  For the full crosstable of all possible matches, we consider the random variable Xi,j which represents player i's value in a match against opponent j where j > i and j is in the set of opponents S (with more than two players this generalizes to j being a subset of opponents).  Since we don't know any information about Xi,j, we need to estimate it by playing M sample matches and computing their sample mean and variance in the standard way.  While this is an unbiased estimator for Xi,j we can do something smarter: duplicate poker.  In duplicate poker instead of sampling M matches where the cards for each match are sampled completely randomly, we can instead play M/2 matches where each match has a duplicate with the private cards swapped.  The sample mean and variance of the duplicate matches is also an unbiased estimator of Xi,j (provided agents forget the events of previous matches) but the variance of the estimator is much better while still using only M total matches since there is correlation between the duplicate cards.

When computing the total bankroll and bankroll instant run-off results we run into a similar situation.  Consider the random variable Xi representing the value of player i against an opponent chosen uniformly at random from S.  One way to compute Xi would be to use the Xi,j and compute the mean and variance of (?j in S Xi,j)/N.  This is an unbiased estimator of Xi but we can still do better.  Instead of computing the mean of the Xi,j, we can think about estimating Xi directly.  By using the same set of cards when playing the mth match for each different opponent j, we can exploit the correlation in the cards to get a lower variance estimator of Xi.  While we always use duplicate poker for our matches, the lower variance estimate of Xi relies on having each opponent play the same number of matches.  Since this isn't always the case in the competition, the competition results use both techniques.