Reinforcement Learning for Blackjack

Saqib A Kakvi Supervisor: Dr Marco Gillies April 30, 2009

ii

Abstract
This report explores the development of an Artificial Intelligence system for an already existing framework of card games, called SKCards. This system was developed by the author as a way of learning Java. The old Artificial intelligence is highly flawed. Reinforcement Learning was chosen as the method to be employed. Reinforcement Learning attempts to teach a computer certain actions, given certain states, based on past experience and numerical rewards gained. The agent either assigns values to states, or actions in states. This will initially be developed for Blackjack, with possible extensions to other games. Blackjack is one of the simpler games and the only game in the SKCards package which needs an Artificial Intelligence agent. All the other games are single player.

iii

iv

Acknowledgements
I am hugely grateful to my project supervisor, Dr. Marco Gillies, and all the members of staff, for all the support and assistance they gave me. I would also like to thank my colleagues, whose help was invaluable. In particular I would like to thank David Jarret, Rohan Sherrard, Daniel Smith, Benoy Sen, Jusna Begum, Arron Donnely & Josephene Amirthanayagam. And a very special thank you to Nidhi Shah and Fatima Nassir for their help in proofreading this report.

v

vi

Contents
1 Introduction 1.1 The problem I am solving . . . . . . . . 1.1.1 Reinforcement Learning Agent . 1.1.2 SKCards . . . . . . . . . . . . 1.2 The layout of the report . . . . . . . . . Background 2.1 Blackjack . . . . . . . . . . . . . . . 2.1.1 Overview . . . . . . . . . . . 2.1.2 Game play . . . . . . . . . . 2.2 Reinforcement Learning . . . . . . . 2.2.1 Introduction . . . . . . . . . . 2.2.2 Rewards . . . . . . . . . . . . 2.2.3 Policies . . . . . . . . . . . . 2.2.4 Agent Behaviour . . . . . . . 2.3 Blackjack in Reinforcement Learning 1 1 1 1 1 3 3 3 3 5 5 5 6 6 7 9 9 9 15 15 15 15 16 16 20 21 21 22 22 23 30 30 30 31 31 31

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

2

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

3

Project Description 3.1 Technologies involved . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System Development 4.1 Prototyping . . . . . . . . . . . . . . . . . . 4.1.1 Prototype 1 - Next card higher/lower . 4.1.2 Prototype 2 - GridWorld . . . . . . . 4.2 Final System . . . . . . . . . . . . . . . . . 4.2.1 AI . . . . . . . . . . . . . . . . . . . 4.2.2 SKCards . . . . . . . . . . . . . . . System Evaluation 5.1 System testing . . . . . . . . . 5.2 Results . . . . . . . . . . . . . 5.2.1 Table . . . . . . . . . 5.2.2 Graphs . . . . . . . . 5.3 Result Analysis . . . . . . . . 5.4 Possible Future Improvements 5.4.1 Card Counting . . . . 5.4.2 Probabilities . . . . . 5.4.3 Busting . . . . . . . . 5.4.4 SKCards . . . . . . .

4

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

5

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

vii

viii 6 Summary and Conclusions 6.1 Summary of the Project 6.1.1 Goals . . . . . 6.1.2 Outcomes . . . 6.2 Conclusion . . . . . . Project Self-Evaluation 7.1 Introduction . . . . 7.2 Project outcomes . 7.3 Project planning . . 7.4 Self-evaluation . .

CONTENTS 33 33 33 33 33 35 35 35 35 35 37 37 42 59 59 67 83 83 83 83 84 89 93 109 109 109 109 109 109 109 110 114 119 122 125

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

7

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

A Card High/Low prototype A.1 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B GridWorld prototype B.1 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C Blackjack AI C.1 Source Code . . . C.1.1 Face . . . C.1.2 Suit . . . C.1.3 Card . . . C.1.4 Pack . . . C.1.5 Blackjack D Final System D.1 Source Code . . . D.1.1 Face . . . D.1.2 Suit . . . D.1.3 Card . . . D.1.4 Pack . . . D.1.5 Shoe . . D.1.6 BJPlayer D.1.7 BJUI . . D.1.8 BJRules . D.1.9 RLPlayer

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

E Sample Blackjack Game

List of Figures
3.1 3.2 3.3 3.4 4.1 5.1 5.2 5.3 5.4 5.5 A.1 A.2 A.3 A.4 A.5 A.6 A.7 A.8 A.9 A.10 A.11 A.12 A.13 A.14 B.1 B.2 B.3 B.4 B.5 B.6 B.7 B.8 B.9 B.10 B.11 B.12 B.13 Old System Architecture New System Architecture Use Cases . . . . . . . . Single Episode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 12 13 14 17 25 26 27 28 29 43 44 45 46 47 48 49 50 51 52 53 54 55 56 69 70 71 72 73 74 75 76 77 78 79 80 81

The frequency of the extended actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . Q-Learing Policies . Win Percentages . . Loss Percantages . . Bust Percantages . . Net Win Percentages Values for Joker . Values for Ace . . Values for 2 . . . Values for 3 . . . Values for 4 . . . Values for 5 . . . Values for 6 . . . Values for 7 . . . Values for 8 . . . Values for 9 . . . Values for 10 . . Values for Jack . Values for Queen Values for King . Episode 1 . . Episode 25 . Episode 50 . Episode 100 . Episode 150 . Episode 200 . Episode 250 . Episode 500 . Episode 750 . Episode 1000 Episode 1500 Episode 2000 Episode 2500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

x

LIST OF FIGURES

List of Tables
5.1 Final Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 57 68

A.1 Final Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.1 Predicted values for the GridWorld problem . . . . . . . . . . . . . . . . . . . . . . . . .

xi

xii

LIST OF TABLES

Chapter 1

Introduction
1.1 The problem I am solving

This project has two goals. The major goal is the development of a Reinforcement Learning agent for Blackjack, using parts of an existing system. The second and less important goal is to improve the design of the existing system, so as to comply with software engineering norms. These goals may be solved independently and then combined to make the final system.

1.1.1

Reinforcement Learning Agent

When the initial system was built, it lacked a proper Artificial Intelligence Agent for the player to play against. This project aims to build an AI agent using Reinforcement Learning for Blackjack. This agent can be the dealer, or it can be another player in the game. Reinforcement Learning is very suited for the task of Blackjack due to the easily represented states. And the agent could learn to match a user’s style of play, over time.

1.1.2

SKCards

SKCards was built a as framework for all card games. It provides the basic Cards, Packs of Cards and a standard interface. The concept is that any programmer could create their own card game and easily integrate it into SKCards. This, would give us an infinitely extendable suite of card games. This was created as an experiment with the Java programming language.

1.2

The layout of the report

This report will aim to break down the problem and present it in several smaller pieces. Each piece will logically follow from the next and will try to build on top of the last. In this way this report should give a smooth transition from theory to implementation and finally the results and conclusions. The next chapter will discuss the separate parts of the project, namely, Blackjack and Reinforcement Learning. Then it discusses the two together. The following chapter will discuss the design aspects of the system. After that the development of the system will be looked at. This will detail the implementation of the system. Following that will the evaluation of the system. This will detail the experiments, the results and an analysis of these results. Finally, the last chapter will cover the self-evaluation.

1

2

CHAPTER 1. INTRODUCTION

Chapter 2

Background
2.1
2.1.1

Blackjack
Overview

Blackjack is a very simple game. The object is to get as close to, but not over 21 points in a maximum of 5 cards. This is achieved is by assigning values to each of the cards in the deck, irrespective of suit, as listed below: 2 - 10 ⇒ Face value (i.e. a 2 is 2 points, a 3 is 3 points) Courts (Jack, Queen, King) ⇒ 10 points A ⇒ 1 or 11 points. The player decides if he wants to the ace “high” (11) or “low” (1). This can be changed in a situation where a “high” ace would cause a player to be bust.

2.1.2

Game play

The player is dealt 2 cards face down. The dealer then deals themselves 2 cards, 1st face up then face down. The player places a bet on his hand and turns over his cards. The player may then take any of several actions, listed below. These have been divided into core and extended rules, where the core rules make for the basis of the game, and the extended rules give additional gameplay. Core Rules HIT: The player may ask for another card from the pack to be added to their hand, this is known as a “hit” or “twist” (the term “hit” shall be used henceforth). The dealer “hits” the player’s hand. The player’s score is increased by the point value of the card added. If the player’s score exceeds 21, they are “bust” and loses on that hand. If not, they can continue “hitting” until they are bust or have 5 cards in their hand. STAND: If the player is satisfied with their hand they may “stand”. This means that he is done with the current hand. The player then moves on to play any other hands they have(see Split in section 2.1.2). After “standing” on all their hands, the player ends their turn. Extended rules These rules are extensions of the rules in the previous section. They provide further functionality, however they are mostly based on the core rules. BLACKJACK: The quickest way to win. A blackjack is an Ace and a Court Card (King, Queen or Jack) bringing the player up to a total of 21, known as a “natural”. The player receives 2.5 times the money he was supposed to receive. This situation does not apply if the player has an Ace and a Ten. This rule was introduced in casinos to increase popularity of the game. Originally it only applied to an Ace and the Jack of Clubs, the “Black Jack”, which is where the game derives it name from. 3

4

CHAPTER 2. BACKGROUND

INSURANCE: If the dealer’s exposed card is an ace , threatening a Blackjack, the player may take out “insurance” against a Blackjack. They place another bet on the table. If the dealer has Blackjack, they gets a 2-to-1 payout. If the dealer doesn’t have a Blackjack, the player loses that money. As the name states, this serves as the player’s backup, should they lose to a dealer blackjack. SPLIT: If both the cards have the same face value (two Queens, two Fives, two Sevens. . . ) the player may opt to “split” their hand into two separate hands. The cards are separated and one card is added to each hand to make two cards in each hand. The player then places the same bet on the second hand. A player may have up to 5 hands. DOUBLE DOWN: If the total points of the players cards is greater than or equal to 11 the player may opt to “double down”. In this case the player doubles their bet and the player’s hand is hit, after which they are forced to stand, provided the player is not bust. This rule is basically a gamble by the player that they will need no more than 1 Card to reach a favourable score. This increased risk is why there is an increased reward. After the player is finished, the next player plays in the same manner. After all players have finished playing, play moves to the dealer, who plays in a similar manner. After the dealer plays, the winner is the hand which is closest to 21. It must be noticed that the dealer generally has some restrictions imposed on them to make it fair on the players.

Dealer restrictions normally come in the form of a hitting threshold, a score above which they can no longer hit. It must be noted that a dealer can hit on high counting hands that are “soft” i.e. they have an Ace counting high. The standard policy in casinos tends to be a threshold score of 17. Rule variations The rules listed above are mostly in effect in the United States of America. There are some variations, specifically on the British version of the game, know as Pontoon. The details of this game are described in Parlett (1994). Although the basic principles of the game are the same, there are several rule variations. These variations are described below. In this game, the role of dealer rotates between the players. The other players are called punters. The dealer also places a stake and plays as normal. Listed below are some of the other rule variations: BUY: This is a variation on the hit rule. Under this rule, the player may request for a card to be dealt face-down to them. This allows the player to keep their score secret. To do this the player must pay an amount greater than what they paid for the last card, where applicable, and a less than their total stake. The fifth card of the hand may not be bought. 5-CARD TRICK: If a player has a hand of 5 cards and has not gone bust, then they automatically win. They receive a payment equivalent double to their stake. PONTOON: In Pontoon Tens, Jacks, Queens & Kings are called tenths. A regular Pontoon is any tenth and an Ace. A Royal Pontoon is three sevens. The payouts for a Pontoon and Royal Pontoon are 2 times and 3 times respectively. Royal Pontoons can only be claimed by a punter. If the dealer holds a pontoon, then each punter pays the dealer double their stake. If the punter holds a Pontoon, they only pay an amount equal to their stake. BANKER’S PLAY: The banker plays in a similar way as the punters, however there are some restrictions on their play. The banker is not allowed to split pairs. The players may only beat the dealer if they hold a score of 16 or higher. After the dealer has stood and is not bust, at a score of s < 21, they make a call of “pay s + 1“. Any players holding hands scored at s + 1 or greater must reveal their hands and claim payment. A Royal Pontoon is unbeatable. A sample game which illustrates most of the gameplay and the basic and extended rules can be found in Appendix E.

2.2. REINFORCEMENT LEARNING

5

2.2
2.2.1

Reinforcement Learning
Introduction

Reinforcement learning is an unsupervised Artificial Intelligence learning method in which the agent teaches itself. The central principle is based on rewards for a sequence of actions taken or sequence of state transitions. Given a state S, the agent will take an action A, based on its knowledge of the environment. This will lead to a change in the environment, and eventually a reward, either positive or negative, will be given. The agent will then take this into consideration and accordingly adjust its knowledge. It must be noted that rewards are not given after every action, but may be after a series of actions. The agent has a preference to each possible action for a given state. This is based on what is known as the action’s value. This is a numeric value attached to each action A, given a state S. Every reward will adjust the action’s value and hence make it more or less desirable. There are several ways to value action and the choice is mainly arbitrary. It depends mainly on the developers understanding of the system and their preference as to how the agent should behave. Reinforcement Learning agents can act based on 3 things: values of states, values of actions or values of actions in states. The choice of which method to employ is left to the developer. This choice is made based on the developer’s understanding of the environment and his ability to represent it accurately. It must be noted that an agent can learn in more than one way from the same environment. Action valuing is only applicable in very trivial tasks and hence is not used as much as state or state-action valuing. The agent learns based on 2 things, namely the current reward and the previous rewards for A, S, or A & S. To simplify calculations and storage, we simply take the average of the previous rewards and update it. As the rewards are arbitrary, although mostly people use +1, -1 and 0, the agent will not know how good a reward is. By comparing it to the average of the past rewards, it can judge how high or low the reward was. A “high” reward would increase the value of the action and a “low” reward would reduce it. How exactly rewards are allocated is a bit of a mathematical quandary. Some methods allow rewards to be “passed back” to previous actions. This follows directly from the fact that rewards are allocated after a series of actions. This is so a low valued action which can be followed by a high valued action seems more attractive. If this were not done then the action values would not be balanced and would only reflect immediate rewards.

2.2.2

Rewards

The basic value method is based on discounted expected future returns. As mentioned above, when an agent receives a reward, it passes back some of that reward. However this is not always back all the way and may just be to the previous state. By doing this, when the agent next encounters the newly rewarded state, it passes back to the previous state and so on and so forth. Based on these values, and the probability of the states occurring, we calculate the expected reward. So in effect we are summing the rewards of the given possible states over their probability distribution, from 0 → ∞. This is not always possible to do, due the large problem space. Instead whenever a reward is received, all previous states or state-action pairs are updated by a fraction of that reward. This fraction is a constant call the discount rate and is denoted by γ. It follows that a state S1 n steps from state S2, which receives a reward of r, will get a reward of γ n × r, where 0 < γ < 1. It is very important, especially for stochastic tasks, to minimise the effect noise will have on the learning algorithm. To do this, we use what is call the step-size parameter, denoted by α. This means that only a small portion of any reward will be passed back, mitigating any effects of noise in the data. valid data

6

CHAPTER 2. BACKGROUND

will be observed several times, which will eventually add up to its true value. The value of the step-size parameter is 0 < α < 1.

2.2.3

Policies

By assigning rewards to states or state-action pairs, the agent learns what is known as a policy, denoted by P. The policy is the set of actions the agent will take in a given state. The aim of Reinforcement Learning is to learn a policy which is optimal or as close to the optimal policy as possible, denoted by P ∗ . For some problems, such as the grid world, the optimal policy is defined. However for stochastic and/or non-stationary problems, this is much harder and some other statistic must be used to determine if the agent has learnt the policy it was required to. Most Reinforcement Learning techniques have been mathematically proven to converge on P ∗ for the limit n → ∞, where n is the number of runs. However as it is not always possible to learn the optimal policy in a reasonable amount of time, so the focus on is to learn a policy that is as close as possible to P ∗ , of not exactly P ∗ . Reinforcement learning is a very powerful Artificial Intelligence tool. This very closely mimics the way humans learn by association. For every set of actions taken, there is a reward and this is associated to those actions, relative to past experience. However not every action is rewarded independently, but instead the entire series of actions receives a reward, which is mathematically calculated based on the whole reward. Hence it is not as powerful as human association, but it is none the less, very close to it. What makes it even more powerful is that it is an unsupervised technique. So the agent can run through thousands of plays on its own and learn some policy P. With supervised learning, the user would be required to tell the agent something after every play. This way it saves time and completely eliminates any human error in the learning process. Assuming the implementation is done correctly, as stated above, it will, in the limit n → ∞, converge on P ∗ .

2.2.4

Agent Behaviour

Agents can behave in several ways. The primary question with Reinforcement Learning is ”exploitation vs. exploration.” An agent can either exploit what it believes to be the best action, or it can explore new options, or ideally it can do some of both. The real trick is striking the correct balance between the two factors. Exploration is best for learning, as the agent will cover as much of the state space as possible, whereas exploitation is very useful after the agent ahs learnt a good policy. The simplest exploitative and most intuitive behaviour is called a greedy agent. This agent will always choose the action with the highest reward value, known as exploitation. In this way it is guaranteed large immediate rewards, but possibly lower rewards in the long run. I believe given a small number of actions needed to reach a goal, this is the best approach. But it must still be said that some lowly valued actions may lead to much higher rewards than greedy actions. In slight contrast to a greedy agent, we have what is known as an ε-greedy agent, which is more explorative. This agent explores other actions with lower reward values with a small probability ε. All non-greedy actions have an equal chance of being chosen. This allows the agent to explore more alternatives and possibly find a better action than the current greedy action. However in my opinion, this is not a very balanced way of having the agent behave as it may take a very poor action as likely as the next best action.

2.3. BLACKJACK IN REINFORCEMENT LEARNING

7

A more explorative agent behaviour is called a softmax selection agent. It takes all actions with a probability proportionate to their value. It creates a probability distribution according to the formula below:

eQt (a)\τ n Qt (a)\τ b=1 e
(Sutton and Barto, 1998) This ensures that the agent picks better actions more often than not. This is, in my opinion, very close to human behaviour and can be perceived as the agent being able to take risks, as it will not always take the greediest action, but it tends to take good actions rather than bad. Also this strikes a fine balance between exploitation and exploration. The parameter τ is called the temperature and decides how much the agent explores. The values of the temperature are bounded as 0 < τ < 1 High temperatures cause the actions to be all (nearly) equiprobable. Low temperatures cause a greater difference in selection probability for actions that differ in their value estimates. In the limit as τ → 0 , softmax action selection becomes the same as greedy action selection. (Sutton and Barto, 1998) Most agents start off with initial values all set to 0. However based on human knowledge of the system, optimistic initial values can be set. As they systems interacts with its environment, it will update the values based on formula defined by the user. This gives the agent a statistical bias. However Sutton and Barto (1998) make the point that it provides “an easy way to supply some prior knowledge about what level of rewards can be expected.”

2.3

Blackjack in Reinforcement Learning

Taking the existing Card/Pack system, which works very well, I intend to rebuild the functionality of the Player and Main classes. I will re-implement the basic GUI I had before, but the Player class will now be redefined. Each game will have 2 types of players, a human player, an improved version of the previous system, and a reinforcement learning agent. This will be the main concentration of my project. Blackjack is a fairly simple game with minimal mathematical complexity both in terms of rules and betting. Also a relatively small number of actions are required to reach and outcome and hence a reward. Both these factors, make Blackjack suitable as a reinforcement learning problem. After having made a successful agent for blackjack, it should be possible to abstract the basic functionality of the agent and hence create AIPlayer, which would integrate with my old framework and make an infinitely extensible suite of card games, with AI playing capabilities. Blackjack is what is known as a Markov Decision Process (MDP). Blackjack is said to be have the Markov property because, given any state, you can easily trace back previous states, as well as the actions leading to those states. This is the case with a large majority of Reinforcement Learning problems. The basic state representation will be the sum of the value of the cards the player has. This is the most important information, as far as the game is concerned. It would be wasteful to have a separate state for each combination, as they are irrelevant, a 6 and 7 is the same is a 5 and 8.

8

CHAPTER 2. BACKGROUND

However there are some complexities involved in this. The most prominent case is the fact that an Ace can count high (11) or low (1). Also deriving from this is the fact that when a high ace would cause a player to be bust, it is automatically switched to a low ace. Another complexity would be enabling the agent to split. Considering the state is only the sum, we can miss on a split, as a 5 and 5 is treated the same as a 2 and 8. Also the double down rule would be a minor complication, but this would not be due to the state representation, but because of the inherent complexity of the rule itself.

Chapter 3

Project Description
3.1 Technologies involved

The system will be developed using the Java programming language, as the existing system is written in Java. From this fact, it follows that any additions to the system must be in Java. To change to any other programming language would require rebuilding the entire system, after it has been refactored. Due to the time constraints, it was decided to continue using Java. In terms of Reinforcement Learning, the algorithms provided in Sutton and Barto (1998) are all in pseudo-code. Hence we can take these algorithms and implement them in any given programming language. As mentioned above, the existing system was developed in Java, hence the learning algorithms will all be implemented in Java to be added to the system.

3.2

System Design

The old system is based on 3 main components. Firstly the Card/Pack system, which is currently the only fully operational part. This may still be modified to improve on the current functionality. The second part is Player interface, and its subclasses, which for the representation for the Player. Thirdly we have the Main interface, its subclasses and SKMain which provide the GUI elements. SKMain is the Main Menu of the program, but does not implement Main. This can be illustrated in the class diagram (see figure 3.1). This system is not very stable, as the dependencies and interactions between Main and Player are very complicated and tend to cause unexpected errors. The new system uses a less densely connected system to ensure a more streamlined system and easier communication between objects. The interface “GameRules” will be introduced to act as a conduit for communication. This is illustrated in the class diagram (see figure 3.2). There are two basic Use Cases for the system, which a Player playing the game. This case is implemented in the old system. However this case is specific for a Human Player. The second Use Case is the Reinforcement Learning agent actually learning how to play the game. This case actually includes the first Use Case. This can be illustrated by the Use Case Diagram (see figure 3.3). Based on the reading and research, it has become apparent that Blackjack is an episodic task. The agent will learnt something each time it plays an episode, which in this case is a game of Blackjack. For Blackjack, there can be no more than 2 states between the initial state and the final state. This is because a Player may not have more than 5 cards in his hand. The initial state is 2 cards. Between these states we have 3 and 4 cards as the two possible intermediate states. The agent will operate as illustrated in the sequence diagram (see figure 3.4). 9

10

CHAPTER 3. PROJECT DESCRIPTION

However it must be noted that in some problems, there may be a reward at an intermediate state. This is not true for Blackjack, hence this was not included in the diagram.

3.2. SYSTEM DESIGN

1 Figure 3.1: Old System Architecture

11

12 CHAPTER 3. PROJECT DESCRIPTION

Figure 3.2: New System Architecture

3.2. SYSTEM DESIGN

Figure 3.3: Use Cases 13

14

CHAPTER 3. PROJECT DESCRIPTION

Figure 3.4: Single Episode

Chapter 4

System Development
4.1 Prototyping

To better understand Reinforcement Learning, two prototypes were built before the final system was built. These were built chosen in such a way that they shared some features with the final system, thus enabling code reuse. Both of these prototypes are discussed below.

4.1.1

Prototype 1 - Next card higher/lower

This system trains an agent to guess if the next Card is higher or lower than the current card. This was chosen so as to see how the Card/Pack system would be used in an agent. No proper Reinforcement Learning algorithm was used, but merely a sum of all past rewards. The agent used the points() method in the Card class to determine of a card was higher or lower. This gives a Joker 0 points, Ace 1 point, number cards face value and Jack, Queen and King get 11, 12, and 13 respectively. The agent received a reward of +1 for guessing correctly, and -1 for guessing incorrectly. The source code for this prototype can be found in Appendix A.1. The prototype did not take into account when two cards where equal and just counted this as an incorrect guess. This situation does not occur very often and was not considered for the prototype as it was simply a proof of concept as opposed to a proper system. Results from this prototype can be found in Appendix A.2. This prototype learns a policy which is fairly consistent with what is expected. However as this does not use a proper Reinforcement Learning method, it is not guaranteed to converge on P ∗ . Despite this, it uses the softmax probability as described in section 2.2.3. From this prototype, code that calculated softmax probabilities was made. This was carried over to the next prototype and the final system. Also it showed how to correctly use the Card/Pack system in a Reinforcement Learning Agent. Parts of this code was carried over to the final system.

4.1.2

Prototype 2 - GridWorld

This prototype solves what is known as the GridWorld problem. This problem has an agent in a gridbased environment. Each square is either a goal, a penalty or empty. The agent begins in a random square and must navigate its way to the goal(s) and avoid any penalty(ies). If the agent moves to a penalty or goal square, the episode ends. It receives a reward of +100 for a goal and -100 for a penalty. The source code for this prototype can be found in Appendix B.1. 15

16

CHAPTER 4. SYSTEM DEVELOPMENT

For this prototype, the Monte Carlo method was used (see Sutton and Barto (1998) chapter 5). This is mathematically proven to converge on P ∗ , which this system does. For an easier graphical representation, states were valued instead of state/action pairs. As stated before either approach is valid and can be chosen based on the users requirements. Some results of this can be found in Appendix B.2. From the previous prototype, code that calculated softmax probabilities was used. This prototype created the code for reward functions and value calculation. The code was modified, as the Monte Carlo method was abandoned in favour of Q-Learning for the final system. However, the learning parameters were carried over.

4.2
4.2.1

Final System
AI

As the Reinforcement Learning Agent was the primary objective of the project, it was built first. It was decided that the agent should be built using the Temporal Difference Q-Learning Algorithm. Although Sutton and Barto (1998) cover Blackjack under the MonteCarlo methods, Temporal Difference is a more powerful method in general. For this agent, only hit and stand were considered as valid actions. The rest of the actions (as listed in 2.1.2) were not considered for the system, as it was thought that the agent would not learn anything of substance. These reasons are enumerated below: • BLACKJACK: This would not help the agent learn anything except that a starting pair adding up to 21 is greatly desired. Hence the logical action would be to stand. The system will already learn it, as it will always get a negative reward for hitting at 21. Although the increased payout was added into the system, however the agent does not learn anything from this. • INSURANCE: Taking out an insurance would not be beneficial in anyway to an AI agents policy, as the payout of this is based on probability. Also this occurs so infrequently, that it was not considered worth adding in. • SPLIT: Splitting hands would only give the agent more hands to learn on. Although it would lead to more learning, it would not reduce the learning time enough to justify its inclusion. Also due to the complexity of being able to split many times it was not included in the final system. • DOUBLE DOWN: Doubling down would lead to a hit and then stand. The agent would effectively learn a policy for hitting at a score sn and standing at score sn+1 . This is would not lead to any additional learning and may also impede the learning of the agent. As a separate action, it would not add anything useful, as it is just a hit and a stand. For further clarification, a graph indicating how frequently each action may have been taken is included below. As can be seen, all the actions occur less than 10% of the time. This would not have lead to any significant learning. Although double down occurs over 70% of the time, as stated above it would not add anything meaningful to the learning.

4.2. FINAL SYSTEM 17

Figure 4.1: The frequency of the extended actions

18 1 2 3 4 5 double double double double double hit = stand sum = hprob sprob

CHAPTER 4. SYSTEM DEVELOPMENT Math . exp ( v a l [ HIT ] [ s c o r e − 1 ] /TAU ) ; = Math . exp ( v a l [STAND ] [ s c o r e − 1 ] /TAU ) ; hit + stand ; = h i t / sum ; = s t a n d / sum ;

The first step was to build the graphical user interface. This was built with the same design as the previous system. This was to ensure that the maximum of code could be re-used in the new system, so as to avoid any complications of porting the code from one interface to another. After the interface was built, the underlying game mechanics were built. For the AI, only hit and stand were considered for actions. The reasons for this have already been discussed above. To facilitate this, a class called BPlayer was created as an inner class, to facilitate easy access. This was used instead of using accessor methods to manipulate the object for simplicity and quick implementation. The BPlayer was created with the basic functionality required for hit and stand. The BPlayer had no real functionality, apart from the Reinforcement Learning, as most of the gameplay is done in the Blackjack object. After this was established, the Reinforcement Learning methods were added. The player has an exploration method, which uses softmax, a greedy method and a fixed method, for the dealer. Finally the Reinforcement Learning agent was developed. This mostly used code from the prototype, with modifications to fit into the system. The completion of the Reinforcement Learning agent makes both system equal in functionality. Although it must be noted that new AI works, where as the old AI does not. To achieve this, we need a state representation. For Blackjack, this is simple as it just the score of the player minus 1. The actions were stored as numbers. The values of the state action pairs were stored in a 2-dimensional array called val. The value for action A in state (score) S is given by val[A][S-1]. The states are represented as S-1 as Java uses 0 as the first array index and n − 1 as the nth index, hence the S-1. The softmax method was fairly simple to implement as there are only 2 possible actions, hence only two probabilities. These were calculated using the following code:

After these probabilities are calculated, we update the last state-action pair, if any. For the case where it is the first action, lastaxn is set to DEAL(-1). Then a random r number is generated to choose the action. After the action is chosen, we set the values of lastaxn and lastscore. This is achieved with the code given below:

The above code is an implementation of the general formula:

Q(st ,at ) = Q(st ,at ) + α(rt+1 + γmaxa Q(st+1 ,a) − Q(st ,at ) )
(Sutton and Barto, 1998) Here the reward rt+1 is 0, as no reward is given. After the learning method, a greedy method was devised. This is for use when the agent has learnt a policy P ≈ P ∗ . As stated previously, after exploring, the agent should exploit what it learnt. This was implemented using the following code:

4.2. FINAL SYSTEM 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 d o u b l e maxval = Math . max ( v a l [STAND ] [ s c o r e −1] , v a l [ HIT ] [ s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = A ∗ ( ( G ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } d o u b l e r = Math . random ( ) ; lastscore = score ; i f ( r > hprob ){ stand ( ) ; l a s t a x n = STAND ; } else{ hit (); l a s t a x n = HIT ; }

19

1 2 3 4 5 6

i f ( v a l [ HIT ] [ s c o r e −1] > v a l [STAND ] [ s c o r e −1]){ hit (); } else{ stand ( ) ; }

Finally the reward function was created. This gives a reward to the agent, based on the game outcome. After the dealer stands, the outcome is calculated. There are 5 possible outcomes. Based on the outcome, the player was judged to have achieved one of the 4 results and based on that it was given a reward. Listed below are the 5 outcomes and their end state and reward in brackets: 1. Player Busts (Bust -10) 2. Dealer Busts (Win +10) 3. Player and Dealer bust (Bust -10) 4. Player Wins (Win -10) 5. Player Loses (Lose -10) 6. Push (Push +5) This was achieved using the code below: 1 2 3 4 5 6 7 8 9 10 11 12 i f ( p l a y e r . s c o r e > 21 && d e a l e r . s c o r e <= 2 1 ) { / / player bust p l a y e r . reward ( −20); } e l s e i f ( p l a y e r . s c o r e <= 21 && d e a l e r . s c o r e > 2 1 ) { / / dealer bust p l a y e r . f u n d s += b e t ∗ 2 ; p l a y e r . reward ( 1 0 ) ; } e l s e i f ( p l a y e r . s c o r e > 21 && d e a l e r . s c o r e > 2 1 ) { / / d e a l e r and p l a y e r b u s t p l a y e r . reward ( −10);

20 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 } e l s e i f ( player . score > d e a l e r . score ){ / / p l a y e r wins p l a y e r . f u n d s += b e t ∗ 2 ; ; p l a y e r . reward ( 1 0 ) ; } e l s e i f ( player . score < d e a l e r . score ){ / / player loses p l a y e r . reward ( −10); } else{ / / push p l a y e r . f u n d s += b e t ; p l a y e r . reward ( 5 ) ; }

CHAPTER 4. SYSTEM DEVELOPMENT

The reward method uses the same formula as employed by the learning method, except the reward is explicitly passed to it as can be seen above. The code for this method is given below: 1 2 3 4 5 6 7 p r i v a t e void reward ( double r ){ d o u b l e maxval = Math . max ( v a l [STAND ] [ l a s t s c o r e −1] , v a l [ HIT ] [ l a s t s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = A ∗ ( r + (G ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } } It must be noted that the final class has more code than listed above, but this code was for testing purposes only and does not interfere with the learning algorithm. For the final code see Appendix C.1.5.

4.2.2

SKCards

The first feature that was added to the SKCards was the Shoe class (see Appendix D.1.5). This class is used to represent what is known as a dealer’s shoe. This is in effect a Pack, which is not fully used. A cut card is inserted at random in the Pack. Once this cut card is dealt, the Shoe is discarded and a new one is brought into play. This is used mainly in casinos for Blackjack to counter card counting. This was implemented for completeness of the system. The dealer does not distinguish between “soft” and “hard” scores. This is as dealer play was not the objective of the experiments. Then the Player interface was reworked, to fit in the new system. This included eliminating all relations to the Main interface, which was renamed to UI. Simultaneously, the UI interface was modified to eliminate all relations to the Player interface. The Rules abstract class was then created and the two interfaces where joined using this. With this the framework was completed. After completing the framework, the existing BJPlayer and BJUI (formerly BJMain) were built up. Where possible, code from the original system was used, in its original form or modified. Where it was not possible, code from the prototype was used, or it was made from scratch. This now makes the Old system, equivalent to the old system in functionality, barring the AI.

Chapter 5

System Evaluation
5.1 System testing

There are 4 basic tests that have been devised, all of which depend on the agent being able to react to a dealer policy effectively. As previously stated, we wish to come as close as possible to P ∗ , which in this case would be a policy better than its opponent (the dealer). Each test uses different values of rewards to achieve a different purpose. The first test uses the normal values as listed below: • Win = +10 • Push = +5 • Lose = -10 • Bust = -10 The subsequent tests involve doubling the reward for win, lose and bust and keeping the rest at their normal values. The push value was not modified as its low frequency of occurrence would mean its effect is minimal. These values were tested against 7 dealer policies, namely dealer hits below 11-17. As the dealer plays a more aggressive strategy, the agent should also adjust its policy, within the parameters provided. This range of policies is that the two boundary values represent significant policies. The lower bound (11) represent the policy a player must play to avoid going bust. The upper bound (17) represents the policy that is employed by several casinos, making it the de facto standard. All other policies were included for completion. The constants in the learning equation were kept the same as listed below: • Softmax temperature τ = 0.5 • Step size parameter α = 0.6 • Discount rate γ = 0.75 21

22

CHAPTER 5. SYSTEM EVALUATION

5.2
5.2.1

Results
Table

This table provides the results of running the agent with different parameters. The data and therefore the results are highly susceptible to noise, due to the random element in the system. However separate data sets with the same parameters tend show a consistency of approximately ±2.5%, giving a total error range of 5%. This error can be attributed to noise in the data, as well as the random factor within the game. Due to the nature of card games, even an optimal action can lead to a sub-optimal result, causing interference in the data. This is mitigated to a certain extent by the Law of Large numbers.

5.2. RESULTS DEALER POLICY 11 12 13 14 15 16 17 11 12 13 14 15 16 17 11 12 13 14 15 16 17 11 12 13 14 15 16 17 WIN REWARD 10 10 10 10 10 10 10 20 20 20 20 20 20 20 10 10 10 10 10 10 10 10 10 10 10 10 10 10 LOSE REWARD -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -20 -20 -20 -20 -20 -20 -20 BUST REWARD -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -10 -20 -20 -20 -20 -20 -20 -20 -10 -10 -10 -10 -10 -10 -10 WIN(%) 44.62 43.38 37.58 34.76 27.86 26.2 19.08 45.98 42.76 37.62 35.02 28.14 26.12 17.92 43.02 37.1 30.06 27.04 19.48 18.3 12.44 29.26 27.52 25.28 25.28 20.66 20.46 16.14 LOSE(%) 17.6 19.88 21.98 22.28 23.08 23.9 17.4 23.1 19.54 20.96 21.92 24.06 23.88 25.46 47.22 52.5 56.22 57.42 58.88 58.26 54.34 3.92 5.48 5.02 5.02 5.86 5.8 6.04 BUST(%) 30.66 29 29.06 29.34 30.22 29.26 38.74 22.02 29.82 29.12 29.66 28.72 29.88 29.54 0 0 0 0 0 0 0 60.08 59.56 59.86 59.32 60.84 59.86 59.02

23 NET WINS(%) -3.64 -5.5 -8.96 -10.76 -12.46 -11.74 -12.86 0.86 -6.6 -7.92 -11.04 -12.26 -12.82 -13.18 -4.2 -15.4 -22.02 -24.4 -26.26 -25.72 -16.76 -34.74 -37.52 -35.44 -32.9 -32.28 -31.12 -25.7

Table 5.1: Final Results

5.2.2

Graphs

All the data from table 5.1 is illustrated graphically in the graphs below. Each graphs show the comparison between the 4 separate testbeds. For each run the following values were calculated: 1. Final Policy 2. Winning Percentage 3. Losing Percentage 4. Bust Percentage 5. Net Winning Percentage The final policy is the highest score at which a greedy agent will hit and will stand on any higher score, based on the values learnt. It is expected that as the dealer policy increases, the agent will also increase its policy to try and stay ahead of it. It is also expected as agent’s reward values are adjusted, its policy will also be adjusted.

24

CHAPTER 5. SYSTEM EVALUATION

The winning percentage is the percentage of games in which neither dealer nor agent has gone bust and the player has won the game. Although when the dealer goes bust, it is considered that the agent has won, this was not considered as this situation arises from poor play from the dealer and not necessarily good play from the agent. Similarly, the losing percentage is the percentage of the games in which the dealer has won. Again the case of the player going bust was not considered under here as it can be attributed to poor play by the agent and not necessarily good play from the dealer. However, busting is an essential part of the game and must not be ignored. To this effect, the percentage of busts was recorded. This number is simply the percentage of times the player has been bust during the course of the learning. When considering all of these figures, we must remember that there is a large random factor that affects the game and therefore the results. Although the step-size parameter can smooth the learning process, there is really no way to compensate for the noise in the final results data. To consolidate all these figures, the net winning percentage was also calculated. This can be found using the formula below:

net wins = (wins + dealer busts) - (losses + player bust)

5.2. RESULTS 25

Figure 5.1: Q-Learing Policies

26

CHAPTER 5. SYSTEM EVALUATION

Figure 5.2: Win Percentages

5.2. RESULTS 27

Figure 5.3: Loss Percantages

28

CHAPTER 5. SYSTEM EVALUATION

Figure 5.4: Bust Percantages

5.2. RESULTS 29

Figure 5.5: Net Win Percentages

30

CHAPTER 5. SYSTEM EVALUATION

5.3

Result Analysis

As can be seen from Figure 5.1, when the agent is given a reward of -20 for busting, it learns the most effective policy to counter it, that is hitting at 11 or below. Here we can see the agent prefers to lose (-10), rather than go bust. The best possible reward is +10 for it winning. However, it chooses not to pursue this, as it is simply tries to avoid going bust and receiving the -20 reward. However, when it is given a reward of -20 for losing, it plays a very aggressive strategy, specifically 20, hitting on everything but a 21. Here the agent tries to avoid losing, even at the risk of busting. The best way to do that is play a very aggressive strategy, which it does. By increasing the win reward to +20, the agent plays a slightly more defensive strategy. However, on several points, both agents pick the same strategy. Cleary, the negative rewards have a greater bearing on the learning than the positive rewards. Hence any tuning of the agent would concentrate more on the negative rewards than the positive. One point that must be noted is that the agent consistently learns a threshold score at which to stop hitting. This is not specified on the algorithm. The algorithm merely provides the agent the ability to value the actions and hence build a policy based on its experience. This can be said to be one of the major successes of this project, in terms of the Reinforcement Learning Agent. Furthermore, from Figure 5.2 it can clearly be seen that as the dealer plays a more aggressive strategy, the player will lose more often. This is as expected, because as the dealer plays a more aggressive policy, it will score higher, hence making it harder for the player to win. A point worth noting is that having a higher bust and lose penalty results in a lower winning percentage, where as increasing the win reward has a minimal effect. As we proceed to Figure 5.3, we can see that this graph shows trends opposite to Figure 5.1. This is expected as the better policy the agent plays, it should lose less. The lines are not as smooth, due to the noise in the data, but the basic trend can be seen. Also it must be noted that the dealer policy has a minimal effect on the loss percentage of the agent. Figure 5.4 shows the same trend we saw in Figure 5.1. The more aggressive policy the agent plays, the more it will bust. As it chooses to hit on higher scores, there is a larger probability of it going bust. Hence, the same trend repeats itself, as expected. However, it must be noted that the agent busts more than the dealer. The reason for this is not known, but it has been observed that the player busts about 5% more than the dealer playign the same policy. From the above graphs and analysis it is clear that the best results are obtained with the winning set at +10 or +20. Changing the penalties may improve the performance in certain areas, but the overall performance is degraded. Hence the best agent under these circumstances would have a win reward of +10 or +20.

5.4

Possible Future Improvements

As can be seen from Figure 5.5 the agent tends to lose more than it wins when playing a policy of 12 or greater, no matter what the learning parameters. As this is the primary performance benchmark, any non-trivial improvement to the system would have to increase this figure. There are two possible ways this can be done, card counting and probability calculations. Both of these are discussed below.

5.4.1

Card Counting

Card counting is a tactic used by some players to try and “beat the system”. This tactic involves keeping a running total of what is know as the “count” of the deck. Each low card “counts” as +1 and each high card “counts” as +1. By adding the “count” of all the cards observed so far, the player can tell how “heavy”

5.4. POSSIBLE FUTURE IMPROVEMENTS

31

or “light” a deck is, indicating a large number of high or low cards respectively. Based on this, the player can adjust their strategy. The best way to combat this is to use a Shoe, which has been implemented (see Appendix D.1.5). Once the shoe is changed out, the player no longer has a count of the deck. This would mean that they must now begin to count again. While they are counting, they would presumably play a fixed strategy. By keeping the cut-off point as low as possible, card counting can be countered.

5.4.2

Probabilities

Another possible way to improve the system is by using probabilities. This assumes knowledge of how many decks are in play. Normally this player information is kept hidden from the player. However, if the number of the decks is known, the number of each card is known. Based on cards already played out, the agent can calculate the probability of getting the required card(s). Using these values, the agent can adjust its policy. The simplest counter to this is to simply not reveal the number of decks. Although a well programmed agent can calculate several probabilities at the same time. It could simply maintain multiple variables and update each with based on Cards observed. The number of decks could be inferred from the frequency of Cards appearing, e.g. 3 occurrences of the King of Spades would indicate at least 3 decks. Again, if we use a Shoe instead of a Pack, we can also quite effectively counter this.

5.4.3

Busting

It has been noted that the agent seems to bust too much. There is no apparent reason for this. Based on analysis of the code in Appendix C.1.5, the system should perform equally to the dealer. However this is not the case. The software needs to be thoroughly investigated to determine the cause of this error. Once found the error needs to be rectified. This would improve the net win percentage of the system, which as stated above is the measure of improvement in this system.

5.4.4

SKCards

There is scope for improvement in the SKCards framework. The Blackjack only implemnts the core rules. The extended rules can be implemented for the player and make it a complete Blackjack game. Also, the GUIs are very functional but do not look very attractive. These could be made more attractive. There is also the possibility of adding more games to the framework.

32

CHAPTER 5. SYSTEM EVALUATION

Chapter 6

Summary and Conclusions
6.1
6.1.1

Summary of the Project
Goals

There were two main aims of this project. The main goal was to develop a Reinforcement Learning agent for Blackjack. The second goal was to refactor the SKCards system and incorporate the Reinforcement Learning agent into it.

6.1.2

Outcomes

As we can see, the Reinforcement Learning agent was successfully developed. Although it was unable to beat the dealer, this is due to the nature of the game and not any shortcomings in the learning agent. However, it must also be noted that the large number of busts lead it perform lower than expected. The refactoring was successful. The new system was created, and conforms the the model shown in Figure 3.2. However due the large amount of time spent on trying to implement the AI, there was little time for the final system. Althought the system was built, it has only the most basic functionality and supports only the core rules as stated in Section 2.1.2. Although it was planned to have all the rules included in the system, this was not possible.

6.2

Conclusion

Based on the results gained from the system, it can be said that the Reinforcement Learning Agent learns an effective policy. Although the algorithm does not specify that the agent should learn a threshold score for hitting, the agent does learn this. This can be seen a large success of the learning algorithm. From figure 5.1 we can see that modifying the values of the bust and lose reward has a much greater effect than the win parameter. It can be concluded that the agent does learn to play Blackjack and that Reinforcement Learning is suitable for Blackjack. Also we notice that as the dealer plays more aggressively, even though the player is playing an equal or better policy, they tend to lose more. It can be inferred from here that through pure strategy alone, the casino policy is very difficult to beat. Also we can see there is some noise affecting the system, as can be seen in Figure 5.4. The graph is fairly consistent with what is expected as the number of busts relates to the policy the agent plays. Both figure 5.4 and 5.1 show the same basic trend. However in Figure 5.4 the last point for the agent with -20 reward for loss is significantly higher than the others. This is the noise and the random element of the game clouding the results. 33

34

CHAPTER 6. SUMMARY AND CONCLUSIONS

Although the Law of Large Numbers mitigates this effect to a certain extent, it is not sufficient. The Pack is shuffled using the Math.random() function in Java, which is not very effective at generating very large numbers of data. This factor is not very significant and does not affect most of the tests. However, this, combined with noise and the random element in Blackjack, can explain erroneous results such as this.

Chapter 7

Project Self-Evaluation
7.1 Introduction

This project aimed to build a Reinforcement Learning agent for Blackjack and build it into an existing framework. The framework had to be refactored to make it more usable and conform with software engineering norms. It can be said that this was very successful to a large extent.

7.2

Project outcomes

The first outcome, which was to develop a Reinforcement Learning agent, was successful. The agent does learn a policy of hitting until a threshold score. This score varies on the rewards it is given. However it can be seen from Figure 5.1 that the agent does learn a suitable policy based on what it is told to do. There is however, the problem of the agent busting more than the dealer. The second outcome was to refactor the old system and include the Reinforcement Learning agent. This was succesfully achieved. However, it was only built to support the core rules of Blackjack. This was due to a planing oversight on my part, which meant that I spent too much time on the AI development.

7.3

Project planning

The initial planning of my project was quite challenging. There were several tasks to be completed and these all needed to be scheduled. The 3 main tasks were: • Background reading and research • Prototyping • System development Although I had allocated time for all these tasks, the prototyping stage spilled over into the system development stage. This gave me less time to work on the system. Barring this, the project went according to plan. There was a maximum deviation from plan of 1 or 2 days. I now realise that I should have allowed a larger buffer time in between tasks for spill over and unforeseen errors

7.4

Self-evaluation

During the course of this project, I learnt the limitations of my programming skills. Although I consider my self highly competent in terms of programming, developing graphical user interfaces is an area where 35

36

CHAPTER 7. PROJECT SELF-EVALUATION

there is great room for improvement. My interfaces tend to lean towards functionality at the expense of appearances. This makes the interface less aesthetically pleasing. Also, I have realised that my planning tends to be to optimistic and does not account for errors and other spill over. This is due to my underestimation of buffer time between tasks, which would be taken up by error correction. Finally I have learnt that I am quite competent at analysis of numerical data. Based on the prototypes I have built and the results thereof, (see Appendix A.2 & B.2) it has become apparent to me that my mathematical skills are better than I thought before the project. However, if I work with the same data for too long, I tend to miss the bigger picture.

Appendix A

Card High/Low prototype
A.1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

Source Code

/ / A d d i n g HighLow t o SKCards package SKCards ; /∗ ∗ Importing the ArrayList class ∗ and t h e IO c o m p o n e n t s ∗/ import j a v a . u t i l . A r r a y L i s t ; import j a v a . i o . ∗ ;

/∗ ∗ ∗ This i s the 1 s t prototype for the Reinforcement Learining agent ∗ f o r t h e SKCards s y s t e s m , s p e c i f i c a l l y S K B l a c k J a c k . T h i s a g e n t ∗ a t t e m p t s t o guess o f t h e n e x t card i s h i g h e r or lower than t h e ∗ c u r r e n t c a r d . I t a c h e i c e v e s t h i s by u s i n g a s i m p l e sum o f a l l ∗ the past rewards . ∗ ∗ @author S a q i b A K a k v i ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @see SKCards . Card Card ∗ @SK . bug s y s t e m d o e s n o t c o n s i d e r t h e c a s e when t h e Cards a r e e q u a l . ∗/ p u b l i c c l a s s HighLow{ p u b l i c f i n a l s t a t i c i n t HIGH = 1 ; p u b l i c f i n a l s t a t i c i n t LOW = 0 ; double [ ] highv , highp ; d o u b l e [ ] lowv , lowp ; double t a u ; Pack p a c k ; int cscore , hscore ;

37

38 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90

APPENDIX A. CARD HIGH/LOW PROTOTYPE /∗ ∗ ∗ The main method . C r e a t e s a new HighLow . ∗/ p u b l i c s t a t i c v o i d main ( S t r i n g [ ] a r g s ) { new HighLow ( ) ; }

/∗ ∗ ∗ D e f a u l t c o n s t r u c t o r . C r e a t e s a new l e a r n i n g a g e n t , w i t h no p r e v i o u s ∗ knowledge . I t b e g i n s l e a r n i n g f o r 10 ,000 e p i s o d e s . A f t e r i t has ∗ i t h a s l e a r n t , i t p l a y s a g a i n s t t h e u s e r . As p a r t o f t h e t e s t i n g , ∗ the values of the l e a r n i n g are w r i t t e n to s e p e r a t e f i l e s . ∗/ p u b l i c HighLow ( ) { p a c k = new Pack ( 2 , t r u e ) ; h i g h v = new d o u b l e [ 1 4 ] ; h i g h p = new d o u b l e [ 1 4 ] ; lowv = new d o u b l e [ 1 4 ] ; lowp = new d o u b l e [ 1 4 ] ; tau = 0.1; f o r ( i n t i = 0 ; i < h i g h v . l e n g t h ; i ++){ highv [ i ] = 0 . 0 ; highp [ i ] = 0 . 0 ; lowv [ i ] = 0 . 0 ; lowp [ i ] = 0 . 0 ; } System . o u t . p r i n t l n ( ” ∗∗∗∗∗LEARNING∗∗∗∗∗ ” ) ; l o n g s t a r t = System . c u r r e n t T i m e M i l l i s ( ) ; try { B u f f e r e d W r i t e r [ ] wr = new B u f f e r e d W r i t e r [ 1 4 ] ; f o r ( i n t i = 0 ; i < wr . l e n g t h ; i ++){ S t r i n g name = ” v a l u e s ” + i + ” . c s v ” ; wr [ i ] = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( name ) ) ; wr [ i ] . w r i t e ( ” low ” + I n t e g e r . t o S t r i n g ( i ) ) ; wr [ i ] . w r i t e ( ” \ t ” ) ; wr [ i ] . w r i t e ( ” h i g h ” + I n t e g e r . t o S t r i n g ( i ) ) ; wr [ i ] . newLine ( ) ; } f o r ( i n t i = 0 ; i < 1 0 0 0 0 ; i ++){ episode ( ) ; f o r ( i n t j = 0 ; j < wr . l e n g t h ; j ++){ wr [ j ] . w r i t e ( Double . t o S t r i n g ( lowv [ j ] ) ) ; wr [ j ] . w r i t e ( ” \ t ” ) ;

A.1. SOURCE CODE 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 wr [ j ] . w r i t e ( Double . t o S t r i n g ( h i g h v [ j ] ) ) ; wr [ j ] . newLine ( ) ; } } f o r ( i n t i = 0 ; i < wr . l e n g t h ; i ++){ wr [ i ] . f l u s h ( ) ; wr [ i ] . c l o s e ( ) ; }

39

} c a t c h ( E x c e p t i o n ex ) { } l o n g s t o p = System . c u r r e n t T i m e M i l l i s ( ) ; long time = s to p − s t a r t ; System . o u t . p r i n t l n ( ” ∗∗∗∗∗DONE LEARNING∗∗∗∗∗ ” ) ; System . o u t . p r i n t l n ( ” ∗∗∗∗∗TOOK ” + t i m e + ” ms TO LEARN∗∗∗∗∗ ” ) ; try { B u f f e r e d W r i t e r bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” v a l u e s . t x t ” ) ) ; bw . w r i t e ( ”num” + ” \ t ” + ” h i g h ” + bw . newLine ( ) ; ” \ t ” + ” low ” ) ;

f o r ( i n t i = 0 ; i < h i g h v . l e n g t h ; i ++){ bw . w r i t e ( i + ” \ t ” + h i g h v [ i ] + ” \ t ” + lowv [ i ] ) ; bw . newLine ( ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } System . o u t . p r i n t l n ( ) ; System . o u t . p r i n t l n ( ”NOW LETS PLAY ! ! ! ” ) ; System . o u t . p r i n t l n ( ”You n e e d t o g u e s s i f t h e n e x t c a r d i s h i g h e r o r ” + ” lower than the c u r r e n t card ” ) ; System . o u t . p r i n t l n ( ” E n t e r 1 f o r h i g h e r , 0 f o r l o w e r , ” + ” e q u a l s i s n o t an o p t i o n ! ” ) ; cscore = 0; hscore = 0; try {

B u f f e r e d R e a d e r b r = new B u f f e r e d R e a d e r ( new I n p u t S t r e a m R e a d e r ( System . i n ) ) ; f o r ( i n t i = 1 ; i < 1 0 ; i ++){ Card c a r d 1 = p a c k . n e x t C a r d ( ) ; Card c a r d 2 = p a c k . n e x t C a r d ( ) ; i n t c1 = c a r d 1 . g e t P o i n t s ( ) ; i n t c2 = c a r d 2 . g e t P o i n t s ( ) ;

40 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198

APPENDIX A. CARD HIGH/LOW PROTOTYPE

i n t caxn ; i f ( h i g h v [ c1 ] > lowv [ c1 ] ) { caxn = 1 ; } else{ caxn = 0 ; } System . o u t . p r i n t l n ( ” Number ” + i + ” : ” + c a r d 1 . t o S t r i n g ( ) ) ; System . o u t . p r i n t ( ” E n t e r y o u r g u e s s : ” ) ; i n t haxn = I n t e g e r . p a r s e I n t ( b r . r e a d L i n e ( ) ) ; System . o u t . p r i n t l n ( ” I g u e s s e d ” + c a x n ) ; c p l a y ( caxn , c1 , c2 ) ; h p l a y ( haxn , c1 , c2 ) ; System . o u t . p r i n t l n ( ” The a c t u a l c a r d i s : ” + c a r d 2 . t o S t r i n g ( ) ) ; } System . o u t . p r i n t l n ( ”YOU SCORED : ” + h s c o r e ) ; System . o u t . p r i n t l n ( ” I SCORED : ” + c s c o r e ) ; } c a t c h ( E x c e p t i o n ex ) { } }

/∗ ∗ ∗ T h i s method g o e s t h r o u g h a s i n g l e e p i s o d e o f t h e l e a r n i n g . T h i s ∗ i n v o l v e s t h e a g e n t p i c k i n g an a c t i o n b a s e d on t h e v a l u e o f t h e ∗ c a r d . The s e l e c t e d a c t i o n i s t h e n t a k e n . ∗/ public void episode ( ) {

Card c a r d 1 = p a c k . n e x t C a r d ( ) ; Card c a r d 2 = p a c k . n e x t C a r d ( ) ; i n t c1 = c a r d 1 . g e t P o i n t s ( ) ; i n t c2 = c a r d 2 . g e t P o i n t s ( ) ; / / Jo < A < 2 < 3 < 4 < 5 < 6 < 7 < 8 < 9 < T < J < Q < K / / 0 < 1 < 2 < 3 < 4 < 5 < 6 < 7 < 8 < 9 < 10 < 11 < 12 < 13 d o u b l e hv = Math . exp ( h i g h v [ c1 ] / t a u ) ; d o u b l e l v = Math . exp ( lowv [ c2 ] / t a u ) ; d o u b l e sum = hv + l v ; h i g h p [ c1 ] = hv / sum ; lowp [ c1 ] = l v / sum ; d o u b l e r = Math . random ( ) ; i f ( r > lowp [ c1 ] ) {

A.1. SOURCE CODE 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 a c t i o n ( HIGH , c1 , c2 ) ; } else{ a c t i o n (LOW, c1 , c2 ) ; } }

41

/∗ ∗ ∗ C a r r i e s o u t a s i n g l e a c t i o n , h i g h o r low , b a s e d on t h e p a r a m e t e r s ∗ r e c i e v e d f r o m t h e e p i s o d e method . The v a l u e s o f t h e two c a r d s a r e ∗ a l s o p a s s e d f o r v e r i f i c a t i o n . The s c o r e i s a d j u s t e d a c c o r d i n l y , ∗ b a s e d on t h e v a l u e s o f t h e c a r d s and t h e axn t a k e n . ∗ ∗ @param axn t h e axn t o be t a k e n ∗ @param c1 t h e v a l u e o f Card1 ∗ @param c2 t h e v a l u e o f Card2 ∗/ p u b l i c v o i d a c t i o n ( i n t axn , i n t c1 , i n t c2 ) { i f ( axn == HIGH ) { i f ( c1 < c2 ) { h i g h v [ c1 ] += 1 ; } else{ h i g h v [ c1 ] −= 1 ; } } e l s e i f ( axn == LOW) { i f ( c1 > c2 ) { lowv [ c1 ] += 1 ; } else{ lowv [ c1 ] −= 1 ; } } }

/∗ ∗ ∗ T h i s method i s u s e d f o r a s i n g l e g u e s s made by t h e c o m p u t e r i n t h e ∗ head t o head game w i t h t h e human . I f t h e c o m p u t e r g u e s s e d c o r r e c t l y ∗ i t gets a point . ∗ ∗ @param axn t h e axn t o be t a k e n ∗ @param c1 t h e v a l u e o f Card1 ∗ @param c2 t h e v a l u e o f Card2 ∗/ p u b l i c v o i d c p l a y ( i n t axn , i n t c1 , i n t c2 ) { i f ( axn == 1 ) { i f ( c1 < c2 ) { c s c o r e ++; }

42 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 } e l s e i f ( axn == 0 ) { i f ( c1 > c2 ) { c s c o r e ++; } } }

APPENDIX A. CARD HIGH/LOW PROTOTYPE

/∗ ∗ ∗ T h i s method i s u s e d f o r a s i n g l e g u e s s made by t h e human i n t h e ∗ head t o head game w i t h t h e c o m p u t e r . I f t h e human g u e s s e d c o r r e c t l y ∗ i t gets a point . ∗ ∗ @param axn t h e axn t o be t a k e n ∗ @param c1 t h e v a l u e o f Card1 ∗ @param c2 t h e v a l u e o f Card2 ∗/ p u b l i c v o i d h p l a y ( i n t axn , i n t c1 , i n t c2 ) { i f ( axn == 1 ) { i f ( c1 < c2 ) { h s c o r e ++; } } e l s e i f ( axn == 0 ) { i f ( c1 > c2 ) { h s c o r e ++; } } } }

A.2

Results

This section contains the results of one run of the High/Low prototype. Several runs were conducted and the results were found to be fairly consistent. The reason for variations in the results was the fact that no proper learning algorithm was used, hence there was no supporting proof of convergence. The following graphs illustrate the values of guessing high or low for a given card, with respect to the number of occurrences.

A.2. RESULTS 43

Figure A.1: Values for Joker

44

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.2: Values for Ace

A.2. RESULTS 45

Figure A.3: Values for 2

46

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.4: Values for 3

A.2. RESULTS 47

Figure A.5: Values for 4

48

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.6: Values for 5

A.2. RESULTS 49

Figure A.7: Values for 6

50

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.8: Values for 7

A.2. RESULTS 51

Figure A.9: Values for 8

52

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.10: Values for 9

A.2. RESULTS 53

Figure A.11: Values for 10

54

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.12: Values for Jack

A.2. RESULTS 55

Figure A.13: Values for Queen

56

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Figure A.14: Values for King

A.2. RESULTS VALUE Joker 2 3 4 5 6 7 8 9 10 Jack Queen King FINAL POLICY HIGH HIGH HIGH HIGH HIGH HIGH LOW LOW LOW LOW LOW LOW LOW

57

Table A.1: Final Policy

As we can see, The agent learns policies at the extremities (Joker and King) very easily. However with the middle values, it has some problems. Using a proper learning algorithm would help smooth out the learning in the middle values. It can bee seen from Table A.1 below that the that the agent has learnt a fairly good policy.

58

APPENDIX A. CARD HIGH/LOW PROTOTYPE

Appendix B

GridWorld prototype
B.1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

Source Code

/ / A d d i n g G r i d W o r l d t o SKCards package SKCards ; /∗ ∗ importing the ArrayList class ∗ and t h e s w i n g / awt c o m p o n e n t s ∗ and t h e b o r d e r c o m p o n e n t s ∗ and t h e B u f f e r e d I m a g e c l a s s ∗ and t h e IO c o m p o n e n t s ∗ and t h e ImageIO c l a s s ∗/ import j a v a . u t i l . A r r a y L i s t ; import j a v a . awt . ∗ ; import j a v a x . s w i n g . ∗ ; import j a v a x . s w i n g . b o r d e r . ∗ ; import j a v a . awt . image . B u f f e r e d I m a g e ; import j a v a . i o . ∗ ; import j a v a x . i m a g e i o . ImageIO ;

/∗ ∗ ∗ T h i s i s t h e s e c o n d p r o t o t y p e f o r t h e SKCards s y s t e m . T h i s ∗ c l a s s a t t e m p t s t o s o l v e t h e G r i d W o r l d problem , u s i n g t h e ∗ R e i n f o r c m e n t L e a r n i n g Monte C a r l o m e t h o d s . ∗ ∗ @author S a q i b A K a k v i ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @SK . bug none ∗/ p u b l i c c l a s s G r i d W o r l d e x t e n d s JFrame { private private private private JLabel [ ] [ ] grid ; Point loc ; d o u b l e t a u , gamma , a l p h a ; double [ ] [ ] val , prob ; 59

60 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 private private private private private private b o o l e a n [ ] [ ] pen ; A r r a y L i s t <P o i n t > p a t h ; JFrame v a l u e s ; JLabel [ ] [ ] vlab ; String filename ; i n t h , w, p ;

APPENDIX B. GRIDWORLD PROTOTYPE

/∗ ∗ ∗ main method . C r e a t e s a new i n s t a n c e o f GridWorld , u s i n g t h e d e f a u l t ∗ constructor . ∗ ∗ @param a r g s n o t u s e d ∗/ p u b l i c s t a t i c v o i d main ( S t r i n g [ ] a r g s ) { new G r i d W o r l d ( ) . r u n ( ) ; }

/∗ ∗ ∗ D e f a u l t c o n s t r u c t o r . C r e a t e s a GridWorld i n s t a n c e b p a s s i n g t h e ∗ p a r a m e t e r s 10 and 10 t o t h e main c o n s t r u c t o r . ∗ ∗ @see G r i d W o r l d ∗/ p u b l i c GridWorld ( ) { t h i s ( 1 0 , 1 0 ) ; }

/∗ ∗ ∗ Main C o n s t r u c t o r . C r e a t e s a g r i d o f s i z e <i >n </ i >x<i >m</ i >. ∗ A l l t h e g r i d s q u a r e s a r e c r e a t e d and p e n a l t i e s a s s i g n e d t o t h e ∗ a p p r o p r i a t e s q u a r e s . The s y s t e m i s t h e n r u n f o r 2500 e p i s o d e s . ∗ ∗ @param n t h e l e n g t h o f t h e g r i d ∗ @param m t h e w i d t h o f t h e d r i d ∗/ p u b l i c G r i d W o r l d ( i n t n , i n t m) { super ( ” GridWorld ” ) ; Dimension d = ( T o o l k i t . g e t D e f a u l t T o o l k i t ( ) ) . g e t S c r e e n S i z e ( ) ; h = ( int )( d . getHeight ( ) ) / 4 ∗ 3; w = ( i n t ) ( d . getWidth ( ) ) / 2; s e t S i z e (w, h ) ; s e t D e f a u l t C l o s e O p e r a t i o n ( DISPOSE ON CLOSE ) ; g r i d = new J L a b e l [ n ] [m ] ; tau = 0.75; gamma = 0 . 7 5 ; alpha = 0.5; v a l = new d o u b l e [ n ] [m ] ; pen = new b o o l e a n [ n ] [m ] ; for ( int i = 0 ; for ( int j = grid [ i ][ grid [ i ][ i < n ; i ++){ 0 ; j < m; j ++){ j ] = new J L a b e l ( ) ; j ] . s e t B o r d e r ( new L i n e B o r d e r ( C o l o r . BLUE ) ) ;

B.1. SOURCE CODE 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144

61 g r i d [ i ] [ j ] . s e t H o r i z o n t a l A l i g n m e n t ( S w i n g C o n s t a n t s . CENTER ) ; i n t r = new Double ( Math . random ( ) ∗ 1 0 ) . i n t V a l u e ( ) ; i f ( ( i + j == ( ( n+m) / 2 ) − 1 ) && ( i ! = n / 2 ) ) { pen [ i ] [ j ] = t r u e ; g r i d [ i ] [ j ] . s e t F o r e g r o u n d ( C o l o r . RED ) ; } else{ pen [ i ] [ j ] = f a l s e ; }

} } C o n t a i n e r cp = g e t C o n t e n t P a n e ( ) ; cp . s e t L a y o u t ( new G r i d L a y o u t ( n ,m ) ) ; f o r ( i n t i = 0 ; i < n ; i ++){ f o r ( i n t j = 0 ; j < m; j ++){ cp . add ( g r i d [ i ] [ j ] ) ; } } g r i d [ 0 ] [ 0 ] . s e t F o r e g r o u n d ( C o l o r . GREEN ) ; g r i d [ n −1][m− 1 ] . s e t F o r e g r o u n d ( C o l o r . GREEN ) ; for ( int i = 0; i < for ( int j = 0; i f ( pen [ i ] [ j val [ i ][ } else{ val [ i ][ } } } v a l . l e n g t h ; i ++){ j < v a l [ i ] . l e n g t h ; j ++){ ]){ j ] = −100;

j ] = 0;

val [ 0 ] [ 0 ] = 100; v a l [ n −1][m−1] = 1 0 0 ; values values values values = new JFrame ( ”VALUES” ) ; . s e t S i z e (w, h ) ; . s e t L o c a t i o n (w , 0 ) ; . s e t D e f a u l t C l o s e O p e r a t i o n ( DISPOSE ON CLOSE ) ;

Container c = values . getContentPane ( ) ; c . s e t L a y o u t ( new G r i d L a y o u t ( n , m ) ) ; v l a b = new J L a b e l [ n ] [m ] ; f o r ( i n t i = 0 ; i < v l a b . l e n g t h ; i ++){ f o r ( i n t j = 0 ; j < v l a b [ i ] . l e n g t h ; j ++){ v l a b [ i ] [ j ] = new J L a b e l ( Double . t o S t r i n g ( v a l [ i ] [ j ] ) ) ; v l a b [ i ] [ j ] . s e t H o r i z o n t a l A l i g n m e n t ( S w i n g C o n s t a n t s . CENTER ) ;

62 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198

APPENDIX B. GRIDWORLD PROTOTYPE vlab [ i ] [ j ] . setOpaque ( true ) ; v l a b [ i ] [ j ] . s e t F o r e g r o u n d ( C o l o r . WHITE ) ; v l a b [ i ] [ j ] . s e t B a c k g r o u n d ( C o l o r . BLACK ) ; c . add ( v l a b [ i ] [ j ] ) ; } } reset (); t h i s . s e t V i s i b l e ( true ) ; values . s e t V i s i b l e ( true ) ; this . repaint (); values . repaint ( ) ; } /∗ ∗ ∗ Runs t h e G r i d W o r l d program . ∗/ public void run ( ) { f o r ( i n t i = 1 ; i <= 2 5 0 0 ; i ++){ f i l e n a m e = new S t r i n g ( ”TEST” + I n t e g e r . t o S t r i n g ( i ) + ” . j p g ” ) ; s e t T i t l e ( ” Episode ” + i ) ; episode ( ) ; reset (); screenShot ( ) ; } System . o u t . p r i n t l n ( p + ” PENALTIES” ) ; }

/∗ ∗ ∗ S e t s t h e G r i d W o r l d and t h e v a l u e s f r a m e ’ s v i s i b l i t y u s i n g ∗ the value passed . ∗ ∗ @param v i s t h e v i s i b i l i t y o f t h e windows . ∗/ public void s e t V i s i b i l i t y ( boolean v i s ){ this . setVisible ( vis ); values . setVisible ( vis ) ; }

/∗ ∗ ∗ T h i s method r u n s a s n i n g l e e p i s o d e t h r o u g h t h e s y s t e m . I t ∗ c o n t i n u e s t o move a s i n g l e s t e p u n t i l i t r e a c h e s a t e r m i n a l ∗ s q a u r e . When t h i s happens , a s c r e e n s h o t o f t h e windows i s t a k e n . ∗ ∗ @see s t e p ∗/ private void episode ( ) { i n t x = ( i n t ) ( ( Math . random ( ) ∗ 1 0 0 ) % g r i d . l e n g t h ) ; i n t y = ( i n t ) ( ( Math . random ( ) ∗ 1 0 0 ) % g r i d [ 0 ] . l e n g t h ) ;

B.1. SOURCE CODE 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252

63

l o c = new P o i n t ( x , y ) ; while ( c h e c k l o c ( ) != 0){ x = ( i n t ) ( ( Math . random ( ) ∗ 1 0 0 ) % g r i d . l e n g t h ) ; y = ( i n t ) ( ( Math . random ( ) ∗ 1 0 0 ) % g r i d [ 0 ] . l e n g t h ) ; l o c = new P o i n t ( x , y ) ; } p a t h = new A r r a y L i s t <P o i n t > ( ) ; g r i d [ x ] [ y ] . s e t T e x t ( ”X” ) ; w h i l e ( c h e c k l o c ( ) == 0 ) { step ( ) ; } reward ( ) ; }

/∗ ∗ ∗ T e s t i n g method . Used t o t a k e a s c r e e n s h o t o f t h e s y s t e m ∗ a f t e r e a c h e p i s o d e . T h i s a l l o w s t h e p r o g r e s s and r e s u l t s ∗ o f t h e s y s t e m t o be d o c u m e n t e d . ∗/ private void s cr ee nS ho t ( ) { / / CODE TAKEN FROM / / www . j a v a − t i p s . o r g / j a v a −se− t i p s / j a v a . awt / how−t o −c a p t u r e −s c r e e n s h o t . h t m l try { Robot r o b o t = new Robot ( ) ; / / Capture the screen s ho t of the area of the screen / / d e f i n e d by t h e r e c t a n g l e B u f f e r e d I m a g e b i = r o b o t . c r e a t e S c r e e n C a p t u r e ( new R e c t a n g l e (w∗ 2 , h ) ) ; ImageIO . w r i t e ( b i , ” j p g ” , new F i l e ( f i l e n a m e ) ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } }

/∗ ∗ ∗ ∗/ private void r e s e t ( ) { for ( int i = 0; i < grid for ( int j = 0; j < i f ( pen [ i ] [ j ] ) { grid [ i ][ j ] . } else{ grid [ i ][ j ] . . l e n g t h ; i ++){ g r i d [ i ] . l e n g t h ; j ++){ s e t T e x t ( ”PEN” ) ;

setText ( ”” );

64 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 } } }

APPENDIX B. GRIDWORLD PROTOTYPE

g r i d [ 0 ] [ 0 ] . s e t T e x t ( ”GOAL” ) ; g r i d [ g r i d . l e n g t h −1][ g r i d [ 0 ] . l e n g t h − 1 ] . s e t T e x t ( ”GOAL” ) ; f o r ( i n t i = 0 ; i < v l a b . l e n g t h ; i ++){ f o r ( i n t j = 0 ; j < v l a b [ i ] . l e n g t h ; j ++){ v l a b [ i ] [ j ] . s e t T e x t ( Double . t o S t r i n g ( v a l [ i ] [ j ] ) ) ; i f ( val [ i ] [ j ] > 0){ i n t v = ( i n t ) v a l [ i ] [ j ] ∗ 255 / 1 0 0 ; v l a b [ i ] [ j ] . s e t B a c k g r o u n d ( new C o l o r ( 0 , v , 0 ) ) ; } else{ i n t v = ( i n t ) v a l [ i ] [ j ] ∗ 255 / 100 ∗ −1; v l a b [ i ] [ j ] . s e t B a c k g r o u n d ( new C o l o r ( v , 0 , 0 ) ) ; } } } repaint (); }

/∗ ∗ ∗ ∗/ private int checkloc (){ i f ( g e t x ( ) == 0 && g e t y ( ) == 0 ) { return 1; } e l s e i f ( g e t x ( ) == g r i d . l e n g t h −1 && g e t y ( ) == g r i d [ 0 ] . l e n g t h −1){ return 1; } e l s e i f ( pen [ g e t x ( ) ] [ g e t y ( ) ] ) { r e t u r n −1; } else{ return 0; } }

/∗ ∗ ∗ T h i s method moves t h e a g e n t a s i n g l e s t e p . The d i r e c t i o n ∗ i s d e c i d e d on u s i n g t h e s o f t m a x p r o b a b i l i t y d i s t r i b u t i o n s . ∗ ∗ @see a c t i o n ( i n t ) ∗/ private void s t e p ( ) { double lim = 0 . 0 ;

B.1. SOURCE CODE 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 d o u b l e sum = 0 . 0 ; int x = getx ( ) ; int y = gety ( ) ; d o u b l e [ ] p r o b = new d o u b l e [ 4 ] ; d o u b l e up , down , l e f t , r i g h t ; b o o l e a n uv = t r u e , l v = t r u e , dv = t r u e , r v = t r u e ; i f ( g e t x ( ) == 0 ) { uv = f a l s e ; } e l s e i f ( g e t x ( ) == g r i d . l e n g t h −1){ dv = f a l s e ; } i f ( g e t y ( ) == g r i d [ 0 ] . l e n g t h −1){ rv = f a l s e ; } e l s e i f ( g e t y ( ) == 0 ) { lv = false ; } i f ( uv ) { up = Math . exp ( v a l [ x −1][ y ] / t a u ) ; sum += up ; } i f ( rv ){ r i g h t = Math . exp ( v a l [ x ] [ y + 1 ] / t a u ) ; sum += r i g h t ; } i f ( dv ) { down = Math . exp ( v a l [ x + 1 ] [ y ] / t a u ) ; sum += down ; } i f ( lv ){ l e f t = Math . exp ( v a l [ x ] [ y − 1 ] / t a u ) ; sum += l e f t ; } i f ( uv ) { prob [ 0 ] } i f ( rv ){ prob [ 1 ] } i f ( dv ) { prob [ 2 ] } i f ( lv ){ prob [ 3 ] }

65

= Math . exp ( v a l [ x −1][ y ] / t a u ) / sum ;

= Math . exp ( v a l [ x ] [ y + 1 ] / t a u ) / sum ;

= Math . exp ( v a l [ x + 1 ] [ y ] / t a u ) / sum ;

= Math . exp ( v a l [ x ] [ y − 1 ] / t a u ) / sum ;

d o u b l e r = Math . random ( ) ;

66 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414

APPENDIX B. GRIDWORLD PROTOTYPE

f o r ( i n t i = 0 ; i < p r o b . l e n g t h ; i ++){ l i m += p r o b [ i ] ; i f ( r < lim ){ action ( i ); break ; } } repaint (); }

/∗ ∗ ∗ T a k e s an a c t i o n , t h a t i s moving 1 s t e p i n any o f t h e ∗ valid directions . ∗ ∗ @param axn t h e d i r e c t i o n t o move . ∗/ p r i v a t e v o i d a c t i o n ( i n t axn ) { p a t h . add ( new P o i n t ( l o c ) ) ; s w i t c h ( axn ) { c a s e 0 : / / up loc . t r a n s l a t e break ; case 1: / / r i g h t loc . t r a n s l a t e break ; c a s e 2 : / / down loc . t r a n s l a t e break ; case 3: / / l e f t loc . t r a n s l a t e break ; }

( −1 ,0);

(0 ,1);

(1 ,0);

(0 , −1);

g r i d [ g e t x ( ) ] [ g e t y ( ) ] . s e t T e x t ( ”X” ) ; }

/∗ ∗ ∗ ∗/ private void reward ( ) { i f ( c h e c k l o c ( ) == −1) p ++; double reward = v a l [ g e t x ( ) ] [ g e t y ( ) ] ; f o r ( i n t i = p a t h . s i z e ( ) − 1 ; i >=0; i −−){ r e w a r d ∗= gamma ;

B.2. RESULTS 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446

67

Point p = path . get ( i ) ; i n t x = ( i n t ) p . getX ( ) ; i n t y = ( i n t ) p . getY ( ) ; v a l [ x ] [ y ] += a l p h a ∗ ( r e w a r d − v a l [ x ] [ y ] ) ; reward = val [ x ] [ y ] ; } }

/∗ ∗ ∗ R e t u r n s t h e x co−o r d i n a t e o f t h e a g e n t ’ s c u r r e n t ∗ p o s i t i o n , a s an i n t s o i t can be u s e d a s an ∗ index in the grid array . ∗ ∗ @return t h e x co−o r d i n a t e ∗ @see g e t y ∗/ p r i v a t e i n t g e t x ( ) { r e t u r n ( i n t ) l o c . getX ( ) ; }

/∗ ∗ ∗ R e t u r n s t h e y co−o r d i n a t e o f t h e a g e n t ’ s c u r r e n t ∗ p o s i t i o n , a s an i n t s o i t can be u s e d a s an ∗ index in the grid array . ∗ ∗ @return t h e y co−o r d i n a t e ∗ @see g e t x ∗/ p r i v a t e i n t g e t y ( ) { r e t u r n ( i n t ) l o c . getY ( ) ; } }

B.2

Results

The GridWorld problem can be said to be mathematically trivial, as the values can be calculated very easily. Given the final reward r and the discount rate γ, we can calculate the value v of a square n steps away from the goal using the formula: n

v =r×γ

Based on this formula, we can calculate the values of all the grid squares based on their distance from the goal square as given in the table below: Included below are some screen shots taken during the execution of the program. These screen shots selected were before episodes 1, 25, 50, 100, 150, 200, 250, 500, 750, 1000, 1500 & 2500.

68

APPENDIX B. GRIDWORLD PROTOTYPE

STEPS TO GOAL 0 1 2 3 4 5 6 7 8 9 10 11 12

VALUE 100 75 56.25 42.1875 31.6406 23.7305 17.7979 13.3484 10.0113 7.5085 5.6314 4.2235 3.1676

Table B.1: Predicted values for the GridWorld problem

B.2. RESULTS

Figure B.1: Episode 1 69

70

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.2: Episode 25

B.2. RESULTS

Figure B.3: Episode 50 71

72

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.4: Episode 100

B.2. RESULTS

Figure B.5: Episode 150 73

74

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.6: Episode 200

B.2. RESULTS

Figure B.7: Episode 250 75

76

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.8: Episode 500

B.2. RESULTS

Figure B.9: Episode 750 77

78

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.10: Episode 1000

B.2. RESULTS

Figure B.11: Episode 1500 79

80

APPENDIX B. GRIDWORLD PROTOTYPE

Figure B.12: Episode 2000

B.2. RESULTS

Figure B.13: Episode 2500 81

82

APPENDIX B. GRIDWORLD PROTOTYPE

Appendix C

Blackjack AI
C.1
C.1.1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Source Code
Face

/ / A d d i n g Face t o SKCards package SKCards ;

/∗ ∗ ∗ D e f i n e s an enum o f f a c e v a l u e s . The v a l u e s a r e 2 −10( t h e words ) , ∗ Jack , Queen , King , Ace and J o k e r . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 1 . 0 A ∗ @see SKCards . S u i t S u i t v a l u e s ∗ @SK . bug none ∗/ p u b l i c enum F a c e {TWO, THREE , FOUR, FIVE , SIX , SEVEN , EIGHT , NINE , TEN , JACK , QUEEN, KING , ACE, JOKER } ;

C.1.2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Suit

/ / A d d i n g S u i t t o SKCards package SKCards ;

/∗ ∗ ∗ D e f i n e s an enum o f t h e p l a y i n g c a r d s u i t s . The v a l u e s a r e Diamonds , Clubs , ∗ H e a r t s , S p a d e s and N u l l f o r j o k e r s . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 1 . 0 A ∗ @see SKCards . Face Face v a l u e s ∗ @SK . bug none ∗/ p u b l i c enum S u i t {DIAMONDS, CLUBS , HEARTS , SPADES , NULL} 83

84

APPENDIX C. BLACKJACK AI

C.1.3
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

Card

/ / A d d i n g Card t o SKCards package SKCards ;

/∗ ∗ Importing the ArrayList class ∗ and t h e s w i n g / awt c o m p o n e n t s ∗/ import j a v a . u t i l . A r r a y L i s t ; import j a v a . awt . ∗ ; import j a v a x . s w i n g . ∗ ;

/∗ ∗ ∗ Card r e p r e s e n t s a p l a y i n g c a r d and h a s two s a t e s − s u i t and f a c e v a l u e . ∗ S u i t i s d e f i n e d i n an enum and can be Diamonds , Clubs , H e a r t s , S p a d e s o r ∗ N u l l f o r J o k e r s . The f a c e v a l u e i s a l s o d e f i n e d i n and enum and i s t h e ∗ s t r i n g r e p r e s e n t a t i o n of the card ’ s f a c e value . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 1 . 0 A ∗ @SK . bug none ∗/ p u b l i c c l a s s Card { private Suit s u i t ; p r i v a t e Face f a c e V a l u e ; /∗ ∗ ∗ C o n s t r u c t s a new c a r d w i t h t h e v a l u e s p a s s e d a s a r g u m e n t s . ∗ @param v t h e f a c e v a l u e o f t h e c a r d . ∗ @param s t h e s u i t o f t h e c a r d . ∗ @see SKCards . Face Face V a l u e s ∗ @see SKCards . S u i t S u i t V a u l e s ∗/ p u b l i c Card ( F a c e v , S u i t s ) { faceValue = v ; suit = s ; }

/∗ ∗ ∗ R e t u r n s t h e f a c e v a l u e o f t h e a c t i v e Card . ∗ @return t h e c o d e o f t h e Face v a l u e . ∗/ public String getValue (){ i f ( t h i s == n u l l ) { return ” ” ; }

C.1. SOURCE CODE 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 e l s e i f ( f a c e V a l u e == return ”2” ; } e l s e i f ( f a c e V a l u e == return ”3” ; } e l s e i f ( f a c e V a l u e == return ”4” ; } e l s e i f ( f a c e V a l u e == return ”5” ; } e l s e i f ( f a c e V a l u e == return ”6” ; } e l s e i f ( f a c e V a l u e == return ”7” ; } e l s e i f ( f a c e V a l u e == return ”8” ; } e l s e i f ( f a c e V a l u e == return ”9” ; } e l s e i f ( f a c e V a l u e == r e t u r n ”T” ; } e l s e i f ( f a c e V a l u e == return ” J ” ; } e l s e i f ( f a c e V a l u e == r e t u r n ”Q” ; } e l s e i f ( f a c e V a l u e == r e t u r n ”K” ; } e l s e i f ( f a c e V a l u e == r e t u r n ”A” ; } e l s e i f ( f a c e V a l u e == return ” Joker ” ; } else{ return ” ” ; } } F a c e .TWO) {

85

F a c e . THREE) {

F a c e . FOUR) {

F a c e . FIVE ) {

F a c e . SIX ) {

F a c e . SEVEN) {

F a c e . EIGHT ) {

F a c e . NINE ) {

F a c e . TEN) {

F a c e . JACK ) {

F a c e . QUEEN) {

F a c e . KING ) {

F a c e . ACE) {

F a c e . JOKER ) {

/∗ ∗ ∗ R e t u r n s t h e f a c e v a l u e o f t h e a c t i v e Card . ∗ @return t h e c o d e o f t h e S u i t v a l u e . ∗/ public String getSuit (){

86 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160

APPENDIX C. BLACKJACK AI i f ( t h i s == n u l l ) { return ” ” ; } e l s e i f ( s u i t == S u i t . NULL) { return ” ” ; } else{ r e t u r n ( new C h a r a c t e r ( s u i t . t o S t r i n g ( ) . c h a r A t ( 0 ) ) ) . t o S t r i n g ( ) ; } }

/∗ ∗ ∗ G e t s t h e S t r i n g v a l u e o f t h e c a r d i n t t h e f o r m FS ( F a c e S u i t ) , e x c e p t t h e ∗ j o k e r . eg t h e 2 o f H e a r t s i s 2H, t h e 10 o f Diamonds i s TD , t h e King o f ∗ S p a d e s i s KS , t h e J o k e r i s J o k e r . ∗ @return t h e S t r i n g v a l u e o f t h e c a r d ∗/ public String t o S t r i n g (){ i f ( t h i s == n u l l ) { return ” ” ; } e l s e i f ( f a c e V a l u e == F a c e . JOKER ) { return ” Joker ” ; } else{ return ( getValue ( ) + g e t S u i t ( ) ) ; } }

/∗ ∗ ∗ G e t s t h e image a s s o c i a t e d w i t h t h e g i v e n card , u s i n g t h e c a r d S e t p r o v i d e d . ∗ @param c s t h e c a r d S e t ∗ @return t h e c a r d ’ s image ∗/ public ImageIcon toImage ( S t r i n g cs ){ r e t u r n new I m a g e I c o n ( ” i m a g e s / c a r d s / ” + c s + ” / ”+ t o S t r i n g ( ) +” . j p g ” ) ; }

/∗ ∗ ∗ R e t u r n s t h e image o f t h e c a r d f a c e down , u s i n g t h e b a c k p r o v i d e d ∗ @param b t h e b a c k ∗ @return t h e c a r d ’ s b a c k image ∗/ public ImageIcon toBackImage ( S t r i n g b ){ r e t u r n new I m a g e I c o n ( ” i m a g e s / b a c k s / ” + b ) ; }

C.1. SOURCE CODE 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 /∗ ∗ ∗ P r i n t s out the d e s c r i p t i o n of the a c t i v e card . ∗/ public void p r i n t ( ) { System . o u t . p r i n t l n ( t h i s . t o S t r i n g ( ) ) ; }

87

/∗ ∗ ∗ Returns t h e p o i n t value of t h e card f o r g e n e r a l uses . This methods c o u n t s ∗ Ace a s 1 , number c a r d s a t f a c e v a l u e , J a c k a s 1 1 , Queen a s 1 2 , King a s 13 ∗ and J o k e r a s 0 . ∗ @return t h e p o i n t v a l u e o f t h e c a r d . ∗/ public int getPoints (){ i f ( f a c e V a l u e == F a c e . ACE) { return 1; } e l s e i f ( f a c e V a l u e == F a c e .TWO) { return 2; } e l s e i f ( f a c e V a l u e == F a c e . THREE) { return 3; } e l s e i f ( f a c e V a l u e == F a c e . FOUR) { return 4; } e l s e i f ( f a c e V a l u e == F a c e . FIVE ) { return 5; } e l s e i f ( f a c e V a l u e == F a c e . SIX ) { return 6; } e l s e i f ( f a c e V a l u e == F a c e . SEVEN) { return 7; } e l s e i f ( f a c e V a l u e == F a c e . EIGHT ) { return 8; } e l s e i f ( f a c e V a l u e == F a c e . NINE ) { return 9; } e l s e i f ( f a c e V a l u e == F a c e . TEN) { return 10; } e l s e i f ( f a c e V a l u e == F a c e . JACK ) { return 11; } e l s e i f ( f a c e V a l u e == F a c e . QUEEN) { return 12; } e l s e i f ( f a c e V a l u e == F a c e . KING ) {

88 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 return 13; } else{ return 0; } }

APPENDIX C. BLACKJACK AI

/∗ ∗ ∗ Gets the p o i n t value of a fa c e value card . ∗ @param f a c e V a l u e t h e f a c e v a l u e o f t h e c a r d s . ∗/ p u b l i c s t a t i c i n t g e t P o i n t s ( Face f a c e V a l u e ){ i f ( f a c e V a l u e == F a c e . ACE) { return 1; } e l s e i f ( f a c e V a l u e == F a c e .TWO) { return 2; } e l s e i f ( f a c e V a l u e == F a c e . THREE) { return 3; } e l s e i f ( f a c e V a l u e == F a c e . FOUR) { return 4; } e l s e i f ( f a c e V a l u e == F a c e . FIVE ) { return 5; } e l s e i f ( f a c e V a l u e == F a c e . SIX ) { return 6; } e l s e i f ( f a c e V a l u e == F a c e . SEVEN) { return 7; } e l s e i f ( f a c e V a l u e == F a c e . EIGHT ) { return 8; } e l s e i f ( f a c e V a l u e == F a c e . NINE ) { return 9; } e l s e i f ( f a c e V a l u e == F a c e . TEN) { return 10; } e l s e i f ( f a c e V a l u e == F a c e . JACK ) { return 11; } e l s e i f ( f a c e V a l u e == F a c e . QUEEN) { return 12; } e l s e i f ( f a c e V a l u e == F a c e . KING ) { return 13; } else{ return 0;

C.1. SOURCE CODE 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 } } /∗ ∗ ∗ R e t u r n s t h e p o i n t s v a l u e o f t h e Card f o r B l a c k j a c k . T h i s ∗ method i s o n l y u s e d by t h e B l a c k J a c k p r o t o t y p e . The SKCards ∗ s y s t e m u s e s t h e B J R u l e s . g e t B J P o i n t s ( Card ) method , w h i c h i s ∗ almost i d e n t i c a l . ∗ ∗ @return t h e B l a c k j a c k p o i n t s v a l u e o f t h e Card . ∗/ public int getBJPoints (){ int points = getPoints ( ) ;

89

i f ( p o i n t s == 10 | | p o i n t s == 11 | | p o i n t s == 12 | | p o i n t s == 1 3 ) { return 10; } else{ return p o i n t s ; } } }

C.1.4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Pack

/ / A d d i n g Pack t o SKCards package SKCards ;

/∗ ∗ Importing the ArrayList class ∗/ import j a v a . u t i l . A r r a y L i s t ;

/∗ ∗ ∗ R e p r e s e n t s a p a c k o f c a r d s . A p a c k i s composed o f a number o f d e c k s , w i t h ∗ o r w i t h o u t j o k e r s , b o t h t o be s p e c i f i e d by t h e u s e r . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 1 . 0 A ∗ @see SKCards . Card Card ∗ @SK . bug none ∗/ p u b l i c c l a s s Pack { private final s t a t i c Suit [] s u i t s = Suit . values ( ) ; p r i v a t e f i n a l s t a t i c Face [ ] v a l s = Face . v a l u e s ( ) ; p r i v a t e A r r a y L i s t <Card> p a c k ; p r o t e c t e d i n t decks , c u r r e n t C a r d ; p r i v a t e boolean j o k e r s ;

90 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82

APPENDIX C. BLACKJACK AI

/∗ ∗ ∗ The d e f a u l t c o n s t r u c t e r . C r e a t e s a p a c k o f Cards , w i t h ∗ 1 d e c k and no j o k e r s ∗/ p u b l i c Pack ( ) { this (1 , false ) ; } /∗ ∗ ∗ C r e a t e s a new p a c k o f c a r d s . T h i s p a c k i s d e f i n e d by ∗ the parameters passed to i t . ∗ ∗ @param d e c k s t h e number o f d e c k s i n t h e p a c k . ∗ @param j o k e r s w h e t h e r t h e d e c k s h a v e j o k e r s o r n o t . ∗ @see SKCards . Card # makeDeck ∗/ p u b l i c Pack ( i n t d , b o o l e a n j ) { decks = d ; j o k e r s = j ; p a c k = new A r r a y L i s t <Card > ( ) ; A r r a y L i s t <Card> d e c k = new A r r a y L i s t <Card > ( ) ; f o r ( i n t i = 1 ; i <= d e c k s ; i ++){ d e c k = makeDeck ( j o k e r s ) ; deck = s h u f f l e ( deck ) ; f o r ( Card c : d e c k ) { p a c k . add ( c ) ; } } p a c k = Pack . s h u f f l e ( p a c k ) ; currentCard = 0; }

/∗ ∗ ∗ C r e a t e s a new o r d e r e d d e c k o f c a r d s , w i t h o r w i t h o u t j o k e r s d e p e n d i n g ∗ on t h e a r g u m e n t s p a s s e d . ∗ @param j o k e r s a b o o l e a n w h i c h d e t e r m i n e s i f t h e d e c k h a s j o k e s o r n o t . ∗ @return d e c k an A r r a y L i s t o f t h e c a r d s . ∗/ p u b l i c s t a t i c A r r a y L i s t <Card> makeDeck ( b o o l e a n j o k e r s ) { A r r a y L i s t <Card> d e c k = new A r r a y L i s t <Card > ( ) ; f o r ( i n t i = 0 ; i < s u i t s . l e n g t h − 1 ; i ++){ f o r ( i n t j = 0 ; j < v a l s . l e n g t h − 1 ; j ++){

C.1. SOURCE CODE 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136

91

d e c k . add ( new Card ( v a l s [ j ] , s u i t s [ i ] ) ) ; } }

i f ( j o k e r s ){ d e c k . add ( new Card ( F a c e . JOKER , S u i t . NULL ) ) ; d e c k . add ( new Card ( F a c e . JOKER , S u i t . NULL ) ) ; } return deck ; }

/∗ ∗ ∗ P r i n t s out a l l the cards in the deck . ∗/ public void p r i n t ( ) { f o r ( Card c : p a c k ) { c . print (); } }

/∗ ∗ ∗ T a k e s a p a c k o f c a r d s and s h u f f l e s them . ∗ @param p a c k t h e p a c k o f c a r d s t o be s h u f f l e . ∗ @return t h e s h u f f l e d P a c k . ∗/ p u b l i c s t a t i c A r r a y L i s t <Card> s h u f f l e ( A r r a y L i s t <Card> p a c k ) { A r r a y L i s t <Boolean > s h u f f l e d = new A r r a y L i s t <Boolean > ( ) ; A r r a y L i s t <Card> s h u f f l e d P a c k = new A r r a y L i s t <Card > ( ) ; f o r ( i n t i = 0 ; i < p a c k . s i z e ( ) ; i ++){ s h u f f l e d . add ( f a l s e ) ; } while ( s h u f f l e d . c o n t a i n s ( f a l s e ) ) { Double numb = Math . random ( ) ; numb ∗= 1 2 5 ; i n t numbr = numb . i n t V a l u e ( ) ; i n t num = numbr % p a c k . s i z e ( ) ; i f ( ! s h u f f l e d . g e t ( num ) ) { s h u f f l e d P a c k . add ( p a c k . g e t ( num ) ) ; s h u f f l e d . s e t ( num , t r u e ) ; }

92 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 } return sh uffle dPac k ; }

APPENDIX C. BLACKJACK AI

/∗ ∗ ∗ R e g e n e r a t e s t h e Pack , by c r e a t i n g a new Pack . T h i s i s i d e n t i c a l t o t h e ∗ p r e v i o u s o n l y i n t h e number o f d e c k s and t h e p r e s e n c e o f j o k e r s , b u t ∗ t h e o r d e r o f t h e Cards i s d i f f e r e n t . ∗/ p r o t e c t e d v o i d newPack ( ) { p a c k = new A r r a y L i s t <Card > ( ) ; A r r a y L i s t <Card> d e c k = new A r r a y L i s t <Card > ( ) ; f o r ( i n t i = 1 ; i <= d e c k s ; i ++){ d e c k = makeDeck ( j o k e r s ) ; deck = s h u f f l e ( deck ) ; f o r ( Card c : d e c k ) { p a c k . add ( c ) ; } } p a c k = Pack . s h u f f l e ( p a c k ) ; currentCard = 0; } /∗ ∗ ∗ G i v e s t h e n e x t Card i n t h e Pack . I f t h e Pack h a s b e e n u s e d ∗ up , a new Pack i s g e n e r e a t e d . ∗ ∗ @return t h e n e x t Card ∗/ p u b l i c Card n e x t C a r d ( ) { i f ( c u r r e n t C a r d >= s i z e ( ) ) { newPack ( ) ; } c u r r e n t C a r d ++; return pack . g e t ( c u r r e n t C a r d −1); }

/∗ ∗ ∗ R e t u r n s t h e s i z e o f t h e Pack ∗ ∗ @return s i z e o f t h e Pack ∗/ public int size (){

C.1. SOURCE CODE 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217

93

return pack . s i z e ( ) ; } /∗ ∗ ∗ R e t u r n s t h e number o f d e c k s i n t h e Pack ∗ ∗ @return t h e number o f d e c k s i n t h e Pack ∗/ public i n t decks ( ) { return decks ; }

/∗ ∗ ∗ Returns a boolean to i n d i c a t e i f t h e r e are ∗ J o k e r s i n t h e Pack o r n o t ∗ ∗ @rerurn t r u e i f t h e r e a r e J o e r s ∗ @return f a l s e i f t h e r e a r e no J o k e r s ∗/ public boolean j o k e r s ( ) { return j o k e r s ; } }

C.1.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Blackjack

/ / A d d i n g B l a c k J a c k t o SKCards package SKCards ; /∗ ∗ I m p o r t i n g t h e s w i n g / awt c o m p o n e n t s and t h e e v e n t m o d e l s ∗ and t h e I / O c l a s s e s ∗/ import j a v a . awt . ∗ ; import j a v a . awt . e v e n t . ∗ ; import j a v a x . s w i n g . ∗ ; import j a v a x . s w i n g . e v e n t . ∗ ; import j a v a . i o . ∗ ;

/∗ ∗ ∗ B l a c k j a c k game w i t h T e m p o r a l D i f f e r e n c e AI . ∗/ p u b l i c c l a s s B l a c k J a c k e x t e n d s JFrame {

/∗ ∗ ∗ T h i s c l a s s r e p r e s e n t s a p l a y e r i n t h e B l a c k J a c k game . I t can be e i t h e r a human ∗ p l a y e r or a R e i n f o r c e m e n t Learning player , u s i n g Temporal D i f f e r a n c e . ∗ @author S a q i b A K a k v i ( 3 3 1 0 6 0 8 7 ) ∗ @version 1 . 0

94 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 ∗ @rubish y e s i t i s ∗/ private c l a s s BPlayer { / / ACTIONS p r i v a t e s t a t i c f i n a l i n t HIT = 1 ; p r i v a t e s t a t i c f i n a l i n t STAND = 0 ; p r i v a t e s t a t i c f i n a l i n t DEAL = −1; / / CONSANTS p r i v a t e f i n a l d o u b l e TAU = 0 . 5 ; p r i v a t e f i n a l double G = 0 . 7 5 ; p r i v a t e f i n a l double A = 0 . 6 ;

APPENDIX C. BLACKJACK AI

/ / t h e p l a y e r ’ s hand p r i v a t e Card [ ] hand ; / / the player ’ s funds private int funds ; / / t h e p l a y e r s s c o r e s c o u n t i n g low / h i g h when a p p l i c a b l e private int score , hscore ; / / t h e v a l u e s a t t a c h e d o f t h e s t a t e −a c t i o n p a i r s p r i v a t e double [ ] [ ] v a l ; / / ∗∗∗∗∗ TESTING ∗∗∗∗∗∗ t h e number o f t i m e s t h e a g e n t t a k e s a c e r t a i n a c t i o n private int [ ] [ ] count ; / / t h e p l a y e r s l a s t a c t i o n and s c o r e private int lastaxn , l a s t s c o r e ; / / f l a g s t o i n d i c a t e i f t h e p l a y e r i s p l a y i n g an a c e a s h i g h , i s a g r r e d y a g e n / / or i s p l a y i n g a f i x e d h e u r i s t i c p o l i c y boolean high , greedy , f i x e d ; / / ∗∗∗∗∗ TESTING ∗∗∗∗∗∗ w r i t i n g a l l t h e a c t i o n d a t a t o f i l e BufferedWriter p ;

/∗ ∗ ∗ Constructor . ∗/ public BPlayer ( ){ hand = new Card [ 5 ] ; funds = 1000000; score = 0; hscore = 10; v a l = new d o u b l e [ 2 ] [ 2 1 ] ; c o u n t = new i n t [ 2 ] [ 2 1 ] ; l a s t a x n = DEAL ; l a s t s c o r e = 0; greedy = f a l s e ; fixed = false ; try { p = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” p l a y e r d u m p . c s v ” ) ) ; p . w r i t e ( ”SCORE ” ) ; p . w r i t e ( ”HITPROB ” ) ; p . w r i t e ( ”STANDPROB ” ) ; p . w r i t e ( ”RNDM ” ) ; p . w r i t e ( ”AXN ” ) ;

C.1. SOURCE CODE 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 p . newLine ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } }

95

/∗ ∗ ∗ T h i s method u s e s t h e s o f t m a x method o f e x p l o r a t i o n t o p i c k an a c t i o n i n t h ∗ current s t a t e . A f t e r the action i s picked , the previous s t a t e i s updated b ∗ on t h e Temporal−D i f f e r e n c e Q−L e a r n i n g a l g o r i t h m . ∗/ private void e x p l o r e ( ) { greedy = f a l s e ; fixed = false ; i f ( h i g h && h s c o r e <= 2 1 ) { score = hscore ; } e l s e i f ( h i g h && h s c o r e > 21 && s c o r e == h s c o r e ) { high = f a l s e ; s c o r e −= 1 0 ; } d o u b l e h i t = Math . exp ( v a l [ HIT ] [ s c o r e − 1 ] /TAU ) ; d o u b l e s t a n d = Math . exp ( v a l [STAND ] [ s c o r e − 1 ] /TAU ) ; d o u b l e sum = h i t + s t a n d ; d o u b l e h p r o b = h i t / sum ; d o u b l e s p r o b = s t a n d / sum ; d o u b l e maxval = Math . max ( v a l [STAND ] [ s c o r e −1] , v a l [ HIT ] [ s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = A ∗ ( ( G ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } d o u b l e r = Math . random ( ) ; lastscore = score ; try { p. p. p. p. if write ( Integer . toString ( score ) ) ; p . write ( ” ” ) ; w r i t e ( Double . t o S t r i n g ( h p r o b ) ) ; p . w r i t e ( ” ” ) ; w r i t e ( Double . t o S t r i n g ( s p r o b ) ) ; p . w r i t e ( ” ” ) ; w r i t e ( Double . t o S t r i n g ( r ) ) ; p . w r i t e ( ” ” ) ; ( r > hprob ){ stand ( ) ; l a s t a x n = STAND ; p . w r i t e ( ”STAND” ) ;

}

96 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 else{ hit (); l a s t a x n = HIT ; p . w r i t e ( ” HIT ” ) ; } p . newLine ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } c o u n t [ l a s t a x n ] [ l a s t s c o r e −1]++; }

APPENDIX C. BLACKJACK AI

/∗ ∗ ∗ T h i s method p r o v i d e s a g r e e d y a g e n t . T h i s a g e n t i s d e f e n s i v e i n i t b e h a v i o ∗ a s i t s t a n d s when t h e v a l u e o f h i t t i n g i s e q u a l t o t h e v a l u e o f s t a n d i n g . ∗/ private void greedy ( ) { greedy = true ; fixed = false ; i f ( h i g h && h s c o r e <= 2 1 ) { score = hscore ; } e l s e i f ( h i g h && h s c o r e > 21 && s c o r e == h s c o r e ) { high = f a l s e ; s c o r e −= 1 0 ; } lastscore = score ; i f ( v a l [ HIT ] [ s c o r e −1] > v a l [STAND ] [ s c o r e −1]){ hit (); l a s t a x n = HIT ; } else{ stand ( ) ; l a s t a x n = STAND ; } c o u n t [ l a s t a x n ] [ l a s t s c o r e −1]++; }

/∗ ∗ ∗ T h i s method p r o v i d e s a f i x e d h e u r i s t i c a g e n t f o r t h e R e i n f o r c e m e n t L e a r n i n ∗ agent to learn against . ∗/ private void f i x e d ( ) { greedy = f a l s e ; fixed = true ;

C.1. SOURCE CODE 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 i f ( h i g h && h s c o r e <= 2 1 ) { score = hscore ; } e l s e i f ( h i g h && h s c o r e > 21 && s c o r e == h s c o r e ) { high = f a l s e ; s c o r e −= 1 0 ; } i f ( s c o r e < 17){ hit (); } else{ stand ( ) ; } }

97

/∗ ∗ ∗ T h i s method i s u s e d t o a l l l o c a t e f i n a l r e w a r d s t o t h e l a s t s t a t e v i s i t e d . ∗ a g e n t s do n o t r e c i e v e r e w a r d s , a s t h e y a r e n o t l e a r n i n g . T h i s r e w a r d f u n c t ∗ b a s e d on t h e T e m p o r a l D i f f e r a n c e Q−L e a r n i n g a l g o r i t h m ∗/ p r i v a t e void reward ( double r ){ i f ( greedy | | fixed ){ return ; }

d o u b l e maxval = Math . max ( v a l [STAND ] [ l a s t s c o r e −1] , v a l [ HIT ] [ l a s t s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = A ∗ ( r + (G ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } } }

/ / the playing agents private BPlayer player , d e a l e r ; / / t h e pack o f card p r i v a t e Pack p a c k ; / / t h e p l a y e r s ’ hands private JLabel [ ] pcards , dcards ; / / display of the users funds private JLabel fundsLabel ; / / command b u t t o n s private JButton hit , stand , betButton , c l e a r ; / / panels used to d i s p l a y user components private JPanel buttonPanel , playerPanel , dealerPanel ; / / t h e c o n t e n t pane p r i v a t e C o n t a i n e r cp ; / / t e x t f i e l d f o r u s e r t o e n t e r amount t o b e t private JTextField betField ;

98 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294

APPENDIX C. BLACKJACK AI / / amount t h e u s e r h a s b e t private int bet ; / / u s e d t o i n d i c a t e c u r r e n t p o s i t i o n i n t h e p l a y e r s hand private int currentCard ; / / f l a g t o i n d i c a t e i f t h e p l a y e r i s p l a y i n g or not p r i v a t e boolean play ; / / ∗∗∗ TEST ∗∗∗ s t o r e s number o f t i m e s e a c h r e s u l t i s r e c o r d e d . private int [] r e s u l t ; / / ∗∗∗ TEST ∗∗∗ s t o r e s number o f t i m e s e a c h r e s u l t i s r e c o r d e d . private int [] deals ; / / ∗∗∗ TEST ∗∗∗ r e c o r d s t h e s c o r e s o f e a c h p l a y e r when t h e draw . p r i v a t e B u f f e r e d W r i t e r push ; / / ∗∗∗ TEST ∗∗∗ s t o r e s number o f t i m e s t h e p l a y e r b u s t s a t e a c h s c o r e . private int [] busts ; /∗ ∗ ∗ The main method . C r e a t e s a new B l a c k J a c k i n t e r f a c e . ∗ @param a r g s n o t u s e d ∗/ p u b l i c s t a t i c v o i d main ( S t r i n g [ ] a r g s ) { new B l a c k J a c k ( ) ; }

/∗ ∗ ∗ S o l e c o n s t r u c t o r . C r e a t e s t h e i n t e r f a c e and t h e p l a y e r s . B e g i n s t h e l e a r n i n g ∗ for the agents ( s ) . ∗/ public BlackJack (){ super ( ” BlackJack ” ) ; setSize (600 ,600); Dimension d = ( T o o l k i t . g e t D e f a u l t T o o l k i t ( ) ) . g e t S c r e e n S i z e ( ) ; int h = ( int )( d . getHeight ( ) ) ; i n t w = ( i n t ) ( d . getWidth ( ) ) ; i n t x = (w − 6 0 0 ) / 2 ; i n t y = ( h − 600) / 2; setLocation (x , y ); s e t D e f a u l t C l o s e O p e r a t i o n ( EXIT ON CLOSE ) ; p a c k = new Pack ( ) ; p l a y e r = new B P l a y e r ( ) ; d e a l e r = new B P l a y e r ( ) ; play = true ; r e s u l t = new i n t [ 6 ] ; d e a l s = new i n t [ 2 1 ] ; b u s t s = new i n t [ 2 1 ] ; currentCard = 0; player . score = 0; try { p u s h = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” p u s h . t x t ” ) ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; }

C.1. SOURCE CODE 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348

99

cp = g e t C o n t e n t P a n e ( ) ; cp . s e t L a y o u t ( new B o r d e r L a y o u t ( ) ) ; h i t = new J B u t t o n ( ” HIT ” ) ; h i t . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { hit (); } }); hit . setEnabled ( false ) ; s t a n d = new J B u t t o n ( ”STAND” ) ; s t a n d . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { stand ( ) ; } }); stand . setEnabled ( false ) ; b e t F i e l d = new J T e x t F i e l d ( ” 10 ” ) ; b e t B u t t o n = new J B u t t o n ( ”BET” ) ; b e t B u t t o n . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { bet ( ) ; } }); c l e a r = new J B u t t o n ( ”CLEAR” ) ; c l e a r . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { clear (); } }); b u t t o n P a n e l = new J P a n e l ( ) ; b u t t o n P a n e l . s e t L a y o u t ( new G r i d L a y o u t ( 5 , 1 ) ) ; buttonPanel buttonPanel buttonPanel buttonPanel buttonPanel . add ( h i t ) ; . add ( s t a n d ) ; . add ( b e t F i e l d ) ; . add ( b e t B u t t o n ) ; . add ( c l e a r ) ;

p c a r d s = new J L a b e l [ 5 ] ; d c a r d s = new J L a b e l [ 5 ] ; f u n d s L a b e l = new J L a b e l ( I n t e g e r . t o S t r i n g ( p l a y e r . f u n d s ) ) ; p l a y e r P a n e l = new J P a n e l ( ) ; p l a y e r P a n e l . s e t L a y o u t ( new G r i d L a y o u t ( 1 , 6 ) ) ;

100 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402

APPENDIX C. BLACKJACK AI d e a l e r P a n e l = new J P a n e l ( ) ; d e a l e r P a n e l . s e t L a y o u t ( new G r i d L a y o u t ( 1 , 6 ) ) ; f o r ( i n t i = 0 ; i < 5 ; i ++){ p c a r d s [ i ] = new J L a b e l ( ) ; d c a r d s [ i ] = new J L a b e l ( ) ; p l a y e r P a n e l . add ( p c a r d s [ i ] ) ; d e a l e r P a n e l . add ( d c a r d s [ i ] ) ; } p l a y e r P a n e l . add ( f u n d s L a b e l ) ; d e a l e r P a n e l . add ( new J L a b e l ( ) ) ; cp . add ( d e a l e r P a n e l , B o r d e r L a y o u t . NORTH ) ; cp . add ( b u t t o n P a n e l , B o r d e r L a y o u t . EAST ) ; cp . add ( p l a y e r P a n e l , B o r d e r L a y o u t . SOUTH ) ; s e t V i s i b l e ( true ) ; int c = 0; f o r ( i n t i = 0 ; i < 1 0 0 0 ; i ++){ bet ( ) ; while ( play ){ player . explore ( ) ; } while ( ! play ){ dealer . fixed ( ) ; } System . o u t . p r i n t l n ( ” p l a y e d ” + c ) ; c ++; } write (1); f o r ( i n t i = 0 ; i < 9 0 0 0 ; i ++){ bet ( ) ; while ( play ){ player . explore ( ) ; } while ( ! play ){ dealer . fixed ( ) ; } System . o u t . p r i n t l n ( ” p l a y e d ” + c ) ; c ++; } write (2); f o r ( i n t i = 0 ; i < 4 0 0 0 0 ; i ++){ bet ( ) ; while ( play ){ player . explore ( ) ; } while ( ! play ){

C.1. SOURCE CODE 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 dealer . fixed ( ) ; } System . o u t . p r i n t l n ( ” p l a y e d ” + c ) ; c ++; } write (3); r e s u l t = new i n t [ 6 ] ; b u s t s = new i n t [ 2 1 ] ; p l a y e r . c o u n t = new i n t [ 2 ] [ 2 1 ] ; f o r ( i n t i = 0 ; i < 5 0 0 0 ; i ++){ bet ( ) ; while ( play ){ player . greedy ( ) ; } while ( ! play ){ dealer . fixed ( ) ; } System . o u t . p r i n t l n ( ” p l a y e d ” + c ) ; c ++; } write (4); writecount ( ) ; try { push . f l u s h ( ) ; push . c l o s e ( ) ; player . p . flush ( ) ; player . p . close ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } }

101

/∗ ∗ ∗ ∗∗∗∗∗ TESTING METHOD∗∗∗∗∗∗ <b r /> ∗ W r i t e s a l l t h e d a t a c o l l e c t e d t o . c s v f i l e s , w h i c h can be i m p o r t e d i n t o an ∗ s p r e a d s h e e t package f o r f u r t h e r data a n a l y s i s . ∗ @param x t h e number t a g f o r t h e f i l e . ∗/ private void w r i t e ( i n t x ){ try {

B u f f e r e d W r i t e r bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” r e s u l t s ” + x + ” . c bw . bw . bw . bw . bw . w r i t e ( ” P l a y e r Bust ” ) ; write ( ”\ t ” ) ; w r i t e ( ” Dealer Bust ” ) ; write ( ”\ t ” ) ; w r i t e ( ” Both B u s t ” ) ;

102 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” P l a y e r w i n s ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” P l a y e r L o s e s ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” Push ” ) ; bw . newLine ( ) ;

APPENDIX C. BLACKJACK AI

f o r ( i n t i = 0 ; i < r e s u l t . l e n g t h ; i ++){ bw . w r i t e ( I n t e g e r . t o S t r i n g ( r e s u l t [ i ] ) ) ; bw . w r i t e ( ” \ t ” ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ; bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” p v a l s ” + x + ” . c s v ” ) ) ; bw . w r i t e ( ” s c o r e ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” s t a n d ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” h i t ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . newLine ( ) ; f o r ( i n t i = 0 ; i < p l a y e r . v a l [ 0 ] . l e n g t h ; i ++){ bw . w r i t e ( I n t e g e r . t o S t r i n g ( i ) ) ; bw . w r i t e ( ” \ t ” ) ; f o r ( i n t j = 0 ; j < p l a y e r . v a l . l e n g t h ; j ++){ bw . w r i t e ( Double . t o S t r i n g ( p l a y e r . v a l [ j ] [ i ] ) ) ; bw . w r i t e ( ” \ t ” ) ; } bw . newLine ( ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ; bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” d v a l s ” + x + ” . c s v ” ) ) ; bw . w r i t e ( ” s c o r e ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” s t a n d ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( ” h i t ” ) ; bw . w r i t e ( ” \ t ” ) ; bw . newLine ( ) ; f o r ( i n t i = 0 ; i < p l a y e r . c o u n t [ 0 ] . l e n g t h ; i ++){ bw . w r i t e ( I n t e g e r . t o S t r i n g ( i ) ) ; bw . w r i t e ( ” \ t ” ) ; f o r ( i n t j = 0 ; j < p l a y e r . c o u n t . l e n g t h ; j ++){ bw . w r i t e ( Double . t o S t r i n g ( p l a y e r . c o u n t [ j ] [ i ] ) ) ;

C.1. SOURCE CODE 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 bw . w r i t e ( ” \ t ” ) ; } bw . newLine ( ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ;

103

bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” b u s t s ” + x + ” . c s v ” ) ) ; f o r ( i n t i = 0 ; i < b u s t s . l e n g t h ; i ++){ bw . w r i t e ( I n t e g e r . t o S t r i n g ( i ) ) ; bw . w r i t e ( ” \ t ” ) ; bw . w r i t e ( I n t e g e r . t o S t r i n g ( b u s t s [ i ] ) ) ; bw . newLine ( ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } } public void w r i t e c o u n t ( ) { try { B u f f e r e d W r i t e r bw = new B u f f e r e d W r i t e r ( new F i l e W r i t e r ( ” c o u n t . t x t ” ) ) ; f o r ( i n t i = 0 ; i < 2 ; i ++){ f o r ( i n t j = 0 ; j < 2 1 ; j ++){ bw . w r i t e ( new I n t e g e r ( p l a y e r . c o u n t [ j ] [ i ] ) . t o S t r i n g ( ) ) ; bw . w r i t e ( ” ” ) ; } bw . newLine ( ) ; } bw . f l u s h ( ) ; bw . c l o s e ( ) ; } c a t c h ( E x c e p t i o n ex ) { } }

/∗ ∗ ∗ T h i s method i s u s e d t o u p d a t e t h e GUI t o t h e m o s t c u r r e n t s t a t e . E v e r y c a l l o f ∗ m e t h o d s r e d r a w s a l l e l e m e n t s on s c r e e n and t h e n r e p a i n t s t h e JFrame . ∗ @see c l e a r ∗/ private void r e f r e s h ( ) { i f ( play ){ f o r ( i n t i = 0 ; i < 5 ; i ++){ i f ( p l a y e r . hand [ i ] ! = n u l l ) {

104 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618

APPENDIX C. BLACKJACK AI p c a r d s [ i ] . s e t I c o n ( p l a y e r . hand [ i ] . t o I m a g e ( ” Normal ” ) ) ; } } d c a r d s [ 0 ] . s e t I c o n ( d e a l e r . hand [ 0 ] . t o I m a g e ( ” Normal ” ) ) ; d c a r d s [ 1 ] . s e t I c o n ( d e a l e r . hand [ 1 ] . t o B a c k I m a g e ( ” T r i c k s t e r ” ) ) ; } else{ f o r ( i n t i = 0 ; i < 5 ; i ++){ i f ( d e a l e r . hand [ i ] ! = n u l l ) { d c a r d s [ i ] . s e t I c o n ( d e a l e r . hand [ i ] . t o I m a g e ( ” Normal ” ) ) ; } } } fundsLabel . setText ( I n t e g e r . t o S t r i n g ( player . funds ) ) ; ( ( JComponent ) ( cp ) ) . r e v a l i d a t e ( ) ; repaint (); }

/∗ ∗ ∗ B e t s money f o r t h e p l a y e r . The amount t o be b e t i s p a r s e d f r o m b e t F i e l d . b e t F i ∗ i s s e t t o 10 by d e f a u l t . T h i s i s s o t h e AI d o e s n o t h a v e t o be made t o b e t , a s ∗ can s i m p l y c a l l t h e method . ∗/ private void bet ( ) { bet = Integer . parseInt ( betField . getText ( ) ) ; i f ( b e t <= p l a y e r . f u n d s ) { p l a y e r . f u n d s −= b e t ; deal ( ) ; h i t . setEnabled ( true ) ; stand . setEnabled ( true ) ; betButton . setEnabled ( false ) ; } refresh (); }

/∗ ∗ ∗ D e a l s a new hand . The p l a y e r / d e a l e r h a n d s a r e r e i n i t i a l i s e d and t h e p o i n t e r a r ∗ a l l r e s e t . Both s c o r e s are t h e n updated . ∗ @see h i t ∗ @see s t a n d ∗/ private void d e al ( ) { d e a l e r . hand [ 0 ] = p a c k . n e x t C a r d ( ) ; d e a l e r . s c o r e += d e a l e r . hand [ 0 ] . g e t B J P o i n t s ( ) ; i f ( d e a l e r . hand [ 0 ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { d e a l e r . high = true ; dealer . hscore = dealer . score + 10;

C.1. SOURCE CODE 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 } d e a l e r . hand [ 1 ] = p a c k . n e x t C a r d ( ) ; d e a l e r . s c o r e += d e a l e r . hand [ 0 ] . g e t B J P o i n t s ( ) ; i f ( d e a l e r . hand [ 1 ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { d e a l e r . high = true ; dealer . hscore = dealer . score + 10; } d e a l e r . l a s t a x n = B P l a y e r . DEAL ; i f ( d e a l e r . h i g h && d e a l e r . h s c o r e <= 2 1 ) { dealer . score = dealer . hscore ; } p l a y e r . hand [ 0 ] = p a c k . n e x t C a r d ( ) ; p l a y e r . s c o r e += p l a y e r . hand [ 0 ] . g e t B J P o i n t s ( ) ; i f ( p l a y e r . hand [ 0 ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { player . high = true ; player . hscore = player . score + 10; } p l a y e r . hand [ 1 ] = p a c k . n e x t C a r d ( ) ; p l a y e r . s c o r e += p l a y e r . hand [ 1 ] . g e t B J P o i n t s ( ) ; i f ( p l a y e r . hand [ 1 ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { player . high = true ; player . hscore = player . score + 10; } i f ( p l a y e r . h i g h && p l a y e r . h s c o r e <= 2 1 ) { player . score = player . hscore ; } d e a l s [ p l a y e r . s c o r e −1]++; p l a y e r . l a s t a x n = B P l a y e r . DEAL ; currentCard = 2; }

105

/∗ ∗ ∗ T h i s method a d d s a Card t o t h e a c t i v e B P l a y e r ’ s hand . The p o i n t e r i s moved t o ∗ n e x t Card . The p l a y e r i s f o r c e d t o s t a n d i f t h e y e x c e e d a s c o r e o f 21 o r h a v e ∗ cards . ∗ @see d e a l ∗ @see s t a n d ∗/ private void h i t ( ) { i f ( play ){ p l a y e r . hand [ c u r r e n t C a r d ] = p a c k . n e x t C a r d ( ) ; p l a y e r . s c o r e += p l a y e r . hand [ c u r r e n t C a r d ] . g e t B J P o i n t s ( ) ; i f ( p l a y e r . hand [ c u r r e n t C a r d ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { player . high = true ; p l a y e r . h s c o r e += p l a y e r . hand [ c u r r e n t C a r d ] . g e t B J P o i n t s ( ) ; } refresh (); c u r r e n t C a r d ++; i f ( p l a y e r . s c o r e > 21 | | c u r r e n t C a r d == 5 ) { stand ( ) ; }

106 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726

APPENDIX C. BLACKJACK AI } else{ d e a l e r . hand [ c u r r e n t C a r d ] = p a c k . n e x t C a r d ( ) ; d e a l e r . s c o r e += d e a l e r . hand [ c u r r e n t C a r d ] . g e t B J P o i n t s ( ) ; i f ( d e a l e r . hand [ c u r r e n t C a r d ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { d e a l e r . high = true ; d e a l e r . h s c o r e += d e a l e r . hand [ c u r r e n t C a r d ] . g e t B J P o i n t s ( ) ; } refresh (); c u r r e n t C a r d ++; i f ( d e a l e r . s c o r e > 21 | | c u r r e n t C a r d == 5 ) { stand ( ) ; } } }

/∗ ∗ ∗ S t a n d s f o r t h e a g e n t t h a t c a l l e d t h e method . I f i t i s t h e p l a y e r , t h e n t h e p l a ∗ i s p a s s e d on t o t h e d e a l e r . I f i t i s t h e d e a l e r , t h e n t h e r e s u l t i s c a l c u l a t e d ∗ and t h e hand e n d s ∗ @see d e a l ∗ @see h i t ∗/ private void s t a n d ( ) { i f ( play ){ play = f a l s e ; currentCard = 2; refresh (); } else{ i f ( p l a y e r . s c o r e > 21 && d e a l e r . s c o r e <= 2 1 ) { / / player bust p l a y e r . reward ( −20); r e s u l t [0]++; b u s t s [ p l a y e r . l a s t s c o r e −1]++; } e l s e i f ( p l a y e r . s c o r e <= 21 && d e a l e r . s c o r e > 2 1 ) { / / dealer bust p l a y e r . f u n d s += b e t ∗ 2 ; p l a y e r . reward ( 1 0 ) ; r e s u l t [1]++; } e l s e i f ( p l a y e r . s c o r e > 21 && d e a l e r . s c o r e > 2 1 ) { / / d e a l e r and p l a y e r b u s t p l a y e r . reward ( −10); r e s u l t [2]++; b u s t s [ p l a y e r . l a s t s c o r e −1]++; } e l s e i f ( player . score > d e a l e r . score ){ / / p l a y e r wins p l a y e r . f u n d s += b e t ∗ 2 ; ; p l a y e r . reward ( 1 0 ) ; r e s u l t [3]++;

C.1. SOURCE CODE 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 } e l s e i f ( player . score < d e a l e r . score ){ / / player loses p l a y e r . reward ( −10); r e s u l t [4]++; } else{ / / push p l a y e r . f u n d s += b e t ; p l a y e r . reward ( 5 ) ; r e s u l t [5]++; try { push . w r i t e ( I n t e g e r . t o S t r i n g ( p l a y e r . s c o r e ) ) ; push . w r i t e ( ” ” ) ; push . w r i t e ( I n t e g e r . t o S t r i n g ( d e a l e r . s c o r e ) ) ; p u s h . newLine ( ) ; } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } } bet = 0; clear (); play = true ; } }

107

/∗ ∗ ∗ C l e a r s a l l GUI e l e m e n t s and r e s e t s a l l v a l u e s t o t h i e r ∗ @see r e f r e s h ∗/ private void c l e a r ( ) { p l a y e r P a n e l . removeAll ( ) ; d e a l e r P a n e l . removeAll ( ) ; f o r ( i n t i = 0 ; i < 5 ; i ++){ p l a y e r . hand [ i ] = n u l l ; d e a l e r . hand [ i ] = n u l l ; p c a r d s [ i ] = new J L a b e l ( ) ; d c a r d s [ i ] = new J L a b e l ( ) ; p l a y e r P a n e l . add ( p c a r d s [ i ] ) ; d e a l e r P a n e l . add ( d c a r d s [ i ] ) ; }

i n t i t i a l values .

p l a y e r P a n e l . add ( f u n d s L a b e l ) ; bet = 0; currentCard = 0; player . score = 0; dealer . score = 0; d e a l e r . high = f a l s e ; player . high = f a l s e ; f u n d s L a b e l . s e t T e x t ( ( new I n t e g e r ( p l a y e r . f u n d s ) ) . t o S t r i n g ( ) ) ;

108 781 782 783 784 785 786 betButton . setEnabled ( true ) ; hit . setEnabled ( false ) ; stand . setEnabled ( false ) ; repaint (); } }

APPENDIX C. BLACKJACK AI

Appendix D

Final System
D.1
D.1.1

Source Code
Face

See Appendix C.1.1

D.1.2

Suit

See Appendix C.1.2

D.1.3

Card

See Appendix C.1.3

D.1.4

Pack

See Appendix C.1.2

D.1.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Shoe

/ / A d d i n g Shoe t o t h e SKCards p a c k a g e package SKCards ;

/∗ ∗ ∗ T h i s o b j e c t r e p r e s e n t s a D e a l e r ’ s s h o e . T h i s i s a number o f d e c k s ∗ o f c a r d s i n a c l o s e d c o n t a i n e r . T h e r e i s a s p e c a i a l c u t card , w h i c h ∗ i s i n s e r t e d r a n d o m l y i n t o t h e Shoe . When t h i s i s d e a l t , t h e s h o e i s ∗ d i s c a r d e d and a new one i s u s e d . S h o e s a r e u s e d t o c o u n t e r Card c o u n t i n g . ∗ ∗ @author S a q i b A K a k v i ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @see SKCards . Pack Pack ∗ @SK . bug none ∗/ p u b l i c c l a s s Shoe e x t e n d s Pack { / / the position of the c u t t o f f 109

110 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 int cutoff ; / / t h e s i z e o f t h e Shoe int size ;

APPENDIX D. FINAL SYSTEM

/∗ ∗ ∗ C o n s t r u c t o r . C r e a t e s a p a c k w i t h t h e s p e c i f i e d p a r a m e t e r s by c a l l i n g ∗ s u p e r ( d , j ) . The c u t o f f Card ’ s p o s i t i o n i s c a l c u l a t e d and i s s t o r e d . ∗ No e x t r a Card i s added t o t h e Pack , s o t h e s i z e r e m a i n s t h e same . ∗ ∗ @param d t h e number o f d e c k s ∗ @param j i f t h e r e a r e j o k e r s i n t h e d e c k ∗/ p u b l i c Shoe ( i n t d , b o o l e a n j ) { super ( d , j ) ; s i z e = super . s i z e ( ) ; calculateCutoff (); }

/∗ ∗ ∗ C a l c u l a t e s t h e c u t o f f Card p o s i t i o n . The Card w i l l be i n t h e r a n g e o f ∗ 0−85% o f t h e Shoe s i z e . Hence a maximum o f o f 85% o f t h e Pack i s ∗ played . ∗/ private void c a l c u l a t e C u t o f f ( ) { c u t o f f = new Double ( Math . random ( ) ∗ 100 ∗ 0 . 8 5 ) . i n t V a l u e ( ) ∗ s i z e ; }

/∗ ∗ ∗ C h e c k s i f t h e c u r r e n t Card i s t h e c u t o f f Card . I f i t i s t h e n a new Pack ∗ i s c r e a t e d and t h e c u t o f f Card ’ s p o s i t i o n i s r e c a l c u l a t e d . Then t h e ∗ method r e t u r n s t h e Card f r o m s u p e r . n e x t C a r d ( ) . ∗ ∗ @ o v e r r i d e SKCards . Pack n e x t C a r d ∗ @return t h e n e x t Card i n t h e Shoe ∗/ p u b l i c Card n e x t C a r d ( ) { i f ( c u r r e n t C a r d == c u t o f f ) { newPack ( ) ; calculateCutoff (); } return super . nextCard ( ) ; } }

D.1.6
1 2 3 4

BJPlayer

/ / A d d i n g P l a y e r t o SKCards package SKCards ;

D.1. SOURCE CODE 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 /∗ ∗ Importing the ArrayList class ∗ and t h e A r r a y s c l a s s ∗ and t h e s w i n g / awt c o m p o n e n t s ∗/ import j a v a . u t i l . A r r a y L i s t ; import j a v a . u t i l . A r r a y s ; import j a v a . awt . ∗ ; import j a v a x . s w i n g . ∗ ;

111

/∗ ∗ ∗ T h i s i s a p l a y e r i n t h e B l a c k J a c k game ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 1 . 0 C ∗/ public c l a s s BJPlayer extends Player { protected int score , hscore ; p r o t e c t e d Card [ ] hand ; protected BJRules r u l e s ; boolean high ; /∗ ∗ ∗ C r e a t e s a new p l a y e r o b j e c t w i t h a g i v e n d e t a i l s . ∗ ∗ @param i d t h e p l a y e r name ∗ @param c a s h t h e amount o f money t h e p l a y e r s t a r t s w i t h ∗ @param s e t P a c k t h e p a c k o f c a r d s t o be u s e d i n t h e game ∗ @param b c k t h e c a r d b a c k t h e p l a y e r u s e s ∗ @param c s t h e c a r d s e t u s e d by t h e p l a y e r ∗/ p u b l i c B J P l a y e r ( S t r i n g i d , i n t c a s h , S t r i n g bck , S t r i n g cs , BJRules r ){ rules = r ; funds = cash ; name = i d ; b a c k = bck ; cardSet = cs ; high = f a l s e ; score = 0; hscore = 10; hand = new Card [ 5 ] ; } /∗ ∗ ∗ B a s i c c o n s t r u c t o r . S i m p l y t a k e s i n t h e r u l e s . Used f o r

112 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 ∗ BJRLPlayer ∗ ∗ @param r t h e r u l e s ∗/ public BJPlayer ( BJRules r ){ rules = r ; }

APPENDIX D. FINAL SYSTEM

/∗ ∗ ∗ C r e a t e s a new hand o f c a r d s f o r t h e c u r r e n t p l a y e r and one f o r t h e c o m p t e r ∗ and t h e n d e a l s two c a r d s t o e a c h hand ∗/ p u b l i c v o i d newHand ( ) { hand = new Card [ 5 ] ; score = 0; } /∗ ∗ ∗ H i t s t h e p l a y e r ’ s hand ∗ ∗ @param c u r r e n t C a r d t h e c u r r e n t c a r d ∗ @param n e x t C a r d t h e c a r d t o be added ∗/ p u b l i c i n t h i t ( i n t c u r r e n t C a r d , Card n e x t C a r d ) { hand [ c u r r e n t C a r d ] = n e x t C a r d ; s c o r e += r u l e s . g e t B J P o i n t s ( hand [ c u r r e n t C a r d ] ) ; i f ( hand [ c u r r e n t C a r d ] . g e t V a l u e ( ) . e q u a l s ( ”A” ) ) { high = true ; h s c o r e += r u l e s . g e t B J P o i n t s ( hand [ c u r r e n t C a r d ] ) ; } i f ( h i g h && h s c o r e <= 2 1 ) { score = hscore ; } e l s e i f ( h i g h && h s c o r e > 2 1 ) { i f ( h s c o r e == s c o r e ) { s c o r e −= 1 0 ; } high = f a l s e ; } return score ; }

/∗ ∗ ∗ Returns the players current score ∗ ∗ @return s c o r e ∗/

D.1. SOURCE CODE 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 public int getScore (){ return score ; } /∗ ∗ ∗ Ends t h e c u r r e n t hand ∗/ p u b l i c v o i d endHand ( ) { i n t comp = 0 ; int player = 0;

113

System . o u t . p r i n t l n ( ” p l a y e r −−>” + p l a y e r ) ; System . o u t . p r i n t l n ( ” c o m p u t e r −−>” + comp ) ; i f ( comp > 0 | | p l a y e r > 0 ) { i f ( comp > p l a y e r ) { JOptionPane . showMessageDialog ( null , rules . display ( ) ; } e l s e i f ( p l a y e r > comp ) { JOptionPane . showMessageDialog ( null , f u n d s += b e t ; rules . display ( ) ; } e l s e i f ( comp == p l a y e r ) { JOptionPane . showMessageDialog ( null , f u n d s += b e t ; rules . display ( ) ; } else{ JOptionPane . showMessageDialog ( null , } } else{ J O p t i o n P a n e . s h o w M e s s a g e D i a l o g ( n u l l , ”NO rules . display ( ) ; } }

”HOUSE WINS ! ! ! ” ) ;

”PLAYER WINS ! ! ! ” ) ;

”PUSH ! ! ! ” ) ;

”NO RESULT FOUND” ) ;

RESULT FOUND” ) ;

/∗ ∗ ∗ C l e a r s a l l t h e h a n d s and c l e a r s t h e i n t e r f a c e ∗/ public void c l e a r ( ) { f o r ( i n t i = 0 ; i < hand . l e n g t h ; i ++){ hand [ i ] = n u l l ; } rules . clear (); }

114 167 168 169 170 171 172 173 174 175 176

APPENDIX D. FINAL SYSTEM

/∗ ∗ ∗ Gets t h e p l a y e r ’ s c u r r e n t hands ∗ @return t h e p l a y e r ’ s h a n d s ∗/ p u b l i c Card [ ] g e t H a n d ( ) { r e t u r n hand ; } }

D.1.7
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

BJUI

/ / a d d i n g BJUI t o SKCards package SKCards ; /∗ ∗ ∗/ import import import import

j a v a . awt . ∗ ; j a v a . awt . e v e n t . ∗ ; j a v a x . swing . ∗ ; j a v a x . swing . e v e n t . ∗ ;

/∗ ∗ ∗ T h i s i s t h e User I n t e r f a c e f o r t h e B l a c k j a c k Game . ∗ I t a l l o w s t h e human p l a y e r t o a c c e s s t h e f u n c t i o n s ∗ a n s shows t h e c u r r e n t s t a t e o f t h e game . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @SK . bug none ∗/ p u b l i c c l a s s BJUI e x t e n d s JFrame implements UI{ / / t h e p l a y e r s ’ hands private JLabel [ ] pcards , dcards ; / / display of the users funds private JLabel fundsLabel ; / / command b u t t o n s private JButton hit , stand , bet , c l e a r , e x i t ; / / panels used to d i s p l a y user components private JPanel buttonPanel , playerPanel , dealerPanel ; / / t h e c o n t e n t pane p r i v a t e C o n t a i n e r cp ; / / t e x t f i e l d f o r u s e r t o e n t e r amount t o b e t private JTextField betField ; / / t h e game r u l e s BJRules r u l e s ; /∗ ∗ ∗ C r e a t e s a new UI . T a k e s i n t h e r u l e s f o r t h e c u r r e n t game . ∗ T h i s a l l o w s t h e two o b j e c t s t o c o m m u n i c a t e .

D.1. SOURCE CODE 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 ∗ ∗ @param r t h e R u l e s f o r t h e game ∗/ p u b l i c BJUI ( B J R u l e s r ) { super ( ” BlackJack ” ) ; setSize (600 ,600); Dimension d = ( T o o l k i t . g e t D e f a u l t T o o l k i t ( ) ) . g e t S c r e e n S i z e ( ) ; int h = ( int )( d . getHeight ( ) ) ; i n t w = ( i n t ) ( d . getWidth ( ) ) ; i n t x = (w − 6 0 0 ) / 2 ; i n t y = ( h − 600) / 2; s e t D e f a u l t C l o s e O p e r a t i o n ( DO NOTHING ON CLOSE ) ; setLocation (x , y ); rules = r ; cp = g e t C o n t e n t P a n e ( ) ; cp . s e t L a y o u t ( new B o r d e r L a y o u t ( ) ) ; h i t = new J B u t t o n ( ” HIT ” ) ; h i t . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { rules . hit (); } }); hitEnabled ( false ); s t a n d = new J B u t t o n ( ”STAND” ) ; s t a n d . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { rules . stand ( ) ; } }); standEnabled ( false ) ; b e t F i e l d = new J T e x t F i e l d ( ” 10 ” ) ; b e t = new J B u t t o n ( ”BET” ) ; b e t . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { rules . bet ( Integer . parseInt ( betField . getText ( ) ) ) ; } }); betEnabled ( true ) ; c l e a r = new J B u t t o n ( ”CLEAR” ) ; c l e a r . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { clear (); }

115

116 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 });

APPENDIX D. FINAL SYSTEM

e x i t = new J B u t t o n ( ” EXIT ” ) ; e x i t . a d d M o u s e L i s t e n e r ( new MouseAdapter ( ) { p u b l i c v o i d m o u s e C l i c k e d ( MouseEvent e ) { BJUI . t h i s . d i s p o s e ( ) ; } }); b u t t o n P a n e l = new J P a n e l ( ) ; b u t t o n P a n e l . s e t L a y o u t ( new G r i d L a y o u t ( 6 , 1 ) ) ; buttonPanel buttonPanel buttonPanel buttonPanel buttonPanel buttonPanel . add ( h i t ) ; . add ( s t a n d ) ; . add ( b e t F i e l d ) ; . add ( b e t ) ; . add ( c l e a r ) ; . add ( e x i t ) ;

p c a r d s = new J L a b e l [ 5 ] ; d c a r d s = new J L a b e l [ 5 ] ; f u n d s L a b e l = new J L a b e l ( ) ; playerPanel playerPanel dealerPanel dealerPanel = new J P a n e l ( ) ; . s e t L a y o u t ( new G r i d L a y o u t ( 1 , 6 ) ) ; = new J P a n e l ( ) ; . s e t L a y o u t ( new G r i d L a y o u t ( 1 , 6 ) ) ;

f o r ( i n t i = 0 ; i < 5 ; i ++){ p c a r d s [ i ] = new J L a b e l ( ) ; d c a r d s [ i ] = new J L a b e l ( ) ; p l a y e r P a n e l . add ( p c a r d s [ i ] ) ; d e a l e r P a n e l . add ( d c a r d s [ i ] ) ; } p l a y e r P a n e l . add ( f u n d s L a b e l ) ; d e a l e r P a n e l . add ( new J L a b e l ( ) ) ; cp . add ( d e a l e r P a n e l , B o r d e r L a y o u t . NORTH ) ; cp . add ( b u t t o n P a n e l , B o r d e r L a y o u t . EAST ) ; cp . add ( p l a y e r P a n e l , B o r d e r L a y o u t . SOUTH ) ; s e t V i s i b l e ( true ) ; }

/∗ ∗ ∗ R e f r e s h e s t h e GUI t o show t h e c u r r e n t s i t u a t i o n . ∗ @override d i s p l a y ∗/ public void d i s p l a y ( ) { boolean play = r u l e s . g e t P l a y ( ) ;

D.1. SOURCE CODE 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 Card [ ] phand = r u l e s . g e t P l a y e r H a n d ( r u l e s . PLAYER ) ; Card [ ] c h a n d = r u l e s . g e t P l a y e r H a n d ( r u l e s . DEALER ) ; S t r i n g c s = r u l e s . g e t P l a y e r C a r d S e t ( r u l e s . PLAYER ) ; S t r i n g bk = r u l e s . g e t P l a y e r B a c k ( r u l e s . PLAYER ) ; f o r ( i n t i = 0 ; i < phand . l e n g t h ; i ++){ i f ( phand [ i ] ! = n u l l ) { p c a r d s [ i ] . s e t I c o n ( phand [ i ] . t o I m a g e ( c s ) ) ; } } d c a r d s [ 0 ] . s e t I c o n ( chand [ 0 ] . toImage ( cs ) ) ; i f ( play ){ d c a r d s [ 1 ] . s e t I c o n ( c h a n d [ 1 ] . t o B a c k I m a g e ( bk ) ) ; } else{ f o r ( i n t i = 1 ; i < c h a n d . l e n g t h ; i ++){ i f ( chand [ i ] != n u l l ){ d c a r d s [ i ] . s e t I c o n ( chand [ i ] . toImage ( cs ) ) ; } } }

117

f u n d s L a b e l . s e t T e x t ( r u l e s . g e t P l a y e r F u n d s ( r u l e s . PLAYER ) . t o S t r i n g ( ) ) ; repaint (); }

/∗ ∗ ∗ D i s p l a y s a new hand on t h e GUI . ∗ @override newDisplay ∗/ public void newDisplay ( ) { Card [ ] phand = r u l e s . g e t P l a y e r H a n d ( r u l e s . PLAYER ) ; Card [ ] c h a n d = r u l e s . g e t P l a y e r H a n d ( r u l e s . DEALER ) ; S t r i n g c s = r u l e s . g e t P l a y e r C a r d S e t ( r u l e s . PLAYER ) ; S t r i n g bk = r u l e s . g e t P l a y e r B a c k ( r u l e s . PLAYER ) ; f o r ( i n t i = 0 ; i < phand . l e n g t h ; i ++){ i f ( phand [ i ] ! = n u l l ) { p c a r d s [ i ] . s e t I c o n ( phand [ i ] . t o I m a g e ( c s ) ) ; } } d c a r d s [ 0 ] . s e t I c o n ( chand [ 0 ] . toImage ( cs ) ) ; d c a r d s [ 1 ] . s e t I c o n ( c h a n d [ 1 ] . t o B a c k I m a g e ( bk ) ) ; f u n d s L a b e l . s e t T e x t ( r u l e s . g e t P l a y e r F u n d s ( r u l e s . PLAYER ) . t o S t r i n g ( ) ) ; repaint ();

118 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 }

APPENDIX D. FINAL SYSTEM

/∗ ∗ ∗ C l e a r s t h e GUI ∗ @override c l e a r ∗/ public void c l e a r ( ) { System . o u t . p r i n t l n ( ) ; p l a y e r P a n e l . removeAll ( ) ; d e a l e r P a n e l . removeAll ( ) ; f o r ( i n t i = 0 ; i < 5 ; i ++){ p c a r d s [ i ] = new J L a b e l ( ) ; d c a r d s [ i ] = new J L a b e l ( ) ; p l a y e r P a n e l . add ( p c a r d s [ i ] ) ; d e a l e r P a n e l . add ( d c a r d s [ i ] ) ; } p l a y e r P a n e l . add ( f u n d s L a b e l ) ; betEnabled ( true ) ; hitEnabled ( false ); standEnabled ( false ) ; repaint (); } /∗ ∗ ∗ G e t s t h e amount b e t f r o m t h e b e t F i e l d . ∗ ∗ @return b e t ∗/ public int getBet (){ return ( I n t e g e r . p a r s e I n t ( b e t F i e l d . getText ( ) ) ) ; } /∗ ∗ ∗ Enables or d i s a b l e s t h e h i t b u t t o n ∗ ∗ @boolean h i t b u t t o n s t a t e ∗/ p u b l i c v o i d h i t E n a b l e d ( b o o l e a n en ) { h i t . s e t E n a b l e d ( en ) ; } /∗ ∗ ∗ Enables or d i s a b l e s t h e s t a n d b u t t o n ∗ ∗ @boolean s t a n d b u t t o n s t a t e ∗/ p u b l i c v o i d s t a n d E n a b l e d ( b o o l e a n en ) { s t a n d . s e t E n a b l e d ( en ) ; }

D.1. SOURCE CODE 258 259 260 261 262 263 264 265 266 267

119

/∗ ∗ ∗ Enables or d i s a b l e s t h e b e t b u t t o n ∗ ∗ @boolean b e t b u t t o n s t a t e ∗/ p u b l i c v o i d b e t E n a b l e d ( b o o l e a n en ) { b e t . s e t E n a b l e d ( en ) ; } }

D.1.8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

BJRules

/ / a d d i n g B J R u l e s t o SKCards package SKCards ; /∗ ∗ i m p o r t i n g t he swing c l a s s e s . ∗/ import j a v a x . s w i n g . ∗ ; /∗ ∗ ∗ The r u l e s o f t h e B l a c k j a c k game . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @SK . bug none ∗/ public c l a s s BJRules extends Rules { p u b l i c s t a t i c f i n a l i n t PLAYER = 0 ; p u b l i c s t a t i c f i n a l i n t DEALER = 1 ; p r i v a t e boolean play ; int currentCard , pscore , dscore , bet ; p u b l i c B J R u l e s ( S t r i n g i d , i n t c a s h , Pack s e t P a c k , S t r i n g bck , S t r i n g cs , SKMain m) { main = m; u i = new BJUI ( t h i s ) ; pack = s e t P a c k ; p l a y e r s = new P l a y e r [ 2 ] ; p l a y e r s [PLAYER] = new B J P l a y e r ( i d , c a s h , bck , cs , t h i s ) ; p l a y e r s [DEALER] = new B J R L P l a y e r ( t h i s ) ; play = true ; }

120 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 p u b l i c v o i d newGame ( ) {

APPENDIX D. FINAL SYSTEM

clear (); ( ( BJUI ) u i ) . h i t E n a b l e d ( t r u e ) ; ( ( BJUI ) u i ) . s t a n d E n a b l e d ( t r u e ) ; ( ( BJUI ) u i ) . b e t E n a b l e d ( f a l s e ) ; p l a y e r s [PLAYER ] . newHand ( ) ; p l a y e r s [DEALER ] . newHand ( ) ; ( ( B J P l a y e r ) ( p l a y e r s [PLAYER ] ) ) . h i t ( 0 , p a c k . p s c o r e = ( ( B J P l a y e r ) ( p l a y e r s [PLAYER ] ) ) . h i t ( ( B J P l a y e r ) ( p l a y e r s [DEALER ] ) ) . h i t ( 0 , p a c k . d s c o r e = ( ( B J P l a y e r ) ( p l a y e r s [DEALER ] ) ) . h i t currentCard = 2; display ( ) ; } p u b l i c v o i d endGame ( ) {

nextCard ( ) ) ; ( 1 , pack . nextCard ( ) ) ; nextCard ( ) ) ; ( 1 , pack . nextCard ( ) ) ;

Card [ ] phand = ( ( B J P l a y e r ) p l a y e r s [PLAYER ] ) . g e t H a n d ( ) ; Card [ ] c h a n d = ( ( B J P l a y e r ) p l a y e r s [DEALER ] ) . g e t H a n d ( ) ; f o r ( i n t i = 0 ; i < phand . l e n g t h ; i ++){ i f ( phand [ i ] ! = n u l l ) { System . o u t . p r i n t ( phand [ i ] . t o S t r i n g ( ) ) ; } } System . o u t . p r i n t l n ( ” PLAYER ” + p s c o r e ) ; f o r ( i n t i = 0 ; i < c h a n d . l e n g t h ; i ++){ i f ( chand [ i ] != n u l l ){ System . o u t . p r i n t ( c h a n d [ i ] . t o S t r i n g ( ) ) ; } } System . o u t . p r i n t l n ( ” DEALER ” + d s c o r e ) ; i f ( p s c o r e > 21 && d s c o r e <= 2 1 ) { JOptionPane . showMessageDialog ( null , ” P l a y e r i s b u s t ! ” ) ; ( ( B J R L P l a y e r ) ( p l a y e r s [DEALER ] ) ) . r e w a r d ( 1 0 ) ; } e l s e i f ( d s c o r e > 21 && p s c o r e <= 2 1 ) { JOptionPane . showMessageDialog ( null , ” De al e r i s b u s t ! ” ) ; p l a y e r s [PLAYER ] . s e t F u n d s ( b e t ∗ 2 ) ; ( ( B J R L P l a y e r ) ( p l a y e r s [DEALER ] ) ) . r e w a r d ( − 1 0 ) ; } e l s e i f ( p s c o r e > 21 && d s c o r e > 2 1 ) { J O p t i o n P a n e . s h o w M e s s a g e D i a l o g ( n u l l , ” Both b u s t ! ” ) ; p l a y e r s [PLAYER ] . s e t F u n d s ( b e t ) ; ( ( B J R L P l a y e r ) ( p l a y e r s [DEALER ] ) ) . r e w a r d ( − 1 0 ) ; } e l s e i f ( pscore > dscore ){ JOptionPane . showMessageDialog ( null , ” P l a y e r wins ! ” ) ; p l a y e r s [PLAYER ] . s e t F u n d s ( b e t ∗ 2 ) ; ( ( B J R L P l a y e r ) ( p l a y e r s [DEALER ] ) ) . r e w a r d ( − 1 0 ) ; }

D.1. SOURCE CODE 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 e l s e i f ( p s c o r e == d s c o r e ) { J O p t i o n P a n e . s h o w M e s s a g e D i a l o g ( n u l l , ” Push ! ” ) ; p l a y e r s [PLAYER ] . s e t F u n d s ( b e t ) ; ( ( B J R L P l a y e r ) ( p l a y e r s [DEALER ] ) ) . r e w a r d ( 5 ) ; } else{ JOptionPane . showMessageDialog ( null , ” P l a y e r l o s e s ! ” ) ; } clear (); play = true ; }

121

public void h i t ( ) { i f ( play ){ p s c o r e = ( ( B J P l a y e r ) ( p l a y e r s [PLAYER ] ) ) . h i t ( c u r r e n t C a r d , p a c k . n e x t C a r d ( ) ) ; c u r r e n t C a r d ++; i f ( p s c o r e > 21 | | c u r r e n t C a r d == 5 ) { display ( ) ; stand ( ) ; } } else{ d s c o r e = ( ( B J P l a y e r ) ( p l a y e r s [DEALER ] ) ) . h i t ( c u r r e n t C a r d , p a c k . n e x t C a r d ( ) ) ; c u r r e n t C a r d ++; i f ( p s c o r e > 21 | | c u r r e n t C a r d == 5 ) { display ( ) ; stand ( ) ; } } display ( ) ; } public void s t a n d ( ) { System . o u t . p r i n t l n ( p l a y ) ; i f ( play ){ play = f a l s e ; display ( ) ; currentCard = 2; while ( ! play ){ ( ( B J R L P l a y e r ) p l a y e r s [DEALER ] ) . l e a r n ( ) ; } } else{ play = true ; endGame ( ) ; } display ( ) ; } p u b l i c v o i d b e t ( i n t amt ) { i f ( amt <= p l a y e r s [PLAYER ] . g e t F u n d s ( ) ) { b e t = amt ; p l a y e r s [PLAYER ] . s e t F u n d s (−1 ∗ b e t ) ;

122 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178

APPENDIX D. FINAL SYSTEM } else{ J O p t i o n P a n e . s h o w M e s s a g e D i a l o g ( n u l l , ”INSUFFICENT FUNDS ! ! ! ” ) ; } newGame ( ) ; display ( ) ; } p u b l i c i n t g e t B J P o i n t s ( Card c ) { int points = c . getPoints ( ) ; i f ( p o i n t s == 10 | | p o i n t s == 11 | | p o i n t s == 12 | | p o i n t s == 1 3 ) { return 10; } else{ return p o i n t s ; } } public boolean g e t P l a y ( ) { return play ; } p u b l i c Card [ ] g e t P l a y e r H a n d ( i n t p ) { return ( ( B J P l a y e r ) p l a y e r s [ p ] ) . getHand ( ) ; }

}

D.1.9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

RLPlayer

/ / a d d i n g B J R L P l a y e r t o SKCards package SKCards ; /∗ ∗ i m p o r t i n g t h e IO c l a s s e s ∗/ import j a v a . i o . ∗ ;

/∗ ∗ ∗ The RL P l a y e r f o r t h e B l a c k j a c k game . ∗ ∗ @author SK ∗ @version 2 . 0A ∗ @since 2 . 0 A ∗ @SK . bug none ∗/ p u b l i c c l a s s BJRLPlayer extends B J P l a y e r { p r i v a t e f i n a l d o u b l e ALPHA = 0 . 6 ; p r i v a t e f i n a l d o u b l e GAMMA = 0 . 7 5 ; p r i v a t e f i n a l d o u b l e TAU = 0 . 5 ;

D.1. SOURCE CODE 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

123

p r i v a t e s t a t i c f i n a l i n t HIT = 1 ; p r i v a t e s t a t i c f i n a l i n t STAND = 0 ; p r i v a t e s t a t i c f i n a l i n t DEAL = −1; p r i v a t e double [ ] [ ] v a l ; private int lastaxn , l a s t s c o r e ; /∗ ∗ ∗ S o l e c o n s t r u c t o r . T a k e s i n t h e r u l e s and p a s s e s i t t o ∗ super ( r ) . ∗ ∗ @param r t h e r u l e s ∗/ p u b l i c BJRLPlayer ( BJRules r ){ super ( r ) ; v a l = new d o u b l e [ 2 ] [ 2 1 ] ; try { B u f f e r e d R e a d e r b r = new B u f f e r e d R e a d e r ( new F i l e R e a d e r ( ” r l v a l s . t x t ” ) ) ; S t r i n g data = br . readLine ( ) ; int i = 0; w h i l e ( d a t a ! = n u l l && i < v a l [ 0 ] . l e n g t h ) { System . o u t . p r i n t l n ( d a t a ) ; S t r i n g [ ] ds = d a t a . s p l i t ( ” , ” ) ; v a l [STAND ] [ i ] = Double . p a r s e D o u b l e ( d s [ 0 ] ) ; v a l [ HIT ] [ i ] = Double . p a r s e D o u b l e ( d s [ 1 ] ) ; data = br . readLine ( ) ; i ++; } } c a t c h ( E x c e p t i o n ex ) { ex . p r i n t S t a c k T r a c e ( ) ; } } /∗ ∗ ∗ S t a r t s a new hand . ∗/ p u b l i c v o i d newHand ( ) { l a s t a x n = DEAL ; s u p e r . newHand ( ) ; } /∗ ∗

124 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124

APPENDIX D. FINAL SYSTEM ∗ Plays a greedy p o l i c y , but u p d a t e s t h e v a l u e s as i t p l a y s ∗/ public void l e a r n ( ) { i f ( s c o r e > 21){ return ; } System . o u t . p r i n t l n ( ” HIT = ” + v a l [ HIT ] [ s c o r e − 1 ] ) ; System . o u t . p r i n t l n ( ”STAND = ” + v a l [STAND ] [ s c o r e − 1 ] ) ; d o u b l e maxval = Math . max ( v a l [STAND ] [ s c o r e −1] , v a l [ HIT ] [ s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = ALPHA ∗ ( (GAMMA ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } lastscore = score ; i f ( v a l [ HIT ] [ s c o r e −1] > v a l [STAND ] [ s c o r e −1]){ rules . hit (); l a s t a x n = HIT ; System . o u t . p r i n t l n ( ” HIT ” ) ; } else{ rules . stand ( ) ; l a s t a x n = STAND ; System . o u t . p r i n t l n ( ”STAND” ) ; } }

/∗ ∗ ∗ A l l o c a t e s t h e f i n a l reward t o t h e agent . ∗ ∗ @param r t h e r e w a r d ∗/ p u b l i c void reward ( double r ){ d o u b l e maxval = Math . max ( v a l [STAND ] [ l a s t s c o r e −1] , v a l [ HIT ] [ l a s t s c o r e − 1 ] ) ; i f ( l a s t a x n ! = DEAL) { d o u b l e r d = ALPHA ∗ ( r + (GAMMA ∗ maxval ) − v a l [ l a s t a x n ] [ l a s t s c o r e − 1 ] ) ; v a l [ l a s t a x n ] [ l a s t s c o r e −1] += r d ; } } }

Appendix E

Sample Blackjack Game
The following notation is used in the sample game: [] - a face down card — - Separation of two hands Cards - Face values - 2 -9 ⇒ 2 - 9, T ⇒ 10, J ⇒ Jack, Q ⇒ Queen, K ⇒ King Suits - D - Diamonds, C - Clubs, H - Hearts, S - Spades e.g. KS ⇒ King of Spades, 2D ⇒ 2 of Diamonds, TH ⇒ 10 of Hearts

Dealer AC [] Bet Player [] [] Player bets $10, and takes an insurance Dealer AC [] Bet $10 $10 Player AD AH Player splits Dealer AC [] Bet $10 $10 $10 Player AD QS | AH AS (BLACKJACK) Player stands on hand 1, splits hand 2 Dealer AC [] Bet $10 $10 $10 $10 Player AD QS | AH 2C | AS 4S Player hits hand 2 Dealer AC [] Bet $10 $10 $10 $10 Player AD QS | AH 2C 4D | AS 4S Player hits hand 2 Dealer AC [] Bet $10 $10 $10 $10 Player AD QS | AH 2C 4D KD| AS 4S (Ace now counts ‘‘low" (1)) Player hits hand 2 Dealer AC [] Bet $10 $10 $10 $10 Player AD QS | AH 2C 4D KD 5S | AS 4S (BUST!) Player doubles down on hand 3 (counting ace ‘‘high") 125

126

APPENDIX E. SAMPLE BLACKJACK GAME

Dealer AC [] Bet $10 $10 $10 $20 Player AD QS | AH 2C 4D KD 5S | AS 4S 6C Player stands on hand 3, dealer plays by turning over his second card Dealer AC JC (BLACKJACK!) Bet $10 $10 $10 $20 Player AD QS | AH 2C 4D KD 5S | AS 4S 6C The dealer has hit a Blackjack, so the player receives their $10. The player loses the $10 from hand 2 because they went bust. As for hands 1 and 3, the dealer and player have tied. However some casinos enforce the rule the “House wins a push”. Under this rule, the player would also lose the $30. If the house treats pushes normally, then the player will get his $30 back. Although there are many different variations on the rules, known as “house rule”, they are all well known and accepted by the players.

Bibliography
Parlett, D. (1994). Teach Yourself Card Games. Hodder Headline Plc, Euston Road, London, NW1 3BH. Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA.

127

Sign up to vote on this title
UsefulNot useful