Quality of State Improvisation Through Evaluation
Function Optimization in Genetic Application
Learning
Dharm Singh
College of Technology and Engineering
Maharana Pratap University of Agriculture and Technology
Udaipur ( India )
dharm@mpuat.ac.in
Abstract— Artificial intelligence algorithms have been applied on
computer board games since 1950s. Board Games provide
competitive, cognitive learning and dynamic environments that
make them ideal area of computational intelligence theories,
architectures, and algorithms. Natural Evolution can also be
considered to be a game play in which the rewards for a test
organism that plays a good game of life are the propagation of its
genetic material to its successors and its continued survival. In
natural evolution, the fitness of an individual is defined with
respect to its competitors and collaborators, as well as to the
environment. Evolutionary algorithms follow the same path to
evolve game playing programs. Among all computer board
games, because of its low branching property, Reversi (Game of
Othello) playing program can easily defeat humans by designing
with strategies based game moves. Here on, the goal of computer
Reversi game is no longer to challenge or defeat human players
but to compete and evolve against other computer programs.
This paper mainly highlights optimization of Reversi program
fitness values by applying genetic operators through linear
evaluation function.
Keywords- Game Playing, Board Games, Genetic algorithm,
Reversi (Game of Othello)
I. INTRODUCTION
Games have long dominated popular area of artificial
intelligence (AI) research, and for all good reasons they are
very challenging on one part and yet easy to formalize and
develop new AI solution exploration methods. Through games
efficiency of AI working can be measured in terms of
capability to acquire intelligence without putting human lives
or property at risk. Most research based work so far has
focused on games that can be described and solved-won in a
compact form using symbolic representations, such as board
and card games. The so-called “good old-fashioned artificial
intelligence” techniques work well with them, and to a large
extent, such techniques were developed, tested and improvised
for such games [1][2].
Since the 1990s, and early dawn of 21st century the field of
programming has radically changed. Availability of
inexpensive yet powerful computer hardware has made it
possible to simulate complex physical learning environments,
resulting in an exploration of artificially improved soft
978-1-4577-0240-2/11/$26.00 ©2011 IEEE
Chirag S Thaker, Sanjay M Shah
Department of Computer Science & Engineering
Sursesh Gyan Vihar University
Jaipur (India)
chiragthaker@yahoo.com, sanjay_shah_r@yahoo.com
cognitive moves by computer programs in all sorts of board
games. Game playing programs have become a facet of many
people’s day-to-day lives. [3]
Almost all traditional board games involves great amount
of AI research as they all has very high space complexity to
traverse before making good moves. These games poses main
challenges in form of guiding the evolution using human
knowledge and in achieving game playing behaviour that is not
only successful, but visibly intelligent to a human observer.
[4][5].
II. HISTORY OF REVERSI (GAME OF OTHELLO)
The exact origins of Othello are not known, although it is
now believed that it has originated from the Chinese game ‘Fan
Mian’, which translates to ‘reverse’ (hence the alternative name
Reversi has come up). Until recently the game has been
primarily played in Japan, where it is the second most popular
game next to in 1890’s a game called Game of Go.
Reversi was being marketed in England by Jacques and
Sons of London. This game was very similar to the modern
game except for two differences:
• Each player was given a set number of discs at the start of
the game. If a player ran out of discs, his opponent
completed the remainder of the game.
• The game began with an empty board.
In 1975 Goro Hasegawa introduced the modern version of
Othello which has been adopted across the world. In the
modern version, players take from a central pool of discs and
the initial layout of four discs is provided. Othello is fast
growing in popularity with major international tournaments
held every year. It has also been a popular test-bed for AI
research.
Formally speaking, Othello is a two-person zero-sum
deterministic finite board game with perfect information. Two-
person zero-sum games are characterized by the fact that either
one player wins and the other loses, or neither player wins
resulting in a draw or tie. It is not possible for both players to
win.
93
 Deterministic games are games where there are no random
events based on event of chance or luck such as the roll of a
dice. In addition to that Othello is a finite game, because there
are a finite number of moves: at most 60. Othello has perfect
information, because all the game information such as piece
position is available to all players. [6]
III. RULES OF THE GAME
There is a saying about the game that Othello is one of
those games that take minutes to learn, but years to master.
Great expertise is required to play well, since board positions
can change drastically in just a few moves, especially towards
the end of the game. The game is played on a chess-like 8 x 8
board with a set of dual-coloured black and white discs. The
initial setup of the board contains four discs placed in the
centre; 2 white and 2 black. The game begins by black making
the first move. A valid move involves placing a disc into an
empty square on the board. Legal moves are those moves that
result in capture of opponent’s pieces.
Figure 1. Othello Board – Initial Configuration
Opponent pieces become captured if they are surrounded,
in a straight line, between the disc being played and another
disc belonging to the current player. Once a disc is captured it
is flipped over to the opponent’s colour. If a player cannot
make a legal move then he has to pass and the move is
transferred to his opponent. The game finishes when neither
player has a legal move, usually when all 64 discs have been
placed. At the end of the game all discs are counted and the
player with the most discs is declared to be the winner; unless
both players have the same number of discs in which case it is
a tie.
A. Reversi Playing Strategies
Like any other brain involving games Othello players use a
number of strategies to maximize their chances of winning.
The formation of this game playing programs is based on
below mentioned strategies to maximize the power of
evaluation function which is the product of square weights and
holder of the square.
94
B. Disc Difference (Greedy)
The most basic strategy is to maximize the number of one’s
pieces at every move. This strategy is often used by novice
Othello players. Although this strategy is very important in the
later stages of the game, it can become disastrous when used
earlier in the game. As the opponent can maximize on his
positions by making a couple of good moves in middle or end
part of the game which may dramatically cost the game.
C. Positional Strategy
A player using the positional strategy recognizes that some
locations on the Othello board are more valuable than others.
For example, the corners are the most valuable, because a disc
placed in a corner cannot be captured. Such discs are called
stable.
Stable discs are important because they are guaranteed to
remain to the end of the game and hence add to the total score
of their owner. Furthermore, they serve as a means for
capturing opponent’s non-stable discs. Thus the stability of the
player’s discs refers to the number of stable discs that the
player possesses and their locations. It is often the case that
good stability leads to a winning situation; however this does
not necessarily imply that controlling all four corners leads to a
win. In Reversi there exists a hierarchy of stable discs. Placing
such discs help the player to maximize his winning chances.
[7].
D. Mobility Strategy
Mobility or move-ability is one of the best strategies in
Othello, but is also one of the hardest to master. The mobility
of a player refers to the number of legal moves available to that
player. Lack of mobility can lead to severe difficulties. A
player with poor mobility may be forced to choose ‘bad’ moves
that lead to a detrimental situation.
On the other hand, a player with good mobility has a large
choice of moves, allowing him to direct the game towards an
advantageous situation. Mobility is important during the
middle part of the game where both players are fighting for a
good position, which will provide easy capture of corners. It is
believed that the mobility strategy was discovered in Japan in
the mid 1970s. The strategy slowly spread to America and
Europe through personal contact between American and
Japanese players. It has been shown that games played with
mobility strategy were more difficult to recall, suggesting that
the mobility strategy is much harder to learn.
Othello was chosen for the following reasons:
• It is a well-studied game, researched by many authors
using a variety of techniques.
• The rules of the game are simple and easy to implement.
• It is easy to implement basic AI opponents, such as a
random player and a greedy player.
• Othello has a reasonable complexity. It is not as simple as
Checkers and not as complex as Chess or Go.
• Since every game is guaranteed to finish in 60 moves or
less, Othello is perfect game for optimization algorithms
like Genetic Algorithms
E. Game Complexity
State-space complexity of any board game represents the
number of possible board states in the game. For example, in
Othello there are 64 board locations where each location can
take one of three values, giving approximately 364 § 1028 total
states. Game tree complexity represents the total number of
nodes in a fully-expanded game tree. Othello has a game tree
complexity of 1058. Based on that figure, the average branching
factor is about 9.26. To check this, we made two random
players play against each other, recording the number of
available moves for each stage of the game. The branching
factor rises slowly from 4 and peaks at 12 on the 30th move.
[8].
IV. GENETIC ALGORITHMS
Genetic algorithm being an effective and efficient subset of
evolutionary algorithms is a natural contestant for learning in
games. A genetic algorithm provides an algorithmic and logical
framework for exploiting all possible board fame scenarios
through natural-evolution processes like selection, cross over
and mutation. It evolves intermediate candidate solutions to
problem domains that have large solution search spaces and are
not open to exhaustive search or conventional optimization
techniques. Since ages Genetic algorithms have been applied to
a broad range of learning and optimization problems [9][10].
When genetic evolution process is run for many numbers of
generations through a series of experiments, the program
acquires a novel set of evaluation function parameters are
gathered for subsequent population.
Normally, a genetic evolution process starts with a random
set of population candidate solutions, called chromosomes.
Through a cross over process and mutation operators, it
evolves the population towards an optimal solution. GAs does
not guarantee an optimal solution hence the challenge is to
design a “genetic” process that maximizes the likelihood of
generating such an optimized solution [11].
The first step is typically to evaluate the fitness of each
candidate solution in the current population based on the fitness
values and to select the fittest candidate solutions to act as
parents of the next generation of candidate solutions. After
being selected for reproduction, parents are recombined
through a crossover operator and then mutated using a
mutation operator to generate offspring. The fittest parents and
their new offspring form a new population, from which the
process is repeated to create new populations.
The operations of evaluation, selection, recombination and
mutation are usually performed many times in a genetic
algorithm. Selection, recombination, and mutation are generic
operations in any genetic algorithm. Evaluation is problem
specific and relates directly to the structure of the solutions So
in a genetic algorithm, a major issue is the choice of the
structure of solutions and of the method of evaluation (fitness
function).Other important parameters include the size of the
population, the portion of the population taking part in
recombination, and the mutation rate. The mutation rate defines
the probability with which a bit is flipped in a chromosome that
is produced by a crossover [12].
95
A. Fitness Function
An important parameter of a genetic algorithm is the fitness
function that defines the fitness of each chromosome where the
values of genetic parameters are adapted as the genetic
evolution progresses. At every generation the best and worst
chromosome fitness are kept track of. In some cases if both
fitness values become equal, the mutation rate is increased, in
order to help the genetic evolution get out of issues like local
maxima. Once there is an improvement in the overall fitness,
the original mutation rate is restored to continue evolution as
normal. Here if evolution stabilizes by the fitness does not
seem to be improving for several generations and the search
does not find any error, the genetic algorithm with the initial
default parameter values and a new randomly generated seed to
generate a new random initial population. [13]
This paper uses genetic algorithms in game-playing
learning programs by constructing a static evaluation function
based on the features and strategies of an individual game
(Game of Reversi). To attain better results we modify genetic
algorithm as per the complexity of evaluation function
parameters. Constructing an evaluation function and updating
its associated weight ages is always a challenging task.
V. REVERSI-GENETIC COALESCE
The program was implemented using classical Genetic
algorithm in which first population is randomly generated, and
a random Evaluation Function serves as a "fitness finder" for
the entire population. And the populations are non-overlapping
populations in nature. When a simple genetic algorithm is
created, it specifies either an individual or a population of
individuals. The new genetic algorithm performs cloning for
the individual(s) that are specify to make its own population.
The simple genetic algorithm creates an initial population
by cloning the individual or population you pass when you
create it. In each generation, the algorithm creates an entirely
new population of individuals by selecting from the previous
population then mating to produce the new offspring for the
new population. This process continues until the stopping
criteria are met (determined by the terminator). [14]
This genetic string will play each and every string from the
population, and the system will assign a fitness function to this
last string with respect to his performance in this match. Then
the next generation is built according to this fitness, with a
certain crossover probability and some mutation probability,
the previously random string (the "fitness function") is now one
of the population’s strings, this can be seen as a cooperation
scheme, having a population of 1 on one side, and another
population on another side. The "fitness giver" can also be a
random string for each generation.
So the most important question here is the "weight" of
evaluation function. If the weights are set correctly, our
computer program will play well. If they're not set correctly,
the program may play very badly. Genetic algorithms learn
good weights. The way to do this is to start with a random set
of weights in the program, and use them to test the program. If
the program does well, we keep the weights, and use those
(making small changes) in the next version of the program. If
they do badly, we change them a lot or throw them out and
start again.
But, just like with animals, evolution happens slowly over a
large population of individuals. Likewise with different
weights of board squares the evolution take place with every
successive generations. Instead of having just one weight age
value of the generation at a time, many values of generations
existed. All of them are to be tested and then keep a few of the
best ones. In this way we hope to get better and better
individual weight sets each time [15].
In this process, different versions of the Evaluation
Function play against each other. The weight set which gives
best results is declared as a winner.
During the training process, hopefully, the genetic
algorithm will find the best suitable weights for each inner
node. That means, the program will know how much influence
a node's value should have on the whole final position decision
to place a new move on board.
This evaluation usually works just by calculating simple
numerical features of the game position (for example in
Othello, whether one player has more pieces or is controlling
the corners of the board). The final evaluation of the position
is a number; you get it by computing a linear function of the
position features. It shows how the evaluation function uses 6
features of the position, it calculates a number for each one,
and then multiplies each numbers by its own "weight" value.
This is how it is been implemented.
VI. GAME PROGRAM IMPLEMENTATION
The board is represented as a vector of length 64. Black
disc is represented as 1 and white disc as -1. Empty space is 0.
The relevance of the board is calculated using weighted piece
counter. The weighted piece counter is a vector of length 64.
Each element of the vector is corresponding to one square in
the Othello board, which means the weights of the board
position. The cumulative fitness value of the board squares is
calculated using dot product of two vectors (weights and
functional features) as follows:
Fitness Weight Value f = (W1 x F1) + (W2 x F2) + . . . +
(Wn x Fn)
Where the Fitness Weight Value f is called the static
evaluation of a game board configuration
• The Wi's are weights that indicate the relative importance
of the features.
• The Fi's are functional features that play important roles
in game-playing strategies
The weights of each position are initialized as a value
between -1 to 1. Until out-of-opening, the weightings in the
population use opening knowledge. After out-of-opening, the
weightings are evaluated in the relevance of board and the next
move is decided. Among possible moves, the one with the
highest fitness weight value (f) is selected.
After doing the optimization for each parameter, the weight
values are set for each parameter manually. Then give each
parameter a weighted coefficient and optimize these
96
coefficients by genetic algorithm again before manually reset
the weight for each of them. Selection from one generation to
next generation is chosen using Roulette-wheel selection with
probability of 20% and 1-point crossover with probability of
80% is applied to the converted 1-dimensonal array of the 8x8
board. Mutation operator with the value of 0.01 changes an
element of the vector as a new value ranging from -1 to 1.
The Othello program provides three sets of initial value (1,
-1, and 0) assign to each parameter. When the Fitness
weight value (f) reaches the best value in the generation, the
best weights are added to the adapted function to make a new
adapted function. After many times of evolution, it has a big
chance that the parameter assignment in the system can come
up with excellent weight assignment ability. The advantage of
this strategy is that it can precede the evolution without any
standard information and the parameters got by evolution have
a good performing ability. The time needed to evolve game
playing genetic parameters for evolution is the disadvantage of
this strategy.
VII. RESULT
The program is not looking for a global optimum but to
achieve genetic evolution of an evaluation function is to adjust
its genetic parameters so that the overall performance of the
program is enhanced.
Figure 2. Othello Program – Fitness Value Comparison (Generation 1 to 97)
At first quick look, automatic tuning of the evaluation
function appeared like an optimization task, well suited for
GA. The many parameters and sub parameters of the
evaluation function can be encoded as a bit-string. They were
randomly initialized as board square “chromosomes”, each
representing one weight age value for evaluation function.
Then the population was evolved until highly tuned “fit”
evaluation functions emerged. During the experiments, one
major challenge was faced in the form of fitness function for
the application of GA. For every set of input parameters, a set
of evaluation parameters as encoded as a chromosome to
calculate its fitness value. Here the solution was derived by
individuals’ fitness value in each generation. The main
drawback of genetic evolutionary approach is the unacceptably
large amount of time needed to evolve each generation. As a
result, severe limitations are imposed on the length of the
games played after each generation, and also on the size of the
population involved.
The Othello program is performed on a Pentium IV
machine with the RAM size of 512 MB. The results are
collected for two sets of Generations. In the first set which has
97 generations the population size was 100 and in the second
set has 100 generations with 200 members per population. For
both the sets the minimum fitness weight value (f) and average
fitness value for each generation is collected and graphs are
plotted accordingly as shown in figures below mentioned. As
per fig.2 the Minimum Fitness values are increasing as the
program proceeds from Gen 00 to Gen 97. But the average
fitness value tends to remain steady the program evolves with
major low values in Generation No. 11,48,61,86 and 96.
Fig. 3 is clearly indicative of the positive evolution of
Othello program as the Function Fitness values increases from
generation 01 to generation 100. In a nutshell both the graphs
serve the purpose of genetic optimization in evolutionary
Reversi game learning.
Figure 3. Othello Program – Fitness Analysis (Generation 1 to 100)
VIII. CONCLUSION
Machine learning has been applied in many computer board
games; the application of genetic operators is fit to enhance an
Othello game playing program. This paper used the genetic
parameter to speed up the efficiency of machine learning using
self-learning genetic features. Othello program was used to
enhance the game playing strategy using fitness function
evaluation using Othello game playing strategies. The purpose
of making simple evaluation function is served as the program
evolves.
After practical test, it has proved that the way of self-
learning proposed in this paper is possible to enhance the
efficiency of learning greatly. The optimization of the genetic
algorithm can improvise fitness functions in order to evaluate
the board state precisely and improve the computer Othello.
97
REFERENCES
[1] Hong, J.-H. and Cho, S.-B. (2004). Evolution of emergent behaviors for
shooting game characters in robocode. In Evolutionary Computation,
2004. CEC2004. Congress on Evolutionary Computation, volume 1,
pages 634–638, Piscataway, NJ. IEEE.
[2] J. Clune. Heuristic evaluation functions for general game playing. In
Proc. of AAAI, 1134–1139, 2007.
[3] J¨org Denzinger, Kevin Loose, Darryl Gates, and John Buchanan.
Dealing with parameterized actions in behavior testing of commercial
computer games. In Proceedings of the IEEE 2005 Symposium on
Computational Intelligence and Games (CIG), pages 37–43, 2005.
[4] Matt Gilgenbach. Fun game AI design for beginners. In Steve Rabin,
editor, AI Game Programming Wisdom 3, 2006.
[5] S. Schiffel and M. Thielscher. A multiagent semantics for the game
description language. In Proc. of the Int.’l Conf. on Agents and Artificial
Intelligence, Porto 2009. Springer LNCS.
[6] T. Srinivasan, P.J.S. Srikanth, K. Praveen and L. Harish Subramaniam,
“AI Game Playing Approach for Fast Processor Allocation in Hypercube
Systems using Veitch diagram (AIPA)”, IADIS International
Conference on Applied Computing 2005, vol. 1, Feb. 2005, pp. 65-72.
[7] Thomas P. Runarsson and Simon M. Lucas. Co-evolution versus self-
play temporal difference learning for acquiring position evaluation in
small-board go. IEEE Transactions on Evolutionary Computation, 9:628
– 640, 2005.
[8] Yannakakis, G., Levine, J., and Hallam, J. (2004). An evolutionary
approach for interactive computer games. In Evolutionary Computation,
2004. CEC2004. Congress on Evolutionary Computation, volume 1,
pages 986–993, Piscataway, NJ. IEEE.
[9] A. Hauptman and M. Sipper. Evolution of an efficient search algorithm
for the Mate-in-N problem in chess. In Proceedings of the 2007
European Conference on Genetic Programming, pages 78–89. Springer,
Valencia, Spain, 2007.
[10] P. Aksenov. Genetic algorithms for optimising chess position scoring.
Master’s Thesis, University of Joensuu, Finland, 2004. Y. Bjornsson and
T.A. Marsland. Multi-cut alpha-beta-pruning in game-tree search.
Theoretical Computer Science, 252(1-2):177–196, 2001.
[11] O. David-Tabibi, A. Felner, and N.S. Netanyahu. Blockage detection in
pawn endings. Computers and Games CG 2004, eds. H.J. van den Herik,
Y. Bjornsson, and N.S. Netanyahu, pages 187–201. Springer-Verlag,
2006.
[12] A. Hauptman and M. Sipper. Using genetic programming to evolve
chess endgame players. In Proceedings of the 2005 European
Conference onGenetic Programming, pages 120–131. Springer,
Lausanne, Switzerland, 2005.
[13] G. Kendall and G. Whitwell. An evolutionary approach for the tuning of
a chess evaluation function using population dynamics. In Proceedings
of the 2001 Congress on Evolutionary Computation, pages 995–1002.
IEEE Press, World Trade Center, Seoul, Korea, 2001.
[14] Holland, J. H. (1975). Adaptation in Natural and Artificial Systems: An
Introductory Analysis with Applications to Biology, Control and
Artificial Intelligence. Ann Arbor, MI: University of Michigan Press.
[15] Goldberg, D. E. (1989). Genetic Algorithms in Search,Optimization and
and Machine Learning. Reading, MA: Addison-Wesley.