BASIC SEARCHING

TECHNIQUE- PART 1

TC6544 – Advanced AI

Prof Dr Masri Ayob

masri@ukm.edu.my
http://www.ftsm.ukm.my/masri/
 MAIN COMPONENTS

Objective Function
• Max (Min) some function of decision variables
Subject to (s.t.)
• equality (=) constraints
• inequality (≤,≥,>,<) constraints
Search Space
• Range or values of decisions variables that
will be searched during optimization. Often a
calculable size in combinatorial optimization
2
 Types of Solutions
A solution for an optimization problem specifies the values of the
decision variables, and therefore also the value of the objective
function.

A feasible solution satisfies all constraints.

An optimal solution is feasible and provides the best objective

function value.

A near-optimal solution is feasible and provides a superior objective

function value, but not necessarily the best.
3
 Tractability

Some problems are intractable:

as they grow large, we are unable to solve them in reasonable
time

What constitutes reasonable time?

• Standard working definition: polynomial time

• On an input of size n the worst-case running time is O(nk) for some constant k
• O(n2), O(n3), O(1), O(n lg n), O(2n), O(nn), O(n!)
• Polynomial time: O(n2), O(n3), O(1), O(n lg n)
• Not in polynomial time: O(2n), O(nn), O(n!)
Polynomial-Time Algorithms

Are some problems solvable in polynomial time?

• Of course: many algorithms we’ve studied provide polynomial-time
solutions to some problems

Are all problems solvable in polynomial time?

• No: Turing’s “Halting Problem” is not solvable by any computer, no
matter how much time is given

Most problems that do not yield polynomial-time

algorithms are either optimization or decision problems.
 Optimization/
Decision Problems
Optimization Problems
• An optimization problem is one which asks, “What is the
optimal solution to problem X?”
• Examples:
• 0-1 Knapsack
• Fractional Knapsack
• Minimum Spanning Tree
Decision Problems
• An decision problem is one with yes/no answer
• Examples:
• Does a graph G have a MST of weight  W?
 Optimization/
Decision Problems
An optimization problem tries to find an optimal solution

A decision problem tries to answer a yes/no question

Many problems will have decision and optimization versions

• Eg: Traveling salesman problem
• optimization: find hamiltonian cycle of minimum weight
• decision: is there a hamiltonian cycle of weight  k

Some problems are decidable, but intractable:

as they grow large, we are unable to solve them in reasonable time
• Is there a polynomial-time algorithm that solves the problem?
 Continuous vs Combinatorial

Optimization problems can be continuous (an infinite

number of feasible solutions) or combinatorial (a finite
number of feasible solutions).

Continuous problem generally maximize or minimize a

function of continuous variables such as min 4x + 5y
where x and y are real numbers

Combinatorial problems generally maximize or minimize

a function of discrete variables such as min 4x + 5y where
x and y are countable items (e.g. integer only).

8
Combinatorial Optimization

Combinatorial optimization is the mathematical

study of finding an optimal arrangement,
grouping, ordering, or selection of discrete objects
usually finite in numbers.
• - Lawler, 1976

In practice, combinatorial problems are often

more difficult because there is no derivative
information and the surfaces are not smooth.

9
 Constraints

Constraints can be hard (must be

satisfied) or soft (is desirable to satisfy).

• Example: In your course schedule a hard constraint

is that no classes overlap. A soft constraint is that
no class be before 10 AM.

Constraints can be explicit (stated in the

problem) or implicit/embedded(obvious
to the problem).

10
 Constraints: EXAMPLE

• Given the coordinates of n

cities, find the shortest
TSP closed tour which visits
each once and only once
(Traveling (i.e. exactly once).
Salesman • Example: In the TSP, an
Problem) implicit constraint is that
all cities be visited once
and only once.

11
 Aspects of an Optimization
Problem

Continuous or Combinatorial

Size – number of decision variables, range/count of possible

values of decision variables, search space size

Degree of constraints – number of constraints, difficulty of

satisfying constraints, proportion of feasible solutions to search
space

Single or Multiple objectives

12
 Aspects of an Optimization
Problem

Deterministic (all variables are deterministic) or

Stochastic (the objective function and/or some decision
variables and/or some constraints are random variables)

Decomposition – decomposite a problem into series

problems, and then solve them independently

Relaxation – solve problem beyond relaxation, and then

can solve it back easier

13
 SIMPLE VS HARD Problem
Simple Hard

• Few decision variables • Many decision

• Differentiable variables
• Single modal • Discontinuous,
• Objective easy to combinatorial
calculate • Multi modal
• No or light constraints • Objective difficult to
• Feasibility easy to calculate
determine • Severely constraints
• Single objective • Feasibility difficult to
• Deterministic determine
• Multiple objective
• Stochastic
14
 SIMPLE VS HARD Problem

Simple • Enumeration or exact methods such

as differentiation or mathematical
problems programming or branch and bound
will work best.

• Differentiation is not possible and

Hard enumeration and other exact
methods such as math programming
problems are not computationally practical.
• heuristics are preferably used.

15
Heuristic Search Techniques

Direct techniques (blind search) are not

always possible (they require too much
time or memory).

Weak techniques can be effective if

applied correctly on the right kinds of
tasks: Heuristic.
• Typically require domain specific information.

16
Heuristic Search Techniques
Blind search techniques used an arbitrary ordering (priority) of operations.

Heuristic search techniques make use of domain specific information - a heuristic.

A Heuristic is a function that, when applied to a state, returns a number that is an

estimate of the merit of the state, with respect to the goal.
• tells us approximately how far the state is from the goal state.

Heuristics are rules to search to find optimal or near-optimal solutions. Examples are
FIFO, LIFO, earliest due date first, largest processing time first, shortest distance first,
etc.

IN AI, heuristics are formalized as:

• Rules for choosing those branches in a state space that are most likely to lead to an acceptable problem
solution.
17
 HEURISTICS

A “rule of
thumb” or • guide decision making
relatively • find approximate solutions
simple solution
method to :

for solving complex (combinatorial)

optimization problems
 Why Use Heuristic Search
Techniques

AI problem
• A problem may not have an exact
solvers solution because of inherent
employ ambiguities in the problem
statement or available data
heuristics • A problem may have an exact
in two solution, but the computational
cost of finding it may be
basic prohibitive
situations:
19
Why Use Heuristic Search Techniques

AI problem • faster than exact

optimization algorithms
solvers
employ • simpler and more
readily understood than
heuristics most exact optimization
due to: algorithms

However, heuristics do not, in general,

provide optimal solutions 20
 Search Basics
Search-is the term used for constructing/improving solutions to
obtain the optimum or near-optimum.

Solution-Encoding (representing the solution)

Neighborhood-Nearby solutions (in the encoding or solution space)

Move-Transforming current solution to another (usually

neighboring) solution

Evaluation-The solutions’ feasibility and objective function value

(quality of solution)

21
 Search Techniques
• always arrive at the same final solution through
the same sequence of solutions, although they
may depend on the initial solution.
Deterministic • Examples are LP (simplex method), tabu
search, simple heuristics like FIFO, LIFO, and
greedy heuristics.

• the solutions considered and their order are

different depending on random variables.
Stochastic • Examples are simulated annealing, ant colony
optimization and genetic algorithms.

22
 Search Techniques
• they find the nearest optimum which may not
Local be the real optimum. Example: greedy
heuristic (local optimizers).

• they find the true optimum even if it involves

Global moving to worst solutions during search
(non-greedy).

Constructive • build a solution piece by piece or

• take a solution and alter it to find a better

Improvement solution.

23
 Search Techniques
• work by constructing a solution step by step,
evaluating that solution for (a) feasibility and
(b)value for objective function.
Constructive • Greedy or myopic (narrow-minded), i.e. they take
the best thing next without regard for the rest of
the solution.
• Example: take the nearest city next (for TSP).

• work by constructing a solution, moving to a

neighboring solution, evaluating that solution
Improvement for (a) feasibility and (b) objective function.
• Example: take a tour and swap the order of two
cities (for TSP)

24
Heuristic Search Techniques

• A heuristic is only an informed

Unfortunately, guess of the next step to take.
like all rules • A heuristic can lead a search to
of discovery a suboptimal solution
and invention, • or fail to find any solution at all,
heuristic are because heuristic use limited
imperfect. information of the present
situation

25
 Example: 8 Puzzle

1 2 3 1 2 3
7 8 4 8 4
6 5 7 6 5
26
1 23 GOAL 1 23
8 4 7 84
7 65 6 5

right

1 23 1 23 1 23
7 84 7 84 7 4
65 6 5 6 85

Which move is the best?

27
 8 Puzzle Heuristics

What heurisitic(s) can we use to decide which 8-puzzle

move is the “best” (worth considering first).

For now - we just want to establish some ordering to the

possible moves (the values of our heuristic does not
matter as long as it ranks the moves).

Later - we will worry about the actual values returned by

the heuristic function.

28
A Simple 8-puzzle heuristic

Number • The higher the number

of tiles the better.
• Easy to compute (fast
in the and takes little memory).
correct • Probably the simplest
position. possible heuristic.

29
 Another approach

Number • This can also be considered a

lower bound on the number of
of tiles moves from a solution!
• The “best” move is the one with
in the the lowest number returned by
incorrect the heuristic.
• Is this heuristic more than a
position. heuristic (is it always correct?).

30
1 23 GOAL 1 23
8 4 7 84
7 65 6 5

right

1 23 1 23 1 23
7 84 7 84 7 4
65 6 5 6 85
h=2 h=4 h=3

Which move is the best?

31
Another 8-puzzle heuristic

Count how far away (how many tile movements)

each tile is from it’s correct position.

Sum up this count over all the tiles.

This is another estimate on the number of moves

away from a solution.

32
1 23 GOAL 1 23
8 4 7 84
7 65 6 5

right

1 23 1 23 1 23
7 84 7 84 7 4
65 6 5 6 85
h=2 h=4 h=4

Which move is the best?

33
 Techniques
There are a variety of search
techniques that rely on the estimate
value provided by a heuristic function.

In all cases - the quality of solution

obtained by the heuristic is important
in real-life application of the technique!

34
http://www.ftsm.ukm.my/cait/DMOLab.html
https://www.facebook.com/dmoukm/

35