Intelligent Search Techniques Proof-Number Search

MICC/IKAT Universiteit Maastricht

1

6. March 2006

1 Lecture Team: Prof. J.v.d. Herk, Dr. J. Donkers, Dr. J. Uiterwijk, G. Chaslot, S. Bakkes, JT. Saito MICC/IKAT Universiteit Maastricht () Intelligent Search Techniques Proof-Number Search 6. March 2006 1 / 97

1

Proof-Number Search Beyond standard Proof-Number Search Monte-Carlo Evaluation and Proof-Number Search

2

3

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

2 / 97

1

Proof-Number Search Basics Algorithm Example Exercise 1 Properties of PNS Beyond standard Proof-Number Search Proof-Number Search Variations Df-pn Monte-Carlo Evaluation and Proof-Number Search Combining Monte-Carlo Evaluation and Proof-Number Search Life-and-death problems in Go Experimental Outcome

2

3

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

3 / 97

The And/Or Tree

Def. Two-level alternating tree
Given two sets L1 and L2 a two-level alternating tree is a tree TL1 ,L2 = (R, N, E ) with root node R ∈ N, nodes N = L1 ∪ L2 and edges E = {(n1 , n2 ) ∈ Li × Lj : i, j = 1, 2 with i = j}.

Remarks
Deﬁnition allows empty trees Alternating levels Is the MIN/MAX tree an AND/OR tree?

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

4 / 97

The And/Or Tree
Def. AND/OR tree
An AND/OR tree is a two level alternating tree TAnd,Or = (R, N, E ). Each node n ∈ N is attributed with one feature value v (n) ∈ {true, false, unkown}. Each OR node nor has value true if there is a child n of nor with v (n) = true. nor has value false if v (n) = false for each child n of nor . Each AND node nand has value false if there is a child n of nand with v (n) = false. nand has value true if v (n) = true for each child n of nand .

Remarks
Used as search trees with search values true, false, or unknown. If all termnial nodes are evaluated the root’s value is either true or false If R ∈ Nand (∈ Nor ), we call T AND tree (OR tree) Is the MIN/MAX tree an AND/OR tree?
MICC/IKAT Universiteit Maastricht () Intelligent Search Techniques Proof-Number Search 6. March 2006 5 / 97

The AND/OR tree as game tree

Alternating levels represent player MAX or MIN respectively Find value of root (or variation to desired value) Start with unkown at root Two-valued (or three-valued) goal (instead of continuous estimates, e.g. in α − β search) Direct global search with goal: Win or Loose Local search with sub-goal, e.g. captrue piece in chess

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

6 / 97

Proof and disproof numbers

Proof number (pn) and disproof number (dn) for each node The pn represents the minimal number of nodes required to be expanded to prove the goal The dn represents the minimal number of nodes required to be expanded to disprove the goal Assume one expansion minimally for each unexpanded node D.A. McAllester (1988): Conspiracy numbers for min-max search. Artiﬁcial Intelligence, 134:145-179.

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

7 / 97

Proof-Number Search: Characteristics

Best ﬁrst search for AND/OR trees Uses structure of AND/OR tree for heuristical evaluation Heuristic: Expand node wich requires least further expansions for proving or disproving

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

8 / 97

Proof-Number Search: Outline

Expand nodes, updating proof and disproof numbers Start with root node There is always a node with lowest pn There is always a node with lowest dn Find this node and expand it Propagate chages of pn and dn back root Repeat whole procedure until root node has pn=0 or dn=0

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

9 / 97

Proof-Number Search: Details
Expanding leaves
Expanded leaf is proof: pn = 0 and dn = ∞ Expanded leaf is disproof: pn = ∞ and dn = 0 Otherwise pn = dn = 1

Backpropagation
AND node: pn = OR node: dn =
c∈children

c.pn

AND node: dn = (argminc∈children (c.dn)).dn
c∈children

c.dn

OR node: pn = (argminc∈children (c.pn)).pn

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

10 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

11 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

12 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

13 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

14 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

15 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

16 / 97

Toy Example

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

17 / 97

Proof-Number Search: Algorithm
void traverse down( node ) if( is leaf( node ) ) return node end Node child; if( is AND node( node ) ) node.children.each( Node ch ) child = get argmin( ch, end else node.children.each( Node ch ) child = get argmin( ch, end end return traverse down( child ) end

do ch.dn )

do ch.pn )

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

18 / 97

Proof-Number Search: Algorithm
void backpropagate( Node node ) if( node = roote ) if( is AND node( node ) ) node.pn = sum( node.children node.dn = min( node.children else node.pn = min( node.children node.dn = sum( node.children end backpropagate( node.parent ) end end proof value pns( Node root ) while( root.pn > 0 AND root.dn > 0 ) Node leaf = traverse down( root ) expand( leaf ) backpropagate( leaf ) end if( root.pn = 0 ) return proof else return disproof end
MICC/IKAT Universiteit Maastricht () Intelligent Search Techniques Proof-Number Search 6. March 2006 19 / 97

) ) ) )

Exercise
Write down pseudocode for the epxand() function.

Consider the game of Tic-Tac-Toe with the board situation xox x-o o– Player Max moves next with Xs. Player min moves second with Os. Write down the trees produced by proof-number search for solving this problem. Proceed your order of expansion from left to right and form top to bottom. Note: Choose your goal for proving carefully.

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

20 / 97

Properties of PNS

Not heuristic Faster then α − β search on certain problems Endgames: chess, checkers Sub-Problems: Shogi, Go Solving games: Checkers

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

21 / 97

1

Proof-Number Search Basics Algorithm Example Exercise 1 Properties of PNS Beyond standard Proof-Number Search Proof-Number Search Variations Df-pn Monte-Carlo Evaluation and Proof-Number Search Combining Monte-Carlo Evaluation and Proof-Number Search Life-and-death problems in Go Experimental Outcome

2

3

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

22 / 97

Opportunities for enhancing Proof-Number Search

Memory: all nodes are stored Time: cost for traversion and back-propagation Implementation details open Domain knowledge ignored

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

23 / 97

Proof-Number Search Variations

1994 Allis et al: Proof-Number Search. Mating problems in chess, Awari, Go-Moku 2001 Seo: PN* - Iterative deepening, proof only. Tsume-Shogi. 2002 Nagai: PDS - Iternative deepening, proof and disproof. Tsume-Shogi 2002 Nagai: Df-pn - Generalizing PDS, speed up. One-Eye Go. Shogi. 2003 Winands et al: PDS-PN. LOA. 2005 Shaeﬀer et al: Solving Checkers. Checkers. 2006 Jakub Pawlewicz and Lukasz Lew. Faster depth–ﬁrst PN–Search in low memory 2006 Saito et al: Monte-Carlo Proof-Number Search for Computer Go

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

24 / 97

Basics Depth-ﬁrst Proof-Number Search

Iterative Deepening Less moving up and down the tree Depth-ﬁrst: Smaller memory usage Thresholds for pn and dn values Thresholds local for each recursive call A. Nagai (2002): Df-pn Algorithm for Searching AND/OR Trees and its Applications. Ph.D. thesis. University of Tokio.

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

25 / 97

Idea

Avoid unnecessary updating of pn and dn values Hash table look-up faster then tree traversion Want to deﬁne thresholds φ, δ for each node n, such that: p < φ and d < δ exactly if there is a most promising move in the subtree of n

Notation
For a Node n, we deﬁne two thresholds n.φ and n.δ dn(n) ORnode pn(n) ORnode n.φ = n.δ = dn(n) ANDnode pn(n) ANDnode

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

26 / 97

Df-pn: Algorithm 1
proof value dfpn( Node root ) root.φ = ∞ root.δ = ∞ itertative deepening( root ) if( root.δ = ∞ ) return proof else return disproof end end int sumφ ( Node n ) int sum = 0 n.children.each( Node child ) do lookup( child, φ, δ ) sum += φ end return sum end int minδ ( Node n ) int min = 0 n.children.each( Node child ) do lookup( child, φ, δ ) min = min(min, δ) end return sum end

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

27 / 97

Df-pn: Algorithm 2
void itertative deepening( Node n ) lookup( n,φ, δ ) if( n.φ ≤= φ OR n.δ ≤= δ ) n.φ = φ n.δ = φ return end if( is terminal( n ) ) if( is proof( n ) ) n.φ = 0 n.δ = ∞ else n.φ = ∞ n.δ = 0 end store(n,n.φ,n.δ) return end expand(n) store(n,n.φ,n.δ) while( n.φ > min δ(n) AND n.δ > sum φ(n) ) Node child = selectChild(n) child.φ += n.δ - sum φ(n) child.δ = min(n.φ, childδ + 1 itertative deepening( child ) end n.φ = min δ(n) n.δ = sum φ(n) store(n, n.φ, n.δ) end MICC/IKAT Universiteit Maastricht () Intelligent Search Techniques Proof-Number Search 6. March 2006 28 / 97

Df-pn: Algorithm 3
void select child( Node n ) Node best int δ = φ = ∞ n.children.each( Node Child ) do lookup(child, φ, δ ) if( δ < δ1 ) best = child δ2 = δ1 δ1 = δ φ1 = φ else if( δ < δ2 ) δ2 =child.δ end if( φ = ∞ ) return best end end return best end

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

29 / 97

The Challenge of Computer Go

Complexity (19 × 19 or 9 × 9) Experienced human players still stronger than computer programs Competition (academic and programming challenge)

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

30 / 97

Approaches to Meet the Challange

Methods
Search: Global search move decision or local goal-directed search Knowledge: Patterns, ML, tuning search

Developments
Search: Monte Carlo method for global search Knowledge: Learning patterns by statistical evaluation

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

31 / 97

A Research Question
Can Monte Carlo search be exploited for local search?

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

32 / 97

1

Proof-Number Search Basics Algorithm Example Exercise 1 Properties of PNS Beyond standard Proof-Number Search Proof-Number Search Variations Df-pn Monte-Carlo Evaluation and Proof-Number Search Combining Monte-Carlo Evaluation and Proof-Number Search Life-and-death problems in Go Experimental Outcome

2

3

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

33 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

34 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

35 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

36 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

37 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

38 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

39 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

40 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

41 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

42 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

43 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

44 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

45 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

46 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

47 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

48 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

49 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

50 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

51 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

52 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

53 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

54 / 97

Monte-Carlo Evaluation for Combinatorial Games

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

55 / 97

Research Question
Can Monte Carlo search be exploited for local search?

Operational question
Can Monte Carlo methods be combined with Proof-Number Search methods?

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

56 / 97

Essential MC Evaluation and PN Search

Monte-Carlo Evaluation
Play random games and evaluate their outcome statistically Evaluation of a global game position

Proof-Number Search
AND/OR or tree search Best ﬁrst search: Rank promising moves “Move which requires least number of further expansions” Applicable to local search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

57 / 97

Idea
Apply Monte Carlo for ﬁne-tuning the heuristic ranking in PN search Evaluate leaves with Monte Carlo Weight PN leaves

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

58 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

59 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

60 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

61 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

62 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

63 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

64 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

65 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

66 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

67 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

68 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

69 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

70 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

71 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

72 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

73 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

74 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

75 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

76 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

77 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

78 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

79 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

80 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

81 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

82 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

83 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

84 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

85 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

86 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

87 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

88 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

89 / 97

MC-PN Search

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

90 / 97

MC-PN Search

Risks
Trade-oﬀ: MC evaluation time vs. gain by reﬁned move ordering

Chances
Little domain knowledge (as MC and PN search use little knowledge) Generalization to domains other than Go Could be extended using knowledge Integration into MC driven Go engine

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

91 / 97

MC-PN Search Setup
Experiment
Test on 30 local sub-problems of Go (life-and-death) Independent variables
Number of games per MC evaluation Distribution of MC games per evaluation Length of MC evaluation games MC evaluation function Mapping from MC evaluation to weights (possibly back-propagation function)

Dependent variables
Number of AND/OR nodes expanded ( 25KB per node) Time spent overall and on MC evaluation

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

92 / 97

Life-and-death problem 1

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

93 / 97

Life-and-death problem 2

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

94 / 97

Time consumption
1000 100

+++ × ×× ♦ ♦ × + × +×+ ♦ × 10 +++×+× ♦ × × × × +++××+ +×× ♦ ♦ + × × ++++×× ++ × + ♦ ♦ × og time (ms) 1 ♦ ♦ ♦ ×× +× ♦ ♦♦ × ♦ ♦ × ♦ ♦♦ ♦ ×+ ++ ♦ × 0.1 PNS ♦ + +♦ ♦ ♦ ♦ ♦ MC-PNS k=10 + ♦ +
0.01 0.001 0 5 10 15 20 25 Test (ordered by memory consumption of PNS) 30

♦♦♦

MC-PNS k=50 MC-CUT k=20 cut=7 × MC-SAN k=20 cut=8

Figure: Main experiment: Time complexity of various conﬁgurations. The PNS benchmark is not out-performed.

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

95 / 97

Space consumption
1000 100

+++ × ×× ♦ ♦ × + × +×+ ♦ × 10 +++×+× ♦ × × × × +++××+ +×× ♦ ♦ + × × ++++×× ++ × + ♦ ♦ × og time (ms) 1 ♦ ♦ ♦ ×× +× ♦ ♦♦ × ♦ ♦ × ♦ ♦♦ ♦ ×+ ++ ♦ × 0.1 PNS ♦ + +♦ ♦ ♦ ♦ ♦ MC-PNS k=10 + ♦ +
0.01 0.001 0 5 10 15 20 25 Test (ordered by memory consumption of PNS) 30

♦♦♦

MC-PNS k=50 MC-CUT k=20 cut=7 × MC-SAN k=20 cut=8

Figure: Main experiment: Time complexity of various conﬁgurations. The PNS benchmark is not out-performed.

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

96 / 97

MICC/IKAT Universiteit Maastricht

() Intelligent Search Techniques Proof-Number Search

6. March 2006

97 / 97