International Journal of Mathematics and Computer Applications Research (IJMCAR) ISSN 2249-6955 Vol.

3, Issue 1, Mar 2013, 1-8 TJPRC Pvt. Ltd.

STOCHASTIC REGULAR LANGUAGE: A MATHEMATICAL MODEL FOR THE LANGUAGE OF SEQUENTIAL ACTIONS FOR DECISION MAKING UNDER UNCERTAINTY
KULANDAI THERESE1 & A. JAMES ALBERT2
1

Ph.D scholar, Karpagam University and Associate Professor of Mathematics, Nirmala College for Women, Coimbatore, India

Dean, Faculty of Arts, Science and Humanities, Karpagam University, Coimbatore, Tamil Nadu, India

ABSTRACT
Decision making is complex in nature. In complex decision problems, it is advisable to approach the problem in a sequential manner. The given complex decision problem may be split in to smaller ones without losing the integrity of the original problem. A sequential decision problem consists of n sequential states which are independent or interdependent. Therefore, action must be taken at each state and hence a sequence of actions must be taken to obtain a solution. A decision made at one state due to an action is passed on to next state and the overall decision depends on the decisions made at each state. The state space of the decision problems considered for the study is finite Stochastic Grammar is an ordinary Grammar but paired with a function that assigns probabilities to the application of the productions. The way in which the probabilities assigned to the application of productions is used for computing probabilities of strings. Stochastic language is a set of strings accompanied with their occurrence probability; one of the conditions for a language to be a stochastic language is that the sum of all probabilities attached to its strings is one. Stochastic Regular Grammar which generates Stochastic Regular Language is one of the types of Stochastic Grammar. The study is an attempt to exhibit Stochastic Regular Language as a Mathematical Model for Sequence of Actions for Decision Making under Uncertainty.

KEYWORDS: Sequential Decision Making, Uncertainty, Non Deterministic, Probabilistic Process Algebra, Stochastic
Transition System, Stochastic Regular Grammar, Stochastic Regular Language

INTRODUCTION
Decision problems play a pervasive role not only in many social, economic, political and technological issues but also in personal life. Given the richness of decision problems, today, decision analysis has evolved into a general thinking. Its framework contains theories, methods, and principles aiming at a better understanding of decision problems for a better or optimal solution. The origin of decision theory is derived from economics by using the utility function of payoffs. It suggests that decisions can be made by computing the utility and probability considering the ranges of options. It also lays down strategies for good decisions [2]. In general, decision problems can be classified into single state decision problems and multiple state decision problems. Multiple state decision problems are considered as sequential states and parallel states. This study is related to sequential states.

Kulandai Therese & A. James Albert

SEQUENTIAL DECISION MAKING

Decision making involves sequential decision making as and when necessary. The essence of sequential decision making is that decisions made at one point of time may have both immediate and long-term effects [17]. The choice one makes may depend on the course of future actions. Most of the researchers have used the concept of state and have defined sequential decision making as a Markovian decision process. That is, sequential decision making is a system of moves from one state into another in a stochastic environment [14]. The solution of a sequential decision problem is a sequence of actions

a1 , a 2 ,........a n that generates a state trajectory s 0 , s1 = f ( s0 ),..........s n +1 = f ( si , ai ) such that each action

a i is applicable in si and s n +1 is a goal state, i.e a i A( si ) and s n+1 S G . The goal of the decision-maker is to choose
actions to change the state of the system so that it can maximize its total reward. The function that selects an action for each possible state is called a policy. Therefore, solving a sequential decision-making problem corresponds to finding a policy that maximizes the expected total reward. A state can be defined as a description of a system at a particular point in time in a stochastic environment. It is common to assume that the state captures all information relevant to the decision makers decision-making process. In most cases, a decision maker will not have complete information about the current state and thus causes uncertainty at that state. Uncertainty can be intrinsic and unavoidable in the stochastic environment. It induces beliefs, i.e. graded dispositions that guide our behavior. Beliefs are necessary ingredients for our decisions. But at the same time it does not mean that beliefs cannot be entertained without any revealing behaviour manifestations [19]. Usually, probability functions are used to quantify beliefs [6], [18]. Accordingly, whenever a decision must be made, the decision makers beliefs must induce a probability function on uncertainty and on the set of possible outcomes as well.

DYNAMICAL SEQUENTIAL DECISION PROBLEMS

Dynamical sequential decision problem consists of a state space and probability distribution governing possible state transitions indicating how the next state of the system depends on past states [14]. These distributions constitute a model to show how the system evolves over a period of time in response to actions and exogenous events. The system reflects the fact that the effects of actions and events may not be perfectly predictable even if the prevailing state is known. When dealing with practical problems, sometimes one needs to take sequence of decisions one after the other. On taking an action, the decision maker attains one state. In general, a sequential decision consists of

n sequential states, independent or

interdependent, where decision made at a state is passed on to the next state and the overall decision depends on the decision made at each state. It is to be noted that the action taken at the n + 1 state depends on the outcomes of an action taken at the
th

n th state.

Generally, action transforms the state of the world and hence action can be well said as mapping from one state of the world to another state of the world. That is, actions can be considered as binary relations on a set of states S . Suppose the decision maker is in a state s S . On performing some action set of new states

a1 at the state s he will move on to one of the states of a

s = {s1 , s 2 ,.........s n }

. If

s is empty, which means that the action a1 has aborted in state s Fig.(i).

If s has a single element s1 , which means that the action elements, which means that an action

a1 is deterministic on state s Fig (ii).And if s has two or more

a1 is non-deterministic on state s Fig(iii). That is, an action a1 performed at the

Stochastic Regular Language: A Mathematical Model for the Language of Sequential Actions for Decision Making Under Uncertainty

state

s has a result at the state s where s S . The same can be diagrammatically shown as follows.
action a action a action a

s1 s
s s1

s2
s
sn

Figure (i)

Figure (ii)

Figure (iii)

It is clear that a binary relation is emerged on the states of

S if an action is performed at any state of S . In S can be viewed as the interpretations of

relation the same can be represented as a pair ( s, s ) . Thus, binary relations on action a1 . The set of all pairs taken from relation and

S is called

S S , or S 2 . The action symbol a1 is interpreted as binary

Ra1 on S , and that action symbol a 2 is interpreted as binary relation Ra 2 on S then the action sequence a1

a 2 can be interpreted such that one can move from state s to state s just in case there is some intermediate state

s with the property that a1 gets from s to s and a2 gets from s to s . This is a well-known operation on binary
relations, called relational composition. If binary relation on action

Ra1 and Ra 2 are binary relations on the same set S , then Ra1 Ra 2 is the

S given by Ra1 o Ra 2 = {( s, s ) there is some s S : ( s, s ) Ra1 and ( s , s ) Ra 2 } .If

a1 is interpreted as relation Ra1 , and action a 2 is interpreted as relation Ra 2 , then the sequence of actions a1 and

a 2 can be interpreted as Ra1 Ra 2 . The same can be diagrammatically represented as follows.

action a action b

s1 s2

s11

s12

sn
Similarly action

s1m

a3 may be taken at any one of the states s11 , s12 ,.........s1m . Hence the sequence of actions

a1 , a 2 and a3 can be interpreted as Ra1 o Ra 2 o Ra 3 and so on.

REASONING ABOUT ACTION AND CHANGE
In Sequential Decision Making Problems, the most important feature is that of representing and reasoning action and change. Given a description of the state of the world and some action, one needs the ability to predict the new state/ states after the action has been performed. In reasoning about actions in dynamic real-world environments, one of the most

Kulandai Therese & A. James Albert

crucial problems that one has to face is uncertainty. Dealing with uncertainty is a fundamental issue for most of the sequential decision problems [10], [11]. Uncertainty may refer both to initial state of the world and to the outcome of actions performed by the decision maker. In the study, uncertainty is considered only for the outcomes of actions. The definition of uncertainty as the lack of surety or certainty is readily defined in a statistical or probabilistic context as it implies that uncertainty exists when the probability of an event occurring is not 0 or 1[8]. In most of the real world problems, the way of adding uncertainty to reasoning about actions is based on quantitative models. Hence, one can have a probability distribution on the set of possible alternatives, thus, numerically distinguishing possible alternatives. This is known as probabilistic uncertainty. In the study, it is considered that decision-making under uncertainty can be performed more efficiently when the state of a system is represented by directly predicting its future after a sequence of actions. Intuitively, a decision-maker should not be concerned by the complexity of its environment, but only about how the environment responds to its actions from a subjective point of view. There are a number of formalisms for probabilistic reasoning about actions. In particular, [1] proposes a probabilistic generalization of the situation calculus, which is based on first-order logics of probability. This allows one to reason out the decision makers probabilistic degrees of belief and to what extent these beliefs change when actions are executed through disjunctive knowledge. The mathematical models needed to characterize decision tasks can be defined in terms of the elements of the following basic state model: [3] (1) a discrete and finite state space (2) a non-empty set of initial states

S, S0 S , SG S ,

(3) a goal state or final state given by a non-empty set (4) a non-empty set of actions (5) a state transition function

A( s) A representing the actions applicable in each state s , f ( s, a ) S mapping states s and actions a A( s ) into sets of
states, i.e.

f ( s, a ) 1 , and
(6) a positive action value

C (a, s) for doing a A( s ) in s .

Different models can be defined adding new restrictions or modifying the statements (2), (5) and (6). Those models are: i) Deterministic models

ii) Nondeterministic models iii) Probabilistic models Deterministic Models are where the dynamics are defined by a deterministic state transition function, i.e.,

f ( s, a ) = 1 . This is the basis of the classical decision scenario, where one has additional constraints of initial state S 0 = 1 and final state S G ( s, a ) = 1 , s S ; a A( s ) . The goal is generally to find a sequence of actions that
moves from the initial state to the goal state or final state. Nondeterministic Models are when the actions may result in more than one next states without any preferences. Probabilistic Models are when actions have probabilistic consequences. Not only the function given, but also the model includes a probability distribution

f ( s, a ) 1 is

P ( s , a ) over f ( s, a ) , s S ; a A(s ) .

Stochastic Regular Language: A Mathematical Model for the Language of Sequential Actions for Decision Making Under Uncertainty

Among these models, probabilistic model has produced significant results in Markov Decision Processes in recent years [4], [5], [9].

STOCHASTIC PROCESS ALGEBRA (SPA)

Stochastic process algebras allow to describe the qualitative, quantitative, logical, and temporal behaviour of dynamical systems in a single formalism [20]. It provides a mathematical foundation to precisely define and calculate the behaviors of dynamic systems. Also, it renders support to have a simple language to describe the behaviour of the system. In the study, the system represents the set of actions available for the decision maker. Behaviour is the total actions that the decision maker can perform. The order in which they can be executed depends on timings / probabilities. That is why a process is sometimes called a discrete event system. In order to consider an abstraction of the real behaviour of any system on certain aspects other aspects are disregarded. The grammars as processes proposes to look at grammars from a different point of view. The grammar is not only viewed as a language generators but also as a generators of behaviours [7]. It is common to define the semantics of SPA in terms of Labeled Transition Systems (LTS). These are defined in terms of a set of states and transitions between states. Definition (Probabilistic Transition System): A probabilistic transition system is a quadruple ( S , Act , P, s 0 ) where S is a countable set of states, Act a countable set of actions and P S Act S [0,1] a transition probability function satisfying:

P(s, a, s ) [0,1] for each a Act and s S and P(s, a, s ) = probability to move from state s to
a s S

s by performing an action a . To be more specific ( s, a, p1 , s ) means that the probability for reaching a state s from
the state s on performing an action a is p1 .In SPA the states correspond to behavior expressions, and the transitions to execution of actions. Considering the expression

S 0 a S1 one says that S0 can make an a-transition to S1 .

PHRASE STRUCTURE GRAMMAR [12], [13], [15], [16]

In an informal sense, grammar is a set of rules which governs the combination of words into sentences. The concepts and terminology for describing the syntax of English language are taken from Noam Chomskys works on linguistic structure. His classification of grammar and the related theory is the basis of formal language theory and theory of computation in which finite automata theory is a part. The formal definition of grammar is applicable to all domains. Definition: Phrase Structure Grammar (or Simply Grammar) G is a quadruple G = ( N , T , S , P) Where, N is the non terminal alphabet; rules;

T is the terminal alphabet and N T = ; S is the start symbol; P is the set of productions, or

P is a pair ( y, z ) , usually written y z , where y is a string in ( N T ) containing at least one non terminal

symbol and z is any string in ( N T ) . Definition (Language): A language generated by grammar G , denoted L (G ) , is the set of terminal strings which is derivable from the start symbol S of the grammar:

L(G ) = {z z T *, S z}.

Regular Grammar: The rules of grammar determine the complexity of the language [16]. Based on the production P , grammar is divided in to four types. Among the four types, the simplest type is the regular grammar. It has rules (or) productions of the form n tm ,

n t and n , where n , m N and t (T ) .

Kulandai Therese & A. James Albert

Probabilistic Grammar: The grammar G can be made in to a probabilistic grammar by assigning to each non terminal

n N a probability distribution p over productions of the form

n P for every n N and

p(n ) = 1
(T N )*s .t ( n P )

A process is sequential if it can perform atmost one action at the same time. A class of sequential process can often be conveniently represented as a LTS. The study is an attempt to establish Stochastic regular language as a language of Sequential Actions for Decision Making under Uncertainty. Firstly, we can define stochastic regular grammar as ordinary regular grammar which includes a probability function that assigns probabilities to the application of the productions. For convenience we do not allow productions to have a probability assignment which is 0. Hence the stochastic regular grammar can be defined as regular grammar together with a real function f that assigns probabilities to the productions.

STOCHASTIC REGULAR GRAMMAR

A Stochastic regular grammar is a 5- tuple, G = ( N , T , S , P, f ) where

N , a finite set of non terminal symbols,

T , a finite set of terminal symbols such that N T = , SN, a starting symbol , P , a finite set of production rules
with P N ( N T ) ,

f : P [0,1] , a real function assigns probabilities to production rules. X aY , X b ,

Usually, the non terminals are denoted by capital letters and the terminals by small letters.

X Where a T ; X , Y N and a real function f : P [0,1] giving the probability of the derivation. The sum
of the probabilities of all derivations from a given variable Here, of actions, and ,

X must be equal to 1.

N is treated as a finite non empty set of states of a decision problem, T is treated as a finite non empty set

S is the start state of the decision problem, P is a description of the outcomes of an action taken at each state

f : N T p ( N ) , i.e. for each state and action we specify a probability distribution over next states. Represents p ( s | s, a ) such that p ( N T N ) = 1 .
the Decision

the distribution

Using process algebra, the above productions may be looked as follows i.

X aY can be interpreted as follows. During the process of an action ' a ' from the state X ,
maker moves on to the state

Y . That is, X a Y

ii.

X a can be interpreted as follows. After performing an action ' a ' the process is terminated. That is, from
the state X , on taking an action ' a ' the Decision maker reaches the goal state/ final state. (because ' a ' is not followed by a non terminal). i.e., an agent can perform an action ' a ' and terminate. X SG , where goal state or final state
a

S G is a

iii.

X , this can be interpreted as follows. Sometimes, in a decision problem the outcomes

of an action do not cause any change. Hence there is no change in the state.

Stochastic Regular Language: A Mathematical Model for the Language of Sequential Actions for Decision Making Under Uncertainty

STOCHASTIC REGULAR LANGUAGE

A stochastic language is a probability distribution over T . This means that each string in T* is assigned some probability between zero and one, and that the sum of all the probabilities equals to one. Formally, we can write

p(w) = 1 .
wT *

Every stochastic regular grammar

G defines a stochastic regular language say LG , through the probabilities

p ( w / LG ) = p ( S w) . The probability p ( S w) that the grammar G generates the string w T is defined in

a recursive way. satisfying

p ( X ) = p ( X ) , p( X aw) = p( X aY ) p(Y w) where Y is the only variable

p ( X aY ) 0.

For our convenience, it is important to note that the probability assignment function f does not assign probabilities to the productions instead to the application of productions in the derivations.

DERIVATION OF STRING AS SEQUENCE OF ACTIONS

In sequential decision problem, action taken at one state produces an outcome. Based on the outcome the decision maker moves on to the next state. This could happen at every state of the problem. Ultimately the decision maker attains the goal state / desired state. The same thing occurs in the derivation of stochastic regular grammar. As mentioned earlier, behaviour is the total actions that the decision maker can perform. The behaviour to accept the sequence (string) of actions

B is said

s Act * if there exists a sequence of states {s i , i N n } such that

B = (S , Act, f , s 0 , S G ) , a 0 , a1 , a 2 ....a n Act*, n Z + , s 0 is a initial state, si +1 = f ( si , ai ), i N n 1 , s n S G .Thus, a string of actions is accepted by a behaviour if the complete string results in a sequence of transitions and
if the last state is a goal state. The run or path of a behaviour is defined as a sequence of states and transitions denoted by

s0 s1 s2 ....si +1 = f (si , ai )
CONCLUSIONS
From the above descriptions, it is obvious that a string of actions in sequential decision problems resembles a string of a regular language. A set of string is regular if it can be generated from the empty sets, the set containing the null string, the set containing a single element of the alphabet using union, concatenation, and the Kleene star operation [ 1 2 ] . The regular sets comprises of a family of languages that play an important role in formal languages, pattern recognition and the theory of finite- state machines.

REFERENCES
1. Bacchus, F., Halpern J.Y., and H. Levesque, Reasoning about noisy sensors and effectors in the situation calculus, Artif. Intell., 111(12), 171-208, (1999).. 2. 3. 4. Ben-Haim, Y., Information-gap Decision Theory: Decisions under Severe Uncertainty, Academic Press, 2001. Bertsekas, D., 1995. Dynamic Programming and Optimal Control, Vols 1 and 2. Athena Scientific. Bonet, B., Geffner, H.: Labeled RTDP: Improving the convergence of real-time.

Kulandai Therese & A. James Albert

5.

Bonet, B., Geffner, H.: Learning Depth-First Search: A unified approach to heuristic search in deterministic and non-deterministic settings, and its application to MDPs. In: ICAPS, (2006)

6.

DeGroot, M. H. (1970). Optimal statistical decisions. McGraw-Hill, New York. dynamic programming. In: ICAPS, Trento, Italy, AAAI Press (2003) 1221.

7.

Esparza, J and J. Knoop. An automata-theoretic approach to interprocedural data-ow analysis. Proceedings of FOSSACS '99, LNCS 1578:14-30, (1999)..

8.

Good, I.J (1994). Reliability always depends on probability of course, Journal of Statistical Computation and Simulation 52, 192-193.

9.

Guestrin, C., Koller, D., Parr, R., Venkataraman, S.: Efficient solution algorithms for factored MDPs. J. Artif. Intell. Res. (JAIR) 19 , 399-468, (2003).

10. Halpern, J. Y and Tuttle, M. R. Knowledge, probability, and adversaries, J. ACM, 40(4), 917-962, (1993). 11. Halpern, J.Y.: Reasoning about uncertainty. MIT Press, Cambridge, Massachusetts (2003). 12. Harry R. Lewis and Christos H.Papadimitriou, Elements of the Theory of International, 1981. 13. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman, Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, Reading, Massachusetts, USA, 2001. 14. Houlding, B., and Coolen, F.P.A., Sequential Adaptive Utility Decision Making for System Failure Correction. Journal of Risk and Reliability. Proceedings of the IMech EPART O, 221(4): 285-295, (2007). 15. John C. Martin, Introduction to Languages and the Theory of Computation, McGraw- Hill Inc., 1991 16. Mishra, K.L.P and N. Chandrasekaran, Theory of Computer Science, Prentice-Hall International, 2005 17. Paul J. Schweitzer and Abraham Seidmann. Generalized polynomial approximations in Markovian decision processes. Journal of Mathematical Analysis and Applications, 110(6): 568-582, (1985). 18. Savage, L. J. (1954). Foundations of statistics. Wiley, New York. 19. Smith P and Jones OR (1986) The philosophy of mind, an introduction. Cambridge University Press. Cambridge. 20. Susanna Donatelli. Superposed stochastic automata: a class of stochastic Petri nets with parallel solution and distributed state space. Performance Evaluation, 18:21-26, (1993). Computation, Prentice-Hall