Textbook Markov Decision Processes in Practice 1St Edition Richard J Boucherie Ebook All Chapter PDF

Markov Decision Processes in Practice
1st Edition Richard J. Boucherie

Visit to download the full and correct content document:
https://textbookfull.com/product/markov-decision-processes-in-practice-1st-edition-ric
hard-j-boucherie/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Decision-Making in Veterinary Practice 1st Edition

Kipperman
https://textbookfull.com/product/decision-making-in-veterinary-
practice-1st-edition-kipperman/
Intelligent Decision Technology Support in Practice 1st

Edition Jeffrey W. Tweedale
https://textbookfull.com/product/intelligent-decision-technology-
support-in-practice-1st-edition-jeffrey-w-tweedale/
Understanding Pulmonary Pathology Applying Pathological

Findings in Therapeutic Decision Making 1st Edition
Richard L. Kradin
https://textbookfull.com/product/understanding-pulmonary-
pathology-applying-pathological-findings-in-therapeutic-decision-
making-1st-edition-richard-l-kradin/
Assessment in counseling : practice and applications

1st Edition Richard S. Balkin
https://textbookfull.com/product/assessment-in-counseling-
practice-and-applications-1st-edition-richard-s-balkin/
Decision Sciences: Theory and Practice 1st Edition
Raghu Nandan Sengupta
https://textbookfull.com/product/decision-sciences-theory-and-
practice-1st-edition-raghu-nandan-sengupta/
Eric Hobsbawm: A Life in History Richard J. Evans
https://textbookfull.com/product/eric-hobsbawm-a-life-in-history-
richard-j-evans/
Dictionary of mathematical geosciences 1st Edition

Richard J. Howarth
https://textbookfull.com/product/dictionary-of-mathematical-
geosciences-1st-edition-richard-j-howarth/
Understanding Voltammetry Simulation Of Electrode

Processes 2nd Edition Richard G Compton
https://textbookfull.com/product/understanding-voltammetry-
simulation-of-electrode-processes-2nd-edition-richard-g-compton/
The Research Interview: Reflective Practice and

Reflexivity in Research Processes 1st Edition Steve
Mann (Auth.)
https://textbookfull.com/product/the-research-interview-
reflective-practice-and-reflexivity-in-research-processes-1st-
edition-steve-mann-auth/
International Series in
Operations Research & Management Science
Richard J. Boucherie
Nico M. van Dijk Editors
Markov
Decision
Processes
in Practice
International Series in Operations Research
& Management Science
Volume 248
Series Editor
Camille C. Price
Stephen F. Austin State University, TX, USA
Associate Series Editor

Joe Zhu
Worcester Polytechnic Institute, MA, USA
Founding Series Editor

Frederick S. Hillier
Stanford University, CA, USA
More information about this series at http://www.springer.com/series/6161

Richard J. Boucherie • Nico M. van Dijk
Editors
Markov Decision Processes

in Practice
123
Editors
Richard J. Boucherie Nico M. van Dijk
Stochastic Operations Research Stochastic Operations Research
University of Twente University of Twente
Enschede, The Netherlands Enschede, The Netherlands
ISSN 0884-8289 ISSN 2214-7934 (electronic)

International Series in Operations Research & Management Science
ISBN 978-3-319-47764-0 ISBN 978-3-319-47766-4 (eBook)
DOI 10.1007/978-3-319-47766-4
Library of Congress Control Number: 2017932096
© Springer International Publishing AG 2017

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
Printed on acid-free paper
This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To
Carla,
Fabian, Daphne, Deirdre, and Daniël –
Thanks for being there in difficult times,
Richard
P. Dorreboom and his daughter –

for coping with my passions,
Nico
Foreword
I had the pleasure of serving as the series editor of this series over its first 20 years
(from 1993 through October, 2013). One of the special pleasures of this work was
the opportunity to become better acquainted with many of the leading researchers
in our field and to learn more about their research. This was especially true in the
case of Nico M. van Dijk, who became a friend and overnight guest in our home. I
then was delighted when Nico and his colleague, Richard J. Boucherie, agreed to be
the editors of a handbook, Queueing Networks: A Fundamental Approach, that was
published in 2010 as Vol. 154 in this series. This outstanding volume succeeded in
defining the current state of the art in this important area.
Because of both its elegance and its great application potential, Markov deci-
sion processes have been one of my favorite areas of operations research. A full
chapter (Chap. 19 in the current tenth edition) is devoted to this topic in my text-
book (coauthored by the late Gerald J. Lieberman), Introduction to Operations Re-
search. However, I have long been frustrated by the sparsity of publications that
describe applications of Markov decision processes. This was less true about 30
years ago when D.J. White published his seminal papers on such real applications
in Interfaces (see the November–December 1985 and September–October 1988 is-
sues). Unfortunately, relatively few papers or books since then have delved much
into such applications. (One of these few publications is the 2002 book edited by
Eugene Feinberg and Adam Shwartz, Handbook of Markov Decision Processes:
Methods and Applications, which is Vol. 40 in this series.)
Given the sparse literature in this important area, I was particularly delighted
when the outstanding team of Nico M. van Dijk and Richard J. Boucherie accepted
my invitation to be the editors of this exciting new book that focuses on Markov
decision processes in practice. One of my last acts as the series editor was to work
with these coeditors and the publisher in shepherding the book proposal through the
process of providing the contract for its publication. I feel that this book may prove
vii
viii Foreword
to be one of the most important books in the series because it sheds so much light
on the great application potential of Markov decision processes. This hopefully will
lead to a renaissance in applying this powerful technique to numerous real problems.
Stanford University Frederick S. Hillier

July 2016
Preface
It is over 30 years ago since D.J. White started his series of surveys on practical
applications of Markov decision processes (MDP),1,2,3 over 20 years after the phe-
nomenal book by Martin Puterman on the theory of MDP,4 and over 10 years since
Eugene A. Feinberg and Adam Shwartz published their Handbook of Markov De-
cision Processes: Methods and Applications.5 In the past decades, the practical de-
velopment of MDP seemed to have come to a halt with the general perception that
MDP is computationally prohibitive. Accordingly, MDP is deemed unrealistic and
is out of scope for many operations research practitioners. In addition, MDP is ham-
pered by its notational complications and its conceptual complexity. As a result,
MDP is often only briefly covered in introductory operations research textbooks
and courses. Recently developed approximation techniques supported by vastly in-
creased numerical power have tackled part of the computational problems; see, e.g.,
Chaps. 2 and 3 of this handbook and the references therein. This handbook shows
that a revival of MDP for practical purposes is justified for several reasons:
1. First and above all, the present-day numerical capabilities have enabled MDP
to be invoked for real-life applications.
2. MDP allows to develop and formally support approximate and simple practical
decision rules.
3. Last but not least, MDP’s probabilistic modeling of practical problems is a skill
if not art by itself.
1 D.J. White. Real applications of Markov decision processes. Interfaces, 15:73–83, 1985.
2 D.J. White. Further real applications of Markov decision processes. Interfaces, 18:55–61, 1988.
3 D.J. White. A Survey of Applications of Markov Decision Processes. Journal of the Operational
Research Society, 44:1073–1096, 1993.

4 Martin Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming.
Wiley, 1994.
5 Eugene A. Feinberg and Adam Shwartz, editors. Handbook of Markov Decision Processes:
Methods and Applications. Kluwer, 2002.

ix
x Preface
This handbook Markov Decision Processes in Practice aims to show the power
of classical MDP for real-life applications and optimization. The handbook is struc-
tured as follows:
Part I: General Theory
Part II: Healthcare
Part III: Transportation
Part IV: Production
Part V: Communications
Part VI: Financial Modeling
The chapters of Part I are devoted to the state-of-the-art theoretical foundation of
MDP, including approximate methods such as policy improvement, successive ap-
proximation and infinite state spaces as well as an instructive chapter on approx-
imate dynamic programming. Parts II–VI contain a collection of state-of-the-art
applications in which MDP was key to the solution approach in a non-exhaustive
selection of application areas. The application-oriented chapters have the following
structure:
• Problem description
• MDP formulation
• MDP solution approach
• Numerical and practical results
• Evaluation of the MDP approach used
Next to the MDP formulation and justification, most chapters contain numerical
results and a real-life validation or implementation of the results. Some of the chap-
ters are based on previously published results, some are expanding on earlier work,
and some contain new research. All chapters are thoroughly reviewed. To facilitate
comparison of the results offered in different chapters, several chapters contain an
appendix with notation or a transformation of their notation to the basic notation
provided in Appendix A. Appendix B contains a compact overview of all chapters
listing discrete or continuous modeling aspects and the optimization criteria used in
different chapters.
The outline of these six parts is provided below.
Part I: General Theory
This part contains the following chapters:

Chapter 1: One-Step Improvement Ideas and Computational Aspects
Chapter 2: Value Function Approximation in Complex Queueing systems
Chapter 3: Approximate Dynamic Programming by Practical Examples
Chapter 4: Server Optimization of Infinite Queueing Systems
Chapter 5: Structures of Optimal Policies in MDP with Unbounded Jumps: The
State of Our Art
Preface xi
The first chapter, by H.C. Tijms, presents a survey of the basic concepts underly-
ing computational approaches for MDP. Focus is on the basic principle of policy im-
provement, the design of a single good improvement step, and one-stage-look-ahead
rules, to, e.g., generate the best control rule for the specific problem of interest, for
decomposition results or parameterization, and to develop a heuristic or tailor-made
rule. Several intriguing queueing examples are included, e.g., with dynamic routing
to parallel queues.
In the second chapter, by S. Bhulai, using one-step policy improvement is
brought down to the essence of understanding and evaluating the relative value func-
tion of simple systems that can be used in the control of more complicated systems.
First, the essence of this relative value function is nicely clarified by standard birth
death M/M/s queueing systems. Next, a number of approximations for the relative
value function are provided and applied to more complex queueing systems such as
for dynamic routing in real-life multiskilled call centers.
Chapter 3, by Martijn Mes and Arturo Pérez Rivera, continues the approximation
approach and presents approximate dynamic programming (ADP) as a powerful
technique to solve large-scale discrete-time multistage stochastic control problems.
Rather than a more fundamental approach as, for example, can be found in the excel-
lent book of Warren B. Powell,6 this chapter illustrates the basic principles of ADP
via three different practical examples: the nomadic trucker, freight consolidation,
and tactical planning in healthcare.
The special but quite natural complication of infinite state spaces within MDP
is given special attention in two consecutive chapters. First, in Chap. 4, by András
Mészáros and Miklós Telek, the regular structure of several Markovian models is
exploited to decompose an infinite transition matrix in a controllable and uncontrol-
lable part, which allows a reduction of the unsolvable infinite MDP into a numer-
ically solvable one. The approach is illustrated via queueing systems with parallel
servers and a computer system with power saving mode and, in a more theoretical
setting, for birth-death and quasi-birth-death models.
Next, in Chap. 5, by Herman Blok and Floske Spieksma, emphasis is on struc-
tural properties of infinite MDPs with unbounded jumps. Illustrated via a running
example, the natural question is addressed, how structural properties of the opti-
mal policy are preserved under truncation or perturbation of the MDP. In particular,
smoothed rate truncation (SRT) is discussed, and a roadmap is provided for preserv-
ing structural properties.
6Warren B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality.

Wiley Series in Probability and Statistics, 2011.
xii Preface
Part II: Healthcare
Healthcare is the largest industry in the Western world. The number of operations
research practitioners in healthcare is steadily growing to tackle planning, schedul-
ing, and decision problems. In line with this growth, in recent years, MDPs have
found important applications in healthcare in the context of prevention, screening,
and treatment of diseases but also in developing appointment schedules and inven-
tory management. The following chapters contain a selection of topics:
Chapter 6: Markov Decision Processes for Screening and Treatment of Chronic
Diseases
Chapter 7: Stratified Breast Cancer Follow-Up Using a Partially Observable MDP
Chapter 8: Advance Patient Appointment Scheduling
Chapter 9: Optimal Ambulance Dispatching
Chapter 10: Blood Platelet Inventory Management
Chapter 6, by Lauren N. Steimle and Brian T. Denton, provides a review of MDPs
and partially observable MDPs (POMDPs) in medical decision making and a tuto-
rial about how to formulate and solve healthcare problems with particular focus on
chronic diseases. The approach is illustrated via two examples: an MDP model for
optimal control of drug treatment decisions for managing the risk of heart disease
and stroke in patients with type 2 diabetes and a POMDP model for optimal design
of biomarker-based screening policies in the context of prostate cancer.
In Chap. 7, by J.W.M. Otten, A. Witteveen, I.M.H. Vliegen, S. Siesling, J.B.
Timmer, and M.J. IJzerman, the POMDP approach is used to optimally allocate
resources in a follow-up screening policy that maximizes the total expected number
of quality-adjusted life years (QALYs) for women with breast cancer. Using data
from the Netherlands Cancer Registry, for three risk categories based on differenti-
ation of the primary tumor, the POMDP approach suggests a slightly more intensive
follow-up for patients with a high risk for and poorly differentiated tumor and a less
intensive schedule for the other risk groups.
In Chap. 8, by Antoine Sauré and Martin L. Puterman, the linear programming
approach to ADP is used to solve advance patient appointment scheduling problems,
which are problems typically intractable using standard solution techniques. This
chapter provides a systematic way of identifying effective booking guidelines for
advance patient appointment scheduling problems. The results are applied to CT
scan appointment scheduling and radiation therapy treatment scheduling.
Chapter 9, by C.J. Jagtenberg, S. Bhulai, and R.D. van der Mei, considers the
ambulance dispatch problem, in which one must decide which ambulance to send
to an incident in real time. This chapter develops a computationally tractable MDP
that captures not only the number of idle ambulances but also the future incident
location and develops an ambulance dispatching heuristic that is shown to reduce
the fraction of late arrivals by 13% compared to the “closest idle” benchmark policy
for the Dutch region Flevoland.
Chapter 10, by Rene Haijema, Nico M. van Dijk, and Jan van der Wal, considers
the blood platelet inventory problem that is of vital importance for patients’ sur-
Preface xiii
vival, since platelets have a limited lifetime after being donated and lives may be at
risk when no compatible blood platelets are available for transfusion, for example,
during surgery. This chapter develops a combined MDP and simulation approach to
minimize the blood platelet outdating percentage taking into account special pro-
duction interruptions due to, e.g., Christmas and Easter holidays.
Part III: Transportation
Transportation science is known as a vast scientific field by itself for both the public
(e.g., plane, train, or bus) and private modes of transportation. Well-known research
areas include revenue management, pricing, air traffic control, train scheduling, and
crew scheduling. This part contains only a small selection of topics to illustrate the
possible fruitful use of MDP modeling within this field, ranging from macro-level
to micro-level and from public transportation to private transportation. It contains
the following chapters:
Chapter 11: Stochastic Dynamic Programming for Noise Load Management
Chapter 12: Allocation in a Vertical Rotary Car Park
Chapter 13: Dynamic Control of Traffic Lights
Chapter 14: Smart Charging of Electric Vehicles
Chapter 11, by T.R. Meerburg, Richard J. Boucherie, and M.J.A.L. van Kraaij,
considers the runway selection problem that is typical for airports with a complex
layout of runways. This chapter describes a stochastic dynamic programming (SDP)
approach determining an optimal strategy for the monthly preference list selection
problem under safety and efficiency restrictions and yearly noise load restrictions, as
well as future and unpredictable weather conditions. As special MDP complications,
a continuous state (noise volume) has to be discretized, and other states at sufficient
distance are lumped to make the SDP numerically tractable.
In Chap. 12, by Mark Fackrell and Peter Taylor, both public and private goals
are optimized, the latter indirectly. The objective is to balance the distribution of
cars in a vertical car park by allocating arriving cars to levels in the best way. If no
place is available, a car arrival is assumed to be lost. The randomness is inherent
in the arrival process and the parking durations. This daily life problem implicitly
concerns the problem of job allocation in an overflow system, a class of problems
which are known to be unsolvable analytically in the uncontrolled case. An MDP
heuristic rule is developed and extensive experiments show it to be superior.
Chapter 13, by Rene Haijema, Eligius M.T. Hendrix, and Jan van der Wal, studies
another problem of daily life and both public and private concerns: dynamic con-
trol of traffic lights to minimize the mean waiting time of vehicles. The approach
involves an approximate solution for a multidimensional MDP based on policy it-
eration in combination with decomposition of the state space into state spaces for
different traffic streams. Numerical results illustrate that a single policy iteration
step results in a strategy that greatly reduces average waiting time when compared
to static control.
xiv Preface
The final chapter of this transportation category, Chap. 14, by Pia L. Kempker,
Nico M. van Dijk, Werner Scheinhardt, Hans van den Berg, and Johann Hurink,
addresses overnight charging of electric vehicles taking into account the fluctuating
energy demand and prices. A heuristic bidding strategy that is based on an analytical
solution of the SDP for i.i.d. prices shows a substantial performance improvement
compared to currently used standard demand side management strategies.
Part IV: Production
Control of production systems is a well-known application area that is known to

be hampered by its computational complexity. This part contains three cases that
illustrate the structure of approximate policies:
Chapter 15: Analysis of a Stochastic Lot Scheduling Problem with Strict DueDates
Chapter 16: Optimal Fishery Policies
Chapter 17: Near-Optimal Switching Strategies for a Tandem Queue
Chapter 15, by Nicky D. Van Foreest and Jacob Wijngaard, considers admission
control and scheduling rules for a make-to-order stochastic lot scheduling problem
with strict due dates. The CSLSP is a difficult scheduling problem for which MDPs
seem to be one of the few approaches to analyze this problem. The MDP formulation
further allows to set up simulations for large-scale systems.
In Chap. 16, by Eligius, M.T. Hendrix, Rene Haijema, and Diana van Dijk, a bi-
level MDP for optimal fishing quota is studied. At the first level, an authority decides
on the quota to be fished keeping in mind long-term revenues. At the second level,
fishermen react on the quota set as well as on the current states of fish stock and fleet
capacity by deciding on their investment and fishery effort. This chapter illustrates
how an MDP with continuous state and action space can be solved by truncation and
discretization of the state space and applying interpolation in the value iteration.
Chapter 17, by Daphne van Leeuwen and Rudesindo Núñez-Queija, is motivated
by applications in logistics, road traffic, and production management. This chap-
ter considers a tandem network, in which the waiting costs in the second queue
are larger than those in the first queue. MDP is used to determine the near-optimal
switching curve between serving and not serving at the first queue. that balances
waiting costs at the queues. Discrete event simulation is used to show the appropri-
ateness of the near-optimal strategies.
Part V: Communications
Communications has been an important application area for MDP with particu-
lar emphasis on call acceptance rules, channel selection, and transmission rates.
This part illustrates some special cases for which a (near)-optimal strategy can be
obtained:
Preface xv
Chapter 18: Wireless Channel Selection with Restless Bandits

Chapter 19: Flexible Staffing for Call Centers With Non-stationary Arrival Rates
Chapter 20: MDP for Query-Based Wireless Sensor Networks
Chapter 18, by Julia Kuhn and Yoni Nazarathy, considers wireless channel se-
lection to maximize the long-run average throughput. The online control problem
is modeled as restless multi-armed bandit (RMAB) problem in a POMDP frame-
work. The chapter unifies several approaches and presents a nice development of
the Whittle index.
Chapter 19, by Alex Roubos, Sandjai Bhulai, and Ger Koole, develops an MDP to
obtain time-dependent staffing levels in a single-skill call center such that a service-
level constraint is met in the presence of time-varying arrival rates. Through a nu-
merical study based on real-life data, it is shown that the optimal policies provide a
good balance between staffing costs and the penalty probability for not meeting the
service level.
Chapter 20, by Mihaela Mitici, studies queries in a wireless sensor network,
where queries might either be processed within the sensor network with possible
delay or queries might be allocated to a database without delay but possibly con-
taining outdated data. An optimal policy for query assignment is obtained from a
continuous time MDP with drift. By an exponentially uniformized conversion (as
extension of standard uniformization), it is transformed into a standard discrete-time
MDP. By computation this leads to close-to-optimal simple policies.
Part VI: Financial Modeling
It is needless to say that financial modeling and stochastics are intrinsically related.
Financial models represent a major field with time-series analysis for long-term
financial and economic purposes as one well-known direction. Related directions
concern stock, option, and utility theory. Early decision theory papers on portfolio
management and investment modeling date back to the 1970s; see the edited book.7
From a pure MDP perspective, the recently published book on Markov decision
processes with special application in finance,8 and the earlier papers by Jörn Sass
and Manfred Schäl are recommended.
Chapter 21 by Jörn Sass and Manfred Schäl, gives an instructive review and
follow-up on their earlier work to account for financial portfolios and derivatives
under proportional transactional costs. In particular, a computational algorithm is
developed for optimal pricing, and the optimal policy is shown to be a martingale
that is of special interest in financial trading.
7 Michael A. H. Dempster and Stanley R. Pliska, editors. Mathematics of Derivative Securities.

Cambridge University Press, 1997.
8 N. Bäuerle, U. Rieder. Markov Decision Processes with Applications in Finance. Springer, 2011.
xvi Preface
Summarizing
These practical MDP applications have illustrated a variety of both standard and
nonstandard aspects of MDP modeling and its practical use:
• A first and major step is a proper state definition containing sufficient infor-
mation and details, which will frequently lead to multidimensional discrete or
continuous states.
• The transition structure of the underlying process may involve time-dependent
transition probabilities.
• The objective for optimization may be an average, discounted, or finite-time
criterion.
• One-step rewards may be time dependent.
• The action set may be continuous or discrete.
• A simplified but computationally solvable situation can be an important first
step in deriving a suitable policy that may subsequently be expanded to the
solution of a more realistic case.
• Heuristic policies that may be implemented in practice can be developed from
optimal policies.
We are confident that this handbook is appealing for a variety of readers with a
background in, among others, operations research, mathematics, computer science,
and industrial engineering:
1. A practitioner that would like to become acquainted with the possible value of
MDP modeling and ways to use it
2. An academic or institutional researcher to become involved in an MDP model-
ing and development project and possibly expanding its frontiers
3. An instructor or student to be inspired by the instructive examples in this hand-
book to start using MDP for real-life problems
From each of these categories you are invited to step in and enjoy reading this
hand book for further practical MDP applications.
Preface xvii
Acknowledgments
We are most grateful to all authors for their positive reactions right from the initial
invitations to contribute to this handbook: it is the quality of the chapters and the
enthusiasm of the authors that will enable MDP to have its well-deserved impact on
real-life applications.
We like to deeply express our gratitude to the former editor in chief and series
editor: Fred Hillier. Had it not been for his stimulation from the very beginning in
the first place and his assistance in its handling for approval, just before retirement,
we would not have succeeded to complete this handbook.
Enschede, The Netherlands Richard J. Boucherie

July 2016 Nico M. van Dijk
Contents
Part I General Theory
1 One-Step Improvement Ideas and Computational Aspects . . . . . . . . . 3

Henk Tijms
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The Average-Cost Markov Decision Model . . . . . . . . . . . . . . . . . . . . 4
1.2.1 The Concept of Relative Values . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2 The Policy-Improvement Step . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.3 The Odoni Bounds for Value Iteration . . . . . . . . . . . . . . . . 11
1.3 Tailor-Made Policy-Iteration Algorithm . . . . . . . . . . . . . . . . . . . . . . . 13
1.3.1 A Queueing Control Problem with a Variable Service
Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.4 One-Step Policy Improvement for Suboptimal Policies . . . . . . . . . . 18
1.4.1 Dynamic Routing of Customers to Parallel Queues . . . . . 19
1.5 One-Stage-Look-Ahead Rule in Optimal Stopping . . . . . . . . . . . . . . 24
1.5.1 Devil’s Penny Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.5.2 A Game of Dropping Balls into Bins . . . . . . . . . . . . . . . . . 27
1.5.3 The Chow-Robbins Game . . . . . . . . . . . . . . . . . . . . . . . . . . 30
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2 Value Function Approximation in Complex Queueing Systems . . . . . 33

Sandjai Bhulai
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2 Difference Calculus for Markovian Birth-Death Systems . . . . . . . . 35
2.3 Value Functions for Queueing Systems . . . . . . . . . . . . . . . . . . . . . . . 40
2.3.1 The M/Cox(r)/1 Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3.2 Special Cases of the M/Cox(r)/1 Queue . . . . . . . . . . . . . . . 42
2.3.3 The M/M/s Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.3.4 The Blocking Costs in an M/M/s/s Queue . . . . . . . . . . . . . 45
2.3.5 Priority Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
xix
xx Contents
2.4 Application: Routing to Parallel Queues . . . . . . . . . . . . . . . . . . . . . . 47

2.5 Application: Dynamic Routing in Multiskill Call Centers . . . . . . . . 52
2.6 Application: A Controlled Polling System . . . . . . . . . . . . . . . . . . . . . 60
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3 Approximate Dynamic Programming by Practical Examples . . . . . . . 63

Martijn R.K. Mes and Arturo Pérez Rivera
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2 The Nomadic Trucker Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.1 Problem Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.2.2 MDP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.2.3 Approximate Dynamic Programming . . . . . . . . . . . . . . . . . 69
3.3 A Freight Consolidation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.3.2 MDP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.4 A Healthcare Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.4.2 MDP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.5 What’s More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.5.1 Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.5.2 Value Function Approximations . . . . . . . . . . . . . . . . . . . . . 96
3.5.3 Exploration vs Exploitation . . . . . . . . . . . . . . . . . . . . . . . . . 97
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4 Server Optimization of Infinite Queueing Systems . . . . . . . . . . . . . . . . 103

András Mészáros and Miklós Telek
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.2 Basic Definition and Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.3.1 Optimization of a Queueing System with Two Different
Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.3.2 Optimization of a Computational System with Power
Saving Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.3.3 Structural Properties of These Motivating Examples . . . . 109
4.4 Theoretical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.4.1 Subset Measures in Markov Chains . . . . . . . . . . . . . . . . . . 109
4.4.2 Markov Chain Transformation . . . . . . . . . . . . . . . . . . . . . . . 112
4.4.3 Markov Decision Processes with a Set of Uncontrolled
States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.4.4 Infinite Markov Chains with Regular Structure . . . . . . . . . 115
4.5 Solution and Numerical Analysis of the Motivating Examples . . . . 116
4.5.1 Solution to the Queue with Two Different Servers . . . . . . 116
4.5.2 Solution to the Power-Saving Model . . . . . . . . . . . . . . . . . . 117
Contents xxi
4.6 Further Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

4.6.1 Optimization of a Queuing System with Two Markov
Modulated Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.6.2 Structural Properties of the Example with Markov
Modulated Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.7 Infinite MDPs with Quasi Birth Death Structure . . . . . . . . . . . . . . . . 121
4.7.1 Quasi Birth Death Process . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.7.2 Solving MDPs with QBD Structure . . . . . . . . . . . . . . . . . . 122
4.8 Solution and Numerical Analysis of MDPs with QBD Structure . . 127
4.8.1 Solution of the Example with Markov Modulated
Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.8.2 Markov Modulated Server with Three Background
States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5 Structures of Optimal Policies in MDPs with Unbounded Jumps:

The State of Our Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
H. Blok and F.M. Spieksma
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.2 Discrete Time Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.2.1 Discounted Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.2.2 Approximations/Perturbations . . . . . . . . . . . . . . . . . . . . . . . 146
5.2.3 Average Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.3 Continuous Time Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.3.1 Uniformisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.3.2 Discounted Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
5.3.3 Average Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.3.4 Roadmap to Structural Properties . . . . . . . . . . . . . . . . . . . . 166
5.3.5 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.3.6 Tauberian Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Appendix: Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Part II Healthcare
6 Markov Decision Processes for Screening and Treatment of Chronic

Diseases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Lauren N. Steimle and Brian T. Denton
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2 Background on Chronic Disease Modeling . . . . . . . . . . . . . . . . . . . . 191
6.3 Modeling Framework for Chronic Diseases . . . . . . . . . . . . . . . . . . . 193
6.3.1 MDP and POMDP Model Formulation . . . . . . . . . . . . . . . 193
6.3.2 Solution Methods and Structural Properties . . . . . . . . . . . . 197
6.3.3 Model Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
xxii Contents
6.4 MDP Model for Cardiovascular Risk Control in Patients with

Type 2 Diabetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
6.4.1 MDP Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
6.4.2 Results: Comparison of Optimal Policies Versus
Published Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
6.5 POMDP for Prostate Cancer Screening . . . . . . . . . . . . . . . . . . . . . . . 208
6.5.1 POMDP Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . 210
6.5.2 Results: Optimal Belief-Based Screening Policy . . . . . . . . 214
6.6 Open Challenges in MDPs for Chronic Disease . . . . . . . . . . . . . . . . 215
6.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7 Stratified Breast Cancer Follow-Up Using a Partially Observable

MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
J.W.M. Otten, A. Witteveen, I.M.H. Vliegen, S. Siesling, J.B. Timmer,
and M.J. IJzerman
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
7.2 Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
7.2.1 Optimality Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7.2.2 Alternative Representation of the Optimality Equations . . 230
7.2.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
7.3 Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
7.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
7.4.1 Sensitivity Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
7.5 Conclusions and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8 Advance Patient Appointment Scheduling . . . . . . . . . . . . . . . . . . . . . . . 245

Antoine Sauré and Martin L. Puterman
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
8.2 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.3 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.3.1 Decision Epochs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.3.2 State Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
8.3.3 Action Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
8.3.4 Transition Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
8.3.5 Immediate Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
8.3.6 Optimality Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
8.4 Solution Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
8.5 Practical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.5.1 Computerized Tomography Scan Appointment
Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.5.2 Radiation Therapy Treatment Scheduling . . . . . . . . . . . . . 260
Contents xxiii
8.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

8.7 Open Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9 Optimal Ambulance Dispatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

C.J. Jagtenberg, S. Bhulai and R.D. van der Mei
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.1.1 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.1.2 Our Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
9.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
9.3 Solution Method: Markov Decision Process . . . . . . . . . . . . . . . . . . . 273
9.3.1 State Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
9.3.2 Policy Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
9.3.3 Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
9.3.5 Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.4 Solution Method: Dynamic MEXCLP Heuristic for Dispatching . . 279
9.4.1 Coverage According to the MEXCLP Model . . . . . . . . . . . 279
9.4.2 Applying MEXCLP to the Dispatch Process . . . . . . . . . . . 279
9.5 Results: A Motivating Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
9.5.1 Fraction of Late Arrivals . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9.5.2 Average Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
9.6 Results: Region Flevoland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
9.6.1 Analysis of the MDP Solution for Flevoland . . . . . . . . . . . 285
9.6.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
9.7 Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
9.7.1 Further Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
10 Blood Platelet Inventory Management . . . . . . . . . . . . . . . . . . . . . . . . . . 293

Rene Haijema, Nico M. van Dijk, and Jan van der Wal
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
10.1.1 Practical Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
10.1.2 SDP-Simulation Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 295
10.1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
10.2 Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
10.3 SDP-Simulation Approach for the Stationary PPP . . . . . . . . . . . . . . 296
10.3.1 Steps of SDP-Simulation Approach . . . . . . . . . . . . . . . . . . 296
10.3.2 Step 1: SDP Model for Stationary PPP . . . . . . . . . . . . . . . . 297
10.3.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
10.4 Extended SDP-Simulation Approach for the Non-Stationary PPP . 300
10.4.1 Problem: Non-Stationary Production Breaks . . . . . . . . . . . 300
10.4.2 Extended SDP-Simulation Approach . . . . . . . . . . . . . . . . . 300
10.4.3 Extension: Including Non-Stationary Periods . . . . . . . . . . 301
xxiv Contents
10.5 Case Study: Optimal Policy Around Breaks . . . . . . . . . . . . . . . . . . . 303

10.5.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
10.5.2 Step I: Stationary Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 304
10.5.3 Steps II to IV: Christmas and New Year’s Day . . . . . . . . . . 306
10.5.4 Steps II to IV: 4-Days Easter Weekend . . . . . . . . . . . . . . . . 310
10.5.5 Conclusions: Extended SDP-Simulation Approach . . . . . . 314
10.6 Discussion and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Part III Transportation
11 Stochastic Dynamic Programming for Noise Load Management . . . . 321

T.R. Meerburg, Richard J. Boucherie, and M.J.A.L. van Kraaij
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
11.2 Noise Load Management at Amsterdam Airport Schiphol . . . . . . . . 323
11.3 SDP for Noise Load Optimisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
11.4 Numerical Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
11.4.2 Discretisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
11.5 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
11.5.1 Probability of Exceeding the Noise Load Limit . . . . . . . . . 329
11.5.2 Comparison with the Heuristic . . . . . . . . . . . . . . . . . . . . . . 330
11.5.3 Increasing the Number of Decision Epochs . . . . . . . . . . . . 331
11.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
12 Allocation in a Vertical Rotary Car Park . . . . . . . . . . . . . . . . . . . . . . . . 337

M. Fackrell and P. Taylor
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
12.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
12.2.1 The Car Parking Allocation Problem . . . . . . . . . . . . . . . . . 340
12.2.2 Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . 344
12.3 The Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
12.5 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
12.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
Contents xxv
13 Dynamic Control of Traffic Lights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

Rene Haijema, Eligius M.T. Hendrix, and Jan van der Wal
13.1 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
13.2 Markov Decision Process (MDP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
13.2.1 Examples: Terminology and Notations . . . . . . . . . . . . . . . . 373
13.2.2 MDP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
13.3 Approximation by Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
13.3.1 Policy Iteration (PI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
13.3.2 Initial Policy: Fixed Cycle (FC) . . . . . . . . . . . . . . . . . . . . . . 377
13.3.3 Policy Evaluation Step of FC . . . . . . . . . . . . . . . . . . . . . . . . 377
13.3.4 Single Policy Improvement Step: RV1 Policy . . . . . . . . . . 379
13.3.5 Computational Complexity of RV1 . . . . . . . . . . . . . . . . . . . 379
13.3.6 Additional Iterations of PI . . . . . . . . . . . . . . . . . . . . . . . . . . 381
13.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
13.4.1 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
13.4.2 Intersection F4C2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
13.4.3 Complex Intersection F12C4 . . . . . . . . . . . . . . . . . . . . . . . . 382
13.5 Discussion and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
14 Smart Charging of Electric Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387

Pia L. Kempker, Nico M. van Dijk, Werner Scheinhardt,
Hans van den Berg, and Johann Hurink
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
14.2 Background on DSM and PowerMatcher . . . . . . . . . . . . . . . . . . . . . . 389
14.3 Optimal Charging Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
14.3.1 MDP/SDP Problem Formulation . . . . . . . . . . . . . . . . . . . . . 393
14.3.2 Analytic Solution for i.i.d. Prices . . . . . . . . . . . . . . . . . . . . 395
14.3.3 DP-Heuristic Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
14.5 Conclusion/Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
Part IV Production
15 Analysis of a Stochastic Lot Scheduling Problem with Strict

Due-Dates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
Nicky D. van Foreest and Jacob Wijngaard
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
15.2 Theoretical Background of the CSLSP . . . . . . . . . . . . . . . . . . . . . . . . 409
15.3 Production System, Admissible Policies, and Objective Function . 410
15.3.1 Production System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
15.3.2 Admissible Actions and Policies . . . . . . . . . . . . . . . . . . . . . 411
15.3.3 Objective Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
xxvi Contents
15.4 The Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413

15.4.1 Format of a State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
15.4.2 Actions and Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
15.4.3 Transition Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
15.4.4 Further Aggregation in the Symmetric Case . . . . . . . . . . . 417
15.4.5 State Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
15.4.6 A Heuristic Threshold Policy . . . . . . . . . . . . . . . . . . . . . . . . 417
15.5 Numerical Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
15.5.1 Influence of the Load and the Due-Date Horizon . . . . . . . 419
15.5.2 Visualization of the Structure of the Optimal Policy . . . . . 419
15.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
16 Optimal Fishery Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425

Eligius M.T. Hendrix, Rene Haijema, and Diana van Dijk
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
16.2 Model Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
16.2.1 Biological Dynamics; Growth of Biomass . . . . . . . . . . . . . 427
16.2.2 Economic Dynamics; Harvest and Investment
Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
16.2.3 Optimization Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
16.3 Model Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
16.3.1 Bounds on Decision and State Space . . . . . . . . . . . . . . . . . 430
16.3.2 Equilibrium State Values in a Deterministic Setting . . . . . 431
16.4 Discretization in the Value Iteration Approach . . . . . . . . . . . . . . . . . 432
16.4.1 Deterministic Elaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
16.4.2 Stochastic Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 434
16.4.3 Analysis of the Stochastic Model . . . . . . . . . . . . . . . . . . . . 435
16.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
17 Near-Optimal Switching Strategies for a Tandem Queue . . . . . . . . . . . 439

Daphne van Leeuwen and Rudesindo Núñez-Queija
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
17.2 Model Description: Single Service Model . . . . . . . . . . . . . . . . . . . . . 442
17.3 Structural Properties of an Optimal Switching Curve . . . . . . . . . . . . 444
17.4 Matrix Geometric Method for Fixed Threshold Policies . . . . . . . . . 447
17.5 Model Description: Batch Transition Model . . . . . . . . . . . . . . . . . . . 450
17.5.1 Structural Properties of the Batch Service Model . . . . . . . 451
17.5.2 Matrix Geometric Method with Batch Services . . . . . . . . . 452
17.6 Simulation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
17.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Contents xxvii
Part V Communications
18 Wireless Channel Selection with Restless Bandits . . . . . . . . . . . . . . . . . 463

Julia Kuhn and Yoni Nazarathy
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
18.2 Reward-Observing Restless Multi-Armed Bandits . . . . . . . . . . . . . . 466
18.3 Index Policies and the Whittle Index . . . . . . . . . . . . . . . . . . . . . . . . . 471
18.4 Numerical Illustration and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 476
18.5 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
19 Flexible Staffing for Call Centers with Non-stationary Arrival

Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
Alex Roubos, Sandjai Bhulai, and Ger Koole
19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
19.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
19.3 Solution Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
19.4 Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
19.4.1 Constant Arrival Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
19.4.2 Time-Dependent Arrival Rate . . . . . . . . . . . . . . . . . . . . . . . 495
19.4.3 Unknown Arrival Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
19.5 Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
Appendix: Exact Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
20 MDP for Query-Based Wireless Sensor Networks . . . . . . . . . . . . . . . . 505

Mihaela Mitici
20.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
20.2 Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
20.3 Continuous Time Markov Decision Process with a Drift . . . . . . . . . 508
20.4 Exponentially Uniformized Markov Decision Process . . . . . . . . . . . 510
20.5 Discrete Time and Discrete Space Markov Decision Problem . . . . . 511
20.6 Standard Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . 513
20.7 Fixed Assignment Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
20.7.1 Always Assign Queries to the DB . . . . . . . . . . . . . . . . . . . . 514
20.7.2 Always Assign Queries to the WSN . . . . . . . . . . . . . . . . . . 514
20.8.1 Performance of Fixed Policies vs. Optimal Policy . . . . . . . 515
20.8.2 Optimal Policy Under Different Values
of the Uniformization Parameter . . . . . . . . . . . . . . . . . . . . . 515
20.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
xxviii Contents
Part VI Financial Modeling
21 Optimal Portfolios and Pricing of Financial Derivatives Under

Proportional Transaction Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
Jörn Sass and Manfred Schäl
21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
21.2 The Financial Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
21.3 The Markov Decision Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
21.4 Martingale Properties of the Optimal Markov Decision Process . . . 531
21.5 Price Systems and the Numeraire Portfolio . . . . . . . . . . . . . . . . . . . . 533
21.6 Conclusive Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
Appendix A: Basic Notation for MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
Appendix B: Dichotomy and Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549

List of Contributors
S. Bhulai
Faculty of Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
H. Blok
Eindhoven University of Technology, Eindhoven, The Netherlands
Richard J. Boucherie
Stochastic Operations Research, University of Twente, Enschede, The Netherlands
Brian T. Denton
Department of Industrial and Operations Engineering, University of Michigan, Ann
Arbor, MI, USA
Mark Fackrell
School of Mathematics and Statistics, University of Melbourne, VIC, Australia
Rene Haijema
Operations Research and Logistics group, Wageningen University, Wageningen,
The Netherlands
Eligius M.T. Hendrix
Computer Architecture, Universidad de Málaga, Málaga, Spain
Johann Hurink
Department of Applied Mathematics, University of Twente, Enschede,
The Netherlands
M.J. IJzerman
Department of Health Technology and Services Research, University of Twente,
Enschede, The Netherlands
C.J. Jagtenberg
Stochastics, CWI, Amsterdam, The Netherlands
xxix
xxx List of Contributors
Pia L. Kempker
TNO, Cyber Security & Robustness TNO, The Hague, The Netherlands
Ger Koole
Faculty of Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
Julia Kuhn
The University of Queensland, Brisbane, QLD, Australia
University of Amsterdam, Amsterdam, The Netherlands
T.R. Meerburg
Air Traffic Control The Netherlands, Schiphol, The Netherlands
Martijn Mes
Department of Industrial Engineering and Business Information Systems,
University of Twente, Enschede, The Netherlands
András Mészáros
MTA-BME Information Systems Research Group, Budapest, Hungary
Mihaela Mitici
Faculty of Aerospace Engineering, Air Transport and Operations, Delft University
of Technology, Delft, The Netherlands
Yoni Nazarathy
The University of Queensland, Brisbane, QLD, Australia
Rudesindo Núñez-Queija
CWI, Amsterdam, The Netherlands
J.W.M. Otten
Department of Stochastic Operations Research, University of Twente, Enschede,
The Netherlands
Arturo Pérez Rivera
Department of Industrial Engineering and Business Information Systems,
University of Twente, Enschede, The Netherlands
Martin L. Puterman
Sauder School of Business, University of British Columbia, Vancouver, BC,
Canada V6T 1Z2
Alex Roubos
CCmath, Amsterdam, The Netherlands
Jörn Sass
Fachbereich Mathematik, TU Kaiserslautern, Kaiserslautern, Germany
Antoine Sauré
Telfer School of Management, University of Ottawa, Ottawa, ON, Canada K1N
6N5
List of Contributors xxxi
Manfred Schäl
Institut für Angewandte Mathematik, Universität Bonn, Bonn, Germany
Werner Scheinhardt
The Netherlands
S. Siesling
Department of Health Technology and Services Research, University of Twente,
Enschede, The Netherlands
Department of Research, Comprehensive Cancer Organisation, Utrecht,
The Netherlands
F.M. Spieksma
Leiden University, Leiden, The Netherlands
Lauren N. Steimle
Department of Industrial and Operations Engineering, University of Michigan, Ann
Arbor, MI, USA
Peter Taylor
School of Mathematics and Statistics, University of Melbourne, VIC, Australia
Miklós Telek
Budapest University of Technology and Economics, Budapest, Hungary
Henk Tijms
Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
J.B. Timmer
Department of Stochastic Operations Research, University of Twente, Enschede,
The Netherlands
Nico M. van Dijk
Stochastic Operations Research, University of Twente, Enschede,
The Netherlands
Diana van Dijk
Department of Environmental Social Sciences, Swiss Federal Institute of Aquatic
Science and Technology (EAWAG), Dübendorf, Switzerland
Nicky D. van Foreest
Faculty of Economics and Business, University of Groningen, Groningen,
The Netherlands
M.J.A.L. van Kraaij
Air Traffic Control, Utrecht, The Netherlands
Daphne van Leeuwen
CWI, Amsterdam, The Netherlands
Hans van den Berg
TNO, Cyber Security & Robustness TNO, The Hague, The Netherlands
The Netherlands
Another random document with
no related content on Scribd:
alguno y nos pusiesse en el
mundo donde residia entonçes el
Consejo real. Lo qual hizo con
gran diligençia, que al presente
residia en Valladolid. Y vn dia de
mañana procuramos presentar la
petiçion en el Consejo de la
Inquisiçion de su magestad y vista
por los del Consejo nos
respondieron que se veria y
proueeria lo neçesario y que
conueniesse; y andando por
algunos de aquellos señores por
hablarlos en sus casas nos
dezian que era escusado esperar
prouision, porque hallauan que si
quitassen estas superfluidades de
las sçiençias no se podria el
mundo conseruar, porque los
sabios y maestros no ternian que
enseñar, y por el consiguiente no
podrian ganar de comer.
Miçilo.—Espantado estoy de ver
quanto mejor obedeçen los
diablos que los hombres.
Gallo.—Y ansi[841] como vimos
que yua la cosa tan a la larga lo
dexamos de seguir, y el mi angel
como me hubo guiado en toda
esta xornada me dixo: mira,
Menipo, yo he hecho este camino
por tu contenplacion, por quitarte
de pena; que bien sabía yo en lo
que auia de parar. Agora te quiero
dezir la suma de mi intinçion.
Sabe que el mejor y más seguro
estado de los hombres en el
mundo es de los ydiotas, simples
populares que passan la vida con
prudençia. Por lo qual dexate de
oy más de gastar tienpo en la
vana consideracion de las cosas
altas y que suben de tu
entendimiento, y dexa de inquirir
con especulaçion los fines y
prinçipios y causas de las cosas.
Menospreçia y aborreçe estos
vanos y cautelosos sylogismos
que no son otra cosa sino vurla y
vanidad sin prouecho alguno,
como lo has visto por esperiençia
en esta xornada y peregrinaje; y
de aqui adelante solamente sigue
aquel genero de vida que te tenga
en las cosas que de presente
posees lo mejor ordenado que a
las leyes de virtud puedas; y
como sin demasiada curiosidad ni
soliçitud en alegria y plazer
puedas vibir más sosegado y
contento; y ansi el mi angel me
dexó y yo desperte como de vn
graue y profundo sueño[842]
espantado de lo mucho que auia
visto como te lo he narrado por el
orden que has oydo y yo mejor he
podido.
Miçilo.—¡O gallo! Dios te lo
agradezca el plazer y honrra que
me has hecho en[843] tu
feliçissima narraçion. De oy más
no quiero otro maestro, otro
philosopho, ni[844] otro sabio
consejero que a tí para passar el
discurso de la vida que me queda,
y ruegote que no me dexes, que
juntos passaremos aqui nuestra
vida; que segun me dizes es la
más segura, segun tengo
entendido por tu esperiençia[845].
Gallo.—Ya te he contado, Miçilo,
hasta agora mi dichosa y
admirable peregrinaçion, en la
qual por su espanto y terribilidad
te he tenido suspenso y algo
desasosegado, segun he
hechado de ver[846]; por lo qual
de oy más te quiero contar cosas
graçiosas y suaues, con que en
donayre y plazer passes mejor el
trabajo del dia. Ofreçesseme;
quiero te contar agora vn suaue y
graçioso conbite; vna opulenta y
admirable copiosidad de vna
missa nueua, en que siendo
clerigo en vn tiempo me hallé.
Dezirte he tanto regocijo de
aquellos clerigos, tanto canto,
tanto vayle, tanta alegría que no
se puede encareçer más; y
despues dezirte he vna fragosa y
arriscada tragedia que calentando
el vino las orejas de los abbades
suçedio. Confio que con esto
soldarás el espanto en que te he
puesto hasta aqui. Agora abre la
tienda, que en el canto que se
sigue lo prosiguire.
Fin del deçimo sexto canto del
gallo.
NOTAS:
[816] G., podeis.
[817] G., tanto.
[818] G., oyeren.
[819] G., mi desuentura.
[820] G., graçiosa.
[821] G., antes.
[822] G., atormentaua.
[823] G., salí.
[824] G., verle tan amargamente llorar su.
[825] G., estaua en su real.
[826] G., forçada.
[827] G., y.
[828] G., hauian.
[829] G., Françisco françes.
[830] G., que no.
[831] G., le.
[832] G., respondi.
[833] G., mejorado.
[834] G., condenados.
[835] G., ahumana.
[836] G., frequentauan mucho los.
[837] G., chançelleria.
[838] G., adelantarse a otros enobleçiendose.
[839] G., ouo.
[840] G., numidas.
[841] G., Pues.
[842] sueño muy profundo.
[843] G., con.
[844] G., más.
ARGUMENTO
DEL DEÇIMO
SEPTIMO CANTO
DEL GALLO
En el deçimo septimo canto que

se sigue el auctor imitando a
Luçiano en el dialogo llamado
Conuiuiun philosophorum,
sueña auerse hallado en vna
misa nueua, en la qual
descriue grandes
aconteçimientos que entre
clerigos en ella passaron[847].
Miçilo.—Despierta, gallo, que

pareçe ser hora para que con tu
promesa me restituyas en mi
pristina alegria, porque el
peregrino y nueuo proçeso y
manera de dezir de tu prodigiosa
narraçion infernal me tiene tan
espantado que por ninguna
contraria manera de dezir pienso
poder boluer en mí para oyr y
hablar con mi primera libertad; y
es ansi qve aunque por su
admiraçion el cuento mueue a
atençion contina hazesse más
estimar quando se considera el
credito que se deue a tu ser por
auer sido çelestial. Porque no
pareçe ni se puede dezir que solo
me le has contado por darme
deletaçion, como hazen los
fabulosos inuentores de mentiras
en las prestigiosas y
monstruosas [848] narraçiones que
escriuen solo por agradar y dar a
los lectores oçiosos con que el
tiempo se pueda entretener[849]
aunque sea con vana ocupaçion.
Porque me dizen que han sido
muchos philosophos auctores de
semejantes obras; como Cthesias
y Jamblico[850]; de los quales el
vno ha escripto cosas admirables
de las Indias; y el otro del mar
oçeano[851] sin que ninguno
dellos huuiesse visto, ni en algun
auctor leydo cosa de las que cada
qual dellos escriuió. Pero fue tan
grande su eloquençia y admirable
manera de dezir que quanto
quiera que manifiestamente
escriuian[852] fiçion, por escreuir
en aquel estilo hizieron graçiosa y
estimada su obra. Otros dizen
que ha hauido que con ingenio
espantoso han contado de si
grandes viajes y peregrinaciones,
fiereza de vestias y diuersidad de
tierras y costunbres de hombres,
sin auer ninguna cosa de las que
descriuen en el mundo, que[853]
por la dulçura de hablar[854] los
han tenido en veneraçion. Como
aquel ingenioso inuentor[855]
Homero escriuió de su Ulixes
auer visto animales, y gigantes
monstruosos Poliphemos con solo
vn ojo en la frente que se
tragauan los hombres enteros y
vibos; y esto sin los auer
engendrado hasta oy naturaleza.
Desto estoy bien seguro yo que tú
no imitas a estos en tu passada
historia, porque no es de presumir
que infames los çelicolas como tú
con[856] mentirosa narraçion. Por
tanto despierta y prosigue que yo
te oyré. Cuentame aquella
sangrienta batalla, aquel suçeso
canpal que ayer me
propusiste[857] dezir, pues de tu
promesa no te puedes excusar.
Gallo.—Por çierto Miçilo, mucho
estoy arrepentido en auerte
propuesto esa sacrilega tragedia,
pues en ella hago ser publicos los
desatinos tan excesiuos que el
vinático furor causó en aquellos
religiosos juizios y habito
saçerdotal, lo qual más conuenia
ser callado y sepultado en el
profundo del oluido por auer
aconteçido en personas que
auian de ser exemplo de
templança, prudençia y
honestidad: antes que ser yo
agora relactor de las deshonestas
y desuariadas furias que
passaron entre su beber. Mal
pareçe dar yo ocasion con mi
lengua a que auiendo tú plazer te
rias y mofes de aquella
consagrada caterua que está en
la tierra en lugar de la diuina
magestad[858]. De manera que si
yo me huuiere flaca y friamente
en el persuadir y demostrar este
aconteçimiento corro peligro en
mi persona de tiuio orador; y
quando por el contrario en el
encareçer y esaxerar me
mostrare eloquente será para
más augmentar tu risa y mofa,
haziendo en mayor infamia de
aquella religiosa gente. Por tanto
mira, Miçilo, si es más
conueniente a hombre bien
acostumbrado como tú dexar de
inportunarme que te cuente
semejantes aconteçimientos;
porque a mí me pareçe ser
obligado a los callar.
Miçilo.—¡O gallo! quiero que
sepas que quanto más niegas mi
petiçion tanto más augmentas en
mí el deseo de te lo oyr. Por lo
qual proçediendo en la costumbre
de nuestra buena conuersaçion y
tu graçioso dezir podras
començando luego ganar el
tienpo que se podria con la
dilaçion perder.
Gallo.—Agora, pues ansi quieres
y tanto me importunas yo te
quiero obedeçer: pero con vna
condiçion, que con juramento te
tengo de ligar á ella; y es que no
se ha de[859] publicar fuera de
aqui.
Miçilo.—Agora comiença, que yo
lo prometo, que no sea[860] más
publico por mí, ni seré causa que
otro lo sepa. Dime por orden
todas las cosas: qué fue la
causa[861] de la cena[862]: y qué
personas fueron alli en el combite,
y que passó en el suçeso.
Gallo.—Pues començando por el
prinçipio sabras que la causa fue
vna misa nueua: porque
Aristeneto cambiador, hombre
rico, tiene[863] vn hijo que se
llama[864] Zenon: hombre
estudioso y sabio, como sabes, el
qual[865] por tener ya edad
conueniente para elegir estado
vino a cantar misa y para esto el
padre de su parte combidó todos
sus parientes, vezinos y amigos,
juntamente con sus mugeres, y
Zenon[866] misa cantano de la
suya[867] llamó a todos sus
preceptores que auian sido de las
sçiençias, gramatica, logica,
philosophia y theologia, y
despues con estos combidó a
todos los curas y benefiçiados
casi desta çiudad que eran en
gran copia[868] y con estos auia
dos religiosos de cada orden.
Miçilo.—Yo nunca vi conpañia de
tanto santidad.
Gallo.—Pues viniendo al
proçeso del aconteçimiento[869]
sabras que el dia señalado que
fue vn domingo primero de mayo,
que es el mes más apacible y
graçioso a todos,[870] conuenimos
luego por la mañana todos los
combidados a casa de Aristeneto
para acompañar a Zenon hasta el
templo; fuemos con gran
çelebridad[871] de cançion de
clerigos, y gran musica de
instrumentos, laud, de arco, rabel,
vihuela, psalterio, y otras
agraciadas sonajas que tañian
hombres que para semejantes
autos se suelen alquilar. Quando
fue acabada aquella diuina
celebraçion de la missa, con el
orador que con ingenio discantó
el merito y grandeça de la
dignidad, y ofreçimos todos al
misa cantano, nos boluimos[872]
juntos con la mesma musica a
casa de Aristeneto. Donde
despedidos aquellos que solo
fueron conbidados para el
acompañamiento, se llegó
Aristineto a mi y a la oreja me
dixo que me quedasse a comer
allá[873] con él. Dios sabe quanto
me holgué, porque çierto que
sobraua en mi casa la raçion;
prinçipalmente porque despues
que en el templo ofreçi no fue
mucho lo que en la bolsa me
quedó. Fuemos lançados todos a
vn gran palaçio muy adornado y
dispuesto para el conbite. En el
qual auia dos messas a la larga
de la sala, la vna que yua a la vna
pared, y otra por otra. En la
frontera de la sala yua vna[874]
messa como cabeçera de las
otras dos, en la qual se sento en
el medio Zenon[875] tomando a su
mano derecha a su padre
Aristeneto; y a la izquierda[876]
estaua su padrino que era aquel
Cleodemo, antiguo y honrado
varon que fue cura del abogado
de las estrenas[877] San Julian.
Miçilo.—¡O qué monarcha y
prinçipe de saçerdotes me has
contado!
Gallo.—A los lados ocupauan
esta mesa de la cabeçera, a la
vna mano el guardian y
compañero de San Françisco y á
la otra el Prior de Sancto
Domingo con vn[878] conpañero
de grande[879] auctoridad. En la
mesa de la mano derecha se
sentaron[880] por orden los
maestros de Zenon y clereçia que
fuemos[881] muchos en numero; y
a la otra mano se sentaron los
casados, cada qual con su muger;
y quando fuemos todos sentados
començaronse las mesas a seruir
con grande abundançia de frutas
del tiempo.
Miçilo.—¿Pues entre los dos
perlados de San Françisco y
Sancto Domingo no uvo
differençia sobre la mano a que
cada qual se auia de sentar.
Gallo.—Mucho antes se consultó
con ellos y diffinió. Entre los dos
curas de Sanctesidro y San
Miguel uvo un poco de contienda;
porque preferiendo Aristeneto en
el asiento el de Sanctesidro al de
San Miguel por su mayor
antiguedad[882] se leuantó en pie
el de San Miguel porque era
preçeptor de Gramática y
presumia de philosopho y dixo: sy
a ti, Aristeneto, te pareçe que el
cura de Sanctesidro se ha de
preferir a mí, engañaste; y por no
lo consentir me voy y os dexo
libre el combite. Porque avnque él
sea viejo por dos razones se me
deue a mí la uentaja, pues dize
Salomon que canas muy antiguas
son[883] en el hombre el saber
quanto quiera que sea moço, y
ansi tomó por la mano su
mochacho y començó a fingir
querer caminar y luego el cura de
Sanctesidro dixo: nunca plega a
Dios que por mí dexes de te
holgar; y apartandose afuera le
hizo lugar en la delantera y él se
sentó[884] atrás.
Miçilo.—Conuenieron presto
esos dos por gozar.
Gallo.—Fue a todos ocasion de
gran risa, y no se podiendo[885]
sufrir Zenothemo maestro de
Philosophia[886] dixo en alta voz
ser aquello exemplo de la figura
Antiptosis isteron proteron[887] de
lo qual todos aduertiendo se
rieron mas[888].
Miçilo.—Pues entre los casados
¿no se ofrecio cosa que
pudiesses notar?
Gallo.—Los casados solamente
tenian ojo y atençion en aquellos
hombres sabios y religiosos, su
ambiçion, su puesto, hablar,
beber y comer y conuersaçion; en
fin, todos aquellos seglares se
fingian tener cuenta con el plato,
pero más la tenian con lo que
entre los clerigos passaua[889].
Porque como todos al prinçipio
començamos a comer de aquellos
sabrosos y bien aparejados
manjares, todos mirauamos al
cura de San Miguel que todo
quanto delante le seruian lo daua
al mochacho que tenía junto[890] a
si, pensando que ninguno lo via, y
el mochacho lo echaua en vna
talega. El comia con insaçiable
agonia y lançaua en los pechos y
fatriguera medias limas y
naranjas, y algunas guindas que
andauan rodando[891] por la
messa.
Daua a mochacho piernas de
perdiz y de pato; pedaços de vaca
y de carnero, y algunos suelos de
pasteles[892] y pedaços de pan y
torta. Diole pañizuelo, la copa en
que bebia; hasta el cuchillo y el
salero le dio. Desto reyan todos
los casados y sus mugeres, que
les era muy gran pasatiempo.
Estando pues todos ocupados en
esto con gran solaz y deleyte,
porque ya auia llegado de mano
en mano hasta la mesa de
Aristeneto y missa cantano que
mucho se reyan dello, suçedio
que entró por la puerta de la sala
Alçidamas cura de San Nicolas,
sin ser llamado, y puesto en
medio de todos[893] el rostro a
Zenon y a Aristeneto su padre
dixo: señores, perdonadme que
no vengo más temprano a vuestro
plazer porque agora disiendo la
misa mayor a mis perrochanos,
saliendo[894] a ofreçer en mi
iglesia me dixo vn feligres mio
que haciades esta fiesta; y ansi
luego me apresuré, que no tardé
en lo que restaua de la misa vn
momento; que casi no me
vagaua[895] desnudarme la
casulia por venir a honrraros por
ser tan vuestro amigo; que los
tales no emos de aguardar á ser
combidados, pero sin ser
llamados vengamos[896] de los
primeros.
Miçilo.—Por çierto cosa digna de
risa me cuentas.
Gallo.—Cada qual le començó a
dezir su donayre dando a
entender su desuerguença; pero
él lo disimuló por gozar del
combite; porque luego acudió
Aristeneto encareçiendo su buena
amistad y acusando su descuydo
y el de su hijo pues de combidarle
se auian oluidado, y ansi le
mandó dar vna silla y que se
sentasse en aquellas mesas
junto[897] aquellos hombres
reuerendos y honrrados[898].
Alçidamas era vn mançebo
grande, membrudo, robusto y de
grandes fuerças; y ansi como le
pussieron delante la silla
arroxandola [899] lexos de sí que
casi la quebrara[900] y diera con
ella al cura de Santispiritus y dixo
que las dueñas y hombres
regalados se auian de sentar a
comer en silla, que[901] vn hombre
moço y robusto como él, que por
alli queria comer passeandose; y
que si acaso se cansasse, que él
se sentaria en aquella tierra sobre
su capa. Respondiole Aristeneto:
anssi sea pues te plaze. Todo
esto hazia Alçidamas mostrando
querer regoçijar la fiesta y dar
plazer a los combidados
pensando él de si mesmo ser
graçioso fingiendose loco y
beodo; y ansi Alçidamas
rodeó[902] en pie[903] todas las
mesas mirando por los mejores
manjares, como lo hazen los
musicos chocarreros en los
combites de fiesta. Ansi comia
Alçidamas donde más le plazia si
via cosa que bien le pareçiesse a
su apetito, mezclandose con
aquellos que seruian las copas y
manjares, y como a las vezes se
aprouechasse de las copas que
estauan llenas en la messa, y
otras[904] vezes de las que
passauan en el seruicio,
hallauase beber doblado; y ansi
con el vino demasiado començó a
más salir de sí. Dezia maliçias y
atreuimientos en todos los que en
el combite estauan. A Hermon,
cura de Sancto Thome dixo que a
cabo de su vejez echasse la
mançeba de casa que tenía diez
años auia so color de moça; y a
Eucrito, cura de San Dionisio,
dixo que si pensaua lleuar al otro
mundo los çien ducados que tenía
dados a Aristeneto a cambio.
Mofaua de aquellas copas de
plata, mesas, sillas, tapiçes y
grande aparato llamando a
Aristeneto el gran[905] vsurero;
engrandeçiale con maliçia su
grande injenio y industria pues
por su buena soliçitud tenía por el
cambio[906] tan grande hazienda y
riquezas auiendo sido poco antes
muy pobre. Y Aristeneto ya
mohino y afrontado que
lastimauan los donayres mandó a
dos criados suyos que le
tomassen y echassen fuera de
casa y çerrassen las puertas
porque no los afrontasse más.
Pero como Alçidamas lo sintio
apartóse a vn lado y con vn vanco
que estaua vaçio juró que le
quebraria en la cabeça del que
llegasse; y ansi de consejo de
todos fue que agora le dexassen,
esperando tiempo más oportuno
para hazer la pressa neçesaria.
Pero de cada momento se fue
empeorando, diziendo injurias a
los frayles, y despues passando a
los casados los afrontaua
vituperandolos[907] en sus
mugeres; dijo delante del rico
Menedemo a su muger que quién
le auia dado más faldrillas,
Demócrito, cambiador, su amigo,
ó Menedemo su marido. De lo
qual la dama se afrontó mucho, y
Menedemo reçibio grande injuria;
y ansi Aristeneto, pensandolo
remediar y que le haria su amigo
mandole dar muy bien a beber,
por que pensó que ansi no le
afrontaria más y por esta causa
mandó a vn criado suyo[908] que
tomasse vna gran copa de vino
añejo y muy puro y se la diesse,
no pensando que fuera ocasion
de mayor mal, como fue. Pero
tomando Alçidamas el vaso con
ambas manos porque era grande
se boluio con él a la mesa de los
casados y en alta voz dijo, que
todos con silençio le quisieron
oyr: señora Magençia, muger de
nuestro huesped Aristeneto, y
madre de Zenon nuestro misa
cantano: yo bebo a ty, y mirad,
señora, que aueis de beber otro
tanto del vaso que yo bebiere so
pena que no lo cunpliendo no
ayas más hijo; y si lo cumplieres,
por la bendiçion de mi San
Nicholas, auras un hijo fuerte
gentil hombre sabio como yo; y
alçando la copa bebió della casi
vn azumbre y luego estendiendo
el braço la daua a Magençia
diziendo que si no bebia que

Textbook Markov Decision Processes in Practice 1St Edition Richard J Boucherie Ebook All Chapter PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Textbook Markov Decision Processes in Practice 1St Edition Richard J Boucherie Ebook All Chapter PDF

Uploaded by

Copyright:

Available Formats

Markov Decision Processes in Practice

1st Edition Richard J. Boucherie

Decision-Making in Veterinary Practice 1st Edition

Intelligent Decision Technology Support in Practice 1st

Understanding Pulmonary Pathology Applying Pathological

Assessment in counseling : practice and applications

Eric Hobsbawm: A Life in History Richard J. Evans

Dictionary of mathematical geosciences 1st Edition

Understanding Voltammetry Simulation Of Electrode

The Research Interview: Reflective Practice and

Associate Series Editor

Founding Series Editor

More information about this series at http://www.springer.com/series/6161

Markov Decision Processes

ISSN 0884-8289 ISSN 2214-7934 (electronic)

Library of Congress Control Number: 2017932096

© Springer International Publishing AG 2017

Printed on acid-free paper

This Springer imprint is published by Springer Nature

P. Dorreboom and his daughter –

Stanford University Frederick S. Hillier

Research Society, 44:1073–1096, 1993.

Methods and Applications. Kluwer, 2002.

The outline of these six parts is provided below.

Part I: General Theory

This part contains the following chapters:

6Warren B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality.

Part II: Healthcare

Part III: Transportation

Part IV: Production

Control of production systems is a well-known application area that is known to

Chapter 18: Wireless Channel Selection with Restless Bandits

Part VI: Financial Modeling

7 Michael A. H. Dempster and Stanley R. Pliska, editors. Mathematics of Derivative Securities.

Enschede, The Netherlands Richard J. Boucherie

Part I General Theory

1 One-Step Improvement Ideas and Computational Aspects . . . . . . . . . 3

2 Value Function Approximation in Complex Queueing Systems . . . . . 33

2.4 Application: Routing to Parallel Queues . . . . . . . . . . . . . . . . . . . . . . 47

3 Approximate Dynamic Programming by Practical Examples . . . . . . . 63

4 Server Optimization of Infinite Queueing Systems . . . . . . . . . . . . . . . . 103

4.6 Further Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5 Structures of Optimal Policies in MDPs with Unbounded Jumps:

6 Markov Decision Processes for Screening and Treatment of Chronic

6.4 MDP Model for Cardiovascular Risk Control in Patients with

7 Stratified Breast Cancer Follow-Up Using a Partially Observable

8 Advance Patient Appointment Scheduling . . . . . . . . . . . . . . . . . . . . . . . 245

8.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

9 Optimal Ambulance Dispatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

10 Blood Platelet Inventory Management . . . . . . . . . . . . . . . . . . . . . . . . . . 293

10.5 Case Study: Optimal Policy Around Breaks . . . . . . . . . . . . . . . . . . . 303

Part III Transportation

11 Stochastic Dynamic Programming for Noise Load Management . . . . 321

12 Allocation in a Vertical Rotary Car Park . . . . . . . . . . . . . . . . . . . . . . . . 337

13 Dynamic Control of Traffic Lights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

14 Smart Charging of Electric Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387

15 Analysis of a Stochastic Lot Scheduling Problem with Strict

15.4 The Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413

16 Optimal Fishery Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425

17 Near-Optimal Switching Strategies for a Tandem Queue . . . . . . . . . . . 439

18 Wireless Channel Selection with Restless Bandits . . . . . . . . . . . . . . . . . 463

19 Flexible Staffing for Call Centers with Non-stationary Arrival

20 MDP for Query-Based Wireless Sensor Networks . . . . . . . . . . . . . . . . 505