Professional Documents
Culture Documents
Sponsors
NSF, DARPA, ATD, BBN, Boeing, Cisco, Comverse, GDIS, Experian, Global MT,
Hughes, Kodak, Krones, Lockheed, Lucent, Microsoft, Mitre, Motorola, NASA, Nokia,
Nortel, OCI, Oresis, OTI, QNX, Raytheon, SAIC, Siemens SCR, Siemens MED,
Siemens ZT, Sprint, Telcordia, USENIX
Advanced ACE Tutorial Douglas C. Schmidt
APPLICATIONS
Observations
HARDWARE DEVICES
www.cs.wustl.edu/˜schmidt/PDF/
middleware-chapter.pdf
Vanderbilt University 1
Advanced ACE Tutorial Douglas C. Schmidt
CLIENT
Improve response-time
WORK
REQUEST – e.g., GUIs and network servers
WORK WORK WORK
CLIENT REQUEST REQUEST REQUEST
Simplify program structure
CLIENT CLIENT
Vanderbilt University 2
Advanced ACE Tutorial Douglas C. Schmidt
CD ROM
FILE
SYSTEM Performance ! multi-processing
and locality
!
(1) STAND-ALONE APPLICATION ARCHITECTURE
Reliability and availability
TIME
NAME
SERVICE
SERVICE CYCLE replication
SERVICE
DISPLAY
SERVICE
Scalability and portability !
FI LE modularity
NETWORK
!
SERVICE
PRINT
Extensibility dynamic
SERVICE configuration and reconfiguration
CD ROM
Vanderbilt University 3
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 4
Advanced ACE Tutorial Douglas C. Schmidt
Caveats
OO is not a panacea
– Though when used properly it helps minimize “accidental”
complexity and improve software quality factors
It’s also essential to understand advanced OS features to enhance
functionality and performance, e.g.,
– Multi-threading
– Multi-processing
– Synchronization
– Shared memory
– Explicit dynamic linking
– Communication protocols and IPC mechanisms
Vanderbilt University 5
Advanced ACE Tutorial Douglas C. Schmidt
Tutorial Outline
Vanderbilt University 6
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 7
Advanced ACE Tutorial Douglas C. Schmidt
Sources of Complexity
PRINTER
Inherent complexity
COMPUTER
– Latency
FILE
CD ROM SYSTEM – Reliability
– Synchronization
(1) STAND-ALONE APPLICATION ARCHITECTURE
– Deadlock
TIME
NAME
SERVICE
SERVICE CYCLE
SERVICE Accidental Complexity
DISPLAY
SERVICE
– Low-level APIs
FI LE
NETWORK SERVICE – Poor debugging tools
PRINT – Algorithmic
SERVICE
CD ROM
decomposition
PRINTER FILE SYSTEM
– Continuous
(2) DISTRIBUTED APPLICATION ARCHITECTURE re-invention
Vanderbilt University 8
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 9
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 10
Advanced ACE Tutorial Douglas C. Schmidt
OO Contributions to Concurrent
and Distributed Applications
Concurrent network programming is Patterns and frameworks elevate
traditionally performed using development level to focus on
low-level OS mechanisms, e.g., application concerns, e.g.,
fork/exec Service functionality and
policies
Shared memory and semaphores
Service configuration
Memory-mapped files
Concurrent event
Signals
demultiplexing and event
sockets/select handler dispatching
Low-level thread APIs Service concurrency and
synchronization
Vanderbilt University 11
Advanced ACE Tutorial Douglas C. Schmidt
Overview of Patterns
Vanderbilt University 12
Advanced ACE Tutorial Do
1: METHOD CALL
: BROKER
: QUOTER
PROXY
4: METHOD RETURN
2: FORWARD REQUEST
3: RESPONSE
CLIENT
NETWORK : QUOTER
SERVER
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
A framework is:
– “An integrated collection of components that collaborate to
produce a reusable architecture for a family of related applications”
Frameworks differ from conventional class libraries:
1. Frameworks are “semi-complete” applications
2. Frameworks address a particular application domain
3. Frameworks provide “inversion of control”
Frameworks facilitate reuse of successful networked application
software designs and implementations
– Applications inherit from and instantiate framework components
Vanderbilt University 14
Advanced ACE Tutorial Douglas C. Schmidt
EVENT GLUE
GUI NETWORK
– Reusable building blocks
CLASSES
CODE IPC
LOOP
CLASSES – Domain-independent
(A) CLASS LIBRARY ARCHITECTURE – Limited in scope
– Passive
EVENT
NETWORKING
Frameworks
LOOP
ADT
CLASSES GUI
APPLICATION-
CALL – Reusable, “semi-complete”
INVOKES SPECIFIC BACKS
FUNCTIONALITY EVENT applications
MATH LOOP
CLASSES – Domain-specific
DATABASE
EVENT
LOOP
– Broader in scope
(B) FRAMEWORK ARCHITECTURE
– Active
Vanderbilt University 15
Advanced ACE Tutorial Douglas C. Schmidt
SERVICE CORBA
FRAMEWORK ACCEPTOR CONNECTOR
HANDLER HANDLER
LAYER
PROCESS/ STREAMS
C++ THREAD LOG SHARED
SERVICE
WRAPPER MANAGERS MSG CONFIG- MALLOC
FACADE REACTOR/
PROACTOR
URATOR
LAYER SYNCH SPIPE SOCK SAP/ FIFO MEM FILE
WRAPPERS SAP TLI SAP SAP MAP SAP
OS ADAPTATION LAYER
C PROCESSES/ WIN32 NAMED SOCKETS/ UNIX SELECT/ DYNAMIC SHARED FILE SYS
PIPES & UNIX
APIS THREADS STREAM PIPES TLI FIFOS IO COMP LINKING MEMORY APIS
www.cs.wustl.edu/˜schmidt/ACE.html
Vanderbilt University 16
Advanced ACE Tutorial Douglas C. Schmidt
ACE Statistics
Vanderbilt University 17
Advanced ACE Tutorial Douglas C. Schmidt
Acceptor-
Reactor Proactor
Connector
Service
Streams Task
Configurator
Vanderbilt University 18
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 19
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 20
Advanced ACE Tutorial Douglas C. Schmidt
TAO Statistics
Vanderbilt University 21
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 22
Advanced ACE Tutorial Douglas C. Schmidt
Java ACE
DISTRIBUTED
SERVICES AND
Java ACE
TOKEN LOGGING NAME TIME
COMPONENTS SERVER SERVER SERVER SERVER Overview
FRAMEWORKS
AND CLASS SERVICE
A Java version
ACCEPTOR CONNECTOR
CATEGORIES HANDLER
of ACE
ADAPTIVE SERVICE EXECUTIVE (ASX)
JAVA SYNCH
SOCK_SAP
THREAD LOG TIMER SERVICE – Used for
WRAPPERS WRAPPERS MANAGER CONFIGURATOR
MSG QUEUE
medical
JAVA VIRTUAL MACHINE (JVM) imaging
www.cs.wustl.edu/˜schmidt/JACE.html prototype
www.cs.wustl.edu/˜schmidt/C++2java.
html
www.cs.wustl.edu/˜schmidt/PDF/
MedJava.pdf
Vanderbilt University 23
Advanced ACE Tutorial Do
P1 CONSOLE
CLIENT
LOCAL IPC LOGGING
P2 DAEMON
RE
MO
P3 TE
IPC SERVER LOGGING
DAEMON
HA
OST A B B
HOST
HOST
CLIENT SERVER
A
IPC
TE
MO
NETWORK
RE
STORAGE
DEVICE
HOST P1
CLIENT
B CLIENT
LOCAL IPC LOGGING
P2 DAEMON
P3
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
and
ACE_DEBUG ((LM_DEBUG,
"(%t) sending to server %s", server_host));
Vanderbilt University 25
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 26
Advanced ACE Tutorial Do
maxhandlep1
acceptor
NETWORK read_handles
CLIENT CONNECTION
REQUEST
LOGGING
RECORDS LOGGING LOGGING
RECORDS RECORDS
CLIENT
CLIENT CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
for (;;) {
// struct assignment.
ready_handles = activity_handles;
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
do {
// Beware of subtle bug(s) here...
h = accept (acceptor, 0, 0);
FD_SET (h, &activity_handles);
Vanderbilt University
Advanced ACE Tutorial Do
for (;;) {
// ... get logging records from client
// application processes ...
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 36
Advanced ACE Tutorial Douglas C. Schmidt
return n <= 0 ? -1 : 0;
}
Vanderbilt University 37
Advanced ACE Tutorial Douglas C. Schmidt
OO Contributions to Software
Vanderbilt University 38
Advanced ACE Tutorial Douglas C. Schmidt
STRATEGIC Reactor
PATTERNS
TACTICAL PATTERNS
Strategic and tactical are relative to the context and abstraction level
Vanderbilt University 39
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 40
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 41
Advanced ACE Tutorial Do
Logging_Handler SOCK_Stream
COMPONENTS
APPLICATION-
SOCK_Acceptor Null_Synch
SPECIFIC
1
Logging n
Logging
Acceptor <<activates>> Handler
COMPONENTS
CONNECTION-
SVC_HANDLER PEER_STREAM
ORIENTED
PEER_ACCEPTOR SYNCH_STRAT
Acceptor Svc
Handler
PEER PEER
ACCEPTOR STREAM
COMPONENTS
FRAMEWORK
Stream
Connection
ACE
IPC_SAP
Service
Configurator
Concurrency Reactor
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– Building distributed applications using low-level APIs is hard
Forces
– Low-level APIs are verbose, tedious, and error-prone to program
– Low-level APIs are non-portable and non-maintainable
Solution
– Apply the Wrapper Facade pattern to encapsulate low-level
functions and data structures
Vanderbilt University 43
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 44
Advanced ACE Tutorial Do
2: ACTIVE
ROLE
1: PASSIVE
ROLE
socket()
bind() socket()
(optional) bind()
connect() listen()
send()/recv() accept()
3: SERVICE send()/recv()
close()
PROCESSING
close()
CLIENT
NETWORK SERVER
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 46
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 47
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 48
Advanced ACE Tutorial Douglas C. Schmidt
Problem with Sockets: Poorly Structured
gethostbyname()
getservbyname()
getsockname()
getpeername()
getsockopt()
setsockopt()
sendmsg()
recvfrom()
recvmsg()
connect()
sendto()
socket()
accept()
writev()
readv()
listen()
write()
send()
bind()
read()
recv()
Limitations
Socket API is linear rather than hierarchical
There is no consistency among names...
Non-portable
Vanderbilt University 49
Advanced ACE Tutorial Do
Socket Taxonomy
N/ COMMUNICATION DOMAIN
T IO ON
C TI LOCAL/REMOTE
N NE ICA S IVE LOCAL
CO MU LE N S
M O PA
CO R ER
XF
E
C TIV
A
TYPE OF COMMUNICATION SERVICE
socket(PF_UNIX)/bind() socket(PF_INET)/bind()
GRAM
DATA
sendto()/recvfrom() sendto()/recvfrom()
socket(PF_UNIX)/bind() socket(PF_INET)/bind()
STREAM CONNECTED
DATAGRAM
socket(PF_UNIX) socket(PF_INET)
bind()/connect() bind()/connect()
send()/recv() send()/recv()
socket(PF_UNIX) socket(PF_INET)
bind()/connect() bind()/connect()
socket(PF_UNIX) socket(PF_INET)
bind()/listen()/accept() bind()/listen()/accept()
send()/recv() send()/recv()
socket(PF_UNIX) socket(PF_INET)
bind()/connect() bind()/connect()
1. Connection role
2. Communication domain
3. Type of service
Vanderbilt University
Advanced ACE Tutorial Do
gram
m D
gra K_
GRAM
DATA
D C
K_ SO
OC SOCK_Dgram_Bcast
LS SOCK_Dgram_Mcast
CONNECTED
DATAGRAM
LSOCK_Dgram SOCK_Dgram
LSOCK_CODgram SOCK_CODgram
LSOCK_Acceptor SOCK_Acceptor
STREAM
LSOCK_Stream SOCK_Stream
LSOCK_Connector SOCK_Connector
Vanderbilt University
Advanced ACE Tutorial Do
ACE_SOCK_IO ACE_SOCK
ACE_SOCK_Acceptor ACE_INET_Addr
ACE_SOCK_Stream ACE_SOCK_Connector
Participants
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 53
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 54
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 55
Advanced ACE Tutorial Do
for (;;) {
char buf[BUFSIZ];
// Error caught at compile time!
ssize_t n =
acceptor.recv (buf, sizeof buf);
new_stream.send_n (buf, n);
}
}
Vanderbilt University
Advanced ACE Tutorial Do
for (;;) {
char buf[BUFSIZ];
ssize_t n =
stream.recv (buf, sizeof buf);
stream.send_n (buf, n);
}
}
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
A
ACE
IPC
SAP
C++NPv1 (www.cs.wustl.edu/˜schmidt/ACE/book1/)
Vanderbilt University 58
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
for (;;) {
// object assignment.
ready_handles = activity_handles;
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
OO Client Logging
Daemon Implementation
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 64
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 65
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 66
Advanced ACE Tutorial Douglas C. Schmidt
ACE_SOCK_Acceptor acceptor;
// ...
ACE_Handle_Set ready_handles;
// ...
Vanderbilt University 67
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 68
Advanced ACE Tutorial Do
acceptor.accept (stream);
stream.send_handle (stream.get_handle ());
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 70
Advanced ACE Tutorial Douglas C. Schmidt
ACE ACE
IPC SOCK
SAP
Vanderbilt University 71
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 72
Advanced ACE Tutorial Douglas C. Schmidt
Inlining is time and space efficient since key methods are very short:
class ACE_SOCK_Stream : public ACE_SOCK
{
public:
ssize_t send (const void *buf, size_t n)
{
return ACE_OS::send (this->get_handle (), buf, n);
}
Vanderbilt University 73
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 74
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– The logging server must process several different types of events
simultaneously from different sources of events
Forces
– Multi-threading is not always available
– Multi-threading is not always efficient
– Multi-threading can be error-prone
– Tightly coupling event demuxing with server-specific logic is
inflexible
Solution
– Use the Reactor pattern to decouple event demuxing/dispatching
from server-specific processing
Vanderbilt University 75
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 76
Advanced ACE Tutorial Douglas C. Schmidt
INITIALIZE
register_handler(callback) Also note how
MODE
REGISTER HANDLER
get_handle() long-running
EXTRACT HANDLE
handle_events() event handlers
START EVENT LOOP
select()
can degrade
FOREACH EVENT DO
handle_input()
quality of service
EVENT HANDLING
handle_signal() thread of
SIGNAL ARRIVES
handle_timeout() control...
TIMER EXPIRES
remove_handler(callback)
REMOVE HANDLER
CLEANUP
handle_close()
Vanderbilt University 77
Advanced ACE Tutorial Do
ACE_Timer_Queue ACE_Event_Handler
0..1
ACE_Reactor
Application Event
ACE_Time_Value
Handler
TOKEN
ACE_Select_Reactor_Impl ACE_Select_Reactor_T
«bind»
<ACE_Select_Reactor_Token>
ACE_TP_Reactor ACE_Select_Reactor
Vanderbilt University
Advanced ACE Tutorial Do
:Timer : Handle
Queue Table : Reactor
Benefits Liabilities
Straightforward to Callbacks are “brittle”
program
Can’t leverage
Concurrency control multi-processors
is easy
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 80
Advanced ACE Tutorial Do
APPLICATION-
DEFINED
APPLICATION-
INDEPENDENT
Reactor
www.cs.wustl.edu/˜schmidt/POSA/
Intent of Acceptor Role Forces resolved
Decouple the passive Reuse passive
connection and connection setup
initialization of a peer and service
service in a distributed initialization code
system from the Ensure that
processing performed acceptor-mode
once the peer service is handles aren’t used
connected and to read/write data
initialized
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Event_Handler SYNCH_STRATEGY
SVC_HANDLER,
PEER_CONNECTOR ACE_Task
ACE_Connector
PEER_STREAM,
SYNCH_STRATEGY
SVC_HANDLER,
PEER_ACCEPTOR ACE_Svc_Handler
ACE_Acceptor
«bind»
Application
Service
Framework characteristics
Vanderbilt University
Advanced ACE Tutorial Do
: Logging : Logging
Handler Handler : Logging
: Logging
Handler
Acceptor
: Svc : Svc
Handler Handler : Svc
: Acceptor Handler
: Logging
PASSIVE ACTIVE Handler
LISTENER CONNECTIONS
1: handle_input() : Svc
2: sh = make_svc_handler() Handler
3: accept_svc_handler(sh) : Reactor
4: activate_svc_handler(sh)
Vanderbilt University
Advanced ACE Tutorial Do
protected:
// Factory method creates a service handler.
virtual SVC_HANDLER *make_svc_handler (void);
// Accept a new connection.
virtual int accept_svc_handler (SVC_HANDLER *);
// Activate a service handler.
virtual int activate_svc_handler (SVC_HANDLER *);
private:
// Acceptor IPC connection strategy.
PEER_ACCEPTOR peer_acceptor_;
};
Vanderbilt University
Advanced ACE Tutorial Do
// Shorthand names.
#define SH SVC_HANDLER
#define PA PEER_ACCEPTOR
accept_svc_handler (svc_handler);
activate_svc_handler (svc_handler);
}
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 86
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 87
Advanced ACE Tutorial Douglas C. Schmidt
Context
STRATEGY
Strategy Define a family of
context_interface() algorithm_interface() algorithms, encapsulate
each one, and make
them interchangeable
Gamma et al., Design
Concrete Concrete Patterns: Elements of
Strategy A Strategy C Reusable Object-Oriented
algorithm_interface() algorithm_interface() Software AW, ’94
Concrete
Strategy B
algorithm_interface()
Vanderbilt University 88
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 89
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Acceptor Initialization
Implementation
Vanderbilt University
Advanced ACE Tutorial Do
// ...
private:
PEER_STREAM peer_; // IPC mechanism.
virtual ˜ACE_Svc_Handler (void);
};
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
ACE_Svc_Handler Implementation
#define PS PEER_STREAM By default, a
#define SS SYNCH_STRAT ACE_Svc_Handler
template <class PS, class SS>
object is registered
ACE_Svc_Handler<PS, SS>::ACE_Svc_Handler with the singleton
(ACE_Reactor *r): ACE_Service_Object (r) ACE_Reactor
{}
template <class PS, class SS> – This makes the
int ACE_Svc_Handler<PS, SS>::open service “reactive”
(void *) { so that no other
// Enable non-blocking I/O.
peer ().enable (ACE_NONBLOCK); synchronization
mechanisms are
// Register handler with the Reactor. necessary
reactor ()->register_handler
(this, ACE_Event_Handler::READ_MASK);
}
Vanderbilt University 93
Advanced ACE Tutorial Do
SERVER
LOGGING Service
DAEMON Config
Logging
Acceptor Service
Repository
Logging
Handler Service
Manager
Logging
Handler Reactor
LOGGING
RECORDS CONNECTION SERVER REMOTE
REQUEST CONTROL
OPERATIONS
CLIENT
CLIENT CLIENT
CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
// Logging_Handler factory.
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 96
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University 97
Advanced ACE Tutorial Do
Logging_Acceptor Initialization
and Termination
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– Prematurely committing ourselves to a particular logging server
configuration is inflexible and inefficient
Forces
– It is useful to build systems by “scripting” components
– Certain design decisions can’t be made efficiently until run-time
– It is a bad idea to force users to “pay” for components they do not
use
Solution
– Use the Component Configurator pattern to assemble the desired
logging server components dynamically
Vanderbilt University 99
Advanced ACE Tutorial Douglas C. Schmidt
Intent
LAYER
Concrete
Component Decouples the
implementation of services
from the time when they are
Component
Component Config
configured
CONFIGURATION
suspend() 1 1
resume() n Forces Resolved
LAYER
init() A
fini() 1 1
info() Reduce resource utilization
Component
Repository
Support dynamic
(re)configuration
REACTIVE
LAYER
Event www.cs.wustl.edu/
Handler n 1 Reactor
˜schmidt/POSA/
ACE_Event_Handler ACE_Service_Config
ACE_Service_Object ACE_Service_Repository
Framework characteristics
Vanderbilt University
Advanced ACE Tutorial Do
SERVICE Thread
CONFIGURATOR
Logger
RUNTIME
Reactive
Logger Service
Object
Service Service DLLS
Repository Object
Thread
Pool
Logger
Service
Reactor Service
Config
Object
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Service Configuration
INITIALIZED
IDLE
START EVENT LOOP/
Reactor::run_event_loop()
PERFORM
RECONFIGURE/ CALLBACK
Service_Config::process_directives()
CALL HANDLER/
Event_Handler::handle_input()
Concurrent OO Logging
Server Architecture
1: SOCK
Acceptor
2: accept()
4: handle_input()
5: spawn() 7: recv() 7: recv()
8: write() 8: write()
Logging
Acceptor Logging Logging
Reactor Handler Handler
3: connect()
6: send()
6: send()
CLIENT CLIENT
A NETWORK B
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Concurrency Overview
A thread is a sequence of
SHARED ADDRESS SPACE
instructions executed in one
or more processes
– One process !
THREADS
stand-alone systems
– More than one process !
distributed systems
PROCESS
Modern OS Concurrency
Lightweight Concurrency
Modern operating systems provide lightweight mechanisms that
manage and synchronize multiple threads within a process
– Some systems also allow threads to synchronize across multiple
processes
Benefits of threads
1. Relatively simple and efficient to create, control, synchronize, and
collaborate
– Threads share many process resources by default
2. Improve performance by overlapping computation and
communication
– Threads may also consume less resources than processes
3. Improve program structure
– e.g., compared with using asynchronous I/O
KERNEL
USER
REQUEST
KERNEL
USER
SERVICE
THREAD EXECUTES
BLOCKED
RESPONSE
SINGLE-
THREADED RPC
KERNEL
USER
SERVER
KERNEL
USER
REQUEST
SERVICE
EXECUTES
RESPONSE
SERVER
KERNEL
REQUEST
USER
RESPONSE
SERVICE
EXECUTES
MULTI-
CLIENT THREADED RPC
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
abstractions
LWP LWP LWP LWP LWP LWP LWP 1. Application
threads
2. Lightweight
KERNEL-LEVEL
processes
PE PE PE PE PE PE PE PE
3. Kernel threads
SHARED MEMORY 4. Processing
elements
PE LWP
UNIX
PROCESSING LIGHTWEIGHT PROCESS
THREAD
ELEMENT PROCESS
Application Threads
Most process resources are Each thread also contains unique
equally accessible to all threads information, e.g.,
in a process, e.g.,
Identifier
Virtual memory
Register set (e.g., PC and SP)
User permissions and access
Run-time stack
control privileges
Signal mask
Open files
Priority
Signal handlers
Thread-specific data (e.g.,
errno)
Thread MUTEX
ADVANCED SYNCH LOCK
TYPE
TYPE Condition
Atomic
Token Barrier Op TSS
GUARDS
SYNCH WRAPPERS
RW Semaphore Guard
Mutex Null
Mutex Mutex
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– Multi-threaded logging servers may be necessary when
single-threaded reactive servers inefficient, non-scalable, or
non-robust
Forces
– Multi-threading can be very hard to program
– No single multi-threading model is always optimal
Solution
– Use the Active Object pattern to allow multiple concurrent logging
server operations using an OO programming style
Vanderbilt University
Advanced ACE Tutorial Do
t3 :
Task : TASK
t1 : STATE
Task : TASK
STATE
: Message
Queue
: Message
Queue ACTIVE
ACTIVE
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
ACE_Event_Handler ACE_Service_Object
SYNCH
0..1 *
ACE_Thread_Manager ACE_Task
SYNCH
ACE_Message_Block ACE_Message_Queue
* 1
Framework characteristics
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
SOCK_Stream
APPLICATION-
SOCK_Acceptor NULL_Synch
SPECIFIC
1 Thr n Thr
Logging Logging
<<activates>>
Acceptor Handler
SVC_HANDLER
COMPONENTS
CONNECTION-
PEER_ACCEPTOR PEER_STREAM
ORIENTED
SYNCH
Acceptor Svc
Handler
PEER PEER
ACCEPTOR STREAM
COMPONENTS
FRAMEWORK
Stream
Connection
ACE
IPC_SAP
Service
Configurator
Concurrency Reactor
global
Vanderbilt University
Advanced ACE Tutorial Do
Thr_Logging_Acceptor and
Thr_Logging_Handler Interfaces
Vanderbilt University
Advanced ACE Tutorial Do
Thr_Logging_Handler
Implementation
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
// ...
handle_log_record (ACE_HANDLE in_h, ACE_HANDLE out_h)
{
// in method scope ...
mutex_lock (&lock);
if (ACE_OS::write (out_h, lr.buf, lr.size) == -1)
return -1;
mutex_unlock (&lock);
// ...
}
Solution: Synchronization
Wrapper Facades
class ACE_Thread_Mutex
{
public:
ACE_Thread_Mutex (void) {
mutex_init (&lock_, USYNCH_THREAD, 0);
}
˜ACE_Thread_Mutex (void) { mutex_destroy (&lock_);
int acquire (void) { return mutex_lock (&lock_); }
int tryacquire (void)
{ return mutex_trylock (&lock_); }
int release (void) { return mutex_unlock (&lock_);
private:
// SunOS 5.x serialization mechanism.
mutex_t lock_;
void operator= (const ACE_Thread_Mutex &);
ACE_Thread_Mutex (const ACE_Thread_Mutex &);
};
Vanderbilt University
Advanced ACE Tutorial Do
Porting ACE_Thread_Mutex to
Windows NT
class ACE_Thread_Mutex
{
public:
ACE_Thread_Mutex (void) {
lock_ = CreateMutex (0, FALSE, 0);
}
˜ACE_Thread_Mutex (void) {
CloseHandle (lock_);
}
int acquire (void) {
return WaitForSingleObject (lock_, INFINITE);
}
int tryacquire (void) {
return WaitForSingleObject (lock_, 0);
}
int release (void) {
return ReleaseMutex (lock_);
}
private:
ACE_HANDLE lock_; // Windows locking mechanism.
// ...
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
// ...
handle_log_record (ACE_HANDLE in_h, ACE_HANDLE out_h) {
// in method scope ...
lock.acquire ();
if (ACE_OS::write (out_h, lr.buf, lr.size) == -1)
return -1;
lock.release ();
// ...
private:
LOCK &lock_;
}
Thread-safe handle_log_record()
Function
template <class LOCK = ACE_Thread_Mutex> ssize_t
handle_log_record (ACE_HANDLE in, ACE_HANDLE out) {
// beware static initialization...
static LOCK lock;
ACE_UINT_32 len;
ACE_Log_Record lr;
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Remaining Caveats
Transparently Parameterizing
Synchronization Using C++
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
1: GET ~schmidt
WWW HTTP/1.0 WWW
CLIENT SERVER
2: index.html
PROTOCOL
HANDLERS
HTML
PARSER
GUI
DISPATCHER
REQUESTER
OS KERNEL OS KERNEL
OS I/O SUBSYSTEM OS I/O SUBSYSTEM
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
// Print diagnostic
cout << "got new request"
<< ACE_OS::thr_self ()
<< endl;
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– A naive implementation of MESSAGE_QUEUE will lead to race
conditions
e.g., when messages in different threads are enqueued and
dequeued concurrently
Forces
– Producer/consumer concurrency is common, but requires careful
attention to avoid overhead, deadlock, and proper control
Solution
– Utilize the Monitor Object pattern and condition variables
# rd_ptr_ : size_t
# wr_ptr_ : size_t
# cont_ : ACE_Message_Block *
# next_ : ACE_Message_Block *
# prev_ : ACE_Message_Block *
# data_block_ : ACE_Data_Block *
Class characteristics
Hide messaging implementations from clients
ACE_Message ACE_Message ACE_Message
_Block _Block _Block
cont() cont() cont()
data_block() data_block() data_block()
wr_ptr() wr_ptr() rd_ptr()
rd_ptr() rd_ptr() wr_ptr()
ACE_Data
_Block ACE_Data_Block
PAYLOAD reference_count() = 2
Vanderbilt University
Advanced ACE Tutorial Do
Class characteristics
Vanderbilt University
Advanced ACE Tutorial Do
// Initialize a Message_Queue.
Message_Queue (size_t hwm = DEFAULT_HWM,
size_t lwm = DEFAULT_LWM);
// Check if full or empty (hold locks)
int is_empty (void) const;
int is_full (void) const;
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
private:
// Check boundary conditions & don’t hold locks.
int is_empty_i (void) const;
int is_full_i (void) const;
size_t high_water_mark_;
size_t low_water_mark_;
size_t cur_bytes_;
size_t cur_count_;
};
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
ACE_Message_Queue Class
Implementation
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
ACE_Message_Queue Operations
template <class SYNCH_STRAT> int template <class SYNCH_STRAT> int
ACE_Message_Queue<SYNCH_STRAT>:: ACE_Message_Queue<SYNCH_STRAT>::
enqueue_tail (ACE_Message_Block *item, dequeue_head (ACE_Message_Block *&item,
ACE_Time_Value *tv) { ACE_Time_Value *tv) {
ACE_GUARD_RETURN (SYNCH_STRAT::MUTEX, ACE_GUARD_RETURN (SYNCH_STRAT::MUTEX,
guard, lock_, -1); guard, lock_, -1);
// Wait while the queue is full. // Wait while the queue is empty.
while (is_full_i ()) { while (is_empty_i ()) {
// Release the <lock_> and wait // Release lock_ and wait for timeout,
// for timeout, signal, or space // signal, or a new message being
// to become available in the list. // placed in the list.
if (not_full_cond_.wait (tv) == -1) if (not_empty_cond_.wait (tv) == -1)
return -1; return -1;
} }
// Actually enqueue the message at // Actually dequeue the first message.
// the end of the list. dequeue_head_i (item);
enqueue_tail_i (item);
// Tell blocked threads that list
// Tell blocked threads that // is no longer full.
// list has a new item! if (cur_bytes_ <= low_water_mark_)
not_empty_cond_.signal (); not_full_cond_.signal ();
} }
Problem
– A very strategic design decision for high-performance Web
servers is selecting an efficient concurrency architecture
Forces
– No single concurrency architecture is optimal
– Key factors include OS/hardware platform and workload
Solution
– Understand key alternative concurrency patterns
2: HANDLE INPUT
HTTP 3: CREATE HANDLER
Handler 4: ACCEPT CONNECTION
5: ACTIVATE HANDLER
HTTP
HTTP Acceptor
Handler Reactor
SERVER
1: CONNECT
CLIENT
CLIENT CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
SERVER
1: CONNECT
CLIENT
CLIENT CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
CLIENT CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
HTTP Event
Dispatcher
Handler
2: ACCEPT CONNECTION
HTTP 3: MORPH INTO HANDLER
Handler HTTP
HTTP Acceptor
HTTP
Handler Acceptor
CLIENT CLIENT
Vanderbilt University
Advanced ACE Tutorial Do
Half-Sync/Half-Async Synchronous
Thread Pool Web Server
2: HANDLE INPUT
Message 3: ENQUEUE REQUEST
Queue
Active HTTP
Object Handler HTTP
4: DEQUEUE & HTTP Handler
PROCESS
Active REQUEST Handler
Object HTTP
Acceptor
Active
Active
Object
Object Reactor
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
HTTP HTTP
Handler Handler
Sock Sock
HTTP Stream Stream
Handler HTTP
Acceptor
Sock
Stream Event Sock
Dispatcher Acceptor
Event Dispatcher
– Encapsulates Web server concurrency and
dispatching strategies
HTTP Handlers
– Parses HTTP headers and processes
requests
HTTP Acceptor
– Accepts connections and creates HTTP
Handlers
Vanderbilt University
Advanced ACE Tutorial Do
Thread-per Thread
Pool Acceptor
Request
Active Connector
Thread-per Object
Connection Component
Half-Sync/ Configurator
Half-Async
Asynchronous
Completion Reactor/ Double
Token Proactor Checked
Locking
STRATEGIC PATTERNS
TACTICAL PATTERNS
Wrapper Abstract
Facade Strategy Adapter Singleton
Factory
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
svc_run
svc_run svc_run
svc_run
www.cs.wustl.edu/˜schmidt/PDF/HPL.pdf
Vanderbilt University
Advanced ACE Tutorial Do
An Integrated Reactive/Active
Web Server
svc_run
REGISTERED svc_run
OBJECTS svc_run
4: getq(msg)
5:svc(msg)
HTTP
APPLICATION
Handler
HTTP
LEVEL
HandlerHTTP
Event Handler HTTP
HandlerEvent Processor HTTP
HandlerEvent Acceptor
Handler 2: recv_request(msg)
3: putq(msg) Event
Handler
FRAMEWORK
1: handle_input()
LEVEL
: Handle
Table
: Reactor
OS EVENT DEMULTIPLEXING INTERFACE
KERNEL
LEVEL
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Integrating Multi-threading
Problem
– Multi-threaded Web servers are needed since Reactive Web
servers are often inefficient and non-robust
Forces
– Multi-threading can be very hard to program
– No single multi-threading model is always optimal
Solution
– Use the Active Object pattern to allow multiple concurrent server
operations in an OO-manner
Handler
HTTP
LEVEL
Handler
HTTP
Event Handler HTTP
Handler
Event Processor HTTP
HandlerEvent Acceptor
Handler 2: recv_request(msg)
3: putq(msg) Event
Handler
FRAMEWORK
1: handle_input()
LEVEL
: Handle
Table
: Reactor
OS EVENT DEMULTIPLEXING INTERFACE
KERNEL
LEVEL
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
HTTP_Processor *
HTTP_Processor::instance (void)
{
// Beware of race conditions!
if (instance_ == 0)
// Create the Singleton "on-demand."
instance_ = new HTTP_Processor;
return instance_;
}
HTTP_Processor::HTTP_Processor (void)
{
// Inherited from class Task.
activate (THR_BOUND,
Options::instance ()->threads ());
}
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Problem
– The canonical Singleton implementation has subtle “bugs” in
multi-threaded applications
Forces
– Too much locking makes Singleton too slow...
– Too little locking makes Singleton unsafe...
Solution
– Use the Double-Checked Locking optimization pattern to minimize
locking and ensure atomic initialization
www.cs.wustl.edu/ Caveat!
˜schmidt/POSA/
This pattern assumes atomic
memory access
Problem
– Justifying the hybrid design of our Web server can be tricky
Forces
– Engineers are never satisfied with the status quo ;-)
– Substantial amount of time is spent re-discovering the intent of
complex concurrent software design
Solution
– Use the Half-Sync/Half-Async pattern to explain and justify our
Web server concurrency architecture
SYNC Intent
TASK LAYER
SYNC
TASK 1
TASK 3
ASYNC
TASK
2: interrupt
Ensure efficient I/O
EXTERNAL
EVENT SOURCES www.cs.wustl.edu/
˜schmidt/POSA/
svc_run
SYNC TASK
svc_run
LEVEL
svc_run
4: getq(msg)
5:svc(msg)
QUEUEING
LEVEL
HTTP
Processor
HTTP
Handler
HTTP HTTP
HandlerHTTP Acceptor
Event Handler
HandlerEvent Event
HandlerEvent Handler
Handler
ASYNC TASK
1: handle_input()
: Handle
LEVEL
2: recv_request(msg)
3: putq(msg) Table
: Reactor
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
ACE_Handler
ACE_Asynch_Read_Stream ACE_Asynch_Write_Stream
ACE_Asynch_Result
Framework characteristics
Vanderbilt University
Advanced ACE Tutorial Do
Completion Operating
Dispatcher System
3: handle 5: accept
events complete
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
: HTTP : HTTP
Handler Handler : HTTP
: HTTP
Handler
Acceptor
: Svc : Svc
Handler Handler : Svc
: Acceptor Handler
: HTTP
PASSIVE ACTIVE Handler
LISTENER CONNECTIONS
1: handle_input() : Svc
2: sh = make_svc_handler() Handler
3: accept_svc_handler(sh) : Reactor
4: activate_svc_handler(sh)
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Acceptor
Strategy
Handler Event Dispatcher
Use “gather-write”
Protocol mechanisms
Filter
Minimize logging
Adapter Pre-compute HTTP
Active Object
responses
Protocol Pipeline
Concurrency
Strategy
Avoid excessive
Framework Framework time() calls
Streams Service Configurator Optimize the
www.cs.wustl.edu/˜jxh/research/ transport interface
GROUND
STATION
PEERS
OO Software Architecture
of the Gateway
Routing
Consumer Table Supplier
Handler Handler
Reactor
Consumer
Supplier Handler
Handler Connector Acceptor
INCOMING OUTGOING
MESSAGES GATEWAY
MESSAGES
CONNECTION CONNECTION
REQUEST REQUEST
www.cs.wustl.edu/˜schmidt/PDF/
TAPOS-00.pdf
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Gateway Behavior
Components in the Gateway behave as follows:
1. Gateway parses configuration files that specify which Peers to
connect with and which routes to use
2. Proxy_Handler_Connector connects to Peers, then creates
and activates Proxy_Handler subclasses
(Supplier_Handler or Consumer_Handler)
3. Once connected, Peers send messages to the Gateway
– Messages are handled by an Supplier_Handler
– Supplier_Handlers work as follows:
Receive and validate messages
Consult a Routing_Table
Forward messages to the appropriate Peer(s) via
Consumer_Handlers
Active Object
Non-blocking
Acceptor- Buffered I/O
Connector
Half-Sync/ Component
Half-Async Configurator
STRATEGIC
PATTERNS
Reactor
Vanderbilt University
Advanced ACE Tutorial Do
SOCK_Connector SOCK_Stream
SPECIFIC
Null_Synch
1 Proxy
Handler Supplier/Consumer
Connector n Handler
<<activates>>
COMPONENTS
CONNECTION-
SVC_HANDLER PEER_STREAM
ORIENTED
PEER_CONNECTOR SYNCH
Connector Svc
Handler
PEER
PEER
CONNECTOR
STREAM
COMPONENTS
FRAMEWORK
Stream
Connection
ACE
IPC_SAP
Service
Configurator
Concurrency Reactor
global
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
OO Gateway Architecture
Application-specific components
– Proxy_Handlers route messages among Peers
Connection-oriented application components
– ACE_Svc_Handler
Performs I/O-related tasks with connected clients
– ACE_Connector factory
Establishes new connections with clients
Dynamically creates an ACE_Svc_Handler object for each
client and “activates” it
Application-independent ACE framework components
– Perform IPC, explicit dynamic linking, event demultiplexing, event
handler dispatching, multi-threading, etc.
Consumer HANDLERS
LEVEL
Consumer Handler
Handler 4: send(msg) Supplier
Handler
Event 2: recv(msg)
Event Handler 3: route(msg)
Handler Event
Handler
1: handle_input()
FRAMEWORK
LEVEL
Timer Handle
Queue Table
: Reactor
Benefits Liabilities
Straightforward to Design is “brittle”
program
Can’t leverage
Concurrency control multi-processors
is trivial
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
APPLICATION- Reactor
INDEPENDENT
www.cs.wustl.edu/˜schmidt/POSA/
Intent of Connector Role Forces Resolved:
Decouple the active Reuse connection
connection and code
initialization of a peer
Efficiently setup
service in a distributed
connections with
system from the
many peers or over
processing performed
long delay paths
once the peer service is
connected and initialized
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Synch_Options
(u_long options = 0,
const ACE_Time_Value &timeout
= ACE_Time_Value::zero,
const void *act = 0);
// This is the default synchronous setting.
static ACE_Synch_Options synch;
// This is the default asynchronous setting.
static ACE_Synch_Options asynch;
};
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Synch_Options and
ACE_Connector Semantics
Vanderbilt University
Advanced ACE Tutorial Do
// Demultiplexing hooks.
virtual int handle_output (ACE_HANDLE);// Success.
virtual int handle_input (ACE_HANDLE); // Failure.
virtual int handle_timeout (ACE_Time_Value &,
const void *);
// Table maps I/O handle to an ACE_Svc_Tuple *.
Hash_Map_Manager<ACE_HANDLE, ACE_Svc_Tuple *,
ACE_Null_Mutex> handler_map_;
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Proxy
Handler
Supplier
Consumer Message
Queue Handler
Handler
Vanderbilt University
Advanced ACE Tutorial Do
class Proxy_Handler
: public ACE_Svc_Handler<ACE_SOCK_Stream, SYNCH> {
public:
// Initialize the handler (called by the
// <ACE_Connector> or <ACE_Acceptor>).
virtual int open (void * = 0);
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Detailed OO Architecture
of the Gateway
Reactor
Supplier
Handler Consumer
Handler
Connector
SOCK
Stream
Acceptor
SOCK Message
SOCK Hash Map SOCK Stream Queue
Connector Manager Acceptor
INCOMING OUTGOING
MESSAGES GATEWAY
MESSAGES
CONNECTION CONNECTION
REQUEST REQUEST
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Supplier_Handler Interface
protected:
// Receive and process Peer messages.
virtual int handle_input (ACE_HANDLE);
Vanderbilt University
Advanced ACE Tutorial Do
ACE_Consumer_Handler Interface
protected:
// Perform a non-blocking put().
int nonblk_put (ACE_Message_Block *mb);
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Supplier Consumer
Handler Handler Forces Resolved:
handle_input() handle_output()
put() Keep misbehaving
connections from disrupting
REACTIVE
LAYER
Collaboration in Single-threaded
Gateway Routing
Routing Consumer
Table Handler
Message 5: send_peer(msg)
Queue
ROUTE
ID
Consumer
Handlers
Consumer
Handler
)
3: find()
sg
m
t(
Message
pu
Queue
4:
Vanderbilt University
Advanced ACE Tutorial Do
Supplier_Handler and
Consumer_Handler Implementations
Vanderbilt University
Advanced ACE Tutorial Do
// Determine route.
Routing_Table::instance ()->find (route_id, re);
Vanderbilt University
Advanced ACE Tutorial Do
Peer_Message Schema
class Peer_Message {
public:
// The maximum size of a message.
enum { MAX_PAYLOAD_SIZE = 1024 };
Peer_Header header_; // Fixed-sized header.
char sdu_[MAX_PAYLOAD_SIZE]; // Message payload.
};
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
int Supplier_Handler::recv_peer
(ACE_Message_Block *&route_addr)
{
if (msg_frag_ is empty) {
msg_frag_ = new ACE_Message_Block;
receive fixed-sized header into msg_frag_
if (errors occur) cleanup
else
determine size of variable-sized msg_frag_
} else
determine how much of msg_frag_ to skip
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
int
Consumer_Handler::send_peer (ACE_Message_Block *mb)
{
ssize_t n;
size_t len = mb->length ();
if (n <= 0)
return errno == EWOULDBLOCK ? 0 : n;
else if (n < len)
// Skip over the part we did send.
mb->rd_ptr (n);
else /* if (n == length) */ {
// Decrement reference count.
mb->release ();
errno = 0;
}
return n;
}
Vanderbilt University
Advanced ACE Tutorial Do
int
Consumer_Handler::handle_output (ACE_HANDLE)
{
ACE_Message_Block *mb = 0;
if (msg_queue_->is_empty ()
reactor ()->cancel_wakeup
(this, ACE_Event_Handler::WRITE_MASK);
}
}
Vanderbilt University
Advanced ACE Tutorial Do
APPLICATION-
Proxy Config
SPECIFIC
Handler
Connector Table
SUPPLIER HANDLER
Routing
CONSUMER HANDLER Table
Gateway
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
SERVICE Thread-per
Connection
CONFIGURATOR Gateway
RUNTIME
Reactive
Gateway Service
Object
Service Service DLLS
Repository Object
Thread
Pool
Gateway
Service
Reactor Service
Config
Object
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Handler
EventHandler
Event 4: send(msg) Event
Handler
Event Handler Event
Handler
Handler Handler
2: recv(msg)
3: route(msg)
1: handle_input()
FRAMEWORK
LEVEL
Timer Handle
Signal
Queue Table
Handlers
Reactor
OS EVENT DEMULTIPLEXING INTERFACE
KERNEL
LEVEL
Vanderbilt University
Advanced ACE Tutorial Do
Collaboration in Multi-threaded
Gateway Routing
Routing Consumer
Table Handler
Message 5: send_peer(msg)
Queue
ROUTE
ID
Consumer
Handlers
Consumer
Handler
)
3: find()
sg
m
t(
Message
pu
Queue
4:
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Consumer
TASK LAYER
Supplier 4: enqueue(msg)
Handler Supplier
Supplier 1: dispatch()
Handler
Handler
Reactor
Proxy_Handler
COMPONENTS
APPLICATION-
SOCK_Connector SOCK_Stream
SPECIFIC
MT_Synch
1 Proxy
Handler Supplier/Thr_Consumer
Connector n Handler
<<activates>>
COMPONENTS
CONNECTION-
SVC_HANDLER
PEER_STREAM
ORIENTED
PEER_CONNECTOR
SYNCH
Connector Svc
Handler
PEER
PEER
CONNECTOR STREAM
COMPONENTS
FRAMEWORK
Stream
Connection
ACE
IPC_SAP
Service
Configurator
Concurrency Reactor
global
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Thr_Consumer_Handler Class
Implementation
// Queue up a message for transmission.
int
Thr_Consumer_Handler::put (ACE_Message_Block *mb,
ACE_Time_Value *)
{
// Perform non-blocking enqueue.
msg_queue_->enqueue_tail (mb,
&ACE_Time_Value::zero);
}
int
Thr_Consumer_Handler::svc (void)
{
ACE_Message_Block *mb = 0;
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Event Filter
Module
TELECOM
SWITCHES EVENT
SERVER
Vanderbilt University
Advanced ACE Tutorial Do
Pipes &
Layers
Filters
Acceptor- Component
Connector Configurator
STRATEGIC
PATTERNS
Reactor
TACTICAL Factory
PATTERNS Iterator Composite Method Proxy Wrapper
Facade
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Framework characteristics
Vanderbilt University
Advanced ACE Tutorial Do
STREAM
Head
DOWNSTREAM
UPSTREAM
Multiplexor
open()=0
STREAM close()=0
Tail put()=0
svc()=0
NETWORK INTERFACE
OR PSEUDO-DEVICES
Vanderbilt University
Advanced ACE Tutorial Do
1: put() 1: put()
Module
Module ACTIVE C
C
TASK-BASED MESSAGE-BASED
PROCESS ARCHITECTURE PROCESS ARCHITECTURE
Vanderbilt University
Advanced ACE Tutorial Do
Presentation
Module SUPER
VISORS
Presentation
Module
MD110 ERICSSON
Switch Adapter
MD110 ERICSSON
Module
TELECOM
SWITCHES
www.cs.wustl.edu/˜schmidt/PDF/
DSEJ-94.pdf
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
CLIENT Event
NETWORK Analyzer
DATABASE
ACE Reactor
FRAMEWORK
EXTENSION
MANAGER
SERVER
1: read()
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Do
if (length > 0)
ACE_OS::write (ACE_STDOUT, mb->rd_ptr (),
length);
mb->release ();
if (length == 0) break;
}
}
Vanderbilt University
Advanced ACE Tutorial Do
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Concurrency Strategies
Use threads for independent jobs that must maintain state for the life
of the job
Don’t spawn new threads for very short jobs
Use threads to take advantage of CPU concurrency
Only use “bound” threads when absolutely necessary
If possible, tell the threads library how many threads are expected to
be active simultaneously
– e.g., use thr_setconcurrency()
Don’t hold locks across long duration operations (e.g., I/O) that can
impact performance
– Use ACE_Token instead...
Beware of holding non-recursive mutexes when calling a method
outside a class
– The method may reenter the module and deadlock
Don’t lock at too small of a level of granularity
Make sure that threads obey the global lock hierarchy
– But this is easier said than done...
Locking Alternatives
Code locking
– Associate locks with body of functions
Typically performed using bracketed mutex locks
– Often called a Monitor Object
Data locking
– Associate locks with data structures and/or objects
– Permits a more fine-grained style of locking
Data locking allows more concurrency than code locking, but may
incur higher overhead
Single-lock Strategy
Invariants
Deadlock
private:
ACE_Thread_Mutex nesting_mutex_;
ACE_Condition_Thread_Mutex mutex_available_;
ACE_thread_t owner_;
int nesting_level_;
};
Vanderbilt University
Advanced ACE Tutorial Do
Acquiring an
ACE_Recursive_Thread_Mutex
int ACE_Recursive_Thread_Mutex::acquire (void)
{
ACE_thread_t t_id = ACE_Thread::self ();
ACE_GUARD_RETURN (ACE_Thread_Mutex, guard,
nesting_mutex_, -1);
// If there’s no contention, grab mutex.
if (nesting_level_ == 0) {
owner_ = t_id;
nesting_level_ = 1;
}
else if (t_id == owner_)
// If we already own the mutex, then
// increment nesting level and proceed.
nesting_level_++;
else {
// Wait until nesting level drops
// to zero, then acquire the mutex.
while (nesting_level_ > 0)
mutex_available_.wait ();
owner_ = t_id;
nesting_level_ = 1;
}
return 0;
Vanderbilt University
Advanced ACE Tutorial Do
if (nesting_level_ == 0) {
// Put the mutex into a known state.
owner_ = ACE_OS::NULL_thread;
// Inform waiters that the mutex is free.
mutex_available_.signal ();
}
return 0;
}
ACE_Recursive_Thread_Mutex::
ACE_Recursive_Thread_Mutex (void)
: nesting_level_ (0),
owner_ (ACE_OS::NULL_thread),
mutex_available_ (nesting_mutex_){}
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Avoiding Starvation
Drawbacks to Multi-threading
Performance overhead
– Some applications do not benefit directly from threads
– Synchronization is not free
– Threads should be created for processing that lasts at least
several 1,000 instructions
Correctness
– Threads are not well protected against interference
– Concurrency control issues are often tricky
– Many legacy libraries are not thread-safe
Development effort
– Developers often lack experience
– Debugging is complicated (lack of tools)
Benefits of patterns
– Enable large-scale reuse of software architectures
– Improve development team communication
– Help transcend language-centric viewpoints
Drawbacks of patterns
– Do not lead to direct code reuse
– Can be deceptively simple
– Teams may suffer from pattern overload
Benefits of frameworks
– Enable direct reuse of code (cf patterns)
– Facilitate larger amounts of reuse than stand-alone functions or
individual classes
Drawbacks of frameworks
– High initial learning curve
Many classes, many levels of abstraction
– The flow of control for reactive dispatching is non-intuitive
– Verification and validation of generic components is hard
Benefits of C++
– Classes and namespaces modularize the system architecture
– Inheritance and dynamic binding decouple application policies
from reusable mechanisms
– Parameterized types decouple the reliance on particular types of
synchronization methods or network IPC interfaces
Drawbacks of C++
– Some language features are not implemented
– Some development environments are primitive
– Language has many dark corners and sharp edges
Purify helps alleviate many problems...
Books
– Gamma et al., Design Patterns: Elements of Reusable
Object-Oriented Software AW, ’94
– Pattern Languages of Program Design series by AW, ’95-’99.
– Siemens & Schmidt, Pattern-Oriented Software Architecture,
Wiley, volumes ’96 & ’00 (www.posa.uci.edu)
– Schmidt & Huston, C++ Network Programming: Mastering
Complexity with ACE and Patterns, AW, ’02
(www.cs.wustl.edu/˜schmidt/ACE/book1/)
– Schmidt & Huston, C++ Network Programming: Systematic
Reuse with ACE and Frameworks, AW, ’03
(www.cs.wustl.edu/˜schmidt/ACE/book2/)
Vanderbilt University
Advanced ACE Tutorial Douglas C. Schmidt
Concluding Remarks