• Embed Doc
  • Copy Link
  • Readcast
  • Collections
  • CommentGo Back
Download
 
A Hybrid Publish Subscribe Protocol
Mark Linderman, Norman Ahmed,James Metzler
Air Force Research LaboratoryRome, NY 13441
{
Mark.Linderman, Norman.Ahmed,James.Metzler}@rl.af.mil
Jason Bryant
ITT CoorporationRome, NY 13441
Jason.Bryant.ctr@rl.af.mil
ABSTRACT
Content-based publish/subscribe system performancedepends upon the e
ffi
cient subscription matching and eventdissemination to interested subscribers. We propose a hy-brid content-based publish/subscribe protocol for large sizeevents wherein a centralized brokering system is coupledwith a decentralized BitTorrent-like peer-to-peer (P2P) pro-tocol for scalable event distribution among publishers andsubscribers. Events are mapped to a torrent that grows asnew events are published. Subscribers self-broker on eventmetadata and request content only if interested. Subscriberinterests determine event popularity that the broker esti-mates with sampling. Popular events are disseminated P2P;unpopular events, directly from the broker; and somewhatpopular ones, with P2P and broker-directed pre-seeding.The challenge is the dissemination of popular events withoutoverwhelming centralized resources while e
ffi
ciently dissem-inating unpopular events that lack su
ffi
cient interest to sus-tain gossip-based dissemination. The key advances includenew means of handling variable event popularity inherent incontent-based pub/sub and an adaptive anti-entropy mech-anism for undelivered events.
Categories and Subject Descriptors
C.2.2 [
Computer - Communication Networks
]: Net-work Protocols; C.2.4 [
Computer - Communication Net-works
]: Client/Server, Distributed Applications
General Terms
Performance
Keywords
BitTorrent, Content Distribution, Event-based Systems, Mid-dleware, Peer-to-Peer, Publish and Subscribe, Scalability
1. INTRODUCTION
c
2008 Association for Computing Machinery. ACM acknowledges thatthis contribution was authored or co-authored by a contractor or affiliateof the U.S. Government. As such, the Government retains a nonexclusive,royalty-free right to publish or reproduce this article, or to allow others todo so, for Government purposes only.
 Middleware
’08 Companion, December 1-5, 2008 Leuven, Belgium.Copyright 2008 ACM 978-1-60558-369-3/08/12 ...$5.00.
The publish and subscribe (pub/sub) disseminationparadigm has emerged as a popular means of disseminatingtime-sensitive or filtered information. Through the use of anevent service, or broker, published information is dissemi-nated only to the subscribers interested in that information.One of the major disadvantages of a centralized broker isthat predicate processing and event dissemination can over-whelm the processing, memory, and bandwidth resources of the broker. Therefore, for large numbers of subscribers (say1K-100K) or for large events (e.g., video or imagery) cen-tralized brokers may lack scalability.The system described in this paper reduces the predicateprocessing burden on the broker with a combination of pop-ularity estimation and filtering-upon-receipt. It reduces thedissemination burden by building upon decentralized peer-to-peer (P2P) file sharing protocols, such as BitTorrent [5],to handle the bulk of the dissemination load for content-based pub/sub.The fundamental challenge addressed by this paper is theproblem of simultaneously disseminating popular events forwhich gossip-based P2P protocols work well and unpop-ular events for which they fail. In this paper, we com-bine a centralized pub/sub broker with a BitTorrent-likede-centralized P2P data dissemination protocol to build ahybrid pub/sub system that is highly scalable, and yet stillhandles unpopular events e
ffi
ciently.We have implemented a full-featured information-spacebased information management system [6] on top of the pub-lish and subscribe protocol described in this paper. Eventsof a given type are treated as a single torrent. Each eventcomprises XML metadata and a binary payload. The meta-data tends to be
∼
1Kb in size. Payloads range from 0 bytesto several megabytes.Principal attributes of the hybrid publish and subscribesystem presented are:
•
The use of torrents that start out empty and grow overtime as new events are published,
•
Peer-based self-evaluation of predicates,
•
Broker-based popularity estimation,
•
Aggressive propagation technique for dealing with mid-popular objects that reduces to hub-and-spoke dissem-ination for very unpopular objects
•
A broker-based fallback mechanism for events that faildelivery by one of the other means.
24
 
The key contribution of our work is a new means of handlingvariable event popularity inherent in content-based pub/subover P2P protocols and an adaptive anti-entropy mechanismfor undelivered events.The remaining parts of this paper are structured as fol-lows. Section 2 presents related work and section 3 presentsour Hybrid Approach. We consider performance set up andevaluations in section 4 and summarize our work and discussfuture research directions in Section 5.
2. RELATED WORK
A number of hybrid pub/sub approaches have been pro-posed. For example, Hyber [9] is a hybrid pub/sub schemeaims to minimize both the event matching and forward-ing overhead within pub/sub network. It constructs virtualgroups based upon common subscription interests amongthe clients, and messages are only matched once at the entrypoint for each virtual group as a topic-based pub/sub. Au-thors concede that the overhead of virtual group control mes-sages and the state maintenance is high. Szarowski [8] pro-poses a method of improving e
ffi
ciency of large-scale systemswhile preserving scalability with a combination of topic-based and content-based techniques. It uses heuristics tomaintain local groups of brokers to increase data transferperformance among the members within these groups. Themain drawback is high implementation complexity.Others have attempted to capitalize on BitTorrent s demon-strated success in large-scale content distribution. BASS [7]is similar to our approach; it is a hybrid system that com-bines the client-server model with BitTorrent protocol tobuild a system that is scalable for streaming content dis-tribution. It distributes the load among P2P connectionsto e
ffi
ciently reduce the streaming load on the media server.The server serves up any pieces of the stream that are not de-livered in a timely manner via BitTorrent. The principal dif-ference between BASS and our system is that BASS focuseson streaming video (wherein each participant wants ALL of the video) and we focus on content-based publish/subscribeevent matching and delivery.
3. HYBRID APPROACH
In this section, we present our proposed hybrid protocolwhich is a combination of a pub/sub protocol with slightlymodified BitTorrent protocol. We first start with the pro-tocol overview, followed by a system design and discuss theevent mechanisms that disseminate large events based upontheir popularity.
3.1 Protocol Overview
At a high level, the protocol works as follows. A brokercreates a torrent for a given type of information (event) andregisters the torrent with a tracker. Peers interested in thattype of event receive a set of neighbors from the trackerand attempt to establish connections with them. Subscrib-ing peers will also register their predicates with the broker.When an event (both the metadata and payload) is createdby a publisher and forwarded to the broker, the broker will:1. Parse the metadata and estimate the popularity of theevent by randomly sampling the set of subscriber pred-icates.2. Break the payload into piece blocks similar to BitTor-rent pieces,3. Generate and digitally sign a “bundle” that containsthe size, index, hash, anti-entropy timeout (see below)for each piece (metadata and payload)4. Send the bundle back to the publisher if it is popularor mid-popular for peer-to-peer dissemination. Other-wise the broker sends an acknowledgement indicatingthat it will handle dissemination.For objects disseminated peer-to-peer, metadata pieces floodthe network (similar to normal BitTorrent) originating atthe publisher. Interested subscribers may then request pay-load pieces. This too is similar to BitTorrent with the ex-ception that peers simply neglect to request pieces in whichthey have no interest.
3.2 System Design
We built our hybrid protocol to support an existing hub-and-spoke content-based publish/subscribe system called theJoint Battlespace Infosphere Reference Implementation (RI)[6], and both are implemented in Java. The hub-and-spokeRI can quickly become overwhelmed when the demand forobjects become high, like in a flash crowd. This is the sweetspot for BitTorrent [4]. Our hybrid protocol uses hub-and-spoke for unpopular events, a BitTorrent-like protocol
1
forpopular events and a blend of the two for objects in between.Some of the major modifications we made to the BitTor-rent protocol are:
•
Removing the presumption of fixed-size pieces and fixed-length torrents,
•
Growing torrents,
•
The use of a broker for index assignment and otherchores,
•
Interest-based requests for payload pieces.When a new publication (event) is added to the torrent,the broker simply assigns it piece indexes at the end of thetorrent, thereby increasing the length of the torrent
2
. Usingthis method, when a new event is published, the subscribersmay immediately take advantage of existing peering rela-tionships. Metadata pieces are sent to all neighbors that donot already have them, and payload pieces are sent only tothose that request them based upon whether is metadatamatched their predicates.We know that the scalability of BitTorrent depends onthe popularity of the file. We loosely define three regionsof popularity;
popular 
,
mid-popular 
and
un-popular 
events.We will discuss on the first two in the following sections.For un-popular events, we use hub-and-spoke disseminationbecause there are few recipients; these events are not addedto the torrent.
3.3 Popular Events
BitTorrent e
ff 
ectively disseminates large files because allpeers want the entire content of the file. In contrast, inour content-based pub/sub environment, a popular event
1
The end-game protocol of BitTorrent is not used becauseit is ine
ffi
cient and in a pub/sub environment peers that arekeeping up will be continually in end-game.
2
If the broker is simultaneously handling several large pub-lications, it may choose to interleave the piece indices arbi-trarily.
25
 
may only be of interest to 50+% of the peers/subscribers).As shown in Fig. 1, while BitTorrent operates e
ff 
ectively inthe region at the right (popularity=1), it does not performwell at lower popularities. Note that doubling the numberof unchoked neighbors has a significant impact, but it mayhave the adverse e
ff 
ect of reducing the dissemination ratebecause overhead tra
ffi
c is increased or bandwidth is spreadtoo thinly.
0.0 0.2 0.4 0.6 0.8 1.0 0 50 100 150 200 250 Popularity 
     F    a     i     l    e     d     N    o     d    e    s
Unreached nodes 
Neighbors
25Neighbors
5
Figure 1: Number of nodes, out of a total popula-tion of 1000, that fail event delivery. Each peer hasa fixed random set of neighbors of size 5 (BitTor-rent default) to 25 to which it will upload. Eventoriginates from a single node. 100 random trialswere averaged for each point.
3.4 Mid-popular Events
Mid-popular events (say 10% to 50% popularity) don’tdisseminate well in BitTorrent. To address this, we in-troduce pre-seeding and aggressive propagation techniques.The broker pre-seeds mid-popular events by sending themdirectly to some randomly selected number of interestedpeers rather than relying solely upon the publisher. Pre-seeding creates additional starting points for event propaga-tion that may reach nodes di
ffi
cult to reach from the pub-lisher. Pre-seeding techniques reduce the depth of the dis-semination tree, thereby reducing latency of delivery. Peersthat exhibit habitually long delivery latencies may becomecandidates for pre-seeding. These nodes can be identifiedthrough the anti-entropy mechanism discussed below.Aggressive propagation requires peers to service requestsfor mid-popular objects from neighbors that they would oth-erwise ignore because they are choked. This has the e
ff 
ectof increasing the e
ff 
ective fan-out of each peer. Figure 1demonstrates the impact of increasing the number of neigh-bors to which one is willing to share pieces. One disadvan-tage of aggressive propagation is that it forces a peer to shareits bandwidth among more peers thus delaying propagationof pieces. For lower popularity pieces, however, the addedoverhead is lessened because many neighbors are not inter-ested in the piece in the first place, and the residual overheadis o
ff 
set by greatly improved dissemination of pieces thatmight otherwise have to wait through several optimistic un-choke intervals before an interested neighbor gets unchoked.We are currently investigating means of reducing the over-head of sending and evaluating metadata information topeers that are ultimately disinterested in an object. Ourintention is to perform coarse-grained predicates evaluationat the broker and to encode the potentially interested peersin a Bloom filter. In this way, we can reduce the number of messages and un-wanted metadata exchanged.
 Anti-entropy Mechanism
Even with aggressive propagation and pre-seeding, it is pos-sible that dissemination to some nodes will fail. If eventloss is unacceptable, some sort of repair mechanism (oftenreferred to as an anti-entropy mechanism in gossip proto-cols) is needed. There are two parts to the problem with aBitTorrent-like dissemination model: the first is that a peermay not know any neighbor with a needed piece; the sec-ond is a peer that knows a neighbor with a needed piece,but they refuse to send it. In either case, the peer needsto correct the situation. We accomplish this by allowing apeer to request pieces from the broker after a timeout delay(specified in the bundle) has elapsed. The role of the time-out is to reduce the burden on the broker and to allow thepeer-to-peer dissemination protocol a chance to work.The mechanism is adaptive in several ways. First, thetime delay can be increased or decreased if broker’s load ishigh or low. Second, it can set the timeout time according toestimated popularity, recognizing that unpopular objects aremore likely to fail. Third, it can prioritize responses to re-quests based upon age of event, the requester, the requester’sprior history, or other criteria. If the broker is overwhelmedor a requester repeatedly fails delivery, the broker may issueadditional neighbors to the peer in hope of increasing itsconnectivity.
4. RESULTSExperimental Setup
Experiments are run on computer cluster with 48 dual Xeon2.2 GHz processors connected with gigabit Ethernet. Thenetwork topology comprises two switches (each servicinghalf the nodes) with a single gigabit connection betweenthe switches. Experiments were conducted with the trackerand a publisher residing on dedicated nodes and di
ff 
erentnumber of clients running on other nodes depending on theexperiment.Performance measurements for the system were calculatedby aggregating statistics gathered by each client. A moni-toring module associated with each client periodically (every5s) takes a snapshot of the performance of each torrent onthe client, aggregates the data, and sends it to an aggre-gation node. The performance measurements represent theaverage value among all the clients, with the exception of thetotal download rate. All events have a payload of 2MB, apiece size of 256KB, and a sub-piece (or block) size of 64KB.One BitTorrent configuration parameter of surprising rel-evance is the rechoking interval that specifies how often apeer re-evaluates the set of peers to which it uploads. A longinterval avoids thrashing and allows the overlay topologyto slowly adapt to changes. For example, a peer that getsbogged down for a few minutes may still recover and scorehigh enough to remain unchoked in the next rechoke inter-val. If very short intervals are used, peers may get bounceddue to transient di
ffi
culties.Following similar reasoning, we anticipated that infrequentunchokes would lead to better performance. To our surprise,that was not the case. Infrequent rechokes led to wild swingsin latency; the peaks and troughs corresponding to re- choke
26
of 00

Leave a Comment

You must be to leave a comment.
Submit
Characters: ...
You must be to leave a comment.
Submit
Characters: ...