Kafkacomet Overview

This service relays messages from Kafka topics to a websocket connection. Clients can establish a websocket connection and send a
subscription for one or more Kafka topics. As long as the connection is established, all Kafka messages from the subscribed topic are relayed
to the websocket client. The client may disconnect or unsubscribe from topics at any time.

Table of Contents

Connect

Authentication

How-to subscribe: Offset and consumerId

Protocol V2

Heartbeat
Ping/Pong Frames
Subscribe
Unsubscribe
Message
Error

Protocol V1 (Deprecated)

Heartbeat
Ping/Pong Frames
Subscribe
Unsubscribe
Message
Error

Subscription notifications for application use

GCS Variables
Audit Logging

Connect

#V1 (user_id) acts as consumerId

ws(s)://<host>:<port>/kafkacomet/socket/websocket?user_id=<userId>

#V2

ws(s)://<host>:<port>/kafkacomet/socket/websocket

Authentication
By default authentication must be provided in order to open a websocket connection with kafkacomet.
The client must provide either a uac.authorization cookie with the value set to a valid token, or an
Authorization bearer <token> header with the connection request.
Failure to provide valid authentication will result in a websocket: bad handshake error, along with a 401 Status code.
The requirement for authentication can be disabled using the Global Configuration Service,
by setting the kafkacomet authentication
configuration to false .

How-to subscribe: Offset and consumerId

When subscribing to a topic, you also have to provide a consumerId (v2) or user_id (v1) as well as, an offset of
latest or earliest (default:
latest). If the consumerId is not specified, one will be generated and returned in the response.
When subscribing to a topic for the very first time with a consumerId, the offset determines where you begin to read messages from the
topic.
 Message Message
sent sent
1:00 pm 1:15 pm
Topic

Earliest Latest

You can read all messages from the beginning, or start with the next message to be produced to the topic.
Note: Messages may expire or
compacted based on topic configuration. So the earliest message, is not always the first message ever produced to the topic.
Once subscribed, Kafka keeps track of the topic offset for your consumerId. If you lose connection or unsubscribe and resubscribe at a later
date with the same consumerId, you start where you left off.

Message Message
sent sent
1:00 pm 1:15 pm
Topic

Earliest Latest
Offset ConsumerId/
user_id

Can I read from a specific Offset

No. Exposing this requires the client to do the heavy lifting, tracking offsets per consumer per partition per topic. So far, no use case
justified the resulting complexity.

Reference parameter
Each request has a ref parameter, it is used to associate the request with it’s response.

Nonexistent topics
Kafka comet does not create topics for subscriptions. A subscription to a topic that does not exist will wait 20 minutes by default
(configurable in GCS) for the topic to exist before timing out.
After the timeout a subscription failure message will be sent assuming
websocket connection is still active.

Protocol V2

Heartbeat
If the server does not receive a heartbeat every 30 seconds in the absence of any other messages, the connection will be terminated
Request:
{
"v2" : {
"ref" : 1,
"operation" : "heartbeat"
}
}

Response:
{
"v2" : {
"ref" : 1,
"operation": "heartbeat",
"response" : {
"success" : true
}
}
}

Ping/Pong Frame
As an alternative to Heartbeat messages, one can send a Ping Frames to the websocket server. The payload for this message is arbitrary
data.
This will keep the connection alive and prevent the server from terminating it, just like the Heartbeat message does. The only difference is
that the
server answers with a Pong Frame, which contains the exact same data that was sent with the Ping Frame. The Ping Frame needs to
be send at an interval < 30 seconds.
This implementation is according to RFC6455

Subscribe
Subscribe to a topic. Streaming starts with the most recent message
Request:
 {
"v2" : {
"ref" : 2,
"operation" : "subscribe",
"consumerId" : "myConsumer", # Generated and returned in response, if not specified
"topics" : [
{
"name" : "bp.aeprocessor.v2_0.alarms",
"startAt" : "latest"
}
]
}
}

You will receive one notification per topic you subscribed to

Response:
{
"v2" : {
"ref" : 2,
"operation": "subscribe",
"consumerId" : "<specified or generated consumerId>",
"response" : {
"topic" : "bp.aeprocessor.v2_0.alarms",
"success" : true
}
}
}

Unsubscribe
Request:
{
"v2" : {
"ref" : 4,
"operation" : "unsubscribe",
"topics" : [
{
"name" : "bp.aeprocessor.v2_0.alarms"
}
]
}
}

You will receive one notification per topic you unsubscribed from
If you have multiple subscriptions to a topic with different consumer IDs
then you may specify the consumerID in the unsubscribe in order to close that specific subscription, otherwise all subscriptions for that topic
will be closed
Response:
{
"v2" : {
"ref" : 4,
"operation" : "unsubscribe",
"response" : {
"topic" : "bp.aeprocessor.v2_0.alarms",
"success" : true
}
}
}

Topic Message Notification

The body is the kafka message JSON object
{
"v2" : {
"operation" : "relay",
"message" : {
"topic" : "srcTopic",
"body" : {},
}
}
}

Error Messages

{
"v2" : {
"ref" : 2,
"operation" : "<operation>",
"response" : {
"error" : "Something went wrong"
}
 }
}

Protocol V1 (Deprecated)

Heartbeat
If the server does not receive a heartbeat every 30 seconds in the absence of any other messages, the connection will be terminated
Request:
{"topic":"phoenix","event":"heartbeat","payload":{},"ref":"7"}

Response:
{"topic":"phoenix","ref":"7","payload":{"status":"ok","response":{}},"event":"phx_reply"}

Ping/Pong Frame
As an alternative to Heartbeat messages, one can send a Ping Frames to the websocket server. The payload for this message is arbitrary
data.
This will keep the connection alive and prevent the server from terminating it, just like the Heartbeat message does. The only difference is
that the
server answers with a Pong Frame, which contains the exact same data that was sent with the Ping Frame. The Ping Frame needs to
be send at an interval < 30 seconds.
This implementation is according to RFC6455

Subscribe
Subscribe to a topic. Streaming starts with the most recent message
Request:
{"topic":"topics:bp.aeprocessor.v2_0.alarms","event":"phx_join","payload":{},"ref":"1"}

Response:
{"topic":"topics:equipment","ref":"1","payload":{"status":"ok","response":{}},"event":"phx_reply"}

Unsubscribe
Request:
{"topic":"topics:bp.aeprocessor.v2_0.alarms","event":"phx_leave","ref":"1"}

Response:
{"topic": "topics:bp.aeprocessor.v2_0.alarms", "ref": "2", "payload": {"status": "ok", "response": {}}, "event":
"phx_reply"}

Topic Message Notification

{"topic":"topics:bp.aeprocessor.v2_0.alarms","payload":{"body":{"value":"{\"test\":\"message\"}"}},"event":"shout"}

Error Messages

{"topic": "topics:bp.aeprocessor.v2_0.alarms", "ref": "2", "payload": {"status": "error", "response": "TestError

message"}, "event": "phx_reply"}

Subscription notifications
Applications that care about whether or not their topic is subscribed to can listen on the topic
bp.kafkacomet.v1.channel.administration for subscription notifications.
The notifications consist of a join message sent out
when a topic is subscribed to e.g.:
{"version":1,"header":{"envelopeId":"a2e2d89a-8049-464e-8a3c-8258a878be4d","timestamp":"2017-08-
22T20:41:41.224Z","traceId":"9eef9356-87e5-4d27-a9d8-0c1ee8986585","upstreamId":"a0ae2c64-bdf4-4372-8cc9-
5abf1ec9aba6"},"event":
{"_type":"bp.kafkacomet.v1.admin","op":"CHANNEL_JOINED","channel":"bp.aeprocessor.v2_0.alarms"}}

and a close message sent out when all subscribers for a channel are gone:
{"version":1,"header":{"envelopeId":"4e3a2444-8c72-4d3a-9586-11c97fa53164","timestamp":"2017-08-
22T20:42:40.374Z","traceId":"a43ac7d8-0cd3-4d01-8f78-ab3d4800094e","upstreamId":"147203b2-596a-4ab1-aa64-
3442e3e05e08"},"event":
{"_type":"bp.kafkacomet.v1.admin","op":"CHANNEL_CLOSED","channel":"bp.aeprocessor.v2_0.alarms"}}

Important Consumers of this topic must be idempotent, since it is not guarenteed that each join/close message will be sent only once.
Each host will send a join/close message when a subscription is created or a channel is closed. E.g. with 2 KafkaComet
instances, when bp.aeprocessor.v2_0.alarms is joined, 2 CHANNEL_JOINED messages will be sent on
bp.kafkacomet.v1.channel.administration for bp.aeprocessor.v2_0.alarms .

GCS Variables

authentication : Whether or not authentication is required to open a websocket connection. Default: true .
consumerTimeout : How long (minutes) to wait for a topic to exist after subscribe
heartbeatPingTimeout :The time (seconds) after which the connection will be closed without receiving a heartbeat/ping. Set
to 0 to disable. Minimum value is 30. Default: 30 .
maxLogSize : MaxLogSize is the maximum size in megabytes of the uncompressed log file before it gets rotated. Defaut: 50 .
 maxLogAge : MaxLogAge is the maximum number of days to retain old log files based on the timestamp encoded in their
filename
maxLogBackups : MaxLogBackups is the maximum number of old log files to retain. Default: 10 .
compressAuditLog : CompressAuditLog determines if the rotated log files should be compressed using gzip. The default is to
perform compression. Default: true .
auditLoggingEnabled : Audit logging enabled (by default not enabled with default value 'false'). Default: false .
consumerIDAudited : Consumer ID audited (by default no consumer ID set, empty string used as default value). Please set this
value to the same value as the consumer ID of the subscription you would like to log.
kafkacometDebugLevel : Kafkacomet internal debug level. Set to 0 to disable. Disabled by default.
MessageRateLimit : Max number of messages per second to forward. Minimum of 1, set to 0 to disable. Disabled by default.

See GCS for more information

Audit Logging

When Audit logging enabled ( auditLoggingEnabled set true ), forwarded/relayed kafka message will be logged into
/bp2/log/audit/<consumerID>/<topicName>.log in the Text format of
time=timestamp level=info message=KafkaRawMessage .
timestamp is the time when the message forwarded/relayed in
RFC3339Nano.

Besides setting auditLoggingEnabled to true , consumerIDAudited needs set to the consumerID you would like to log as
following:
curl -X PUT "http://GCS_IP:8080/gcs/api/v1/config" -H "accept: application/json" -H "Content-Type: application/json
Generated log will be rotated according to tunable GCS variables maxLogSize , maxLogAge and maxLogBackups .