Scribd Logo
Upload

Welcome to Scribd!

  • 088 Kafka-Meetup

    Uploaded by

    Pradeep Saraswat
    2 views1 page
    This document provides commands for creating a Kafka topic, starting a Kafka producer and consumer, describing the Kafka topic, starting Kafka Connect, and running Hive queries on data in the topic stored in HDFS. It creates a Kafka topic, starts a producer and consumer, loads data into HDFS via Kafka Connect, creates an external Hive table pointing to the data, and runs queries on the table.

    Original Description:

    Original Title

    088 kafka-meetup

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document provides commands for creating a Kafka topic, starting a Kafka producer and consumer, describing the Kafka topic, starting Kafka Connect, and running Hive queries on data in the topic stored in HDFS. It creates a Kafka topic, starts a producer and consumer, loads data into HDFS via Kafka Connect, creates an external Hive table pointing to the data, and runs queries on the table.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    2 views1 page

    088 Kafka-Meetup

    Uploaded by

    Pradeep Saraswat
    This document provides commands for creating a Kafka topic, starting a Kafka producer and consumer, describing the Kafka topic, starting Kafka Connect, and running Hive queries on data in the topic stored in HDFS. It creates a Kafka topic, starts a producer and consumer, loads data into HDFS via Kafka Connect, creates an external Hive table pointing to the data, and runs queries on the table.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    --Create Topic

    bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --


    partitions 3 --topic hirw-meetup-topic

    --Start producer

    java -cp KafkaMeetup-1.0.jar:/home/ubuntu/confluent-3.3.0/share/java/kafka/*:/


    home/ubuntu/confluent-3.3.0/share/java/confluent-common/*:/home/ubuntu/confluent-
    3.3.0/share/java/schema-registry/*:/home/ubuntu/meetup_libs/*
    com.hirw.kafka.KafkaMeetupProducerAvro

    --Start consumer

    java -cp KafkaMeetup-1.0.jar:/home/ubuntu/confluent-3.3.0/share/java/kafka/*:/


    home/ubuntu/confluent-3.3.0/share/java/confluent-common/*:/home/ubuntu/confluent-
    3.3.0/share/java/schema-registry/*:/home/ubuntu/meetup_libs/*
    com.hirw.kafka.KafkaMeetupConsoleConsumer

    --Describe parition

    bin/kafka-topics --describe --zookeeper localhost:2181 --topic hirw-meetup-topic

    --Start Kafka connect

    bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties
    etc/kafka-connect-hdfs/meetup-hdfs.properties

    --Hive against RSVP data

    SET hive.mapred.supports.subdirectories=TRUE;
    SET mapred.input.dir.recursive=TRUE;

    CREATE EXTERNAL TABLE meetup_avro


    ROW FORMAT
    SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    STORED AS
    INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    LOCATION '/user/ubuntu/topics/hirw-meetup-topic'
    TBLPROPERTIES ('avro.schema.url'='hdfs:///user/ubuntu/meetup.avsc');

    select count(1) from meetup_avro;

    select get_json_object(rsvp, '$.group.group_name') as group_name, count(1) from


    meetup_avro
    group by get_json_object(rsvp, '$.group.group_name');

    You might also like