Scribd Logo
Upload

Welcome to Scribd!

  • Twitter Agent

    Uploaded by

    kushal
    10 views1 page
    The document defines a Flume agent configuration called TwitterAgent that collects tweets from Twitter using keywords like "Hadoop" and "bigdata", stores them temporarily in memory channels, and writes them out in batches to HDFS every 10 minutes at the path hdfs:///user/abhinav9884/Tweets. The Twitter source, memory channel, and HDFS sink are configured along with authentication details to collect tweets.

    Original Description:

    Original Title

    TwitterAgent.docx

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document defines a Flume agent configuration called TwitterAgent that collects tweets from Twitter using keywords like "Hadoop" and "bigdata", stores them temporarily in memory channels, and writes them out in batches to HDFS every 10 minutes at the path hdfs:///user/abhinav9884/Tweets. The Twitter source, memory channel, and HDFS sink are configured along with authentication details to collect tweets.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    10 views1 page

    Twitter Agent

    Uploaded by

    kushal
    The document defines a Flume agent configuration called TwitterAgent that collects tweets from Twitter using keywords like "Hadoop" and "bigdata", stores them temporarily in memory channels, and writes them out in batches to HDFS every 10 minutes at the path hdfs:///user/abhinav9884/Tweets. The Twitter source, memory channel, and HDFS sink are configured along with authentication details to collect tweets.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.

    sinks =
    HDFS TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
    TwitterAgent.sources.Twitter.channels = MemChannel
    TwitterAgent.sources.Twitter.consumerKey = l1pKGTtLnfPsVAo69ZrY6bPcZ
    TwitterAgent.sources.Twitter.consumerSecret=
    CH0wbGvzicTKmuxtfpILdKZFFOh9MbIQSPkpAcr60EIdYCSRH6
    TwitterAgent.sources.Twitter.accessToken = 1060227482228420609-
    MR1RMOoyrYYQgv559gGkL5mP2VhFyY

    TwitterAgent.sources.Twitter.accessTokenSecret=LEhEwP1I2kbIWPnqFTGiENDYsku7pYbtTD
    mUQNIEoNN3

    TwitterAgent.sources.Twitter.keywords = Hadoop, bigdata

    ################## SINK #################################


    TwitterAgent.sinks.HDFS.channel = MemChannel TwitterAgent.sinks.HDFS.type = hdfs
    TwitterAgent.sinks.HDFS.hdfs.path = hdfs:///user/abhinav9884/Tweets
    TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
    TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text TwitterAgent.sinks.HDFS.hdfs.batchSize =
    10 TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600
    TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 #################### CHANNEL
    ######################### TwitterAgent.channels.MemChannel.type = memory
    TwitterAgent.channels.MemChannel.capacity = 100 #default -
    TwitterAgent.channels.MemChannel.capacity = 100
    TwitterAgent.channels.MemChannel.transactionCapacity = 100

    flume-ng agent -n TwitterAgent -Dtwitter4j.streamBaseURL=https://stream.twitter.com/1.1/ -c


    conf -f /home/abhinav9884/flume/flume.conf

    vi flume/flume.conf

    You might also like