Streaming Tweets into Hadoop (Part II)

In the previous part of this series, I discussed a concept of streaming Twitter data into Hadoop. In this part, I will focus on Apache Flume and its settings.

Flume is an Apache project that is capable of collecting data from many different sources into HDFS. Using connectors, Flume works with social media data, log files, email messages and pretty much anything one may think of.

Flume architecture is built on top of a producer-consumer model. The Source is the event producer and the Sink is the consumer. In this post, the Source will be Twitter data feed. Flume is also capable of processing of web server logs or network data are also possible to be captured by Flume.

Data are sent from a Source, received by an asynchronous Channel and drained into a Sink. A Channel drains into a Sink in an asynchronous fashion so the source doesn’t have to wait until data are stored by the Sink.

Getting your keys from Twitter

Go to Twitter Apps and click Create New App. Fill in required information to create a new app. Go to the Keys and Access Tokens tab and save securely following keys:

• Consumer Key (API Key)
• Consumer Secret (API Secret)
• Access Token
• Access Token Secret

Flume configuration

Flume comes preinstalled in HDP sandbox. To get Twitter source installed execute

Download a preconfigured Twitter conf file

Now it’s time to configure Flume. Open the downloaded flume.conf file in your favourite editor. Every entry is in the format [agent_name].[object_type].[object_name].[parameter_name] where agent_name attribute is specified at the beginning of the file and object_type is either the Sink or the Channel. My configured flume.conf file is included bellow.

I’ve made couple of changes

sources:

• set my own hashtags I want to capture
• changed path to point to my NamedNode (as long as we are using HDP sandbox, it is the local DNS name) hdfs://sandbox.hortonworks.com:8020/user/root/flume/test

sinks:

• increase batchSize to 1000 (number of events written to the HDFS in one batch)
• set rollSize to 67108864 (start a new file when the current one reaches 64MB)
• set rollInterval to 3600 (start a new file after 1 hour)
• set rollCount to 0 (0 is for never roll over based on the count of events)

channels configuration

• increase capacity to 1000 (count of events sent to the sink in a single batch)

If I didn’t increase MemChannel.capacity, I would get the error bellow because memory buffer couldn’t keep pace with the incoming data volume

Note to my configuration

I set up lots of hashtags to stresstest my sandbox. I reached 64MB rollsize in about 7 minutes.

Imagine a situation when you don’t capture trending topics. In such a situation, it would be better to lower rollSize or rollInterval to write files into HDFS more frequently. For memory resident sink, any disruption within the roll period would result in data loss.

Create destination folder for Twitter data

Launching flume agent

Take a deep breath and execute a flume agent. Nohup launches Flume in the background so we may keep using the same console for viewing the generated log file.

Execute Tail command to view the generated log file.

If everything was set correctly, the log file will contain following lines and will repeat BucketWriter entries for each file the Sink created.

Checking on created files

To list HDFS’ filesystem execute the following Shell command.

You might also check on files from Hue web console at http://YOURSANDBOXIP:8000/filebrowser/#/user/root/flume/test

To terminate Flume agent, simply issue kill command with the process number.

Next time, I’m going to process all the raw files using Hive.