AWS kinesis configuration: ----

1. Launch EC2 with a programmatic access of the following policies - AmazonS3FullAccess,

AmazonKinesisFullAccess, AdministratorAccess, AmazonEC2FullAccess

2. Download the access key and .pem file if you don't have.

Once EC2 is up using sudo access - install Kinesis agent on the machine

sudo yum install –y aws-kinesis-agent

Writing to Amazon Kinesis Data Streams Using Kinesis Agent - Amazon Kinesis Data Streams

STEP1:-- INSTALL AWS-KINESIS-AGENT ON EC2-INSTANCE AND DO

SOME CONFIGURATION:--
2 sudo yum install –y aws-kinesis-agent

sudo yum install https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-

latest.amzn2.noarch.rpm –y

or

sudo yum install https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-

latest.amzn2.noarch.rpm

3 vim /etc/aws-kinesis/agent.json

"cloudwatch.emitMetrics": true,

"kinesis.endpoint": "kinesis.us-east-1.amazonaws.com",

"firehose.endpoint": "firehose.us-east-1.amazonaws.com",
 "awsAccessKeyId": "AKIA5QYYHX2KNYAL2UOO",

"awsSecretAccessKey": "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",

"flows": [

"filePattern": "/tmp/*.log",

"kinesisStream": "kds-demo",

"partitionKeyOption": "RANDOM"

4 vim /etc/sysconfig/aws-kinesis-agent

# Set AWS credentials for accessing Amazon Kinesis Stream and Amazon Kinesis Firehose

# AWS_ACCESS_KEY_ID="AKIA5QYYHX2KNYAL2UOO"

# AWS_SECRET_ACCESS_KEY="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"

# AWS_DEFAULT_REGION="us-east-1"

# AGENT_ARGS=""

# AGENT_LOG_LEVEL="INFO"

5 sudo service aws-kinesis-agent start

6 sudo service aws-kinesis-agent restart

9 sudo service aws-kinesis-agent restart

10 exit
STEP2:--NOW CREATE KINESIS STREAM:--
STEP 3:--NOW CREATE KINESIS DELIVERY STREAM:--
2 ls -lrt
 3 vi temp.log

4 head /var/log/messages

5 sudo head /var/log/messages

6 vi temp1.log

7 cd /tmp

8 ls -lrt

9 cd

10 ls

11 cp temp1.log /tmp/temp1.log

12 cp temp1.log /tmp/temp2.log

13 cp temp1.log /tmp/temp3.log

14 cp temp1.log /tmp/temp4.log

15 cp temp1.log /tmp/temp5.log

16 tail -40 /var/log/aws-kinesis-agent/aws-kinesis-agent.log

17 history

STEP 4:--NOW GO TO S3 BUCKET FOR LOG FILE OF KINESIS:--