Apache Nifi

You might also like

Download as docx, pdf, or txt
Download as docx, pdf, or txt
You are on page 1of 8

AWS SDK:

 It is used to perform actions on AWS directly from your application code without using CLI.
 SDK means Software Development Kit.
 Official SDKs are JAVA, .NET, Node.js, Python (named boto3/botocore) etc.
 If you don’t specify or configure a default region, then us-east-1 will be chosen by AWS SDK by
default.

BOTO3:
 We have different ways to automate tasks on AWS.
 The first way is to write SHELL SCRIPTS which contains AWS CLI commands.
 The second way is to write PYTHON SCRIPTS which contains AWS CLI commands.
 The third way is to use PYTHON SCRIPTS with BOTO3 (MOST EFFICIENT WAY TO DO ABOVE
TASK).
 BOTO3 is an open-source python module/SDK for AWS.
 It allows you to directly create, delete, and update AWS resources from PYTHON SCRIPTS.
 botocore is written in python, which is the basis for AWS-CLI.
 BOTO3 is written on top of BOTOCORE.
 Compare to BOTOCORE, BOTO3 contains a lot of great objects and methods to easily work with
any AWS services.

AWS S3 (Simple Storage Service):


 AWS S3(Global service) allows people to store objects(files) in “Buckets” (directories)
 Buckets must have a globally unique name.
 Buckets are defined at the regional level.
 Naming convention – No uppercase, No underscore, Not an IP, first lowercase letter, or number.
 Objects have a KEY
 Key is a FULL Path ---- s3://my_bucket/myfile.txt
s3://my_bucket/my_folder/another_folder/myfile.txt
 Prefix -> my_folder/another_folder
 There’s no concept of directories.
 VERSIONING: You can version your file in S3.
 The same key overwrites will increment the “version”: 1, 2, 3…
 It is best practice to version your buckets to protect against unintended deletes(ability to
restore)
 Easy rollback to the previous version.
 Various Encryptions are also available to protect objects from server intrusion (See screenshots).
 Security and Bucket Policy.
 Host S3 Websites by changing Bucket Policies.

AWS LAMBDA:
 Serverless: It’s a new paradigm in which the developers don’t have to manage servers anymore.
 They just deploy code….deploy functions.
 Initially... Serverless == FaaS (Function as a Service).
 But now it includes anything that is managed remotely: “databases, messaging, Etc”.
 Serverless does not mean there are no servers….it means you just don’t manage/provision / see
them.

 Serverless in LAMBDA:
 AWS Lambda
 DynamoDB
 Amazon S3
 AWS Kinesis Data Firehose.
 AWS API Gateway etc.

AWS CLI:
 Goal – How to perform interactions with AWS without using Online Console?
 Goal – How to interact with AWS Proprietary services? (S3, DynamoDB, etc...)
 AWS CLI is written in Python and it uses Boto3 SDK.
 There are various ways to do so; we will be using AWS CLI on the local computer.
 AWS CLI install
 Configuring AWS, Working of S3 commands (Working with root user)
 “ls – lart” command is used to list hidden folder.

Steps performed to get working connectivity.


 Install AWS CLI and configure access id and access key which is stored it .aws folder in home
directory , so that I don’t need to additionally provide aws_access_id and
aws_secret_access_key
 By doing this boto3 automatically look into .aws folder for required credentials file.
 Python Script using boto3 to List Buckets after configuring AWS CLI (Although we could do
this using AWS CLI only -> aws s3 ls )

import boto3
s3_object = boto3.resource('s3')
for each_bucket in
s3_object.buckets.all():
print(each_bucket.name)
 Python Script using boto3 to upload file into AWS Bucket after configuring AWS CLI (Although
we could do this using AWS CLI only -> aws s3 cp souce_file_name s3://bucketname )saved
in s3_script_to_upload_file.py

AWS POLICIES:
 AWS Policies are objects created and attached to the IAM identities (users, group of users, etc)
and AWS Resources which defines their permissions in AWS.
 AWS evaluated these policies to when an IAM user makes a request.
 Permissions in the policy determine whether the request is allowed or not.
 There are several types of policies:

Identity based policy: Identity based policy decides what the actions that an identity can perform
are allowed.

Resource based policy: Resource-based policies are JSON policy documents that you attach to a
resource such as an Amazon S3 bucket. These policies grant the specified principal permission to
perform specific actions on that resource and define under what conditions this applies. Resource-
based policies are inline policies. There are no managed resource-based policies.

ARN: Amazon Resource Name, it is uniquely identified name across AWS platform.

IAM Users: A physical person/ per physical person

IAM Roles: A role to an AWS machine/ per application.

IAM roles are a secure way to grant permissions to entities that you trust. Examples of entities
include the following:

 IAM user in another account


 Application code running on an EC2 instance that needs to perform actions on AWS
resources
 An AWS service that needs to act on resources in your account to provide its features
 Users from a corporate directory who use identity federation with SAML

LAMBDA EXECUTION ROLE:


 Grants the permission to Lambda Functions to AWS services/Resources.
 Sample managed policies for Lambda:
 AWSLambdaBasicExecutionRole – uploads logs to Cloudwatch.
 Use resource based policies to give other accounts and AWS services permissions to use Lambda
Function.

Lambda Asynchronous Invocations.


Used Amazon SQS – DLQ for failed processing.
When we run a Lambda function asynchronously then we don’t know the result of
the execution of our lambda function. If any problem occurs then that event will
be saved in SQS queue. Created a SQS Queue named DeadLetterQueue and then
attach policies to Lambda function role to access the SQS Queue.

S3 Events Notifications.
In our project destination should be a S3 Bucket.

You might also like