NIDHI ABHI

jad48919@gmail.com

Sr. AWS Data Engineer

Summary:

 Expert in seamlessly integrating Python applications with diverse databases, including SQL databases such as PostgreSQL,
MySQL, and NoSQL databases like MongoDB and AWS DynamoDB, optimizing data retrieval, storage, and processing for
Python-centric AWS data engineering projects.
 Extensive experience in creating end-to-end cloud solutions using a range of AWS services including Glue, Step Functions,
EC2, S3, RDS, IAM, DynamoDB, Elastic Beanstalk (EBS), ElasticCache, Auto Scaling, Security Groups, EC2 Container
Service (ECS).
 Implemented efficient Amazon Athena queries to analyze and derive actionable insights from large-scale datasets stored in
Amazon S3, optimizing query performance and minimizing costs through effective data partitioning and query optimization
techniques.
 Comprehensive knowledge of AWS services and tools, including AWS CodeBuild, AWS CodeCommit, AWS CodePipeline,
and API Gateway, for implementing CI/CD pipelines and serverless deployments.
 Implemented Python scripting for automation of data processing tasks, leveraging Apache Spark for distributed data
processing and analysis on AWS cloud infrastructure. Developed efficient Spark jobs in Python to handle large-scale datasets
stored in Amazon S3, optimizing data workflows for performance and scalability while ensuring cost-effective utilization of
AWS resources.
 Expertise in data modeling, ETL processes, and data warehousing concepts, ensuring efficient data storage, retrieval, and
analysis.
 Proficient in full-stack development, spanning frontend in JavaScript, React and backend in Python, Django implementations.
 Extensive experience with asynchronous programming using Python's asyncio library, enhancing API responsiveness and
concurrent request handling.
 Experience in working with relational databases such as Postgres, ensuring data integrity and efficient querying as well as
good Knowledge of Big Data, Data Warehouse Architecture, Star Schema, Snow flake Schema and Dimensional Tables.
 Proficient in developing dynamic and interactive web applications with Angular framework, leveraging its powerful features
for front-end development, as well as creating modern and responsive user interfaces using React library, utilizing its
component-based architecture and virtual DOM for efficient UI rendering.
 Expertise in designing for horizontal scalability, allowing microservices to handle increased load by adding more instances.
 Skilled in CI/CD pipelines using tools like Jenkins and GitLab, automating the build, test, and deployment processes for
faster and more reliable software releases.

Work Experience

Sr. AWS Data Engineer || Netflix

May 2022 - Present
Responsibilities:
 Led the web Application Engineers and used Python scripts to load the data into AWS Cloud database and implemented
background tasks and asynchronous processing using FastAPI's async capabilities and optimized API responsiveness
efficiently by handling time-consuming operations.
 Utilize AWS services such as AWS Lambda, Step Functions, and API Gateway to design and implement serverless
architectures and cloud-native applications.
 Utilized AWS Glue for automated ETL processes, enabling seamless extraction, transformation, and loading of data into
AWS data lakes and data warehouses.
 Designed and implemented relational database solutions using Amazon RDS, ensuring efficient data storage, retrieval, and
management for web applications.
 Optimized SQL queries and data warehouse performance in AWS Redshift, reducing query execution times and improving
overall system efficiency.
  Utilized Amazon Athena to query and analyze large-scale datasets stored in Amazon S3, leveraging SQL-like queries to
extract insights and perform ad-hoc analysis on structured and semi-structured data formats such as CSV, JSON, and Parquet.
 Implemented event-driven workflows using AWS Event Bridge for decoupled and scalable event processing and created
RESTful APIs using Python with ensuring efficient data exchange and seamless integration between systems and step
functions with AWS Lambda functions, S3, and other AWS services to create seamless end-to-end workflows.
 Managed end-to-end AWS cloud solutions, from architecture design to deployment, leveraging services like AWS Glue, Step
Functions, and S3 to ensure scalability and reliability.
 Utilized Python libraries and framework Django for API development by streamlining the creation of endpoints and handling
complex request-response flows.
 Configured and managed Bitbucket repositories to facilitate collaborative version control and code collaboration for Python-
based AWS data engineering projects, ensuring seamless integration with CI/CD pipelines and enabling efficient code
review, branching, and merging workflows.
 Developed Python scripts to automate data processing tasks, including data ingestion, transformation, and cleansing,
optimizing workflows for efficiency and scalability in AWS cloud environments.
 Developed Python scripts to automate deployment and orchestration of Docker containers on AWS ECS and Fargate,
ensuring efficient resource utilization and implemented serverless architecture using AWS Lambda for event-driven
processing, reducing operational overhead and costs.
 Configured AWS S3 buckets for storing and managing static assets and user-generated content, implementing lifecycle
policies for cost optimization.
 Utilized AWS SNS and SQS for decoupling components and enabling asynchronous communication between microservices.
 Designed and implemented CI/CD pipelines using Docker, Jenkins, and AWS services, facilitating continuous integration,
testing, and deployment processes.
 Configured and maintained Jenkins pipelines for Python-based data engineering projects on AWS, automating the build, test,
and deployment processes of ETL (Extract, Transform, Load) workflows and data pipelines using tools such as AWS Glue
and Apache Airflow.
 Implemented Angular components, services, and modules to encapsulate functionality and promote code reusability across
the application..
 Customized SQL scripts and designed the solution to Developed a custom ETL pipeline by using Python. Used log data and
filter required columns by Logstash configuration and send it to Elasticsearch.

Data Engineer || Bank Of America

July 2020 – May 2022
Responsibilities:
 Designed and implemented batch processing workflows using Apache Airflow, orchestrating ETL processes and job
scheduling for data ingestion and transformation.
 Implemented Lambda functions in Python to process data, trigger actions, and integrate with other AWS services and used
Python libraries such as Pandas for data manipulation and transformation by facilitating the processing of complex data
structures within API responses.
 Optimized database schema for RestAPIs and improved data retrieval performance by reducing query response time
significantly by 17%. Optimized Lambda functions and Step Functions for performance and cost efficiency as well as
employed Python's dynamic typing and expressive syntax to implement data validation, transformation, and serialization
within API responses.
 Leveraged Apache Spark for distributed data processing and analysis, implementing Spark jobs in Python to handle large-
scale datasets efficiently and perform complex data transformations and aggregations, enhancing the performance and
scalability of data processing workflows on AWS.
 Tested different modules in Docker and Kubernetes containers and implemented programming automations using Jenkins
and Ansible on Linux based OS over cloud like Docker.
 Utilized Amazon QuickSight to create interactive dashboards and visualizations, providing stakeholders with real-time
insights into data analytics and trends, and enabling informed decision-making based on dynamic data exploration and
analysis.
  Developed and optimized Athena queries for performance and cost-efficiency, utilizing partitioning, data compression, and
query optimization techniques to reduce query execution time and minimize AWS costs associated with data processing and
analysis.
 Designed and implemented Docker containers for Python applications, ensuring portability and consistency across
development, testing, and production environments.
 Utilized Angular CLI for project scaffolding, code generation, and building optimized production bundles for deployment.
 Implement AWS Lambda, Step Functions, and API Gateway to develop and orchestrate serverless workflows, automating
business processes and enhancing operational efficiency.
 Utilized AWS Lambda for serverless computing, implementing event-driven functions for real-time data processing and
integration with S3 and DynamoDB.
 Applied AWS architectural best practices to design and optimize cloud solutions, focusing on cost-efficiency, scalability, and
reliability.
 Configured AWS SNS for notification delivery and SQS for message queuing, enhancing communication between
microservices and external systems.
 Implemented CI/CD pipelines using Docker, GitLab CI/CD, and AWS CodePipeline, automating the build, test, and
deployment processes.
 Integrated Kafka with various AWS services such as Lambda, Kinesis, and S3 for building end-to-end data processing
pipelines.
 Implemented error handling and retry mechanisms within Step Functions to enhance the reliability and fault tolerance of
workflows.
 Managed datasets efficiently using Panda data frames and MySQL. Executed MySQL database queries from Python using
the Python-MySQL connector and MySQL dB package, facilitating seamless data retrieval and manipulation.

Data Engineer || Accenture

May 2018 – July 2020
Responsibilities:
 Worked on server-side applications with Django using Python programming and Used python libraries like matplotlib,
NumPy and generated Python Django forms to record data of online users and used PyTest for writing test cases and
modified various SQL queries and Functions, Cursors and Triggers as per the client requirements.
 Gained basic knowledge of AWS Lambda and Step Functions through training or self-study, demonstrating eagerness to
learn serverless technologies.
 Configured AWS S3 buckets as durable storage for static assets, logs, and backups, implementing encryption and access
control policies.
 Integrated Jenkins with AWS services such as AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy to enable
continuous integration and continuous deployment (CI/CD) of Python scripts and data processing jobs, ensuring efficient and
reliable execution of data engineering tasks in cloud environments.
 Build ETL- Extract, Transform, Load pipelines using AWS Glue for data integration and developed backend services using
Python creating RESTful APIs to handle product management and user authentication.
 Utilized AWS infrastructure as a code using Terraform templates to build staging and production environments. Configured
and networking of Virtual Private Cloud (VPC) Written Cloud Formation templates and deployed AWS resources using it.