Roadmap to becoming Data Engineer

Abhi yah shortcut le lo aur apna life change kar lo

Varna aise hi ghumte firoge , Yahan per free ka dukaan khol kar nahin Rakha hai dost

Remember continues efforts take's you to greater heights

Whatever I m going to tell is from my personal experience 🚀

There are two ways

✔️ java + scala and spark (mandatory) 1st
✔️python and spark (mandatory) 2nd
Today I m going to show you the path of 2nd option:

⚒️This is going to help you to become a spark developer with some data engineer skills
In brief:

1)𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲: Python

Learn from Andrei Neagoie

2)𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸: Spark

Learn from Kedar Nanda

3) 𝗟𝗲𝗮𝗿𝗻 : - Hadoop & Hive

Learn from OnlineLearningCenter Suraz G.

4) 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝘁𝗼𝗼𝗹𝘀: ADF/airflow

Learn from Ramesh Retnasamy / Deepak Goyal for Azure Data Factory
Learn from Marc Lamberti for Airflow

☑️All the Links are available in the comment section

𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹 𝗲𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺: Sqoop,Hbase,oozie,elasticsearch,apache pulsar, apache delta lake

, Databricks ,Azure DevOps /Github

Yes Yes I get you, Ohh Bhai Cloud technology kahan hai 🔥🔥🔥

You can still learn those things from a short course from Pluralsight Linux Academy
I m showing you a path for a data engineer developer, not for a big data cluster admin

✔️In real time you will develop your application in local

✔️When deploying the app you will change your upstream and downstream paths
✔️Most likely it will be object storage like S3, azure blob storage / messaging queue service
like ApachePulsar
✔️To authenticate the path your gonna add vault/credentials details in the config file of the app
✔️Rest of the infra will be taken by the big data cluster admin team/DevOps team
So it is no big deal
🔥🔥🔥
🚀 I will show you the 1st first path in the next post soon
🚀Through this you will have a strong career growth in Big data engineering