Advancement of Computer Technology and its Applications

Volume 5 Issue 3

Data Set of AI Jobs

Divyam Pithawa1, Sarthak Nahar2*, Shivam Sharma3*, Er. Nikhil Chaturvedi4*

1,2, 3
B. Tech Student, CSE, Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore, India
4
Assistant Professor, CSE, Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore, India
*CorrespondingAuthor
E-Mail Id: divyampithawa@gmail.com

ABSTRACT
The automated, targeted extraction of information from websites is known as web
scraping. Similar technology used by search engines is marked as “Web Crawling.”
Although human data collection is a possibility, automation is frequently faster, more
efficient, and less prone to mistakes.
Online job portals frequently collect a substantial amount of data in the form of resumes
and job openings, which may be a useful source of knowledge on the features of market
demand.
Web scraping may be categorized into three steps: the web scraper finds the needed links
on the internet; the data is then scraped from the source links; and finally, the data is
shown in a CSV file. For doing the scrape, the Python language is used.
As part of the job series of datasets, this dataset can be helpful for finding a job as an AI
engineer!

Keywords:- Web Scraping, Job Portal, Dataset, AI Jobs

INTRODUCTION enormous potential it represents, business

The World Wide Web [1] consists of an executives should concentrate on locating
interlinked network of information, and maintaining that special, crucial
which is presented to users through piece of data. ((IDC), 2017) David
websites. Reinsel et. al., “Data Age 2025 [2].”

The way we share, gather, and publish Web / Data Scraping

data has changed substantially as a result
of the World Wide Web. The quantity of
information available continues to grow.
As data volume, diversity, and
importance rise, business executives must
focus on the most important facts.
Not all data is equally significant to
consumers or corporations. Businesses
that can recognize and capitalize on the
crucial fraction of data that will have a
meaningfully beneficial impact on user
experience, resolve complicated issues,
and generate new economies of scale will
prosper throughout this data
transformation. To fully realize the
Web scraping [3], also known as data
scraping, is a term used for automatically
retrieving data from the internet and
structuring it in a useful manner.
Choosing to manually copy and paste
data from a job website will take days as
most of the data is viewed using a
browser. Our aim is to automate this
process with the help of a web scraper,
which will scrape the job postings in
mere seconds to achieve the task.
The use of fully automated algorithms
that can transform entire webpages into
well-organized data sets have replaced

HBRP Publication Page 1-7 2022. All Rights Reserved Page 1

 Advancement of Computer Technology and its Applications
smaller ad hoc, human-assisted
procedures in contemporary web scraping
strategies to adapt to a variety of settings.
Modern online scraping solutions may
parse markup languages and JSON files
in addition to integrating computer visual
analytics and natural language processing
to mimic how people navigate the web.
Butler (2007; Butler) (Yi et al. 2003) [4].
Job information is regarded as important
information in the web scraping industry.
In developed nations, 51% of workers are
searching for new jobs, and 58% of them
do so online, based on the 2017 State of
the American Workplace study by Gallup
[5].
This indicates that there is a significant
online job market, and being able to
monitor the data can benefit a job
aggregator, a business seeking to employ,
or a potential employee.
OBJECTIVE
The objective of this research project is to
develop an AI Jobs dataset that uses the
concept to collect a specific piece of
information from a job website such as
TimesJob. Indeed, and Shine, and
represent this information in a structured
format. The first phase is responsible for
connecting to the World Wide Web.
● TimesJobs Scraper
The second phase collects the
information from the websites using web
scraping techniques, and the third phase
stores the collected information into a
single common CSV file that contains
HBRP Publication Page 1-7 2022. All Rights Reserved
Volume 5 Issue 3
details from the websites.
RELATED WORK
This dataset [6] contains 732 columns
containing 6 classes named as S.No, Title,
Company, Location, Salary, and Details. It
contains data from a single website, and
the size of the dataset is small.
This dataset [7] contains 24300 columns
containing 8 classes named as S.No,
Company, GOC, job_title, primary_ind,
sector_name, revenue, and size.
This dataset is way too big and not
everyone has a high-end system to train a
model with this dataset. It contains data
from two different websites, LinkedIn and
Glassdoor, but it is in two different files.
METHODOLOGY
The project's process involves compiling
all the data that has been gathered from
multiple resources using Python scripts
that make use of the web scraper
Beautiful Soup's rich capabilities.
Coding
The project's fundamental web scraping
script presents the data that was scraped
and saved in the dataset of the
employment websites—in this example,
TimesJobs, Shine, and Indeed.
Using the Web Scraper, we have
extracted Job Title, Company Name,
Experience Required, Salary, Location,
Job Description, and Key Skills for AI
Jobs from TimesJobs.
Page 2
 Advancement of Computer Technology and its Applications
Volume 5 Issue 3

Fig.1:- Code for Implementation of TimesJobs Scraper.

● Shine Scraper
Using the Web Scraper, we have extracted Job Title, Company Name, Experience Required,
and Location for AI Jobs from Shine.

Fig.2:- Code for Implementation of Shine Scraper.

● Indeed Scraper
Using the Web Scraper, we have extracted the Job Title, Company Name, Location, and
Job Description for AI Jobs from Indeed.

Fig.3:- Code for Implementation of Indeed Scraper.

HBRP Publication Page 1-7 2022. All Rights Reserved Page 3

 Advancement of Computer Technology and its Applications
Volume 5 Issue 3

● Creating DataFrame
A DataFrame is created which contains the details of the AI Jobs which we have scraped
from the websites.

Fig 4:- Creation of DataFrame.

● Exporting DataFrame to a file (.csv)

Generating a CSV file which is the DataSet of AI Jobs, is our objective.

Fig.5:- Exporting DataFrame to a file (.csv).

● A Preview of the DataSet Generated

Fig.6:- A Preview of DataSet of AI Jobs.

HBRP Publication Page 1-7 2022. All Rights Reserved Page 4

 Advancement of Computer Technology and its Applications
TOOLS & SOFTWARE USED
We have used the following Tools and
Software in our project –
Python
Python is a general-purpose, high-level
programming language. Its design
philosophy prioritizes code readability
and heavily employs indentation. Using
its language structures and object-
oriented technique, programmers may
create clear, logical code both for small-
and large-scale projects.
In this project, we used the following
Python Libraries: -
● BeautifulSoup
Data from XML and HTML files can
be extracted using the Python library
Beautiful Soup. Along with your
preferred parser, it offers intuitive
ways to browse, search, and modify
the parse tree. It frequently saves
programmers hours or even days of
effort.
● Request
The requests library is the de facto
industry standard for making HTTP
requests [8] in Python. It provides an
easy-to-use API for communicating
with HTTP activities like GET,
POST, etc. The Requests library's
methods use a specific web server
identified by its URL to perform
HTTP operations.
● NumPy
NumPy is the abbreviation for
Numerical Python. It is a Python
library for manipulating arrays.
Additionally, it supports matrices,
the Fourier transform, and linear
algebra functions. NumPy was
originally developed by Travis
Oliphant in the year 2005. It's an
open-source project, so one can use it
without any restrictions.
● Pandas
Pandas is an open-source library
primarily made for using relational
HBRP Publication Page 1-7 2022. All Rights Reserved
Volume 5 Issue 3
or labelled data swiftly and logically.
It provides a variety of data
structures and operations for
handling numerical and time series
data. The NumPy library serves as
the foundation for this library.
Pandas is speedy and provides its
users with great productivity and
performance.
● Re
Python's built-in re-module can be
used to utilize regular expressions. A
regex, or regular expression, is a
group of characters that generates a
search pattern. To determine whether
a text contains a particular search
pattern, use RegEx.
Visual Studio Code
Microsoft created the source-code editor
Visual Studio Code, sometimes known as
VS Code, for Windows, Linux, and
macOS. Among the features are debugging
assistance, syntax highlighting, intelligent
code completion, snippets, code
refactoring, and integrated Git.
Jupyter Notebook
Jupyter Notebook is a free and open-
source tool that enables you to create and
share documents with live code,
equations, visuals, and text. Jupyter
Notebook is maintained by the Project
Jupyter team.
Among some of the features are debugging
assistance, syntax highlighting, intelligent
code completion, snippets, code
refactoring, and integrated Git.
RESULT
A DataSet [9] of AI Jobs contains
approximately 1500 entries from various
job websites; in this case, TimesJobs,
Shine, and Indeed have been successfully
generated. It includes Job Title, Company
Name, Experience Required, Salary,
Location, Job Description, and Key Skills
as features.
Page 5
 Advancement of Computer Technology and its Applications
Volume 5 Issue 3

CONCLUSION relatives, classmates, and friends during

The DataSet that is generated can be used the research project. Last but not least, we
by candidates who are actively looking for wish to thank everyone who has
a job as an AI Engineer, which can help contributed in any manner to our efforts.
them identify the requirements of Being able to learn at SHRI VAISHNAV
recruiters for a particular job, which can VIDYAPEETH VISHWAVIDYALAYA
give them a major competitive advantage. in Indore has been a blessing.
We can also analyze the job trends in the
market. REFERENCES
1. “World Wide Web.”
FUTURE SCOPE SpringerReference, doi:
We can increase the entries in the DataSet 10.1007/springerreference_28639.
by fetching more entries from the same 2. Mellor, C. Seagate sponsors re-
websites or we can also scrape more fashioned IDC digital data-flood
websites for AI Jobs. We can also increase blather. The Register. Apr. 2017.
the features in the DataSet. Machine https://www.theregister.com/2017/
Learning models [10] can also be built to 04/05/seagate_sponsors_refashione
find the important skills required by d_idc_digital_universe_blather/
recruiters. (accessed: Jun.2022).
3. Coble, Z. Collecting Data with
ACKNOWLEDGEMENT Web Scraping.
We are grateful to various folks for their 4. Zhao, B. Web Scraping.
advice and help during the period of Encyclopedia of Big Data: 951-953,
finishing our research project. First and 2022, doi: 10.1007/978-3-319-
foremost, we would like to express our 32010-6_483.
gratitude to Dr. Anand Rajavat, Head of 5. State of the American Workplace.
the Department of Computer Science & GALLUP. 2017.
Engineering at S.V.I.I.T., Indore, and Er. https://www.gallup.com/workplace/
Nikhil Chaturvedi, the research project 238085/state-american-workplace-
mentor, for their invaluable assistance and report-2017.aspx (accessed:
guidance. They also assisted us in Jul.2022).
discovering new technologies by sharing 6. Pawar. H. AI jobs in India. Kaggle.
their technical knowledge. With his 2020.
supervision, direction, and constructive https://www.kaggle.com/datasets/hri
criticism, he provided inspired us. thikpawar/ai-jobs-in-india (accessed:
Jun.2022).
We also want to thank our director, Dr. 7. Bansal. S. 2021 DS / ML related
Anand Rajavat, from the bottom of our Jobs - Linkedin, Glassdoor. Kaggle.
hearts for all of his help. Dec. 2021.
https://www.kaggle.com/datasets/shi
We extend our heartfelt gratitude to the vamb/glassdoor-jobs-data (accessed:
teaching and non-teaching personnel in the Jun.2022).
Computer Science & Engineering 8. Freeman, A (2022). Making HTTP
department at SVVV Indore for their kind Requests. Pro Angular:665-690.
cooperation and provision of the necessary 9. Pluijmaekers, P. , Lelli, F (2022) . A
information. Dataset Containing Job Descriptions
Suitable for NLP and NN
We appreciate the support and helpful Processing. doi:
suggestions provided by our parents, 10.20944/preprints202206.0346.v1.

HBRP Publication Page 1-7 2022. All Rights Reserved Page 6

 Advancement of Computer Technology and its Applications
Volume 5 Issue 3

10. Chen R.H, Chen, C. (2022)

Machine Learning Modeling.
Artificial Intelligence:119-124, doi:
10.1201/9781003214892-14.

HBRP Publication Page 1-7 2022. All Rights Reserved Page 7