Web scraping by using R

Web Scraping
1. Overview/Usefulness
2. Prerequisites
3. HTML Overview
4. Code for web scraping
5. End note
1.Overview/Usefulness:
There has never been a time where information has been more readily available online. Data
growth on the world wide web has continued to exponentially increase over the past decade
and has given no indication of slowing sometime. While the presence of online information is
in clear abundance, accessing that information is not such a simple endeavor. This tutorial is
designed to help those in need of access to online information by providing a method to
extract data from webpages via web scraping. This method can be effectively used with the
programming language R, and a package called rvest.
Web scraping is extracting large amounts of data from resources that are located on the
World Wide Web. This data is extracted and stored on the scraper’s computer or to a
database. Many businesses and organizations across the globe need this technique to maintain
a competitive advantage, increase revenue, or maintain a working knowledge of what their
competition is doing. Government use of web scraping can be viewed in competitor analysis,
as well as providing insight into
personal circumstances facing the country through social media. Applications can also
extend to the acquisition process used by military agencies in procurement research.
Government, however, is not the only entity that benefits from the use of web scraping.
Industry examples of web scraping include companies gathering email addresses to bolster
lead generation, learning what competitors are selling and selling similar or the same
products, an inspection of competitor prices, and scraping information
on social media websites to learn what’s trending. Web scraping, typically, is straightforward
in concept, but presents many challenges that include:
1. Each website has a unique infrastructure and requires a unique script.
2. Unique script languages may be written for each page in a single website.
3. Webpages may be altered regularly by web developers. Slight changes in the code
may require a complete script rewrite for web crawlers.
4. Successfully scraping a specific piece of data from a website does not mean that the
information itself will be imported perfectly. It may be, and often is necessary vital to
purge the data of irregularities.
5. Some web pages have been purposefully designed to prevent actions such as web
scraping. Many professional web crawling companies have come about the provide
businesses with data on their competition.
2. Prerequisites:
library(rvest)
library(tidyverse)
library(stringr)
library(knitr)
3. HTML Overview:
This section covers the foundation of scraping website data from a single webpage.
Moreover, this section will illustrate a method of extracting specific elements of information
embedded within a webpage, with an explicit focus on extracting data from HTML websites.
To begin, must provide a concise explanation of how HTML webpages are typically
arranged. HTML layouts are provided by Cascading Style Sheets (CSS) instructions which
are embedded in the HTML. CSS is a web style sheet. The language used to describe the
presentation of a document written in a markup language. This technology is used amongst
many websites to deliver visually engaging web pages and user interfaces for both web
applications and mobile applications. CSS enables the differentiation of the presentation
aspects of a webpage and the content of the webpage. This permits website developers to
maintain thematic concepts among several webpages while changing the content of each
page. This structure is
governed by a set of rules, housed within each sheet made up of one or more selector. CSS
selectors are used to defining which parts of the HTML style apply to different sections on
the web page by matching tags and attributes in the markup itself. Selectors can be applied to
an entire HTML document and specified components such as headers, for instance. An
example of a defined heading in a CSS selector would be main heading as (h1), sub-headings
as (h2), and sub-sub-headings as (h3). HTML elements are written with a start tag identifying
the section, the content, and an ending tag which identifies the closing of the section. The
start tag identifier is housed between < and > symbols and the desired content would follow
directly after. The end tag identifier is housed between </ and > symbols. An example of a
CSS selector is: Some of the most commonly identified tags present in CSS selectors are

 <h1>, <h2>, …, <hn>: Largest headings, second largest headings, etc.

 <p>: paragraph elements
 <ul>: Unordered bulleted list
 <ol>: Ordered list
 <li>: Individual List item
 <div>: Division or section
 <table>: Table

4. Code for web scraping:

install.packages('XML')
install.packages('rvest')
install.packages('magrittr')
library(rvest)
library(XML)
library(magrittr)
cummuter_complaints <-NULL
complaints <- NULL
url1 <- "provide desired url address "

for(i in n){ # n means number of subpages in the url

url <- as.character(read_html(as.character(paste(url1,i,sep=""))))

complaints <- url %>% as.character() %>% read_html() %>%

html_nodes(".compl-text div") %>% html_text()
# ifelse(i<=0,

#complaints <- url1 %>% html_nodes("div") %>% html_text()

#complaints <- url %>% html_nodes(id = "load-more-trigger") %>%
html_nodes(".show-more__control") %>% html_text()
# )

cummuter_complaints <-c(cummuter_complaints,complaints)
complaints <-NULL

write.table(cummuter_complaints,file= "provide path to save the

file.txt",sep = "",row.names = FALSE)

5. End note:

R language will give greate levaerage and it is robust, you can extract data from website and
can do various types of analysis depending on your requirement.

Note: some websites are having the technology of anti web scraping where web scraping is
not possible. However, you do need to understand the legality of scraping data and whatever
you are doing with the scraped data.