BIG DATA ANALYTICS

Basics of Big Data

• Big data is a term for data sets that are so large or complex that traditional data processing
application software is inadequate to deal with them. Big data challenges include capturing
data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating
and information privacy.
• The data could be from social networks, web server logs, traffic flow sensors, satellite imagery,
broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages,
scans of government documents, GPS trails, telemetry from automobiles, financial market data
and so on.
• It answers specific questions such as the need of the customers, their opinion and image of the
brand.
• For organisations, analysis of this hidden data may give an insight into things which were
previously hidden due to its bulk and the subsequent cost required for its process. This is done
by collecting, organizing and analysing large sets of data to discover patterns and other useful
information.
• For instance, analysis of shoppers’ transactions, social and geographical data gives the analyst
knowledge about peer influence on customers, greatly reducing the time that would otherwise
require for sampling followed by extensive investigations.
• It also enables new products and services, by combining a large number of signals from a user’s
actions and those of their friends, Facebook has been able to craft a highly personalized user
experience and create a new kind of advertising business.
• With the right big data analytics platforms an enterprise can boost sales, increase efficiency,
and improve operations, customer service and risk management.
• One of the fundamental reasons for opposition of Big Data is centred on privacy since massive
amounts of personal data is collected and analysed without a consideration to the person in
question.
• The large volume of information being collected may be used by finance companies to personalise
various schemes for maximisation of their benefits thereby leading to indiscrimination against
a certain group of people.

Characteristics of ‘Big Data’

Science and Technology Satveer Sir 1

Applications based on ‘Big Data’ Technology
• Seed Selection – Big-data businesses can analyse varieties of seeds across numerous fields,
soil types, and climates and select the best.
• Crop disease – Similar to the way in which Google can identify flu outbreaks based on where
web searches are originating, analysing crops across farms helps identify diseases that could
ruin a potential harvest.
• Irrigation – Precision agriculture aids farmers in tailored and effective water management,
helping in production, improving economic efficiency and minimising waste and environmental
impact.
• Weather – Advanced analytics capabilities and agri-robotics such as aerial imagery, sensors
help provide sophisticated local weather forecasts can help increasing global agricultural
productivity over the next few decades.
• Climate change – Since, climate change and extreme weather events will demand proactive
measures to adapt or develop resiliency, Big Data can bring in the right information to take
informed decisions.
• Food processing – They help in streamlining food processing value chains by finding the core
determinants of process performance, and acting to continually improve the accuracy, quality
and yield of production. They also optimise production schedules based on supplier, customer,
machine availability and cost constraints.
• Loss control – In India, every year 21 million tons of wheat is lost, primarily due to scare cold-
storage centres and refrigerated vehicles, poor transportation facilities and unreliable electricity
supply. Big Data has the potential of systematisation of demand forecasting thus reducing such
losses.
• Pricing – A trading platform for agricultural commodities that links small-scale producers to
retailers and bulk purchasers via mobile phone messaging can help send up-to-date market
prices via an app or SMS and connect farmers with buyers, offering collective bargaining
opportunities for small and marginal farmers.

Science and Technology Satveer Sir 2

How can it be a Challenge?
• The challenges and opportunities of data is immense in a country like India with 638,000 villages
and 130 million with 140 million hectares of cultivable land under 127 agro climatic regions
capable of supporting 3,000 different crops and one million varieties.
• Self-driven vehicles can already drive themselves across fields using Global Positioning System
(GPS) signals accurate to less than inch of error thus helping farmers plant more accurately.
• But the real potential is what happens when this data from thousands of tractors on thousands
of farms is collected, grouped and analysed in real time.
• There is need to formulate a business model wherein value can be captured from the scale of
data being captured by different players in the agri-supply chain.
• Companies must act now to focus, simplify and standardise big data through an enterprise-wide
data management strategy.

Role of Big Data in Governance

One of the greatest strengths of big data is its flexibility and universal application to so many
different industries. Along with many other areas, big data in governance can have an enormous impact
— local, national and global. With so many complex issues on the table today, governments have their
work cut out trying to make sense of all the information they receive and make vital decisions that
affect millions of people. Not only is it difficult to sift through all the information, but it’s sometimes
difficult to verify the reality of the information itself. Faulty information can have awful consequences.
By implementing a big data platform, governments can access vast amounts of relevant information
important to their daily functions. The positive effect it can have is nearly endless. It’s so important
because it not only allows the government to pinpoint areas that need attention, but it also gives them
that information in real time. In a society that moves so quickly from one thing to the next, real-time
analysis is vital. It allows governments to make faster decisions, and it allows them to monitor those
decisions and quickly enact changes if necessary. Here are just a few of the areas that big data can
positively affect at the government level.
• Transportation: Every day millions of Americans are on the road driving. There are so many
different nuances to driver safety, from roads to police officers, weather conditions and vehicle
safety that it’s impossible to control everything that might cause an accident. However, with big
data governments can better oversee transportation to ensure better roads, safer roadways,
better routes and new routes.
• Healthcare: Healthcare is a very complicated issue these days, and not just here in the United
States, but also across the world. With so many health systems that rely on government subsidies
and support, there is a potential for resources to be wasted or to be unfairly allocated. With big
data, governments can have a much clearer picture of where the money is going and why. It
means they can also assume better control over resources. They can also analyze more effectively
the needs of the citizens and from there make the necessary changes to provide the citizens
with the best possible services for the best possible prices.
• Education: Education is another extremely hot topic across the country. What can be done to
improve education? There are a lot of different things to be done, and up-to-date, relevant
information is vital to this. Big data helps governments understand more about educational
needs on a local and federal level in order to ensure that the youth of the nation are getting the
best possible education in order to serve the country in the future.
• Agriculture: How do you keep track of so much land and livestock that exists in our country and
across the globe? All the different crops that are grown, the animals that are held and so many
other complicated issues come together in the agriculture world to form a very difficult job for
the government. It’s hard to monitor because of the vast numbers. Big data is changing the ways
governments manage and support the farmers and their resources. It’s ability to gather huge
amounts of information and analyze them quickly makes all the difference.

Science and Technology Satveer Sir 3

 • Poverty: There is so much poverty in the world. It’s extremely difficult to combat, and we’ve
been trying to do so for thousands of years. Big data gives governments tools to discover more
effective and innovative ideas on how to decrease poverty across the world. It’s easier to pinpoint
areas with the greatest need and how those needs can be met.
Big data technology is vitally important for governments across the world. It can’t solve every problem,
but it’s a step in the right direction. It’s giving leaders the tools necessary to enact important changes
that will be of benefit for citizens now and in the future.

Challenges to India for using big data and analytics for better governance:
Information technology systems abound these days. India, the maker of the largest data repository
in the world, now is faced with the challenge of how to use huge data sets for better governance. The
country owns the biggest data complex gathered through digitalization of records for purposes such
as passports, IDs and subsidies payment. Big data is characterized by its variety, volume, speed and
the analytics involves in processing cost-effectively to be able to draw conclusions for their useful
application.

USING LARGE DATA SETS TO IMPROVE THE CUSTOMER EXPERIENCE

Information technology is at the heart of India. For many years it has been the outsourcing
destination of the world, catering to clientele from different parts of the world, offering highly efficient
services and products. With its enormous amount of data, it has opened up numerous opportunities
to apply the data to improve the customer experience. Furthermore, it could also be used to boost the
efficiency of the government, particularly in delivering services and to boost business, to build
capacity to serve domestic and export markets. Big data analytics that merges into fields such as
machine learning, deep learning and artificial intelligence has huge possibilities.
With the internet of things coming into its own after ITeS or information technology enabled
services, a whole new world opens up for data with things such as sensors, for example. Experts
claim that the relevance of big data could be gauged from the fact that ninety percent of digital
information all over the world has been made in the last couple of years, while processing power has
risen by forty percent between the years 2010 and 2016. At the same time, the cost of data storage
has plunged 500 percent.

NEW OPPORTUNITIES AND NEW RISKS

Big data brings new opportunities as well as new risks when it comes to integrity and confidentiality.
Thus, good data governance is essential for organizations in the big data world these days.
Furthermore, companies have to adopt practical steps to manage it effectively. Huge data and
associated analytics are advantageous in numerous areas, such as,
• Resolving traffic problems in cities
• Efficient supply chain management
• Targeting healthcare delivery
• Providing personalized educational experience for students
• Preventive steps to protect the environment
• Enabling security to people and society
• Informed policy making

CRITICAL ASPECTS OF BIG DATA

One of the most critical aspects of data analytics is its effect on how decisions are made and who
makes them. When data is costly to obtain, scarce or not available in digital form, it makes sense to
allow people with experience to make decisions based on relationship and patterns they have observed
and have internalized. Leaders state their opinions on what the future brings, what will happen, how
well something would work and so on as per their intuition and plan accordingly.

Science and Technology Satveer Sir 4

 Nonetheless, in the big data age, managers and leaders in private companies and government
should be data-driven. They must have the courage to ignore their intuition and do what the data
states. This needs a change in mindset as well as effective training to make decisions that are data-
driven. Although businesses have adopted data and analytics in different forms efficiently to boost
business efficiency and personalize offerings, governments have been laggards. The possible benefits
that data analytics could bring to government could vary from transforming the programs of the
government and empowering citizens to boost transparency and to enable participation of all stakeholders.

THE GOVERNMENT IN A BIG DATA WORLD

Governments differ from businesses when it comes to goals, mission and decision making. Although
decision makers in businesses are limited, they are a diverse set in government. The government has
tremendous data in legacy databases and forms that must be curated and migrated for new-age analytic
tools. Also, data collection id a paramount task for government since data is received from numerous
online and offline channels. Data sharing between departments and across ministries is a challenge,
provided the existing jurisdictional boundaries.
Data analytics, when properly managed, could be a blessing to businesses struggling to gain bigger
insights as well as competitive edge in the marketplace in the world. However, it could also be a curse
to organizations that are not prepared to handle the untold x-bytes of unstructured data pouring
unceasingly into their coffers. This is the dilemma most companies wake up to every day, and thus
they capture only a fraction of the value that data analytics brings.
Big data could have a huge impact only when used on a massive scale, with safeguards by the Indian
government for the delivery of public services and goods.

INDIA’S ADVANTAGE
• NASSCOM has set the target of making India one among the top three big data markets in next
three years
• Currently, Big data industry is employing 90k people in various sectors.
• Govt agencies + Big data analyses= Aadhaar and UPI
• CAG has drafted a big data management policy to improve their functions using big data. It is to
exploit the data-rich environment in the state and union govt, building capacity in the Indian
Audit and account debt ·
• DISCOMS- capturing data from sensors that are installed at the last mile of power construction
road and railways safety measures can be taken with the help of big data.
• Reliance Jio and storing of big data, govt can use those data if necessary
• AkshayPatra[G15] foundation in Bangalore that uses data analysis to deliver food to schools in a
cost-effective manner
• keeping Information about terrorist groups, people, their activities, through big data and sharing
those data with other countries, thus building consensus on protection measures
• Defence sector can use big data to build a strong security NITI AYOG in attempt to optimise
private business and public goods and services is furthering the idea of India’s capability in big
data
It creates a new paradigm with data being created by various kinds of satellites, smartphones,
sensors and social sites. With rapid advancement in fields of growth for big data industry in India is
clearly visible.

Big Data for Better Governance

• Collaborative federalism with focus on balanced regional development is a prime objective of the
Niti Aayog.
• Effective use of big data analytics is called for, for making this developmental objective more
meaningful.

Science and Technology Satveer Sir 5

Need
• There is a notable information asymmetry at various administrative levels.
• This is evidently hampering the targeting of various government measures and keeps the
outcomes largely undermined.
• There are entities that have outlived their utility and others that use outdated systems and
processes.
• This has to be eliminated and existing legacy systems need to be analysed with clear data
points.
• The UN, by a resolution on official statistics, expects India to produce quality statistics.
• The purpose is to shed enough light on the true picture of material and human resources and
the needs and demands of the societies.

Steps to be taken
• Data - Statistics must be offered as a public good for the government, enterprises and the general
public.
• The big data analytic centres do have micro data, geo-coded, along with tools for extraction of
relevant information.
• There has to be a quantitative analysis on all these.
• There is also the need for devising a formula for aggregation of data to enforce a code of practice.
• Area of focus - Health, education and demography are the significant and demanding areas in
this regard.
• Web-based reporting for timely collection, collation and dissemination should be taken up.
• Segregating the data under different socio-economic heads would facilitate making appropriate
response for concerns in each of them.
• Localisation - Data at the district level would enable understanding the true picture at the
ground level.
• The impact of developmental schemes gets captured at the smallest administrative level.
• This would be supportive for better implementation of policy initiatives and making course
corrections.
• Organisation - Establishing such a massive data pipeline is indeed highly challenging.
• It should thus be ensured that data once captured are handled in an organised fashion.
• These are essential for making the developmental initiatives meaningful.

Big Data Initiative

Government is trying to collect data of all of its citizens through Aadhaar, CMS (Central Monitoring
System) and NATGRID projects; this in turn may initiate the issue of Right to Privacy.

Government projects that are collecting Big Data

a. Aadhar: People belonging to marginalized sections of society in India often do not have a valid
proof of identity. As a result, they miss out on availing social benefits provided by the government.
To overcome this Indian government launched a scheme to issue a unique 12-digit number,
termed ‘Aadhaar’ (meaning ‘foundation’ or ‘support’) to every resident of India. It is an identification
that a person can carry for a life time and potentially use with any service provider. Aadhaar is
the world’s largest ID platform. It is also the largest biometric programme in the world, as biometric
data of each person is recorded and stored. This unique identification is now being used by
various Government agencies to ensure that services and subsidies are made available only to
the people to whom they are targeted and preventing leakages in the delivery mechanisms.
b. Digilocker: DigiLocker provides a personal storage space in the cloud to Indian citizens.
Organizations that are registered with DigiLocker can push electronic copies of documents and
certificates (e.g. driving license, Voter ID, School certificates) directly into citizens’ lockers.
Citizens can also upload scanned copies of their legacy documents in their accounts. These
legacy documents can be electronically signed using the eSign facility provided in DigiLocker. A
citizen can share these electronic certificates online with various agencies while applying for
the services provided by them, without having to provide paper copies.

Science and Technology Satveer Sir 6

Criticism of Big Data Initiative
The critics of the big data are opposing Aadhar scheme, central monitoring system and NATGRID
project of the government. They are worrying about the safety of the data as it may infringe the privacy
of the people. This may also lead to large increase in state surveillance, which the government takes in
form of data collection, data mining and other such invasive methods to prevent crime, terrorist attacks
and to deliver welfare service.
Further, the NATGRID links multiple government databases. There is no clarity whether the Aadhar
will be connected to NATGRID or not.
The imbalance of power created by the state’s attempts at treating citizens like pawns is dangerously
magnified by advances in digital technology that allow for easy monitoring of communication and access
to large amounts of data.
The graver threat is a digital replay of colonial era exploitation, with data replacing mineral resources
and raw materials as the source of value.

Benefits of Big Data initiative

According to some expert’s big data when combine with Internet of things, where the majority of
gadgets, machines and human will be connected through internet, will open up a future where all
important decisions about business, life and society would be taken on the basis of data. This kind of
decision making is called as ‘evidence-based decision making’.
It is already playing a big role in the management of industry and infrastructure. Some argue that
with the advances in cloud computing will change how humans think and therefore act and live.

Key actions required for successful implementation of initiatives are:

1. Talent Pool– Create industry academia partnership to groom the talent pool in universities as
well as develop strong internal training curriculum to advance analytical depth.
2. Collaborate– Form analytics forum across organization boundaries to discuss the pain-points of
the practitioner community and share best practices to scale analytics organizations.
3. Capability Development– Invest in long term skills and capabilities that forms the basis for
differentiation and value creation. There needs to be an innovation culture that will facilitate IP
creation and asset development.
4. Value Creation– Building rigor to measure the impact of analytics deployment is very critical to
earn legitimacy within the organization.

Conclusion
The balance between big data initiative and the privacy issue arising out is needed.Moreover, while
privacy is necessary for a functional democracy, it is not the only causality of big data. What disturbs
besides privacy concern is a digital replay of colonial era exploitation by replacing mineral resources
and raw materials with data. Particularly India being a developing country and having less stringent
laws regarding cyber security needs to first ensure a safe and secure platform with stringent laws
dealing with data.

Role of Big Data in Disaster Management

• Disaster management is a systematic process with primary aim to reduce the negative
consequences and effect of disasters, hence safeguarding people and social infrastructure.
• Effective management as well as monitoring of disasters is a global challenge.
• As the number and access to different datasets is expanding rapidly, the potential and utility of
big data is growing for disaster management.
• Big Data has already saved lives and proven effective within the emergency management field.
• India and South Korea have signed agreements including cooperation in development of big
data technologies for their diverse applications, like disaster management.

Science and Technology Satveer Sir 7

Need for Big Data in Disaster Management
• The storage and processing of large volumes of disaster data are the biggest challenges faced by
civil defense, police, fire departments, public health and other government organizations.
• It is very crucial for these organizations to get processed real-time disaster data as quick as
possible in order to react and coordinate efficiently.
• Big data tools and techniques can assist disaster management officials to optimize decision-
making procedures.
• Effective planning and management hugely depends on the quality as well as quantity of the
data available.
· Emergency personnel can minimize their search time and maximize their recovery time when
they have access to real-time information.

Applications
• Empower decision-makers to make accurate assessment during a disaster.
• Big data generated from geo-informatics and remote sensing platforms can contribute to early
warning systems for disasters.
• Integration of different data streams, along with data processing and storage is effective for
disaster preparedness.
• Help in the development of effective strategies and contribute to minimize the potential effects
of disasters.
• Significance of big data analytics to predict occurrences of the floods and for flood management.
• It will help for timely humanitarian response to different disasters.
• Using geospatial datasets along with big data paradigm can provide location based services to
avoid hazardous situations.
• It will also benefit in the identification of regions which need the most urgent attention.

Science and Technology Satveer Sir 8

 • Enhance disaster recovery by utilizing community information and connecting victims with
emergency responders and family.
• Connecting Missing People with Their Families
• With big data, safety professionals can better prepare disaster simulations for more accurate
implementations.
• Big Data opens up new career opportunities for those who want to find innovative ways to help
others.

Limitations
• Understanding how to link different datasets with different kinds of disasters.
• The potential of big data technology has not been fully explored for disaster management.
• Not all big data is public and freely available.
• Network security threats and vulnerabilities.
• Challenges related to protection of personal information and privacy.
Way Forward
• Integration of datasets along with providing access to information to agencies managing disasters
is crucial to enable effective decision making.
• It’s important to protect individuals’ identifications and efforts should be put to anonymize the
collected datasets.
• Leverage techniques from artificial intelligence (AI) and machine learning (ML) to understand,
correlate and draw findings from the disaster related data.
• Analysis from processed disasters information can help to identify the most effective strategies
to respond future disasters.
• Ensure data consistency, accuracy and completeness for decision making processes.
• Need to investigate data mining challenges as well for disaster management.
• Security as well as privacy issues in data transmission and storage also need to be under constant
investigation.

Role of Big data analytics in dealing with the social issues faced by India
Big data is characterised by 4 Vs - Volume, Variety, Veracity and Velocity and refers to hugeness in
all the 4 respects. Big Data Analytics refer to the analysis of the Big data to come up with meaningful
inferences, which can be applied in various sectors.
India is a culturally diverse nations and hence conflicts arise between different groups and
communities. It has resulted in some castes exerting superiority and depriving other castes. Hence, in
this socio-economic inequality milieu, Big data has following uses which can help in dealing with such
issues:
• Targeting needs better: ‘One size fits all’ concept while formulating policies fails in Indian
context. Big data analytics will provide the specific needs of the people of different regions, hence
helping government work better for their upliftment.
• Gender parity: Data from schools, surveys, health care centres can be analysed to obtain the
progress made in gender parity at various fronts and success of various women empowerment
schemes.
• Rights of weaker sections: Analysis of employment pattern and educational status of weaker
sections can help amend reservation and other welfare policies to provide equal opportunities to
all.
• Improving agriculture’s state: Big data can help government target agricultural subsidies in a
better way, thus preventing farmers from falling into poverty trap.

Science and Technology Satveer Sir 9

 • Fight against corruption: Big data has the potential to fight corruption in various departments,
black money, hoarding and smuggling by enabling collection and analysis of unofficial data through
social networking and other sites.
Hence, big data which is being intensively used by E-commerce firms to target consumer’s
preferences can provide a way out for government to improve its governance and fight social issues in
an effective manner.

Big Data Ethics

• “Data is the new oil and Big Data the oil reserve”. To exploit the power and wealth behind these
big data, corporates and countries alike, are running after it. Big Data ethics tries to balance
the opportunity created by Big Data with responsible and ethical use of data.
• However, since laws as well as societal values are unable to keep up with the rapid pace with
which Big Data is being utilized, a number of issues have cropped up. Privacy is the first victim
without which Big Data loses its potency. It is used to mine, store, analyze and monetize huge
amount of private data without the knowledge of the users.
• It is also used to predict and influence human behaviour. A revelation of Cambridge Analytica
shows how such data can be used to influence swing voters, election results and critical democratic
decisions, often touted as being the new face of colonialism.
• Big corporates like Facebook when indulged in unethical use of Big Data also leads to breach of
trust of users.These users are often helpless about how to protect their data and extract
accountability from these big corporates.
• A number of applications and websites coerce users to allow them access to personal data without
which the users cannot avail the benefits of the platforms. These are then used to track user
behaviour though cookies. This not only leads to unwanted surveillance but strips users of any
personal choice. Countries like China has used it to unleash huge surveillance systems.
• Big Data has huge benefits but when used in unethical manner, they also have terrible costs,
from undermining individual rights to subverting democracy and sovereignty when data is being
controlled by others. If proper measures are not taken, it may lead to many ugly consequences
like using Big Data by terrorists and demagogues to increase violence and communal flares.
Apart from strengthening individual control over one’s data, state must come up with adequate
control mechanisms to prevent and raise cost of its misuse because it is ultimately the responsibility
of the state to protect its citizens.

Summary
Big Data is the new reality of everyday living and stands for an immensely huge pool of data which
cannot be de-jargoned with ordinary and archaic methods of data interpretation. Such huge data sets
form the basis of all reasonable aspects of society. Data-driven decision-making is gaining leverage and
has lent more authenticity to modern ways of life.
Thus, it is not just big data but the various correlations or linkages which can be drawn between two
completely different data-sets that has led to explosion in value of the unstructured and unsorted datasets.
Such analysis is highly complex and cumbersome as most data comes in unorganised format which
takes painstaking efforts to analyse, organize, retrieve and model the mine of data. Another major
hurdle after all this is crossed is accurate interpretation and presentation of the results. This step is
critical to drawing relevant conclusions and actionable knowledge.
Data management and analysis poses novel challenges to the experts. This creates a need for
appropriate investment of time and resources to streamline the processes further, to create more
economic value for the nations. One has to come up with revolutionary approaches to completely
turnaround traditional modes of data analysis tools and systems.



Science and Technology Satveer Sir 10