Spot On

Journalists struggle to get political opinion

polls right
By KRISHN KAUSHIK | 1 December 2013

ON THE MORNING OF 27 JANUARY 2004the day after Indias 55th

Republic DayPrime Minister Atal Bihari Vajpayee drove to Rashtrapati
Bhavan to meet President Abdul Kalam. He was carrying a letter from his
cabinet, which recommended that the Lok Sabha be dissolved so that the
National Democratic Alliance (NDA) government could face elections eight
months before the expiry of its term. The decision was bolstered by the
confidence gained from its stellar performance in the state elections held
the previous month; the Bharatiya Janata Party had defeated the
Congress in three crucial states: Rajasthan, Madhya Pradesh, and
Chhattisgarh.
Four days later, India Today published an opinion poll predicting that the
NDA would win a whopping 330 seats. No party or alliance had achieved
such a victory since the Congress won 415 seats in 1984. A reassured
Vajpayee called on the president again, on 5 February, and told him that
all consitutional formalitiesincluding passing a finance bill that would
allow the new government to pay for essential goods and services while it
drafted a new budgetwere complete. Kalam dissolved the 13th Lok
Sabha the next day, and on 1 March the Election Commission announced
that voting would begin in April, six months earlier than it was due.
Following the announcement, at least five more polls by major news
networks and pollsters projected that the NDA would win somewhere in
the region of 270 Lok Sabha berths. The Congress and its pre-election
allies were predicted to get anything between 150 and 170. With no
outliers, everyone in the media was confident that they were more or less
right. In mid April, Vir Sanghvi, then an editorial director for HT Media,
which publishes the Hindustan Times, wrote in that paper, I dont know
of a single person who thinks that the Congress will get more than 120
seats and most people say it will get even less. Plus, I suspect that
Vajpayee as Prime Minister is probably unbeatable.
By late April, it was clear the Congress had made some gains, but
pollsters and journalists were still confident of an NDA victory. The cover
of the 26 April edition of India Today peddled an exclusive survey
(commissioned in partnership with Aaj Tak and Dainik Bhaskar, and
conducted by the market research firm ORG-MARG), which it claimed was
the most exhaustive opinion poll ever undertaken in India. After
surveying over 50,600 voters from 185 Lok Sabha constituencies across
the country, it predicted that even as the Congress gained momentum,
the NDA would win 282 seats. Even in early May, after three rounds of the
four-phased election, exit polls had the NDA winning although it was
predicted that the coalition might miss the majority mark of 272 by a
whisker.

Three weeks later, when the Election Commission announced the results,
the bedrock of the NDA, Vajpayees Bharatiya Janata Party, had lost about
1.6 percentage points from the vote share it had garnered in the last Lok
Sabha elections, in 1999, sliding from 23.75 percent to 22.16 percent.
The Congress lost 1.8 percentage points, dropping to 26.53 percent of
votes nationwide. But despite similar declines in vote share, the two
parties achieved contradictory results. While the BJP lost 44 seats, its
main challenger gained 31. In the end, the NDA did not even touch 200,
falling 15 short. Congresss alliance marshalled 217, and eventually
secured enough external support to form a new government.
It wasnt the first time that political polls had been inaccurate in India
but calling the 2004 polls inaccurate is like calling a tsunami a ripple. Not
only did every major pollster and media outlet get the numbers wrong,
they miserably failed to predict the overall trend. Worse, they had
oversold their polls in a way that now seemed disingenuous. India Today,
which had conducted 155 opinion polls since 1978, would have known
that even the most rigorous survey can get the final outcome wrong. But
the magazine, and others in the media, presented their various polls as
definitive forecasts.
Some members of the press, humiliated by the discrepancy between their
headlines and the results, laid the blame on pollsters. I dont see any
reason why this magazine should carry the can for the incompetence and
ineptitude of desi and foreign pollwallahs who use the media in order to
pontificate on their brilliant scientific models, an angry Vinod Mehta,
then the editor-in-chief of Outlook, wrote. Some readers are convinced
Outlook manipulates its polls. I would like to inform them that I only pay
huge sums of money to pollsters so that they can embarrass me with
hopelessly inaccurate predictions.
TODAY, despite the embarrassment of years like 2004, there is little to
suggest that the media has become more cautious in selling opinion polls.
Survey after survey is proclaimed to be the definitive exercise in election
forecasting. Every seat, every state, every alliance and, finally, every
election, is the subject of self-assured prophesying.
Perhaps as a corollary, cynicism about opinion polling seems fairly
widespread. Since 1997, the Election Commission has contemplated
banning opinion polls, claiming, without clear evidence, that voters will be
unfairly swayed by survey results. In a letter written to the Election
Commission on 30 October this year, the Congress supported the proposal
to prohibit the dissemination of surveys once elections are announced.
The party said polls lack credibility and had the potential to be
manipulated.
But the scepticism isnt limited to political parties, which are generally
eager to protect the morale of their cadres from negative survey results.
In the last two general elections, the majority of opinion polls have been
off the mark when it comes to the number of seats the two major parties
will win, and this seems to have fostered public distrust of pollsters and

their methods. If the final number is wrong, many people tend to assume
a poll was either manipulated or based on flawed science.
In reality, however, most of the opinion polls published and broadcast by
national media houses are fairly robust exercises led by social scientists
with decades of experience. Although they sometimes get the final tally of
seats wrong, this isnt necessarily the result of poor polling methods or
bias. Rather, its a reflection of the inherent difficulty of distilling into a
handful of numbers the vastness and diversity of the Indian electorate
and the fickleness of Indian politicsa challenge of which pollsters and
editors are acutely aware.
But opinion polls are nevertheless presented as a sort of political gospel,
and the media often gives viewers and readers only the final outcome of a
surveyseat projections. But these projections are subject to greater
uncertainty than any other product of the entire polling exercise. Behind
the numbers are two complex, expensive processes. The first
comparatively easy but by no means simpleinvolves holding structured
conversations with a large number of people to understand how they are
likely to vote and to determine a party or candidates probable vote share.
The second is to create seat projections by subjecting vote share to
various mathematical models and to adjustments based on historical
precedents and a dizzying array of political factors.
Rarely does the complexity of these processes get conveyed by
journalists, many of whom feel the public are only interested in seat
projections. If we dont give seat number projections based on vote
shares, the viewer feels cheated, Rajdeep Sardesai, the editor-in-chief of
CNN-IBN, told me. Then the viewer says tum cop out kar rahe ho (You
are copping out). You are not telling me who will get how many seats. I
am not interested in the vote share. So after having spent so much
money, if you give only vote share then viewers and readers are upset.
Unfortunately, Sardesai added, most viewers and readers expect it to be
arithmetic, saying, why didnt you get this number right?
Most of the opinion polls available for public consumption are solely
commissioned by the media. But there is little consensus on why they do
it. Almost all the editors I met while reporting this story believe their
audiences are interested in the surveys, but theres no way to measure
that. Sardesai and other television journalists said it does not make much
of a difference to their viewership numbers. Newspaper and magazine
editors told me that in India, publications are largely subscription-driven
and readers seldom buy papers off the stands, so its impossible to detect
short-term fluctuations in readership. But Sardesai claimed that some of
the data can be used to improve reporting on topics that survey
respondents care about, and that, he thinks, is extremely valuable for any
editor.
Some journalists I spoke with called polls a publicity stunt, even though
its not clear that surveys contribute to a media organisations bottom
line. Others said that polls add value to their socio-political understanding

and reporting. A good opinion pollone that is transparent about its

methodology, adequate and representative in its sample, and honest
about its fundingis a tool that can help journalists understand the mood
of broad segments of the population. It can elucidate voting behaviour
even if the seat numbers are off the mark, providing insight not only into
who the electorate wants in power, but also why they are choosing one
candidate over anotherwhether its a local representative they favour or
a national leader, or a party, or an alliance. As Sardesai put it, polls
should only be a starting point, frankly, for you to analyse what is
happening across the country.
Unfortunately, the public appetite for seat projections is fuelled by
journalists who like to talk up their polls and boast about knowing the
pulse of the nation. The gap between the best possible use of surveys and
the reductive way in which theyre presented has led to a widespread
distrust of pollsthe outcome of a sometimes lazy, sometimes careless
and sometimes dishonest approach some journalists take towards their
responsibilities. Too often, the running mantra is: the pollster takes the
blame if the numbers are wrong, the editor takes the credit if theyre
right.
LATE ONE EVENING IN EARLY AUGUST, I bumped into Sardesai in the
foyer of the Noida Film City building that houses CNN-IBNs newsroom.
About a week earlier, the channel had run poll-based forecasts on what
would have happened if general elections had been held in July. The
survey indicated that the NDA could get up to 180 seats, beating out the
incumbent United Progressive Alliance, which was projected to get only
153. Sardesai told me that he had strong opinions on polling, and agreed
to meet to discuss them further.
A week later, on a Sunday evening, I sat across from Sardesai at his
bungalow in a quiet, leafy south Delhi neighbourhood. He was relaxed
wearing shortsand spoke thoughtfully. My sense is, looking at this from
the television I have done for 18 years, first when I was at NDTV and now
at CNN-IBNso I have been privileged in a way to have done it with
people who I consider the best in the businessthe sense I get is that
opinion polls can be done seriously and can lend themselves to serious
political analysis, he said. But opinion polls can also become a business
and a gimmick.
In a way, though, opinion polls have always been about both business and
analysisabout the relationship between the media and its consumers as
much as about the relationship between the citizenry and its political
choices. Although much of the polling thats undertaken today looks more
like statistical modelling than beat reporting, the practice has its roots in a
very journalistic enterprise. Opinion polling is a child of the newspaper
world, the Swedish sociologist and pollster Hans Zetterberg has written.
Only later did the academic world of social science enter as a stern
stepfather.
George Horace Gallups doctoral thesis in journalism at the State

University of Iowa in the late 1920s described a method for gauging

reactions to newspaper features by sampling the opinions of a carefully
selected set of readers. Over the next few years, he developed his
techniques and founded the American Institute of Public Opinion, which
ran what would become the famous Gallup Pollsfortnightly surveys of
American public sentiment that he syndicated to newspapers across the
country. Although the polls were partly meant to gauge the mood of the
nation, Gallup also expected them to improve newspaper sales, and they
soon became popular with editors.
In 1936, when Gallups company was barely a year old, it became famous
for correctly predicting not only that Franklin D Roosevelt would be reelected to the United States presidency, but also that the countrys most
popular survey at the timerun by the magazine Literary Digest, which
collated postcard questionnaires from roughly 2 million peoplewould get
its forecast wrong. Gallups major insight was that the number of people
polled was less important than the extent to which those people
represented the balance of views in the countrys various constituencies. A
humiliating failure to predict the outcome of the 1948 presidential race
taught Gallupwho was so confident of his result that he stopped
collecting data several weeks before election daythat opinion polls also
had to keep up with the dynamic nature of public sentiment.
MANY PEOPLE CONSIDER 1980 to be the dawn of Indian polling, when
two young economists (Prannoy Roy and Ashok Lahiri) teamed up with
two young market researchers (Dorab Sopariwala and KMS Titoo
Ahluwalia) to correctly forecast the general elections on behalf of Aroon
Puries five-year-old fortnightly magazine, India Today. But the father of
Indian polling, Ahluwalia told me, was the one who got it famously
wrong.
Unlike in America, it wasnt journalists who originally felt the need to
conduct opinion polls in India, but an Oxford-trained economist named
Eric da Costa. By the early 1950s, da Costa had left a job in the civil
service and became the editor of a publication called Eastern Economist.
He soon met Gallup and Henry Durant, the director of Social Surveys
Limited (the British counterpart of the American Institute of Public
Opinion). Based on his conversations with them, da Costa founded the
Indian Institute of Public Opinion, in Delhi, which was modelled on
Gallups US operation. IIPO conducted and published its first national poll
before the 1957 general elections.
The challenges facing da Costa in India were much tougher than the ones
Gallup and Durant had to account for in the US and the UK. Polling and
election forecasting is a complicated process in any democracy, but in
India it is compounded by a multitude of ethnic, caste, religious, linguistic
and regional identities; by the size of the electorate and the remoteness
of many of its members; and by the proliferation of major parties
sometimes as many as seven in a racebattling it out in multi-pronged
contests at both the state and national levels. Added to this is the fact
that, although the public is often interested in who will become prime
minister, its ultimately not their votes that decide, but the personalities

and internal politics of the winning party or coalition. Not for nothing has
India been called a pollsters nightmare.
In his later years, Ahluwalia told me, da Costa was really a very
impressive old gentleman, very refined, very erudite. But quite
opinionated, I thought. For the fourth Lok Sabha elections in 1967, da
Costa did several polls that I think were front-paged in the Indian
Express, Ahluwalia said. According to an article by the historian
Ramachandra Guha, published in The Hindu, a 1967 report by da Costa
forecasted the disintegration of the monolothic exercise of power by the
Congress party.
Although the Congress ultimately lost power in many states, Indira was
able to maintain control at the centre for the next ten years. In many
peoples eyes, da Costa got the most important part of his assessment
the overall picturewrong. (In fact, however, da Costas numbers appear
to have been fairly right on.) So I think pollsters went into hiding for a
while, Ahluwalia said, laughing. That got opinion polling on to a very bad
start.
The resurgence in Indian political polling began in 1979, when Roy and
Lahiri, then young professors at the Delhi School of Economics, created an
election forecasting tool called the Index of Opposition Unity. At the time,
Congress (Indira) was the single largest party and, even though it wasnt
in power, Roy and Lahiri theorised that the likelihood of a successful
challenge to the party depended upon the solidarity of its opponents. Roy
approached Purie, whom he knew from their days at the Doon School, to
see if he wanted to publish the results of their predictions.
Before starting India Today, Purie had studied with Sopariwala at the
London School of Economics. Sopariwala was now working with the Indian
Market Research Bureau under the leadership of Ahluwalia, who was only
in his late twenties but already heading one of the largest private market
research firms in the country. Ahluwalia and Sopariwala had previously
done some commercial polls for India Today and some political polls for
other publications. Purie brought them together with Roy and Lahiri. And
we got it right, several times, Ahluwalia said as we chatted in his
apartment overlooking the Arabian Sea in south Mumbai. Including, most
famously, the Rajiv Gandhi victory.
In 1980, the India Today team predicted a solid majority for the Congress
(Indira), which ended up getting 353 seats. In 1984, after Indiras
assassination, they officially gave Rajiv Gandhi up to 400 seats, but said
that these results might be understated and he could actually go on to
win even more. He won 415.
Five years later, they hit the bullseye with an exit poll accurately
predicting the Congress would win 193 seats. Purie threw a party at his
house to celebrate the poll just as the final results were being announced,
Sopariwala told me over email. The results for Uttar Pradeshs Pratapgarh
constituency came in around 10 pm, pushing the final tally to exactly 193.

There was celebration all around and bottles of champagne were

opened, Sopariwala said. The file that contained the polling data was
named Spot On. For years, we referred to that election in our
discussions and hoped that wed get another Spot On, Sopariwala
added. They never did.
It was just bloody good luck! Ahluwalia remembered. We suddenly
began to be seen as whiz kids. This seems to have set the precedent for
the way opinion polling is often viewed todayas an exercise whose
sole purpose is to foretell the number of seats the winning party will bag
on election day, and which is only of value to the extent that it gets this
number right.
ON 7 MARCH 2012, CNN-IBN aired a half-hour show called BATTLE FOR
THE STATES: Ask Yogendra Yadav. In the opening minute, a mans voice
intoned: Uttar Pradesh has given an extraordinary verdict for the
Samajwadi Party. When everyone spoke of a hung assembly, the CNNIBNThe WeekCSDS post-poll survey predicted exactly this verdict.
Then came a video message from Uttar Pradeshs chief minister-elect,
Akhilesh Yadav: I really congratulate the people at CNN-IBN. They
brought in a wave of celebrations to our cadre even before the sixth [of
March]. Reacting to the results for Madhya Pradesh, the Congress leader
Digvijaya Singh appeared on a split screen opposite a very pleased
Sardesai. You won the bet Rajdeep, Singh said. I owe you a dinner. A
graphic popped up on screen: CNN-IBN = ELECTIONS.
The camera cut to a beaming Sagarika Ghose in her Noida studio. Hi
there, good evening. Yes, CNN-IBN equals elections, she said. The
assembly elections of 2012 saw some hits and misses. But once again the
CNN-IBNWeekCSDS post-poll survey got the assembly elections results
of 2012 right.
Over the next 30 minutes, we will put your questions to my old friend
Yogendra Yadav, senior fellow at the Centre for Studies of Developing
Societies [CSDS] someone who many call Indias election oracle, Ghose
continued. Oracle of Delhi, not the oracle of Delphi. She looked over at
Yadav, who sat with her in the studio. Just to take it off, let me begin by
asking you a question, she said. Are you satisfied with this particular
post-poll survey that you have done?
In fact I was thinking about it when you said we got it right, asking
myself, did we really get it right? Yadav said, looking slightly
uncomfortable. I mean, we got the big picture right. Thats important
Did we get the exact things right? No, we did not. We overestimated the
extent of Mulayam Singhs victory in terms of seats, but even more in
terms of votes.
Ghose jumped in: Hes ended up with a 29 percent vote share while we
have given him 34 percent.
Which is a significant difference, Yadav replied. And basically, you see,

our ambition in the long run, Sagarika, is to make polls something which
is not discussed too often. Doctors dont discuss thermometers.
Ghose tried to summarise: It is to catch the trend.
Earlier this year, I met Yadav, one of Indias foremost psephologists, in
his office at the CSDS campus in north Delhi. Yadav founded the Lokniti
Project at CSDS in 1997 to collect data on voters opinions before and
after every major state and national election. (He stepped down as the
head of the project last year, when he joined the Aam Aadmi Party.
Though he is still a part of the CSDS team, he does not participate in
producing opinion polls or seat forecasts now.) The projects goal is partly
to monitor the working of Indian democracy.
My sense is there is unnecessary mystique and a feeling of black magic
about polls in our country, Yadav said to me. Largely because these are
new things and people are not familiar with this stuff. Essentially, opinion
poll is nothing but a very systematic way of holding conversations with a
very large number of people. And the findings are presented not in the
form of quotations from those conversations but in the form of numbers,
because you cant report 5,000 conversations. In that sense they are not
in principle different from news reporting.
The trouble is that in our country opinion polls have been reduced to
election-related polls, election-related polls have been reduced to election
forecasting, he added. He spoke softly, confidently, calculating each
word. Now that is a very limiting way of looking at polls. The poll CSDS
did for CNN-IBN in 2012 estimated that 34 percent of votes in Uttar
Pradesh would go to the Samajwadi Party, and 24 percent to Mayawatis
Bahujan Samaj Party. This 10 percentage point lead was three times the
actual differencewell beyond the surveys stated margin of errorand it
seemed to bother Yadav. But Ghose was happy that her channel had
picked the winning side.
Part of the problem, pollsters feel, is that journalists dont understand the
usefulness of the data that underlie seat projections. They are
uncomfortable with numbers, said Yashvant Deshmukh, the head of
CVoter, one of the largest private socio-political polling agencies in the
country. He told me journalists dont look behind the final numbers to see
what went into the result. That is why media is coming up always with so
superficial, over-simplistic analysis, virtually pedestrian analysis, for every
mandate, he told me.
Since 1993, when Deshmukh founded CVoter after graduating in
journalism from the Indian Institute of Mass Communication, he has
conducted polls for almost all major English- and Hindi-language national
news channels; some regional ones; magazines including The Week and
India Today; the Hindustan Times and Indian Express; and various other
institutions, according to his companys website.
Over the period of all these years the media was not educated about the

polls, Deshmukh said. The numberswhat exactly a poll means, what is

the probability, what is the margin of error, what do you get out of the
poll. Scientifically speaking, what you get out of a good poll is the vote
share.
He gave me an example of how vote share often gets misunderstood.
Suppose if you say today that Congress will get 30 percent of the votes,
that means scientifically speaking it will get anywhere between 27 to 33
percent of the votes. If you are projecting Congress 30 percent and BJP
28 percent, that means your margin of victory or defeat are very much
within the margin of error. So Congress getting 28 and BJP getting 30 is
also a probability. People dont look into this. People dont understand
this.
Projecting how many seats a party will win is an entirely separate affair.
Conversion of those vote shares into seat share has nothing to do with
the polling business. Its not part of the polling at all. Survey stops at the
projection of the vote share. Unfortunately in India nobody wishes to
understand that. Everybody wishes to look at the seat. Seats batao kitni
hain. Vote chhodo, seat batao kitni hain. (Tell us how many seats. Leave
the votes, tell us how many seats.)
Later, I asked Deshmukh if the media was interested in understanding
these complexities of polling and projection. No, not at all, he said, It is
painful. He didnt fault the public for not realising what a good poll
actually tells them. In India, my vote doesnt really get translated into
number of seats, he said. Its not a presidential system. Popularity
ratings dont convert into votes, votes dont convert into seats. Seats
dont convert into alliances. Alliances dont convert into government, in
majority figures. When you have so many intangibles and the entire
population including the media is highly illiterate about what to expect
from the pollsif the entire media is asking, number batao, kisko kitni
seatein? Toh agar media hi yeh pooch raha hai, public kya poochegi?
Public kya poochegi? (Tell us the number, how many seats to whom? So if
the media itself is asking this, what will the public ask? What will the
public ask?)
AT THE START OF TIMES NOWS national poll projection show on 29
July, a methodology note flashed up on the bottom of the screen. It
stayed for three minutesmore than ample time to read the description:
National representative sample of 13052 randomly selected respondents
across all states during 18th July to 24th July 2013; Date weighted to
known population profile; margin of error is +/-3% at national level and
+/-5% at regional level.
Arnab Goswami got the ball rolling for the two-hour programme. Who is
going to form the next government in India if elections were to be held
now? he asked, before introducing his panellists. To my right is
Yashvant Deshmukh, the man who takes all the blame, and we get all the
credit. Thats a terrible clich, but I know Yashvant is used to it.

That Deshmukh takes the blame and Goswamis channel gets the credit
holds true for more than just the final forecasts. Opinion polls are often
criticised for a lack of information on how they were conducted. In June,
for example, the economist Vivek Dehejia criticised polls in India generally
(and a specific poll in particular) for their poor reporting practices. The
GFK poll tells us only that interviews were conducted in respondents
homes and in street corners but gives us no indication that subjects were
picked randomly, Dehejia wrote in Business Standard. Also, as is typical
with Indian polls, we are not told the margin of error, so have absolutely
no way to assess the accuracy of the predictions.
Dehejia seemed to suggest that the lack of transparency in Indian polling
was the fault of pollsters. But methodology notes such as the one aired by
Times Nowalthough it would have failed standards established by
various international polling bodiessatisfy the limited guidelines
published by the Press Council of India (PCI). The PCI recommends that a
newspaper publishing a survey should indicate which institutions carried
out the survey, who commissioned it, the size and nature of the selected
sample, the method of selection of the sample, and the possible margin of
error in the findings. Even though the PCIs jurisdiction is restricted to the
print industry, it is the only institution in the country that produces any
sort of recommendations for publishing polls. Most of the media houses
adhere to them as indifferently as the PCI seems to lay them down.
Yogendra Yadav, who has been conducting polls and surveys since 1996,
said he wasnt aware of these recommendations: As people who carry it
out, at least we have never received any guidelines from anyone. He
agreed that there is a need to make opinion polls more transparent. I
have personally been fighting for it, he said. I have written about it. I
have written to all kinds of people who matter to say please make it
mandatory for every opinion poll to disclose their methodology in great
length. I should exactly know the methodology followed for the survey. I
should exactly know the method for vote-to-seat conversion. I should
know who paid for this survey, who was the customer, who commissioned
it, who paid money for the survey. And what is the track record of the
agency who is doing it? Is there any conflict of interest? These are
absolutely standard procedures.
Accusing just the pollster of opacity in an environment where there is no
code of conduct or monitoring body is unfair. It gives the gatekeepers of
informationthe news channels and publicationsa guilt-free pass. Some
outlets, such as CNN-IBN, which works exclusively with CSDS, do an
admirable job of ensuring the transparency of their surveys, putting
thorough methodological notes on their websites. But others fall far short
of this standard. If a poll is like reporting, only multiplied, its the editors
job to make sure his correspondent isnt relying on shoddy research,
personal bias or an outright plant. Like a bad story, if a bad poll is
published, the editor and the publication have to share the blame.
DORAB SOPARIWALA IS IMMACULATE AND POLITE. He is also the
only person I have ever met who talks about god as the woman up
there. His former colleague Titoo Ahluwalia called him, along with

Yogendra Yadav amongst, I would say, the finest pollsters anywhere on

this planet. They really are extremely knowledgeable, with impeccable
integrities.
I met Sopariwala in Prannoy Roys office. Roy was not present, busy with
one of the many duties of running a media company. The office, hidden
on one side of the NDTV newsroom in south Delhi, is small but
extraordinarily pleasant, lacking the pomp one might expect from the
founder of a major national television network.
As we discussed issues of political polling and election forecasts,
Sopariwala told me about the first poll he got wrong. It was when NTR
came, he said. In a paper Sopariwala and Roy wrote in 1990, they
described how the Telugu cinestar of mythical proportions Nandamuri
Taraka Rama Rao founded the Telegu Desam Party and entered Andhra
Pradesh politics on a platform of his own fame in 1982. Sopariwala was
hired by India Today to do a statewide poll ahead of the January 1983
assembly election. To meet the deadline of the sponsor, the fieldwork
began just after the nominations closed, with three weeks to go before
elections. The poll indicated a victory for the ruling Congress. But Rao
won a massive victory and became the chief minister. In a postmortem by Sopariwalas team, they learnt that the pollsters had gone in
too early and missed a strong late swing.
Sopariwala told me that after Raos Chaitanya Ratham campaignthe
actor drove a converted Chevrolet across the statewhich was quite close
to the election date, he swept the votes. We couldnt catch the wave. I
was so embarrassed I didnt charge my fees. He joked that he should
have charged double, because he had to go back to a lot of the people
they had interviewed in order to understand what went wrong. I dont
work as hard on anything else, he said of election forecasts. I kill myself
doing it. You think I want to get it wrong?
The process of opinion polling starts with understanding the objectives of
the pollthe topics on which its meant to sample opinions. A
questionnaire is then designed that, in most cases, is supposed to solicit
peoples views without distorting them (for instance, by asking leading
questions or using certain loaded terms). Often, several questions are
asked to get at the same issuesuch as preferences for a given
politicianfrom different angles. A method is then devised for selecting
and interviewing a representative sample of the electorate, and a
sufficient sample size is decided. Most pollsters either send teams to
conduct face-to-face interviews, or they call people up on their mobile
phones. The data collected are then used to calculate vote share for each
party, and this is finally converted into the likely number of seats each
party would grab if the elections happened when the poll was carried out.
Every step has its possible pitfalls, all of which are compounded by the
diversity of the countrys population and the complexity of its politics.
Insufficient sample size is perhaps the most frequent criticism put to
pollsters in India. How can a sample size of 30,000 or 50,000 be

representative of more than 1.2 billion people? These sorts of questions,

Ahluwalia told me, are just as old as the polls. And they just show a very
low level of debate on the subject. And unfortunately they still happen to
this day. Sample size, he said, hasnt got anything to do with the size of
the population. Instead, its all about heterogeneity. How much tea do
you need to taste to know whether the sugar is alright or not? If the sugar
is stirred, one sip will tell you. If its not stirred, you can drink it right up
to the end and not know. Pollsters try to solve this problem by identifying
homogenous clusters at sub-regional levels, and then building up a picture
of the the regional, state and national electorate through the
amalgamation of these units.
Sanjay Kumar, who now leads the Lokniti project at CSDS, said that just
having a bigger sample size doesnt help, as it also multiplies the errors.
In theory, the more you spread the sample, the more representative it is.
But spreading the sample also has some restrictions, he said. Ideal
would be to go to all the constituencies. But one has to think about
manageability, whether you can manage it or not.
One of the many problems pollsters face is whether to conduct telephone
or face-to-face interviews. The former is cheaper, but some critics believe
mobile penetration in India is not yet deep enough to yield representative
survey results. In-person interviews are beset by their own difficulties. For
one, the interviewer has to win the confidence of the respondent.
According to both Deshmukh and Kumar, in rural areas, members of
marginalised communities often misrepresent their views out of
intimidation and fear of reprisal from the often urban interviewers. This is
apparently the reason Mayawatis electoral base is always underrepresented in polls.
To overcome some of these distortions, Sopariwala and Ahluwalia
introduced to India a secret ballot technique that was pioneered by
Gallup in America in the late 1940s. Respondents are asked to mark their
candidate preferences on a sheet of paper and put it in a dummy ballot
box. The paper is coded with a serial number to match a serial number on
the respondents main questionnaire.
To try and ensure that their samples are representative, pollsters take a
systematic approach to selecting interviewees at random. Systematic
randomit is kind of a lottery, where there is no bias applied, Kumar told
me. CSDS has a method for randomly picking voters from the electoral
roll. The list is then given to interviewers in the field, who never decide
whom to poll. For telephone surveys, a computer automatically connects
random voters from a given region with interviewers who speak the local
language. In either case, the pollster checks for under-represented
populations, and then does additional polling or weights the data to
account for this.
However a poll is conducted, there are inevitably many rejections, or
people are unavailable, or government data used to generate addresses
and telephone numbers are outdated, so the pollsters have to attempt to

sample more people than they need for a robust poll. The data are then
collated, and the vote share of each party is derived by tallying up the
individual responses.
All this makes polling a very expensive exercise. The price differs from
agency to agency and according to what is expected of the pollster. In the
last general elections, CSDS, which is a non-profit organisation, polled
roughly 30,000 people around the country. The total cost was around Rs 1
crore, Kumar said. About 20 percent of that was covered by CSDSs media
partners; the rest came in the form of grants from academic and research
institutions and government bodies.
Few for-profit agencies, if any, are able to make money from political
opinion polls for the Indian media. CVoter makes most of its money
outside of India, doing polls for foreign governments and international
organisations, according to Deshmukh. Market research firms, such as
ORG (formerly ORG-MARG), use the publicity generated by these surveys
to market themselves to corporate clients interested in understanding
consumer behaviour, which is how they make most of their income.
Sopariwala said that election forecasting is less than one percent of their
business, but ninety percent of the public face.
Many media houses share their costs and data with another partner. CNNIBN had The Week as a print partner for the assembly election polls last
year. This year, their election tracker was shared by The Hindu in July and
The Weekin October. CVoter is in a tri-party agreement with Times Now
and India TV, and is also doing polls for the India Today group. Deshmukh
did not tell me how much it costs him to do his polls, but it is presumably
cheaper than CSDS; the former relies mostly on telephone interviews,
while the latter only interviews their respondents face-to-face.
TO GET FROM THE RESULTS OF THE POLL to the seat numbers is a
mind-bogglingly complex process. Sopariwala and Ahluwalia think that the
biggest challenge is identifying those respondents who give interviews but
do not vote, thus distorting the analysis of the data. But this is only one of
a nearly endless list of factors that can go into seat projections.
Various indices and formulas have been developed over the years by
Indian psephologists. Prannoy Roy and Ashok Lahiri started with their
Index of Opposition Unity. Then, building on Roy and Lahiris work, came
the economist Surjit Bhallas Lying Index, which tried to offset the effects
of electors who dont vote. The process to fine tune these techniques is
ongoing.
Pollsters are continually trying to account for the ways that various layers
of political representationthe local MLA, the states incumbent party, the
member of parliament, the national incumbent party, the governing
alliance, the executive, and the prime ministeraffect voter preferences.
An individual may be unhappy with a politician or partys performance at
any one of these levels, and still vote for the same party at other levels.

For two-way races in first-past-the-post democracies, the basic formula

that governs conversions from vote share to seats won is the cube law,
which states that seats will be divided up in the same proportion as the
ratio of vote shares raised to the power of three. In India, however, there
is often the challenge of accounting for more than two parties, for parties
splitting between elections, and for fluid alliances.
Kumar gave me an example of how unpredictable the results can be. In
Uttar Pradesh in 2012, the Samajwadi Party had 29 percent of the votes
and got 224 seats. In 2007, the Bahujan Samaj Party had over 30 percent
of the votes, but won only 206 seats. Same state, same voters,
same parties. Party gets 1 percent less votes and gets 20 extra seats,
Kumar said. These surpises also happen at the national level: the increase
in Congresss vote share between 2004 and 2009 was only 2.02 percent,
but they picked up 60 extra seats.
Every election I find something new, and fine tune it, Deshmukh said.
It can never be perfect. It is still not foolproof. Arre public opinion is as
dynamic as it can be. Every damn election we learn something or the
other, and it is an unending process.
IN 2009, on the eve of the second phase of Indias general elections,
Prannoy Roy sat in front of an externally lit Hyderabad Public School
building, facing Dorab Sopariwala and Shekhar Gupta, the editor-in-chief
of Indian Express. It was two days prior to the second round of assembly
elections in Andhra Pradesh, and the Election Commission had recently
prohibited the broadcasting or publication of data based on exit polls
before the last phase of the elections was over. All three men were
dressed in kurta-pajama for an NDTV show titled Battleground 2009.
We travelled all across Karnataka, and we also travelled all across
Andhra Pradesh, Roy said. So now these two gentlemen, they know
everything. They are experts on these two states. Theyre gonna tell us,
by the end of the show, whos gonna win in both of these states. Not only
that, by travelling in these areasits amazingthey have an idea of the
entire country. So if you want to know whos gonna win the entire
country, hang on.
They discussed the prospects of three of the states most significant
contenders: the Telugu Desam Party, the Congress, and the former actor
Chiranjeevis Praja Rajyam Party. Lets move on to whats happened in
phase one of this election, Roy continued. Now weve been talking to a
lot of people. This is not really an opinion poll, but its an analysis for
which we sat together and tried to figure out what happened in that phase
one. They examined Chiranjeevis chances in exceptional quantitative
detail, tossing out likely percentages of vote share, then moved on to the
odds bookies were offering on the general election.
Now many people believe that bookies know little more than Dorab, Roy
said in jest. Sometimes. Though most of the time you know more than
them. They discussed a survey, based on lakhs and lakhs of text

messages from NDTVs viewers, which suggested that Manmohan Singh

was the top prime ministerial candidate, followed distantly by LK Advani.
I think thats the general mood, at least in the SMS class, Gupta said.
But SMS classthere are 400 million mobile phones in this country, Roy
offered.
I think the fact that we are here discussing projections of bookies and
SMS polls is a very unfortunate thing, Gupta replied. I would rather go
by theby the figures thrown up by your opinion poll.
Roy laughed and raised a finger to his lips. Shh You cant even mention
the word. Hahaha. No, I know the point you are getting to.
I know I cant, Gupta said. So I am not mentioning the figures. But,
but, but I want to say that Election Commission has done great things,
running up in these elections. But, this is the most terrible decision. Thats
my editorial viewpoint. I think it would have had been fantastic to have
some kind of exit polls now. Figures coming init just enriches the
discussion. I think if the Election Commission is watching, orthe new
Election Commissioner has taken over todayNavin Chawla, if he
watching, I think this is one decision that they have to revisit. Because its
a terrible decision. It doesnt help anybody If bookies can talk about
figures, why cant legitimate people?
We know all over the world, most bans dont work, Yogendra Yadav told
me when I met with him recently. The ban on exit polls still stands, and
the Election Commission is also considering banning surveys during the
period after elections are announced. (At the time of writing, the ban was
under consultation with political parties.)
The proposal to ban the polls, Yadav said, was based on four flawed
assumptions of the Election Commission: most polls are either mistaken
or vicious; they can influence voting behaviour; these problems cannot
be tackled by any other means; and bans will work. While admitting there
are some increasing unprofessional practices, he said most polls are still
infinitely better than any other way of tracking the popular mood, and
whether surveys affect voters in some undue way is an empirical question
that remains unanswered. If we want to ban something, and if we have
been discussing a ban for ten years, should we as a country not gather
evidence about these things? he asked.
Prohibition should be the last resort to tackle any problem, Yadav argued;
ensuring transparency through some basic regulation would be a far
better way to curb malpractice. Bans are counterproductive, he
continued. He said that although political parties conduct their own
surveys, some are now trying to create more opacity on whatever limited
data are available. Prohibiting polls would create a black market of
information So, instead of actually creating public information it would
create an information elite. Second, it would create rumoursI know they

have done it, but I cant release it legally, but let me tell you what the
survey is. It creates, absolutely, a culture of duplicity, rumours and so
on.
He mentioned Battleground 2009: Prannoy Roy, Dorab Sopariwala and
Shekhar Gupta sat and said, According to my intuition Congress will get
23 seats in Andhra Pradesh. Of course they had done a poll. Everyone
knew they had done a poll.
The ramifications of these bans, Yadav suggested, could go well beyond
surveys. The simple fact is you cannot say Prannoy Roy cannot have
intuition. Tomorrow I can have intuition. The question is, will we impose
this ban on people who write on the edit page? Will we impose this ban on
reportersthat they cannot say someone is ahead, someone is behind? If
we dont ban all that, how can we possibly ban this?
EDITORS ACROSS TELEVISION AND PRINT MEDIA told me that while
some bogus, rogue or manipulated polls find space in regional and
local news outlets, it is rare to find them on national broadcasts, or in
national publications. There are malpractices in the Indian industry,
Yadav agreed. But, If you look at some of the more professional polls, I
would say they compare with some of the best in the world. Nowhere in
the world are election forecasts 100 percent correct.
Still, there are valid criticisms to be made even of polls publicised in the
national media. Foremost among these is that pollsters methods and the
calculus that produces seat projections remain outside public scrutiny.
Almost every other suspicion about the technicalities of pollingabout the
size and representativeness of samples, margins of error, statistical
confidence, unresponsive interviewees, and adjustments based on
historical precedentscould be dispelled if pollsters and their media
partners were absolutely transparent about how they conduct their polls.
And this would enhance the credibility of the process. It should be
mandatory, Yadav said, for pollsters to disclose their methodology at
great length. Now with web there is no space problem. You can ask them
to put it up on the web.
Other sceptics question the motivations and political bias of the pollsters.
Recently, on blogs, Twitter and in an article in Open magazine, CVoter has
been called out for gravely overestimating the BJPs prospects in many
forecasts since 2004. Deshmukh, the founder, is a nephew of Nana Saheb
Deshmukh, one of the Rashtriya Swayamsevak Sanghs most revered
ideologues. For their part, Yadav and Sopariwala have faced accusations
of being sympathetic towards Congress. But none of this proves bias.
Here, too, transparency with regard to methods, and clear disclosures
about potential conflicts of interest, would help shed light on the validity
of such claims, and prevent bias from entering polls in the future.
In the end, however, it may be that people only believe the polls that
reinforce their beliefs. Part of this is human psychology, and part of this
may be that whatever lack of trust exists between the public and the

media is extended to surveys and forecasts. Readers or viewers may pay

less attention to a poll than to the medium that brought it to them.
I asked Sardesai why he still paid for polls, despite their uncertain effects
on his networks popularity and the technical and reputational pitfalls that
beset the process. You ask me a question which I have no answer to,
barring saying that I am an election junkie, he said. I still believe its fun
to do.
See this is the mugs game, Sardesai continued. If we are going to go
by what television audiences want, then we are going to end up doing
only naach gaana [dance numbers]. Thatsthats not a factor. I think
you have to do things that are intelligent. I think the viewer respects an
intelligent opinion, or an intelligent poll. A viewer knows when the pollster
is also faking. That I think is becoming increasingly clear.
Its one more weapon available to try and explain the large election
landscape of this country. I still believe in that, which is why we still do
polls. Maybe in the future we may have to stop it altogether. I hope that
day doesnt come.

Correction: The spellings of Rashtriya Swayamsevak Sangh and Bahujan

Samaj Party have been corrected. The Caravan regrets the errors.