New Delhi: Economic Advisory Council to the Prime Minister has issued a working paper by Sanjeev Sanyal Member EAC-PM and Akanksha Arora. The Paper analyses three perception-based indices: Freedom in the World Index, V-DEM indices, and EIU Democracy Index.
The EAC-PM tweeted the following thread which finds serious problems with the methodology used in these perception-based indices. And says ‘questions used by these indices are not an appropriate measure of democracy across all countries.
“In recent years, India’s rankings and scores have declined on a number of global opinion-based indices that deal with subjective issues such as democracy, freedom and so on.
@sanjeevsanyal & @AakankshaArora5 write.
This working paper analyses three perception-based indices: Freedom in the World Index, V-DEM indices, and EIU Democracy Index.
Freedom in the World Index and V-DEM indices have placed India at the same level as during the Emergency of the 1970s. Moreover, India has been placed below countries like Northern Cyprus. Surely, this is not credible.
There are serious problems with the methodology used in these perception-based indices. First, these indices are primarily based on the opinions of a tiny group of unknown “experts”
Second, the questions that are used are subjective and are worded in a way that is impossible to answer objectively even for a country, let alone compare across countries. Third, there are questions that should be asked but are excluded.
Fourth, certain questions used by these indices are not an appropriate measure of democracy across all countries.
Since these indices are inputs into the World Governance Indicators, the World Bank should ensure greater transparency and accountability from these institutions.
Meanwhile, independent Indian think tanks should be encouraged to do similar perception-based indices for the world in order to break the monopoly of a handful of western institutions.”
Following is the Executive Summary of this working paper:
Executive Summary
A noticeable trend in recent years has been the decline of India’s rankings on a number of global indices, specifically in opinion-based indices that deal with subjective issues such as democracy, press freedom etc. One way to respond to this is ignore these as mere opinions. However, the issue is that they have concrete implications. For instance, these indices are inputs into the World Bank’s World Governance Indicators (WGI) that, in turn, have approximately 18-20% weightage in sovereign ratings. So, they can’t be completely ignored.
In this paper we look at 3 such indices which are used by WGI, which are Freedom in the World Index, Economist Intelligence Unit (EIU) Democracy Index and V-DEM indices. All three indices are almost entirely perception based.
The first index in this paper is the Freedom in the World Index which has been published since 1973 by Freedom House. India’s score on Civil Liberties was flat at 42 till 2018 but dropped sharply to 33 by 2022; that for Political Rights dropped from 35 to 33. Thus, India’s total score dropped to 66 which places India in the “partially free” category – the same status it had during the Emergency. Since the publication of the index, the only two previous instances where India was considered as Partially Free was during the time of emergency and then during 1991-96 which were years of economic liberalisation. Clearly, this seems very arbitrary as what did the period of emergency which was a period of obvious curtailment of various activities had similar to the period of economic liberalisation or of the current times.
In contrast, the Freedom in the World report has given the territory of Northern Cyprus a score of 77 which makes it a free democracy. This is a territory only recognized by Turkey, not even by the United Nations. Meanwhile, the think-tank continues to treat Jammu and Kashmir as a separate territory since the early 1990s and now places it in the category of “not free”.
The second index we look at in this paper is the Economist Intelligence Unit (EIU) Democracy Index. This index is published by EIU, which is the research and consulting arm of the firm that publishes the Economist magazine. India is placed in the category of “Flawed Democracy” and its rank deteriorated sharply from 27 in 2014 to 53 in 2020 and then improved a bit to 46 in 2021. The decline in rank has been on account of decline in scores primarily in the categories- Civil Liberties and Political Culture. The maximum decline has been in the category Civil Liberties, for which the score declined from 9.41 in 2014 to 5.59 in 2020. In the same time period, India’s score for Political Culture declined from 6.25 to 5.0. India’s rank then recovered marginally to 46 in 2021, primarily on grounds that the government rolled back farm-sector reforms, which lead to improvement in scores on the categories- Civil Liberties, Functioning of Government and Political participation. 3 The comparison of scores of other countries on this EIU-Democracy Index has some surprising results. India’s latest score for Civil Liberties lags that of Hong Kong (8.53). Similarly, India’s score for Political Culture is much lower than that of Hong Kong (7.5) and Sri Lanka (6.25). Clearly, this seems very arbitrary.
EIU in their methodology mention that they do not only rely on opinion of experts but also take responses for some questions from opinion polls, if available (primarily from World Value Survey (WVS)). If the responses from opinion polls are not available, then the responses for those questions also are answered by experts. In case of India, the latest available WVS report is of 2012. The latest round of Wave 7 (2017-2020) has not been conducted for India yet. This implies that score of EIU Democracy index for India is based only on expert opinion since 2012.
The third is Varieties of Democracy (V-DEM) indices produced by the Varieties of Democracy Institute at the University of Gothenburg in Sweden. It comes up with six indices covering various aspects of democracy – Liberal Democracy, Electoral Democracy, Liberal Component, Egalitarian Component, Participatory Component, and Deliberative Component. These indices are comprised of various sub-indices. Some of the variables that feed into these indices are objective, i.e., which are based on factual data such as election type, minimum voting age, percentage of population with suffrage, occurrence of referendum, head of state name, upper chamber name, Presidential election vote share of largest vote-getter etc. Whereas some other variables are subjective, i.e. evaluative indicators based on ratings provided by experts. The final indices are created combining kind of variables.
A time series analysis of the V-DEM scores show that India does well on objective parameters such as share of population with suffrage, but scores on various subjective subindices have declined sharply since 2014. In fact, India’s has been termed as an “electoral autocracy” in the 2021 report, same as it was during the period of Emergency. The Liberal Democracy Index has declined from 0.567 in 2013 to a 0.357 in 2021. Secondly, the Electoral Democracy Index declined from 0.695 to 0.444 during the same time period. This Electoral Democracy index has 5 sub-indices. The scores remained high on the category- share of population with suffrage and elected official index but declined sharply on the other three- Clean Elections sub-index has dropped from 0.785 in 2013 to 0.552 in 2021, Freedom of Expression from 0.882 to 0.598 and Freedom of Association from 0.863 to 0.72.
The cross-country comparison of this index also throws up some interesting results. India’s rank on Liberal Democracy Index is 93. In contrast, Kingdom of Lesotho which started having democratically elected government only in mid 1990s, and has since faced various 4 disruptions including a military coup and emergencies has a rank of 60. Or Kosovo, which declared independence from Serbia only in 2008 has a rank of 79.
A common thread in all these indices is that they are derived from the perceptions or opinions of few experts. These institutions do not provide any transparency on how the experts were chosen or even their expertise or nationality (expect in case of V-DEM where they clarify that they chose some experts from each country from different fields). For instance, the Freedom House report mentions that report is produced by a team of inhouse staff/analysts/consultants and external analysts and expert advisers from the academic, think tank, and human rights communities. The nationality and expertise of the experts are not clear in the report.
Another common feature of these indices is all these are based on a set of questions. A reading of the questionnaire shows that most of the questions are subjective in nature, hence simply providing the same questions for all countries does not mean getting comparable scores for different countries as the generic questions can be answered very differently by experts. For instance, EIU has a question “How pervasive is corruption?”. This kind of question is not possible to answer objectively even for a country and is impossible to do across countries. Further, the way the questions are framed can have an impact on scores. For instance, if an ostensibly reasonable question like “Is the head of state democratically elected?” is added in the list of questions, it would immediately negatively affect countries such as Sweden, Norway, UK, Denmark, Belgium, Netherlands as these countries are constitutional monarchies. Most readers will agree that asking such a question is not unreasonable in an index trying to assess the democratic situation in a country.
Moreover, there are questions that are not meaningful indicators of democracy. For instance, there is a sub-index called Direct Popular Vote (in V-DEM) based on the following question: “To what extent is the direct popular vote utilized?” This is an indicator in which India scores zero! This is because it relates to use of direct referendum, plebiscites which is obviously not possible for a large country like India; even US scores zero on this. Obviously, this sub-index is suitable only for a small country like Switzerland where direct referendums are feasible. Ironically, countries such as Afghanistan, Belarus, Cuba have score high than zero in this sub-index (In short, they are deemed more democratic on this parameter than India or the US).
To the extent that the think-tanks justify the change in scores in the reports, they are based on a selective use of some reports from media. For instance, EIU in its report mentions that “authorities’ handling of the coronavirus pandemic has also led to a further erosion of civil liberties in 2020”. There is no basis to show how anything different was being done in India beyond what was necessary to contain the pandemic. Most countries had imposed 5 restrictions during the time. Further, Freedom House declares that “informal community councils issue edicts concerning social customs that discriminate against women and minority groups”. However, it does not explain how this has worsened over time. There are many such examples listed in this paper.
Overall, it is important to understand that perception-based indices should not be ignored as mere opinions as they find their way into concrete things such as sovereign ratings via WGI which is based on a combination of lot of these indices. These will become even more important in future as Environmental, Social and Governance (ESG) indices are introduced into global business/investment decisions.
Hence, as a first step, the Indian government should request the World Bank to demand transparency and accountability from think-tanks that provide inputs for the WGI. In the longer term, independent think-tanks in India should be encouraged to research in these areas and come up with their own indices so that comparative indices are available.
The working paper (in full) can be accessed at:
