Varieties of Measurement: A Comparative Assessment of Relatively New Democracy Ratings based on Original Data

Abstract

A series of new democracy measures have been introduced in recent years, many based on previous measures, but some offering original data. This chapter provides critical discussion of three new measures based on original data collection: the Democracy Index (DI) constructed by the Economist Intelligence Unit, the Bertelsmann Transformation Index (BTI) compiled by the Bertelsmann Foundation, and the Political Regimes of the World dataset provided by Boix, Miller, and Rosato (BMR). Our assessment shows that DI and BTI share many features: They have limited temporal coverage, they are based on comprehensive definitions, and they have many expert-coded indicators, which are combined into graded sub-indices and an overall democracy measure. In contrast, BMR is based on a narrow definition of democracy, offers a single, in-house coded, dichotomous variable, and covers most independent countries back to 1800. All three measures suffer from a lack of transparency. Our evaluation concludes with a brief comparison of these datasets with two other new datasets based on original data collection: the Lexical Index of Electoral Democracy (LIED) and Varieties of Democracy (V-Dem). The former provides unmatched coverage of historical polities and annual updates of disaggregate indicators and a series of ordered, crisp regime categories. V-Dem is based on scores from more experts combined with a sophisticated measurement model, and it offers a comprehensive coverage and more indicators on different aspects of democracy than any other dataset. Correlation analyses indicate that it is implausible to consider the examined democracy measures to be interchangeable across the board; in some contexts, they are clearly not.