Browsing by Author "Tzelgov, Eitan"

Now showing 1 - 5 of 5

Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys
(University of Gothenburg, 2015) Pemstein, Daniel; Tzelgov, Eitan; Wang, Yi-ting; V-Dem Institute
The data produced by the Varieties of Democracy (V-Dem) project contains ordinal ratings of a multitude of country-level indicators across space and time, with multiple experts providing judgments for each country-year observation. We use an ordinal item response theory (O-IRT) model to aggregate multiple experts' ratings. The V-Dem data provide a challenging domain for such models because they exhibit little cross-national bridging. That is, few coders provide ratings for multiple countries, making it difficult to calibrate the scales of estimates cross-nationally. In this paper, we provide a systematic analysis of the issue of bridging. We first use simulations to explore how much bridging one needs to achieve scale identification when coders' thresholds vary across countries and when the latent traits of some countries lack variation. We then examine how posterior predictive checks can be used to check cases of extent of scale non-comparability. Finally, we develop and evaluate search algorithms designed to select bridges that are most likely allow one to correct scale incompatibility problems.
The V–Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data
(2022-03) Pemstein, Daniel; Marquardt, Kyle L.; Tzelgov, Eitan; Wang, Yi-ting; Medzihorsky, Juraj; Krusell, Joshua; Miri, Farhad; Römer, Johannes von; V-Dem Institute
The Varieties of Democracy (V–Dem) project relies on country experts who code a host of ordinal variables, providing subjective ratings of latent—that is, not directly observable—regime characteristics over time. Sets of around five experts rate each case (country-year observation), and each of these raters works independently. Since raters may diverge in their coding because of either differences of opinion or mistakes, we require systematic tools with which to model these patterns of disagreement. These tools allow us to aggregate ratings into point estimates of latent concepts and quantify our uncertainty around these point estimates. In this paper we describe item response theory models that can that account and adjust for differential item functioning (i.e. differences in how experts apply ordinal scales to cases) and variation in rater reliability (i.e. random error). We also discuss key challenges specific to applying item response theory to expert-coded cross-national panel data, explain the approaches that we use to address these challenges, highlight potential problems with our current framework, and describe long-term plans for improving our models and estimates. Finally, we provide an overview of the different forms in which we present model output.
The V-Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data
(2015) Pemstein, Daniel; Marquardt, Kyle L.; Tzelgov, Eitan; Wang, Yi-ting; Miri, Farhad; V-Dem Institute
The Varieties of Democracy (V–Dem) project relies on country experts who code a host of ordinal variables, providing subjective ratings of latent—that is, not directly observable— regime characteristics over time. Sets of around five experts rate each case (country-year observation), and each of these raters works independently. Since raters may diverge in their coding because of either differences of opinion or mistakes, we require system- atic tools with which to model these patterns of disagreement. These tools allow us to aggregate ratings into point estimates of latent concepts and quantify our uncertainty around these point estimates. In this paper we describe item response theory models that can that account and adjust for differential item functioning (i.e. differences in how experts apply ordinal scales to cases) and variation in rater reliability (i.e. random error). We also discuss key challenges specific to applying item response theory to expert-coded cross-national panel data, explain the approaches that we use to address these challenges, highlight potential problems with our current framework, and describe long-term plans for improving our models and estimates. Finally, we provide an overview of the different forms in which we present model output.
The V-Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data
(2020-03) Pemstein, Daniel; Marquardt, Kyle L.; Tzelgov, Eitan; Wang, Yi-ting; Medzihorsky, Juraj; Krusell, Joshua; Miri, Farhad; von Römer, Johannes; V-Dem Institute
The Varieties of Democracy (V-Dem) project relies on country experts who code a host of ordinal variables, providing subjective ratings of latent- that is, not directly observable- regime characteristics over time. Sets of around five experts rate each case (country-year observation), and each of these raters works independently. Since raters may diverge in their coding because of either differences of opinion or mistakes, we require systematic tools with which to model these patterns of disagreement. These tools allow us to aggregate ratings into point estimates of latent concepts and quantify our uncertainty around these point estimates. In this paper we describe item response theory models that can that account and adjust for differential item functioning (i.e. differences in how experts apply ordinal scales to cases) and variation in rater reliability (i.e. random error). We also discuss key challenges specific to applying item response theory to expert-coded cross-national panel data, explain the approaches that we use to address these challenges, highlight potential problems with our current framework, and describe long-term plans for improving our models and estimates. Finally, we provide an overview of the different forms in which we present model output.
The Varieties of Democracy Core Civil Society Index
(2015-10) Bernhard, Michael Coppedge; Tzelgov, Eitan; Jung, Dong-Joon; Coppedge, Michael Coppedge; Lindberg, Staffan I.; V-Dem Institute
This paper introduces the Core Civil Society Index (CCSI), a measure of the robustness of civil society, using indicators from the Varieties of Democracy battery on civil society. It begins with a discussion of the reemergence of civil society as a central concept in comparative politics and discussestheexistingstateofdatatocaptureit. Itthendiscussestheconceptualizationbehind the CCSI and the construction of the measure. This is followed by a series of face and discriminant validity checks. The paper closes with an example of the CCSI’s utility by examining the question of whether postcommunist civil society is “weak” compared to other regions. We find no evidence that postcommunist civil society is less robust than civil society in other major world regions via time-series cross-sectional analysis using 2999 country-year observations between 1989 and 2012.