The scientific limits of understanding complex social phenomena

Since Aristotle, the question about the potential relationship between economic inequality and democratic changes has been studied and debated. Scholars have explored whether increased economic inequality may be more or less likely to lead to democratic change? If there may be a causal relationship between these phenomena? But scientifically our ability as researchers to assess and understand how such complex phenomena may be related is much more limited than recognised.

This recent paper * illustrates that the existing literature is laden with contradictory hypotheses and findings that suggest that this potential relationship can be positive or negative, stronger or weaker, differentiated or non-existent and can vary across and within countries and time periods.

However, fundamental methodological and empirical limitations of analysis do not allow us to make such claims robustly. This is partly because the process of democratisation and changes in levels of equality are highly nuanced, idiosyncratic and heterogeneous and thus difficult to capture econometrically. Some of the most prominent authors in this literature claim that high levels of inequality decrease the likelihood of democratisation, and they also talk about “causal effects” and “the impact of democracy” on outcomes. Such conclusions presuppose a number of very demanding assumptions and requisite premises that cannot be rigorously met.

In fact, thousands of academic papers analyse the potential relationship between political variables like democracy and economic variables like inequality by gathering their data, selecting their methods and then going forward with their analysis, interpreting their findings and potentially informing policy, with many other steps along the way that involve making important implicit methodological assumptions.

My approach (as outlined in this paper) is to instead go backwards to analyse whether the data and methods that are applied by the leading authors in this literature are able to produce the robust results that they claim. This emphasises that how we as researchers generate our correlational (or “causal”) claims cannot be viewed independently from how we make everyday, typically unreflective decisions, such as what we decide to analyse, how we construct our variables, how we collect and use our data, which methods we choose to apply, how we interpret our statistical results and so forth.

A better understanding of the methodological and empirical limits of analysing the potential relationship between phenomena like inequality and political regimes is important for both research and policy: leading economists and other social scientists often misguidedly claim to establish causal relationships, and when their studies are used to inform public policy their claims can bring about adverse social outcomes.

Contrary to the existing literature, I argue that “causal mechanisms”, or even a robust correlation, that may potentially link the distribution of economic wealth and different political regimes cannot be identified due to a number of critical constraints.

The main methodological and empirical limitations are outlined in the paper.¹ An example of one of the main limitations is that we are particularly constrained in trying to quantify democracy as a statistical variable in order to analyse it. This is because democracy is an overarching concept encompassing the functioning of a range of difficult to measure institutions together with a number of observable, unobserved and unobservable factors. The variations between democratic institutions across countries are too large to be meaningfully comparable from an econometric and methodological perspective. Further, because democracy is generally a macro variable aggregately captured with a single observation per year or country, such analysis is statistically constrained to very rough correlations.

It is important to stress that dynamic social phenomena like democracy do not have an “intrinsic nature” nor do they abide by “social laws”, and so the data and methods used to measure democracy do not allow us to say anything about a causal relationship. Using new data sources, analysing different time periods or employing new data analysis techniques cannot resolve this question or provide robust, general conclusions about this potential relationship across countries.

Because researchers are restricted to exploring rough correlations over specific time periods and geographic contexts with imperfect data, they need to be more critical and transparent in explicitly outlining the limitations of the data and methods they apply, and about the accuracy and interpretation of their results. The hope of the paper is to possibly provide a useful reminder for researchers against overly ambitious research aims and the overselling of their estimated results.

A version of this post was originally published on the Institute for New Economic Thinking Blog.

Featured image: CC0