In-group bias in the Indian judiciary: Evidence from 5 million criminal cases

Elliott Ash, Sam Asher, Aditi Bhowmick, Sandeep Bhupatiraju, Daniel Chen, Tanaya Devi, Christoph Goessmann, Paul Novosad, Bilal Siddiqi

August 2021

Download the Bias Paper Access the Judicial Data

This project was funded in part by Emergent Ventures, the World Bank Research Support Budget, the World Bank Program on Data and Evidence for Justice Reform, the UC Berkeley Center for Effective Global Action, and the DFID Economic Development and Institutions program. Alison Campion, Rebecca Cai, Nikhitha Cheeti, Kritarth Jha, Romina Jafarian, Ornelie Manzambi, Chetana Sabnis, and Jonathan Tan provided excellent research assistance on this project.


Structural inequalities across groups defined by gender, religion, and ethnicity are seen in almost all societies. Governments often try to remedy these inequalities through policies, such as anti-discrimination statutes or affirmative action, which must then be enforced by the legal system. A challenging problem is that the legal system itself may have unequal representation. It remains an open question whether legal systems in developing countries are effective at pushing back against structural inequality or whether they serve to entrench it.

We studied ingroup bias in the Indian judiciary, where Muslims and women are substantially underrepresented. India is home to 195 million Muslims, and women represent 48% of the population, yet:

Judge composition changes

Does the gender and religious imbalance of the courts affect judicial outcomes? Using data on over five million court cases filed under India’s criminal codes between 2010–2018 across the country, we examined whether defendants receive better judicial outcomes when their cases are heard by judges with the same gender or religious identity (male/female or Muslim/non-Muslim).

We collected data on the outcomes of close to the universe of criminal cases in India from 2010-2018 and took advantage of the fact that cases are effectively randomly assigned to judges. Whether a defendant sees a judge of the same gender/religion as them is basically a coinflip -- albeit a coin flip that is heavily stacked against women and Muslims due to their underrepresentation in the judiciary. We compare defendants charged under the same criminal act and section, in the same month, in the same district court; the only difference is that some are assigned to judges who match their gender or religious identity and some are not.

We find: no evidence of bias! Judges of different genders do not treat defendants differently according to defendant gender, nor do judges display favoritism on the basis of religion. This null is seen both in terms of outcomes (i.e. acquittals and convictions) and in terms of process (i.e. speed of decision). Defendant outcomes are unaffected by whether their identity matches the judge. The tables below show the relationship between judicial decisions and gender group (Table 1) and religious group (Table 2) of judge and defendant. The third row shows the in-group bias in percentage points under a range of empirical specifications: a value of 0.002, for example, means matching the judge's identity raises the acquittal rate by 0.2% -- i.e. not at all, and not statistically significantly.

Table 1. Magnitude of in-group bias by religion is a precisely estimated zero

Zero religion bias
 

Table 2. Magnitude of in-group bias by religion is a precisely estimated zero

Zero religion bias

Why is this surprising? Because a large number of studies have found substantial in-group bias among judges, using random assignment of judge designs very similar to ours. For instance:

  • Shayo and Zussman find that Arab and Israeli defendants get better outcomes in Israeli small claims courts when they are respectively matched to Arab and Israeli judges respectively
  • Anwar, Bayer, and Hjalmarsson (2012) find that all-white juries in the U.S. convict Black defendants 16 percentage points more often than white defendants, a gap which is entirely eliminated when there is just one Black member on the jury.
  • Knepper (2012) finds that women are more likely to win workplace sex discrimination lawsuits when their cases are heard by female judges.

Digging deeper, we explore a subset of cases that might be predicted to make gender and religious identity more salient. First, we examine cases where the defendant and the victim of the crime have different identities; in these cases, the judge has an identity matching either the victim or the defendant, but not both. Second, we examine gender bias in criminal cases categorized as crimes against women, which are mostly sexual assaults and kidnappings. In both of these subset analyses, we continue to find a null bias. In the third case, we examine whether religious bias is activated during Ramadan. Here, we do find a weak effect -- Muslim defendants appear to have slightly lower acquittal rates than non-Muslim defendants, only if it is during the month of Ramadan and they appear before a non-Muslim judge. However, even in this very narrow subset of cases, the bias effect we find is very small relative to the literature.

One reason that all these studies have found statistically significant effects could be publication bias. We surveyed every study we could find that uses randomized judge assignment to study in-group bias. Figure 1 plots the standardized effect size of each study (on the X axis) against the standard error of the main estimated effect. With the standard error axis reversed, this is the standard ``funnel plot'' used to examine publication bias. In the absence of publication bias or a design-based mechanical correlation (such as adaptive sampling), study estimates should form a cone that is centered around the true estimate. The set of studies examined here show a highly non-conic and asymmetric shape, where effect magnitude is highly correlated with effect size, such that many of the studies fall just outside the cone boundary defining statistical significance at the 5% level. Other explanations may be possible (though we haven't thought of any), the funnel plot is at least consistent with there being a substantial degree of publication bias. It is quite possible that studies find null effects of in-group bias may have remained in researchers' file drawers or gone unpublished for other reasons.

Figure 1. Relationship Between Effect Size and Standard Error of In-Group Bias Studies

Effect
               sizes vs standard errors

For more information, please see our paper, In-group bias in the Indian judiciary: Evidence from 5 million criminal cases.

We have made all the data from this project public and open source: a dataset of 80 million cases in the Indian judicial system from 2010-2018 (5 million of which were criminal cases where we could identify judge and defendant religious or gender identity). To access the open data, click here.

To access the paper replication code and data, please visit the paper's public GitHub repo.

For a description of the public data with a small set of examples of potential use cases, see our medium post describing the data release.

If you use these data, please cite our paper:

@Unpublished{     aabbcdgns2021bias,
  author        = {Ash, Elliot and Asher, Sam and Bhowmick, Aditi and Bhupatiraju, Sandeep and Chen, Daniel and Devi, Tanaya and Goessmann, Christoph, and Novosad, Paul and Siddiqi, Bilal},
  note          = {Working Paper},
  title         = {{In-group bias in the Indian judiciary: Evidence from 5 million criminal cases}},
  year          = {2021}
}
            


Contact


Development Data Lab works with governments and private firms to generate bespoke insights from our data platform or your own data. For more information, send us an email.