In-group bias in the Indian judiciary: Evidence from 5 million criminal cases

Elliott Ash, Sam Asher, Aditi Bhowmick, Sandeep Bhupatiraju, Daniel Chen, Tanaya Devi, Christoph Goessmann, Paul Novosad, Bilal Siddiqi

August 2021

Download the Bias Paper Access the Judicial Data

This project was funded in part by Emergent Ventures, the World Bank Research Support Budget, the World Bank Program on Data and Evidence for Justice Reform, the UC Berkeley Center for Effective Global Action, and the DFID Economic Development and Institutions program. Alison Campion, Rebecca Cai, Nikhitha Cheeti, Kritarth Jha, Romina Jafarian, Ornelie Manzambi, Chetana Sabnis, and Jonathan Tan provided excellent research assistance on this project.


Structural inequalities across groups defined by gender, religion, and ethnicity are seen in almost all societies. Governments often try to remedy these inequalities through policies, such as anti-discrimination statutes or affirmative action, which must then be enforced by the legal system. A challenging problem is that the legal system itself may have unequal representation. It remains an open question whether legal systems in developing countries are effective at pushing back against structural inequality or whether they serve to entrench it.

We studied ingroup bias in the Indian judiciary, where Muslims, women, and lower castes are substantially underrepresented. India is home to 195 million Muslims, and women represent 48% of the population, yet:

Judge composition changes

Does the gender and religious imbalance of the courts affect judicial outcomes? Using data on over five million court cases filed under India’s criminal codes between 2010–2018 across the country, we examined whether defendants receive better judicial outcomes when their cases are heard by judges with the same gender, religious, or caste identity.

We collected data on the outcomes of close to the universe of criminal cases in India from 2010-2018 and took advantage of the fact that cases are effectively randomly assigned to judges. Whether a defendant sees a judge of the same gender/religion/caste as them is basically a coinflip--albeit a coin flip that is heavily stacked against women, Muslims, and lower castes due to their underrepresentation in the judiciary. We compare defendants charged under the same criminal act and section, in the same month, in the same district court; the only difference is that some are assigned to judges who match their identity on some dimension and some are not.

In the aggregate, we find no evidence of bias! Judges of different genders do not treat defendants differently according to defendant gender, nor do judges display favoritism on the basis of religion. We do not have caste identifiers in the data, so we proxy social group alignment by situations where the judge and defendant have the same last name. Naturally, this measure has limitations, which are discussed in the paper. On average, being assigned to a judge with the same last name as the defendant does not affect the acquittal rate. These null effects are seen both in terms of outcomes (i.e. acquittals and convictions) and in terms of process (i.e. speed of decision). The tables below show the relationship between judicial decisions and gender group (Table 1) and religious group (Table 2) of judge and defendant. The third row in each table shows the in-group bias in percentage points under a range of empirical specifications: a value of 0.002, for example, means matching the judge's identity raises the acquittal rate by 0.2% -- i.e. neither substantively nor statistically significantly.

Table 1. Magnitude of in-group bias by gender is a precisely estimated zero

Zero religion bias
 

Table 2. Magnitude of in-group bias by religion is a precisely estimated zero

Zero religion bias

Why is this surprising? Because a large number of studies have found substantial in-group bias among judges, using random assignment of judge designs very similar to ours. For instance:

  • Shayo and Zussman find that Arab and Israeli defendants get better outcomes in Israeli small claims courts when they are respectively matched to Arab and Israeli judges respectively
  • Anwar, Bayer, and Hjalmarsson (2012) find that all-white juries in the U.S. convict Black defendants 16 percentage points more often than white defendants, a gap which is entirely eliminated when there is just one Black member on the jury.
  • Knepper (2012) finds that women are more likely to win workplace sex discrimination lawsuits when their cases are heard by female judges.

The literature suggests that some contexts may make identity more salient than others. We explore a subset of cases where we might see this effect. First, we examine cases where the defendant and the victim of the crime have different identities; in these cases, the judge has an identity matching either the victim or the defendant, but not both. Second, we examine gender bias in criminal cases categorized as crimes against women, which are mostly sexual assaults and kidnappings. In both of these subset analyses, we continue to find a null bias.

In the third case, we examine whether religious bias is activated during Ramadan. Here, we find a weak effect-- Muslim defendants appear to have lower acquittal rates than non-Muslim defendants, only if it is during the month of Ramadan and they appear before a non-Muslim judge. Even in this very narrow subset of cases, the bias effect we find is very small relative to the literature.

Finally, we explore the effect of last name similarity in subgroups that have rare last names. When judge and defendant share a last name, but the last name is common, this may not be all that salient to the judge, because it represents shared membership only in a very large social group. If they share a rare last name, this indicates shared membership in a much smaller social group, which may raise the salience of that shared identity. To test this hypothesis, we run the shared last name estimation, limiting the sample to rare last names. Here, we find a substantial degree of in-group bias; sharing a rare last name with a judge raises your acquittal rate by three percentage points. This suggests that in-group bias is indeed activated when the group is very small. But the aggregate effect of this bias will remain small, because individuals with rare last names have by definition a low probability of being randomly assigned to a judge with the same last name.

In the paper, we discuss a number of reasons why the Indian judicial context does not appear to activate in-group bias in the majority of cases. Our findings on salience are suggestive. Most prior work on judicial bias is in the United States (focused on race) and Israel (focused on ethnicity); these may be contexts where group differences are extremely salient, particularly in the judicial system.

However, a second factor to consider is that there could be publication bias in the literature on judicial bias. We surveyed every study we could find that uses randomized judge assignment to study in-group bias. Figure 1 plots the standardized effect size of each study (on the X axis) against the standard error of the main estimated effect. With the standard error axis reversed, this is the standard "funnel plot" used to examine publication bias. In the absence of publication bias or a design-based mechanical correlation (such as adaptive sampling), study estimates should form a cone that is centered around the true estimate. The set of studies examined here show a highly non-conic and asymmetric shape, where effect magnitude is highly correlated with effect size, such that many of the studies fall just outside the cone boundary defining statistical significance at the 5% level. While other explanations may be possible (we haven't thought of any), the funnel plot is at least consistent with a substantial degree of publication bias. It is quite possible that studies finding null effects of in-group bias may have remained in researchers' file drawers or gone unpublished for other reasons.

Figure 1. Relationship Between Effect Size and Standard Error of In-Group Bias Studies

Effect
               sizes vs standard errors

For more information, please see our paper, In-group bias in the Indian judiciary: Evidence from 5 million criminal cases.

We have made all the data from this project public and open source: a dataset of 80 million cases in the Indian judicial system from 2010-2018 (5 million of which were criminal cases where we could identify judge and defendant religious or gender identity). To access the open data, click here.

To access the paper replication code and data, please visit the paper's public GitHub repo.

For a description of the public data with a small set of examples of potential use cases, see our medium post describing the data release.

If you use these data, please cite our paper:

@Unpublished{     aabbcdgns2021bias,
  author        = {Ash, Elliot and Asher, Sam and Bhowmick, Aditi and Bhupatiraju, Sandeep and Chen, Daniel and Devi, Tanaya and Goessmann, Christoph, and Novosad, Paul and Siddiqi, Bilal},
  note          = {Working Paper},
  title         = {{In-group bias in the Indian judiciary: Evidence from 5 million criminal cases}},
  year          = {2021}
}
            


Contact


Development Data Lab works with governments and private firms to generate bespoke insights from our data platform or your own data. For more information, send us an email.