Kleros: Promoting access to (analog) justice?

Response to Kleros team – Written by Jinzhe Tan, Research Assistant at Cyberjustice Laboratory and Ph.D. student at Université de Montréal.

1. Introduction

This post is a response to the Kleros team.

First of all, I would like to thank the Kleros team for their detailed and professional response, which helped us to better understand how their system works, and for their professional approach to the discussion about their product.

Since this post is a quick response, I can only superficially cover some of the points I want to make, and as for more nuanced expressions, I will address them in a paper called “Analog Justice”.

I noticed in the Kleros team’s response some vague misunderstandings about my perceptions. I need to clarify that I am not and will not be one of those who dismissed electricity when it first emerged, but since Kleros is a new system about achieving justice, we must keep an open mind while using a critical perspective to examine it, and clarify its value as well as its potential flaws, believing that after discussion and research, Kleros will become more sophisticated and our understanding of justice will go further.

2. Response to (2) — on the “objective” oracles and “subjective” oracles

Considering that our readers are mainly from the field of law, I attach here a brief introduction to blockchain oracles. The oracles mechanism refers to the blockchain’s access to information from the outside world. The blockchain is a deterministic, closed system, which, in the context of smart contracts, means that it must yield the same result in any situation in which it operates. This means that “from inside, one can only access data that is on the blockchain already”[1], one cannot connect directly with the outside world. And when a smart contract needs external information to trigger its automatic execution, real-world data needs to be fed to the blockchain via a trusted oracle[2].

As Sharma mentions in his post, in the design of the Kleros system, and in the expectations of the disputing parties, jurors should accurately review the materials and relevant evidence and make their decisions in accordance with the applicable norms. Jurors, under the economic incentive to make the right choice and the punitive pressure to make the wrong choice, will tend to choose the Schelling point they predict in their minds, and in some cases (perhaps even most cases), that Schelling point will converge on using the moral standards in the jurors’ minds and achieve a “decision based on subjective reasoning and the applicable norms” effect.

I fully agree with this perception, which is the ingenuity of the Kleros system’s design, and that is why it is worth debating. However, it is also important to recognize that the immediate motivation of the jurors participating in the Kleros system is not to judge based on internal “moral-based” subjectivity criteria, but to obtain economic incentives. The rationale of the Kleros system is to use economic incentives to analogize the effects of the former and expect to achieve a relatively good substitution effect.

Whereas Sharma argues that since Kleros jurors and judges (or arbitrators) both make decisions in a subjective manner, there is not much difference between them. I agree that in the design of Kleros’ system, jurors transmit their subjective judgments to the blockchain via oracles. But the difference between a Kleros juror and a judge (or arbitrator) is not whether the decision is subjective or objective, but rather the motivation behind the subjective judgment, and the motivation behind the subjective judgment of a Kleros juror, motivated by economic incentives to try to predict the most chosen option, is fundamentally different from the motivation behind the subjective decision made by a judge or arbitrator.

3. Response to (3) — on the “simplicity” of Schelling Points

The comparison between the “simplest answer” and the “most correct one” that I referred to in my previous blog is a situation where the complexity of the case has increased to such a degree that even a well-intentioned person reviewing the case materials can easily be blinded by the facts on the surface, resulting in a wrong first judgement.

In a traditional courtroom, people can deepen their understanding of the case by discussing it with each other, and it may eventually redirect people to the right answer (a process like academic arguments). In the Schelling Point system, on the other hand, people have no incentive to analyze the case too deeply, because others may stay at the surface level of judgment, when too much analysis can even bring punishment to jurors. In this case, following one’s own superficial perception is the “simplest choice” for the juror, and it is the choice that will be rewarded, but it is hard to argue that it is the morally “most correct one”.[3]

There are numerous decisions in history that have had a profound impact on society that have been arrived at through a long and painful process. In the process of analyzing cases, judges go through a painful journey, adjusting and even reversing their perceptions in the midst of many arguments, before arriving at a decision that can withstand the challenges of history.

Of course, jurors in the Kleros system have little access to such high-impact cases, but the Kleros system is designed so that jurors have less incentive to rigorously challenge their own intellects.

Sharma argues that the concerns about these risks are based on elitist arrogance and a repackaging of anti-democratic arguments. However, this is not a critique of the competence of the jurors in Kleros, but rather our self-perception of the competence of humankind. Even the most respected and experienced judges may be initially blinded by superficial appearances, but the mechanism of the moral-based trial gives people the determination to search for the truth. And a major and important part of democracy is communication and debate among participants. In the Schelling Point system, such communication may be difficult to achieve.

In his post, Sharma notes in (4) that the platform provides a mechanism for jurors to communicate with each other, and also creates telegram chat groups, which surprised me since Schelling points by definition deny the existence of communication. I look forward to the opportunity in the future to directly observe how this communication system works and try to understand the rationale behind it.

4. Response to (4), (5) — Are we striving for a perfect dispute resolution system?

There is no perfect dispute resolution system, and different dispute resolution mechanisms have different shortcomings, which reinforces the value of pointing out the potential risks of a particular system.

Sharma argues that we have looked at Kleros using unattainable strict standards without doing the same for other approaches. But discussing the shortcomings of a new system does not mean we think the old one is perfect. Rethinking the traditional moral-based courtroom is also an important topic, and as we have mentioned in previous blogs, there are many empirical studies showing that human judges are biased[4] and are influenced by a range of factors unrelated to their decisions[5]:

1. When and what a person has eaten[6]
2. The time of day[7]
3. How many other decisions a person has made that day (decision fatigue)[8]
4. Reliance on intuition[9]
5. The attractiveness of the individuals involved[10]

The existence of risk in a system does not mean that the system should not be used, but before a system can be widely used, it must be ensured that the probability of risk can be controlled. In a traditional justice system, the fact that one judge is influenced by some extraneous factor does not pass quickly to the other judges, because different judges are affected differently by extraneous factors, this stops the massive spread of risk, which may not be the case in the Kleros system.

Sharma, while agreeing with the non-uniqueness of the jurors’ motives, the possibility of collusion at Schelling, and the possibility of an “irrational rule” within the jury, offers some reasons to believe that the probability of these risks is low.

However, if the volume of disputes handled by Kleros increases significantly in the future, the probability that these risks will arise when it becomes profitable to use these vulnerabilities to attack the Kleros system increases dramatically.

The attacker may spread the designed “irrational rules” widely in places such as social media (in the Kleros system, irrational rules are destructive because jurors who choose according to irrational rules are actually rewarded, and this positive feedback can promote undesirable behavior in the system), and the attacker may also hire jurors to deliberately render verdicts that are inconsistent with common motives to prove system failure, etc.

Once such an attack successfully takes on scale, the legitimacy of the platform will be compromised. Therefore, even if these potential risks have not yet become truly significant, Kleros needs to come up with effective solutions for the prevention and control of these risks to improve the robustness of the entire system. If one expects the users to be well-intentioned, then one may suffer an unbearable impact.

We see that Kleros has designed promising risk prevention and control measures, and we will follow up with an analysis of the effectiveness of these measures in subsequent studies.

5. Response to (6) — On attack vectors and design weaknesses

Kleros’ efforts to limit the risk of juror collusion are noteworthy, and as I mentioned in the original blog, recognizing and limiting the risk was the driving force in facilitating the development of the Kleros. We will also observe the actual performance of the Kleros-designed appeal mechanism in future research.

While Kleros’ existing pool of jurors is not large enough,[11] it is exciting to see the efforts Kleros is making to expand its pool of jurors, and I believe that as the number of jurors increases, the potential risks caused by small pool of jurors will be addressed. Due to the limitations of my background knowledge, it is difficult for me to make a proper evaluation of the commercial rollout, and I wish Kleros the best in expanding its pool of jurors with my sincerest expectations.

6. Response to (7) — On the tension between financial incentives and the motivation for justice

6.1 On fairness or justice

Events will sometimes be “just” but “unfair” or “fair” but “unjust”.

If we draw a spectrum of dispute resolution mechanisms, the leftmost part of the graph represents extreme fairness, such as a coin toss to determine the winner, completely random, and the rightmost part of the graph is a flawless decision by an omniscient judge. Then the existing real dispute resolution mechanisms fall between these two. It is difficult to achieve better results by simply emphasizing “fairness” or “justice”.

Sharma claims that Kleros provides a “fair” outcome, not a “just” one, and that if an unjust outcome is fairly distributed to both sides of a dispute, then there are too many cheaper ways for people to come up with results.. If the risk of the system cannot be suppressed to provide a fair outcome (at least in most cases), then fairness is just a ripple in the water and will soon disappear.

I absolutely agree with what Sharma said about “The protocol’s legitimacy thus derives from whether it can keep delivering consistent, effective results for its users that they would otherwise be robbed of under the status quo.” This means that we need to find a place on the spectrum for Kleros with a reasonable degree of fairness and a reasonable degree of justice, while suppressing the threats to the system.

6.2 On the Inability of Schelling Point Systems and Moral-Based Systems to Coexist

In the Schelling point system, the only incentive people receive is the economic reward they receive for choosing the option chosen by the most people. Therefore, without considering people who do not consider economic incentives as a motivation for action, the vast majority will choose the option they believe is most likely to receive an economic incentive.

In some cases, distortions can arise between choices made on the basis of economic incentives and judgments made on the basis of morals. However, to maintain the smooth functioning of the Schelling Point system, the requirement for moral standards must be discarded within the system. That is, even if a juror makes a choice that is not morally correct, it should be considered a normal result produced by the system. We cannot tell jurors that economic incentives are their only goal, while using moral principles to critique the judgements they make.

What is described above is the internal situation of the Schelling Point system. From the design of the whole system, the purpose of the Schelling Point system is to fill the areas that traditional justice and other dispute resolution mechanisms fail to cover, and ultimately facilitate access to justice. Therefore, we need to consider the extent to which choices based on economic incentives can be distorted from judgments based on morality.

Sharma equate judges “motivated by both the pecuniary and non-pecuniary incentives of their jobs” with the pursuit of economic interests by jurors in the Schelling Point system. This is clearly unjustified. Let us take academia as an example. There is a fundamental difference between academics who follow academic ethics and do their best to complete scientific research and then receive a salary from the academy as a result, and those who accept money from a specific client and complete research that proves the client’s specific requirements, which stems from whether money acts directly on people’s motivation to produce academic work. Due to the nature of academia, academic ethics must be followed when conducting scientific research. However, as mentioned above, because economic incentives and ethical principles cannot coexist in the Schelling Point system, jurors in the Schelling Point system should not be criticized for pursuing economic interests.

If Sharma are discussing the reality in which some judges are influenced by monetary or non-monetary bribes to make unethical decisions, my response is that we cannot use the failure of one system to justify the failure of the other. Even if a comparison is needed, the probability of failure and the magnitude of the impact of the two systems should be compared. This requires us to obtain empirical evidence.

6.3 Analog justice

Schelling point systems can produce just results, but this is not a direct expectation in the design of the system. Schelling point systems is an analog of traditional justice systems, as analog signals use current fluctuations to simulate sound. Although the analog signal is transcoded to sound like a human voice, it is actually just the flutter of electrical waves, and we can hear distortion and noise when the analog signal is out of order. The concern we raised in our previous blog was about the possibility of distortion of this analog justice system.

This, of course, involves how we define “justice”, which is a subject of endless debate in legal scholarship. Nor can I explore the concept of justice in a short blog, but I can try to describe here what I mean by true justice: the judgments people make about events based on their inner conscience and good intentions.

Some scholars believe that the concept of justice should be strictly limited, that the court should be the core of the justice system, and that the extension of the concept of justice to ADR and ODR should be guarded[12]. However, even the bold expansion of the concept of justice does not break the limit of “individual ethical characteristics”[13].

It is interesting to note that scholars skeptical of ADR have made the following criticism, “It does not contribute to substantive justice because mediation requires the parties to relinquish ideas of legal rights during mediation and focus, instead, on problemsolving.[14]” With the broad use of ADR and ODR, people gradually accepted the justice provided by the out-of-court system. We can also feel the flow of the concept of justice in the history.

Now, removing moral principles from “justice” still seems too radical, which is why we call the verdict offered by Kleros a “analog justice”, where there is a certain probability that a verdict based on economic incentives will leap to real justice after it is rendered.

As the finding that analog signals are distorted under certain circumstances does not mean denying the potential value of analog signals. I fully analog justice system can help the disputing parties to some extent to arrive at a reasonable solution, but the problem of system failure remains, and our task is to:

a) Understand to what extent the analog justice system can function smoothly

b) What are the special circumstances that have a higher probability of causing the system to fail

c) How to design follow-up mechanisms to suppress the system risk.

6.4 Kleros — Promoting access to (analog) justice?

Dispute resolution has been costly and time-consuming since time immemorial, and many less privileged people have been unable to use the law to protect their legitimate rights due to a variety of constraints. Throughout the history of our judicial system, various efficiency-enhancing methods have been invented to facilitate access to justice, such as qui tam actions, class actions, derivative actions, settlements (where right and wrong give way to efficiency), and pre-litigation exchanges of evidence (discovery).

While I believe that the traditional ethics-based justice system is still the optimal dispute resolution system at this stage, we also see that the existing justice system cannot provide justice for everyone who needs dispute resolution. So, weakening some of the requirements for justice in the dispute resolution system may be another option to promote access to justice.

The fact that the Kleros system produces utility and promotes access to justice does not necessarily require that it be superior to traditional courts or other systems in terms of the degree of justice. The gap in access to justice is often not because the dispute resolution system is not just enough, but because it is too expensive, too time consuming, too complicated to understand, etc. So, Kleros has a wide range of prospects if it can be improved in other aspects while maintaining a significant degree of fairness.

7. Conclusion

After the discussion above, it is clear that we stand on the same ground with Sharma in many aspects. We both agree that the Schelling point system can yield results close to traditional justice to some extent, and we also agree that this system can have some risks in some cases. All systems have risks, we just need to carefully analyze the causes of the risks, and for the underlying risks that cannot be avoided, we can design other mechanisms to avoid them.

The goal of the Schelling point system should not be set to replace traditional justice system, but to be a powerful supplement to the traditional justice system when it is absent. And exactly how much potential the Schelling point system can have for application, as Sharma says, conducting an experiment is a good path to help us find the answer, which requires us to carefully design the experiment to observe the core of the problem.

Blind dismissal of emerging technologies and procedures can discourage innovation, but unguardedly welcoming emerging everything with open arms can also put us at risk, and I believe that using an open mind and a prudent approach to innovation can help us get better on the road to a bright future.

[1] Roman Mühlberger et al, Foundational Oracle Patterns: Connecting Blockchain to the Off-Chain World (Cham: Springer International Publishing, 2020) at 36.

[2] Giulio Caldarelli, “Understanding the Blockchain Oracle Problem: A Call for Action” (2020) 11:11 Information 509 at 4.

[3] This has less relevance to the Occam’s Razor. But I may as well interject here a response to Occam’s reference to “the simplest explanation is usually the correct one”. We are not sure that the trial of morals in court is already the simplest optimal solution. If we over-abbreviate the necessary elements, we may degenerate into using a coin flipping to determine the winner.

[4] Donald C Nugent, “Judicial Bias” (1994) 42:1 Clev St L Rev 1–60; Jon Kleinberg et al, “Human Decisions and Machine Predictions*” (2018) 133:1 The Quarterly Journal of Economics 237–293; John Zhuang Liu & Xueyao Li, “Legal Techniques for Rationalizing Biased Judicial Decisions: Evidence from Experiments with Real Judges” (2019) 16:3 Journal of Empirical Legal Studies 630–670.

[5] Tania Sourdin, “A Broader View of Justice?” in The future of dispute resolution, 1st ed (Chatswood, N.S.W: LexisNexis Butterworths, 2012).

[6] John Tierney, “Do You Suffer From Decision Fatigue?”, New york times (17 August 2011).

[7] Ibid.

[8] Ibid.

[9] Michael Kirby, “Judging : reflections on the moment of decision.” 4:3 The Judicial Review: Selected Conference Papers: Journal of the Judicial Commission of New South Wales 189–216.

[10] Maria Agthe, Matthias Spörrle & Jon K Maner, “Does Being Attractive Always Help? Positive and Negative Effects of Attractiveness on Social Decision Making” (2011) 37:8 Pers Soc Psychol Bull 1042–1054.

[11] According to the Kleros website, Kleros now has 778 active jurors. “Homepage”, online: Kleros <https://kleros.io/> (Accessed: December 8, 2022).

[12] Hazel Genn, “What is Civil Justice for – Reform, ADR, and Access to Justice” (2012) 24:1 Yale JL & Human 397–418 at 397.

[13] Tania Sourdin, A Broader View of Justice? (Rochester, NY, 2012).

[14] Genn, “What is Civil Justice for – Reform, ADR, and Access to Justice Representing and Contesting Ideologies of the Public Spheres”, supra note 13 at 411.

Ce contenu a été mis à jour le 25 avril 2023 à 16 h 07 min.