Chinese apps are letting public juries settle customer disputes

I’m fascinated by these experiments that give power back to the users and let them make decisions on platform rules and in individual disputes. While most of the jury features were discontinued after just a few years, they nevertheless provide interesting insight into basic human behavior on the internet and the nature of grassroots governance.

One of the oldest programs was from Alibaba, the e-commerce giant. From 2012 to 2018, Alibaba allowed any user, whether a seller or a buyer, to vote on inappropriate behaviors on the platform or in transaction disputes from unsatisfied buyers. By the end of the program, this jury system had processed over 16 million cases, with over 1.7 million users casting over 100 million votes. 

The system had some built-in mechanisms to prevent people from abusing it by judging cases where their personal interests were involved. There was a multi-step randomized distribution system that made it impossible to predict what case you’d get assigned, and the app proactively suspended users who kept skipping over cases. (It also gamified its design, in which jurors could get experience points and level up after they judged more cases. It meant nothing in real life, but you got bragging rights for being able to say you were among “the top 100 jurors around the world.”)

Even though the exact program doesn’t exist anymore today, the data it generated can teach us more about how these crowd-voting systems really work. 

Angela Zhang, a law professor at the University of Hong Kong, wrote a paper in 2021 with two fellow academics in which they studied 630,000 jury cases on Alibaba in a 20-month period. Over 150,000 jurors participated in these cases; more than 80% of them were buyers on the platform.

Zhang’s analysis found that crowdsourced dispute resolution is significantly more efficient than the alternative. While a normal complaint would take three to four business days for the platform to process, the online jury normally took about 73 minutes to reach a decision. (However, it can take some time to secure a jury trial; for instance, one merchant told me that could take days on Meituan.)

But the researchers also found a major problem: bias. In a buyer-seller dichotomy, people are more likely to vote for the group they belong to. On average, a juror who sold on Alibaba’s platform was 10% more likely to vote for the seller’s side than a juror who was also a buyer. That bias increased when jurors saw an ambiguous case or after they witnessed a few cases in which their own “side” lost. 

Perhaps that’s not surprising. Participants in the program didn’t need to go through intensive jury training; nor were there any real accountability mechanisms to check their decisions.