Give Me Credit
Zhima Credit: China’s Social Credit System
Numbers are very convenient for a lot of things. We can use them to define a quantity; they are useful for expressing size in a meaningful way; they are unique and so can be used as identifiers; and they are perfect for ranking. In sports and games, a score is used as an easy means to determine who is better.
In the financial world, a credit score is used as an easy measure of who is better, as well as a decent way to track whether one’s financial decision making has improved in the eyes of those who lend money.
In China, the government is instituting a score to rank its citizens and their overall quality in the eyes of the party; this social credit score is used to help quantify the difference between a “good” citizen from a “bad” one.
In the following analysis of Sesame Credit and the Chinese national social credit system, the assumption will be made that, in a vacuum, all humans are equal. Terrell Ward Bynum’s concept of the ethical point of view requires the adoption of the mindset that equality, justice, and respect are basic necessities and rights of every human.
These rights are not always granted in society, but those three rights will be granted to all individuals considered in this post.
Before we begin in earnest, we should ensure we have a firm understanding of what Sesame Credit is. Sesame Credit, or Zhima Credit, is a third party credit check company run by Ant Financial, an affiliate of the Alibaba Group. The range of scores extends from 350 as the lowest possible, up to 950 maximum, with a higher score being more desirable.
An individual’s Sesame Credit score is calculated based off how how “good” of a citizen they are. This is a complex concept to fully understand and quantify, partly because the algorithm used to calculate the score is a secret, but also because “good” is subjective.
Right now, participation in Sesame Credit is voluntary but participation in a national social credit system will be mandatory for all Chinese citizens by 2020.
We will get into the weeds of how to adjust one’s Sesame Credit, but first why should we care; what motivates someone to try to improve their social credit score? People with higher scores receive more real-life benefits than those with low scores.
With a score of 750 or more, for example, someone can use a fast-lane to check in at the Beijing airport. 600 or more allows one to get a short-term rental room with no deposit. Bikes can be rented by users with a high enough score without needing to make a deposit. People on dating websites and apps may have the option to view the score of a prospective date before choosing to talk to them (angrymoo, 2017).
In order to gain those benefits, and to ensure that a low score does not impact success on dating sites, one must build up their score. So how do we get into that fast lane? The Sesame Credit system is built on the idea of rewarding good behavior.
Financial responsibility such as timely bill payments, rent payments, and credit repayments are essential to improving one’s score. This part of the system is very similar to the FICO credit score that we are familiar with in the United States. Our score changes along with our financial decision-making.
The fact that Ant Financial is an affiliate of Alibaba is important because Alibaba also owns Alipay. Alipay is one of two primary apps used in China for payment, and is more of a massive ecosystem than just a single payment app (Hvinstendahl, 2017).
Chinese users can use Alipay for essentially everything. The app also directly integrates with many major services such as Uber and Airbnb. In 2015, Sesame Credit appeared in the Alipay app portal.
With hundreds of millions of users, and such a breadth of options, Alipay has been able to amass a huge quantity of credit data. When everyone’s spending habits are tracked in a single location, a lot of insight into those lives can be gained. According to the Sesame Credit User Agreement, the types of spending data includes: shopping, car rental, transfer/payments, loan transactions, and credit card transactions (angrymoo, 2017).
Which brings us to the other part of being a good citizen: having a positive social presence. Sesame Credit does not exclusively track and calculate its score based on users’ financial choices, it also factors in what you buy, your educational background, government records, and even your friends.
Add friends on Alipay that have high Sesame Credit, and that will help to improve your own. On the other hand, friends that have lower credit will reduce your score.
It is not yet clear whether Sesame Credit itself will be implemented as part of the national credit system, but we can use it as a proxy for what is to come to gain some insights about what impacts that future system may have.
First we will look at Sesame Credit alone, and then expand to the larger plan of the nationwide social credit program. We will look at three ethical issues with this system and try to demonstrate their varying levels of impact on citizens.
According to angrymoo, a research tool for expanding businesses into Chinese markets, the four tiers of data that are collected by Sesame Credit are basic information, products/services, government (payment records), and judiciary/government information. As mentioned previously, users’ scores are calculated from this data along with the scores of their friends on the app.
If the scores existed in a vacuum, then they would be nearly meaningless. Maybe my score is higher than yours but the value in that score is removed. When higher scores are rewarded, it takes on meaning.
This brings us to the first ethical dilemma: our friends’ scores factor into our own, and vice versa. Higher scores are eligible for greater rewards so it is easy to see that participants will want to have friends with the highest scores possible. If one or more friends have lower scores, it would behoove one to give those friends a nudge to try to improve theirs.
Since the scores are calculated by factors that are related to quality of life (staying out of debt, being in good standing with taxes, not breaking the law) then encouraging your friends to improve their scores could be viewed as encouraging them to live a better life. However, the scores are only recalculated monthly and many of the inputs are take a long time to adjust.
For example, paying off a loan is not usually a quick process. The judiciary/government factors of the score are based off of a list of citizens who have committed offenses, though there is no indication of how far back offenses are penalized. If a person stole something a decade ago, does that factor into their score today?
If past transgressions, especially those in the remote past, reduce one’s score then that is a disadvantage when attempting to gain friends in the credit system. And as we have seen, a lower score disincentivizes others with higher scores from becoming your friend, which further puts a ceiling on your own score.
This cycle of low scores being driven lower by virtue of being low is one that we will see multiple times.
Being on a blacklist used by the Supreme Court is another example of the judiciary system’s impact on your score and life. The same blacklist used by the courts is integrated directly into the scoring system; if you are on the list, that factors into your score. Where the score exists to indicate reward eligibility, the blacklist exists to restrict.
One example given by Hvistendahl was of a man who was blacklisted because of a defamation lawsuit he had been involved in. When he tried to pay the fine, he transferred the money to the wrong account number so he remained on the blacklist and was not allowed travel.
This begins to bring future possibilities into focus with how this scoring system may work on a national level. It does not seem to be a huge stretch to see how the scores could be used not just as rewards, but also to restrict freedoms.
We have a travel-restricting blacklist, the same blacklist affects a social credit score, and so we see a social credit score being used to restrict freedom.
It is not just those who have had dealings with the law who may have a low score, obviously. Previously we considered an example of a reformed criminal being pushed to the fringes of society because their lower score did not incentivize others to become their friends.
Again, we must emphasize that friendship and its effects on this app are between approximately 200 million users; China will be enforcing a full social system on the entire country in the next two years. This is incredibly important. It is not simply the ethics of whether to add a friend on Facebook; this system has real life consequences.
Employers want to hire only the best people, and as we know, personal finances are a key player in the score calculation. If an employer wants to hire the most responsible people, a seemingly safe proxy would be their Sesame Credit score. But if you are unemployed, there is a much higher risk of experiencing some sort of financial hardship.
Unfortunately, if you fall behind on your rent payment or other loans due to unemployment your Sesame Credit score will decrease and a lower score will make you less attractive as an employee. We can again see a potential for the score to self-actualize into the person’s life as they fall further behind on their loan payments and become even less attractive as an applicant based off of their score.
“As Lucy Peng, the company’s chief executive, was quoted as saying in Ant Financial, Zhima Credit ‘will ensure that the bad people in society don’t have a place to go, while good people can move freely and without obstruction.’” (Hvinstendahl, 2017).
The end goal is to divide society along moral lines with Sesame Credit, ensuring the successful remain successful and the unsuccessful remain unsuccessful.
The national goal seems to be similar, but with more social data input than just friends’ scores. According again to Hvinstendahl, “[t]he State Council has signaled that under the national social credit system people will be penalized for the crime of spreading online rumors, among other offenses.” By penalizing users with lower scores for transgressions on social media, the state will have control over its citizens without ruling by fear.
People do not want to be friends with others who have lower scores because it reduces their own. This encourages you, not the government, to police what your friends post online. If your friend continues to post things that reduce their score, it is in your score’s best interests to cut ties with that person.
Everyone will have a threshold, and as that person loses more and more friends they will be further marginalized and silenced. If you have no friends to read your anti-government posts, those posts will go unseen.
Thus through those social pressures, the message that the government wants to convey and the reality they want to create becomes so. It is oppression via incentive rather than oppression by fear. By “gamifying” obedience, a person gets a shot of dopamine for increasing their score.
Historically, dictatorships and other oppressive regimes ruled by fear, which can work for a short period but has often ended in revolt. People do not like to be afraid, and that unease is volatile. To make compliance desirable through reward is a different tactic that is unprecedented on such a large scale.
Early indications of how the system is structured by Ant Financial and will be created by the government demonstrate a desire to divide. Keep the low scores low, keep the high scores high and keep those people separate.
Given the potential effects of the scoring system, is this ethical? We will analyze this question under two ethical theories; first, from a utilitarian perspective and then using a Kantian view. In our utilitarian consideration, we will perform a stakeholder analysis to understand the utility of this system for the general population. Our Kantian investigation will involve a professional standards analysis of the creators of the system.
Ant Financial only rewards people with higher scores, there are no explicit penalties for having a lower score. From a utilitarian perspective, this is purely good for the population. If your score goes down, you lose a perk that you had gained rather than having something you owned taken away.
Equalizing to zero rather than a net negative is an important distinction. Losing your upgrade to the fast lane puts you back where you were before, it does not leave you in worse shape than you began.
Since the score can be impacted by friends, if you lose a friend that you had before participating in Sesame Credit then that would be a net negative utility for the lower-score person.
It is impossible to say whether society overall loses happiness from this, but it does demonstrate that it is possible to reduce happiness even in the strictly rewards-based system due to your score.
The implementation of a national social credit system will result in every Chinese citizen receiving a social credit score. Since this is not yet implemented, we have to make some assumptions and ask questions rather than be able to actually analyze the system’s impact. Since we do not know what the distribution of the scores will be for the population, we will assume that they follow a normal distribution.
If the national program were to implement penalties for lower scores and magnitude of the penalties is equal to that of the rewards, we again arrive at a zero sum system.
We will likely not know until 2020 what the pros and cons are of the scores. It is possible that there are no explicit penalties, and that the algorithm is designed in such a way that social pressures are enough of a motivator. If the penalty of a low score is ultimately social exile for the “bad”, maybe as terrible as it sounds, there could be a net positive societal impact.
Through the lens again of employers and employment, an economy depends on an efficient and effective labor force. Should this score prove to be a good proxy for work ethic, employers can make more informed hiring decisions. This could boost company profits, and give the national economy an uptick.
Circling back to solely Sesame Credit, let’s take one final look to assess this scoring mechanism as utilitarians. As we saw above, the only real negative in the system exists as a side effect: un-friending someone.
Does that friendship end in real life, or become strained in any way in the Sesame Credit system? That will depend on the relationship which is outside of the scope of the credit score. If the system places a heavier weight on friends’ scores in its calculation, then it could be less valuable for happiness.
Losing a real friend carries much more emotional weight than reinforcing, or even getting a “+1” as a result of, an existing friendship with an app.
Since Sesame Credit is operated by a financial institution, they will want more people to use their app and so want people to have a positive mental association with it. So it would make sense for the algorithm to be constructed in a way that maximizes user interaction and participation.
If the world of friendships became cutthroat over their Sesame Credit scores, then eventually people will stop participating because getting 50% off your first night at a hotel is not worth losing a friend. In this way, the overall utility of the app seems to be positive. We will have to wait and see what happens at the national level.
As a reminder of Kantian theory, we assume that humans have intrinsic value and do not need anything outside themselves to give them worth. This is the categorical imperative: “Always treat every person… as a being that has worth in itself, never merely as a being to be used to advance someone else’s goals,” (Bynum, 2008).
Sesame Credit and any implementation of the national credit system most certainly violate this ethical theory. Remember the quote by Lucy Peng, stating that Sesame Credit will ensure that the “bad” people in society will be separated from the “good”.
By giving people a higher score, especially for things outside of their control such as the propensity of their friends to confirm and maintain friendship, the social credit system does not give everyone the same value. In fact, it quite literally does the opposite.
Not only does this violate the philosophy of Kant, it also does not hold up to scrutiny under the Data Science Association’s Code of Conduct. Surprisingly, a system designed this way does meet a lot of the criteria of a that code.
Some assumptions must be made about the security of the data, but it seems counter-intuitive that a large corporation or the government would be cavalier with data that is so valuable to them both financially and socially. Ant Financial wants that customer data to help them with their proprietary algorithm and for a competitive advantage.
It seems safe to assume that they would prioritize the security of that data. The system rolled out by the government would want to keep information about their citizens private. We also do not know the quality of the data scientists that have been hired to develop these algorithms, but China is a burgeoning tech giant, and mega-corporations like Tencent and its competitors like Alibaba employ a lot of talent.
There is one very problematic point in the code, but it is morally subjective. Per the DSACC, Rule 8(h) specifies that
“A data scientist shall use reasonable diligence when designing, creating and implementing machine learning systems to avoid harm.”
Furthermore,
“If a data scientist reasonably believes the machine learning system will cause harm, the data scientist shall take reasonable remedial measures, including disclosure to the client, and including, if necessary, disclosure to the proper authorities.”
If we were asked to help develop this system, how do we raise a moral objection to the proper authorities when the government is implementing the program? All that one can do is suggest that there will be a divide that will favor some citizens while causing harm to others.
Unfortunately, the government already knows this and has expressed that it is a desirable outcome. Perhaps culturally this is acceptable in China; maybe even desirable. It is difficult to imagine those with scores that are low being particularly excited about it, but perhaps indifference is sufficient to maintain the status quo.
Because of stated intent of the programs is to separate two groups, driving down the “bad” and raising up the “good”, I feel that these social credit systems as designed are immoral. One may argue that we do not know enough about the future implementation of the national credit system to say for certain that it will severely harm citizens.
The Sesame Credit system currently only offers rewards and does not technically penalize anyone for having a low score. If the national system of the future similarly only rewards higher scores and does not explicitly penalize lower scores, then net happiness and benefit will likely be positive.
While we cannot predict the future with certainty, we also cannot forget that China is not a democracy. Communist countries have a history of civil unrest and there have been many cases of governments overthrown. Those in power in China will want to remain in power as evidenced by President Xi Jinxing’s recent repeal of term limits for the presidency.
By implementing a social credit system, the government is turning model citizenship into a game that has rewards. It disincentivizes free thought and encourages agreement with the government. That is authoritarian and it is dangerous.
We must be vigilant in the future as this system is rolled out and its implications are further revealed. The power of this system makes it ripe for abuse; a simple addition of a rule such as “anyone with a score below 400 must serve a prison sentence” or “may raise their score by performing crippling physical labor” would be violations of human rights. But they could be easily disguised through the game rather than an explicit governmental decree.
It opens many potential scary doors of rule through obfuscation. Laws disguised as rules of the game; adjusting scores to oppress certain individuals without condemning them by name. It is a frightening game and one that has potential to greatly simplify oppression at a massive scale.
Sources
“A Quick Guide to China’s Sesame Credit.” Angrymoo, 22 Nov. 2017, angrymoo.com/sesame-credit-summary/
Bynum, Terrell Ward; Rogerson, Simon (2008-06-09). Computer Ethics and Professional Responsibility (p. 85). Wiley. Kindle Edition.
“Code of Conduct.” About the Data Science Association | Data Science Association, www.datascienceassn.org/code-of-conduct.html.
Hvinstendahl, Mara (2017, December 14) Inside China’s Vast New Experiment in Social Ranking. Retrieved 2018, July 28, from https://www.wired.com/story/age-of-social-credit/