11. Conclusion

This book began with a description of an ethical dilemma: whether an issuer of subprime credit cards should cut in half the credit limits of customers who use their card to pay for marital counseling. The chapters that followed have not suggested what the right answer is, or even whether there is a single “right” answer. Instead, they have described why an organization should take such an issue seriously and how it might go about reaching a considered, responsible decision about it. What have we learned about how an organization should handle the ethical dilemmas that its own use of advanced analytics and AI can create?

Chapter 3 (Risks) helped us to think about ways in which business use of these technologies can harm others. In the subprime credit card scenario, for example, we can see risks to privacy as the card company uses customer purchase information to predict the individual’s credit-worthiness—information that the card holders did not know they were revealing. The risk of error also rears its head. Without sufficient attention to data quality and data science (and perhaps even with such attention), the card company will incorrectly classify some card holders. How will this impact those mistakenly deprived of the credit they need? Opacity and procedural unfairness also affect card holders who may not know why the company has reduced cut their credit limit in half and so do not feel empowered to challenge this decision. We also see how advanced analytics changes the balance of power between card issuer and card holder, giving the issuer ever more potent insights that they can use to their advantage. The card company’s actions may cause harmful bias if turns out that the proxies (pawn shops, massage parlors, marital counseling) correlate to a protected characteristic. Chapter 3 helps us to see that the card company’s actions generate each of these risks.

Chapter 4 (What is Data Ethics Management) suggests that, to address these risks, the credit card company may need to go beyond compliance with the law. No law directly prohibits cutting the credit limit of those who engage in marital counseling. Were the subprime card issuer to forego the use of this insight, it would be doing more than the law required. Chapter 5 (Motivations) goes a step further and suggests that it may be in the company’s long-term interests to go beyond compliance in this way. If it becomes public (as, in fact, it did) that the company was penalizing those who went to a marriage counselor, this could harm the company’s reputation and make potential card holders leery about dealing with it. The growing regulation of algorithmic decision-making, and the company’s need to get ready for it, may also make it wise for the card issuer to consider whether it should find other ways to address the risk of card holder default.

Chapter 6 (Drawing Substantive Lines) concluded that sets of AI ethics principles, while important, can be too broad and internally inconsistent to produce a determinate decision. The subprime credit card case bears this out. Good faith arguments can be made for the beneficence of cutting off the credit of those who go to a marital counselor (it saves them from the pain of default) and for the malevolence of doing so (it will deter people from engaging in marital counseling and so hurt marriages and children). It also shows how principles such as beneficence and justice can conflict with one another since, while it may help borrowers to cut them off when they go to marriage counseling, it hardly seems just to do so. Chapter 6 also discussed the gut-level judgment calls that some organizations use to make decisions about data and AI ethics. In the subprime credit card example, a manager using such an approach might conclude that their grandmother would not approve of cutting the credit of those who go to a marital counselor (public expectations) or that, were the shoe on the other foot, the manager would not want the same policy to be applied to them (the Golden Rule). Such judgment calls may, in fact, keep the company in line with social norms. But they hardly constitute a thoughtful or consistent way of resolving such issues. The subprime credit card issuer should strive to develop more general and prospective policies to guide its actions.

Chapter 7 (Management Structures and Functions) drives home the importance of making a specific person or committee responsible for identifying and managing data ethics issues. The subprime credit card issuer would benefit from this advice. If it fails to spot or handles carelessly the marital counseling issue, that could have an important impact on its goodwill, reputation, and future. It needs to allocate responsibility for managing these critical business issues. It may even want to create a cross-functional data and AI ethics committee to consider these questions from multiple perspectives.

Chapter 8 (Management Processes) describes processes that companies use to spot and resolve their data and AI ethics issues. The use of checklists, consultations with external stakeholders, and other methods for identifying data ethics issues may have sensitized the subprime credit card issuer to concerns about cutting the credit of those who go to a marital counselor. The company will also want to think carefully about its processes for deciding such issues, and who gets the final say.

Chapter 9 (Technical Solutions) identifies key technologies and technical practices that can make an organization’s advanced analytics and AI practices fairer, more privacy protective, and more explainable. The subprime credit card issuer would do well to consider, and perhaps adopt, these techniques. It would also benefit from an audit to determine whether the proxies it uses for determining who gets their credit cut have a disparate impact on one or more protected groups. The technical dimension of data ethics management, while not the focus on this book, is clearly essential.

As we have just illustrated, this book can be of practical use to organizations that confront a data or AI ethics issue. But that is not all that it does. The book also seeks to serve as a resource for legislators and regulators who, in designing new laws and policies, should understand how companies are currently managing these issues. Current practice is not the same as best practice. But it is the starting point for legislation and regulation. This book gives lawmakers a sense of the ground on which they are building.

The book also seeks to spark more research on data ethics management. Scholars should not only update our study with more current information; they should also conduct evaluative research to identify which approaches work best, and which do not work very well at all. Defining such best practices and, ultimately, integrating them into standards, codes of practice, and laws, is key to protecting individuals and society from threats that the algorithmic economy generates. If organizations—both those in the private sector as is the focus on this book, and governmental bodies—fail to use responsibly the power that advanced analytics and AI give them, they may lose their social license to operate. Should that happen, we would all miss out on the promise that these technologies hold for better health, education, and many other such social goods. We all have a stake in building strong and effective AI and data ethics management.