Skip to main content
Erschienen in: Artificial Intelligence and Law 2/2021

24.06.2020 | Original Research

Scalable and explainable legal prediction

verfasst von: L. Karl Branting, Craig Pfeifer, Bradford Brown, Lisa Ferro, John Aberdeen, Brandy Weiss, Mark Pfaff, Bill Liao

Erschienen in: Artificial Intelligence and Law | Ausgabe 2/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Legal decision-support systems have the potential to improve access to justice, administrative efficiency, and judicial consistency, but broad adoption of such systems is contingent on development of technologies with low knowledge-engineering, validation, and maintenance costs. This paper describes two approaches to an important form of legal decision support—explainable outcome prediction—that obviate both annotation of an entire decision corpus and manual processing of new cases. The first approach, which uses an attention network for prediction and attention weights to highlight salient case text, was shown to be capable of predicting decisions, but attention-weight-based text highlighting did not demonstrably improve human decision speed or accuracy in an evaluation with 61 human subjects. The second approach, termed semi-supervised case annotation for legal explanations, exploits structural and semantic regularities in case corpora to identify textual patterns that have both predictable relationships to case decisions and explanatory value.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Anhänge
Nur mit Berechtigung zugänglich
Fußnoten
1
We note that models for legal prediction, as with other inductive models in dynamic domains, can be subject to concept drift (Medvedeva et al. 2020).
 
2
See The EXplainable AI in Law (XAILA) (2018) for a recent exception to this generalization.
 
4
Cases in which decisions consist of numerical awards can be modeled as regression problems. For simplicity, we confine the discussion in this paper to categorical classification.
 
5
Mean.
 
6
SE.
 
7
At the time of writing, we have not yet completed annotation of each individual issue for ever instance in our data set. The experiments described below therefore involve prediction only of the overall outcome of the case without individual issue decisions.
 
8
Decision sections were annotated as well. However, since Decision sections consisted only of brief conclusory text, this portion was useful only for obtaining the decision label—transferred or not transferred—but was not useful for explanation purposes and were therefore not used in tag projection.
 
10
In tenfold cross validation, we observed a mean f-measure of 0.971 and MCC of 0.815 for transfer prediction using an SVM (Platt 1999) applied to the 1133 highest information n-grams \((n=1{-}5)\) occurring in the the stop-word filtered text of the Findings sections of the full corpus.
 
11
We used a 300-dimension word embedding based on 55,975,964 words and a skipgram model, which we found outperformed cbow for our task.
 
12
For this, and each of the tests below we used XGBoost (Chen and Guestrin 2016), an efficient implementation of the gradient boosting algorithm, for prediction. However, we obtained very similar results using the Hall et al. (2009) implementation of Bayesian Network classification (Bouckaert 2005).
 
15
See 15(e) of the Rules for Uniform Domain Name Dispute Resolution Policy for CIBF, https://​www.​icann.​org/​resources/​pages/​udrp-rules-2015-03-11-en.
 
Literatur
Zurück zum Zitat Al-Abdulkarim L, Atkinson K, Bench-Capon TJM, Whittle S, Williams R, Wolfenden C (2017) Noise induced hearing loss: an application of the angelic methodology. In: Legal knowledge and information systems—JURIX 2017: the thirtieth annual conference, Luxembourg, 13–15 December 2017, pp 79–88 Al-Abdulkarim L, Atkinson K, Bench-Capon TJM, Whittle S, Williams R, Wolfenden C (2017) Noise induced hearing loss: an application of the angelic methodology. In: Legal knowledge and information systems—JURIX 2017: the thirtieth annual conference, Luxembourg, 13–15 December 2017, pp 79–88
Zurück zum Zitat Aleven VAWMM (1997) Teaching case-based argumentation through a model and examples. PhD thesis, University of Pittsburgh, Pittsburgh. AAI9821228 Aleven VAWMM (1997) Teaching case-based argumentation through a model and examples. PhD thesis, University of Pittsburgh, Pittsburgh. AAI9821228
Zurück zum Zitat Aleven V, Ashley K (1996) Doing things with factors. In: Proceedings of the 3rd European workshop on case-based reasoning (EWCR-96), Lausanne, pp 76–90 Aleven V, Ashley K (1996) Doing things with factors. In: Proceedings of the 3rd European workshop on case-based reasoning (EWCR-96), Lausanne, pp 76–90
Zurück zum Zitat Ashley KD (2017) Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press, CambridgeCrossRef Ashley KD (2017) Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press, CambridgeCrossRef
Zurück zum Zitat Ashley KD, Aleven V (1997) Reasoning symbolically about partially matched cases. In: Proceedings of the 15th international joint conference on artificial intelligence. Morgan Kauffmann, San Francisco, pp 335–341 Ashley KD, Aleven V (1997) Reasoning symbolically about partially matched cases. In: Proceedings of the 15th international joint conference on artificial intelligence. Morgan Kauffmann, San Francisco, pp 335–341
Zurück zum Zitat Ashley KD, Brüninghaus S (2009) Automatically classifying case texts and predicting outcomes. Artif Intell Law 17(2):125–165CrossRef Ashley KD, Brüninghaus S (2009) Automatically classifying case texts and predicting outcomes. Artif Intell Law 17(2):125–165CrossRef
Zurück zum Zitat Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. Preprint arXiv:1409.0473 Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. Preprint arXiv:​1409.​0473
Zurück zum Zitat Bench-Capon TJM, Dunne PE (2007) Argumentation in artificial intelligence. Artif Intell 171(10–15):619–641MathSciNetCrossRef Bench-Capon TJM, Dunne PE (2007) Argumentation in artificial intelligence. Artif Intell 171(10–15):619–641MathSciNetCrossRef
Zurück zum Zitat Berger AL, Pietra VJD, Pietra SAD (1996) A maximum entropy approach to natural language processing. Comput Linguist 22(1):39–71 Berger AL, Pietra VJD, Pietra SAD (1996) A maximum entropy approach to natural language processing. Comput Linguist 22(1):39–71
Zurück zum Zitat Boles DB, Adair LP (2001) The multiple resources questionnaire (MRQ). Proc Hum Fact Ergon Soci Ann Meet 45(25):1790–1794CrossRef Boles DB, Adair LP (2001) The multiple resources questionnaire (MRQ). Proc Hum Fact Ergon Soci Ann Meet 45(25):1790–1794CrossRef
Zurück zum Zitat Boughorbel S, Jarray F, El-Anbari M (2017) Optimal classifier for imbalanced data using Matthews correlation coefficient metric. PLoS ONE 12(6):E0177678CrossRef Boughorbel S, Jarray F, El-Anbari M (2017) Optimal classifier for imbalanced data using Matthews correlation coefficient metric. PLoS ONE 12(6):E0177678CrossRef
Zurück zum Zitat Branting LK (2000a) An advisory system for pro se protection order applicants. Int Rev Law Comput Technol 14(3):357–369CrossRef Branting LK (2000a) An advisory system for pro se protection order applicants. Int Rev Law Comput Technol 14(3):357–369CrossRef
Zurück zum Zitat Branting LK (2000b) Reasoning with rules and precedents: a computational model of legal analysis. Kluwer, DordrectCrossRef Branting LK (2000b) Reasoning with rules and precedents: a computational model of legal analysis. Kluwer, DordrectCrossRef
Zurück zum Zitat Branting LK, Yeh A, Weiss B, Merkhofer EM, Brown B (2017) Inducing predictive models for decision support in administrative adjudication. In: AI approaches to the complexity of legal systems—AICOL international workshops 2015–2017, revised selected papers. Lecture notes in computer science, vol. 10791. Springer, Berlin, pp 465–477 Branting LK, Yeh A, Weiss B, Merkhofer EM, Brown B (2017) Inducing predictive models for decision support in administrative adjudication. In: AI approaches to the complexity of legal systems—AICOL international workshops 2015–2017, revised selected papers. Lecture notes in computer science, vol. 10791. Springer, Berlin, pp 465–477
Zurück zum Zitat Brooke J (1996) SUS—a quick and dirty usability scale. Usab Eval Ind 189(194):4–7 Brooke J (1996) SUS—a quick and dirty usability scale. Usab Eval Ind 189(194):4–7
Zurück zum Zitat Brüninghaus S, Ashley KD (1999) Toward adding knowledge to learning algorithms for indexing legal cases. In: Proceedings of the 7th international conference on artificial intelligence and law, ICAIL’99. ACM, New York, pp 9–17. https://doi.org/10.1145/323706.323709 Brüninghaus S, Ashley KD (1999) Toward adding knowledge to learning algorithms for indexing legal cases. In: Proceedings of the 7th international conference on artificial intelligence and law, ICAIL’99. ACM, New York, pp 9–17. https://​doi.​org/​10.​1145/​323706.​323709
Zurück zum Zitat Bruninghaus S, Ashley KD (2003) Predicting outcomes of case based legal arguments. In: Proceedings of the 9th international conference on artificial intelligence and law, ICAIL’03. ACM, New York, pp 233–242 Bruninghaus S, Ashley KD (2003) Predicting outcomes of case based legal arguments. In: Proceedings of the 9th international conference on artificial intelligence and law, ICAIL’03. ACM, New York, pp 233–242
Zurück zum Zitat Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD’16. ACM, New York, pp 785–794 Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD’16. ACM, New York, pp 785–794
Zurück zum Zitat Dunn PH (2003) How judges overrule: speech act theory and the doctrine of stare decisis. Yale Law J 113(2):493–532CrossRef Dunn PH (2003) How judges overrule: speech act theory and the doctrine of stare decisis. Yale Law J 113(2):493–532CrossRef
Zurück zum Zitat Ferro L, Aberdeen J, Branting K, Pfeifer C, Yeh A, Chakraborty A (2019) Scalable methods for annotating legal-decision corpora. In: Proceedings of the natural legal language processing workshop 2019. Association for Computational Linguistics, Minneapolis, pp 12–20 Ferro L, Aberdeen J, Branting K, Pfeifer C, Yeh A, Chakraborty A (2019) Scalable methods for annotating legal-decision corpora. In: Proceedings of the natural legal language processing workshop 2019. Association for Computational Linguistics, Minneapolis, pp 12–20
Zurück zum Zitat Hadfield G (2016) Rules for a flat world: why humans invented law and how to reinvent it for a complex global economy. Oxford University Press, Oxford Hadfield G (2016) Rules for a flat world: why humans invented law and how to reinvent it for a complex global economy. Oxford University Press, Oxford
Zurück zum Zitat Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11(1):10–18CrossRef Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11(1):10–18CrossRef
Zurück zum Zitat Henschen B (2018) Judging in a mismatch: the ethical challenges of pro se litigation. Public Integr 20(1):34–46CrossRef Henschen B (2018) Judging in a mismatch: the ethical challenges of pro se litigation. Public Integr 20(1):34–46CrossRef
Zurück zum Zitat Herrera F, Charte F, Rivera A, del Jesus M (2016) Multilabel classification: problem analysis, metrics and techniques. Springer, BerlinCrossRef Herrera F, Charte F, Rivera A, del Jesus M (2016) Multilabel classification: problem analysis, metrics and techniques. Springer, BerlinCrossRef
Zurück zum Zitat Hill F, Cho K, Korhonen A (2016) Learning distributed representations of sentences from unlabelled data. In: 2016 conference of the North American chapter of the association for computational linguistics, pp 1367–1377. Association for Computational Linguistics (ACL). 15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016; Conference date: 12-06-2016 Through 17-06-2016 Hill F, Cho K, Korhonen A (2016) Learning distributed representations of sentences from unlabelled data. In: 2016 conference of the North American chapter of the association for computational linguistics, pp 1367–1377. Association for Computational Linguistics (ACL). 15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016; Conference date: 12-06-2016 Through 17-06-2016
Zurück zum Zitat Katz DM, Bommarito MJ II, Blackman J (2017) A general approach for predicting the behavior of the supreme court of the united states. PLoS ONE 12(4):e0174698CrossRef Katz DM, Bommarito MJ II, Blackman J (2017) A general approach for predicting the behavior of the supreme court of the united states. PLoS ONE 12(4):e0174698CrossRef
Zurück zum Zitat Lauritsen M, Steenhuis Q (2019) Substantive legal software quality: a gathering storm? In: Proceedings of the 17th international conference on artificial intelligence and law, ICAIL’19. ACM, New York, pp 52–62 Lauritsen M, Steenhuis Q (2019) Substantive legal software quality: a gathering storm? In: Proceedings of the 17th international conference on artificial intelligence and law, ICAIL’19. ACM, New York, pp 52–62
Zurück zum Zitat Lawrence J, Reed C (2020) Argument mining: a survey. Comput Linguist 45(4):765–818CrossRef Lawrence J, Reed C (2020) Argument mining: a survey. Comput Linguist 45(4):765–818CrossRef
Zurück zum Zitat Lippi M, Torroni P (2016) Argumentation mining: state of the art and emerging trends. ACM Trans Int Technol 16(2):10:1–10:25CrossRef Lippi M, Torroni P (2016) Argumentation mining: state of the art and emerging trends. ACM Trans Int Technol 16(2):10:1–10:25CrossRef
Zurück zum Zitat Maxwell KT, Oberlander J, Lavrenko V (2009) Evaluation of semantic events for legal case retrieval. In: Proceedings of the WSDM’09 workshop on exploiting semantic annotations in information retrieval, ESAIR’09. ACM, New York, pp 39–41. https://doi.org/10.1145/1506250.1506259 Maxwell KT, Oberlander J, Lavrenko V (2009) Evaluation of semantic events for legal case retrieval. In: Proceedings of the WSDM’09 workshop on exploiting semantic annotations in information retrieval, ESAIR’09. ACM, New York, pp 39–41. https://​doi.​org/​10.​1145/​1506250.​1506259
Zurück zum Zitat McCarty LT (2018) Research handbook on the law of artificial intelligence, Chap. Finding the right balance in artificial intelligence and law. Edward Elgar Publishing McCarty LT (2018) Research handbook on the law of artificial intelligence, Chap. Finding the right balance in artificial intelligence and law. Edward Elgar Publishing
Zurück zum Zitat Medvedeva M, Vols M, Wieling M (2020) Using machine learning to predict decisions of the European court of human rights. Artif Intell Law 28(2):237–266CrossRef Medvedeva M, Vols M, Wieling M (2020) Using machine learning to predict decisions of the European court of human rights. Artif Intell Law 28(2):237–266CrossRef
Zurück zum Zitat Mikolov T, Yih SWT, Zweig G (2013) Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies (NAACL-HLT-2013). Association for Computational Linguistics, New York Mikolov T, Yih SWT, Zweig G (2013) Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies (NAACL-HLT-2013). Association for Computational Linguistics, New York
Zurück zum Zitat Peterson M, Waterman D (1985) Rule-based models of legal expertise. In: Walters C (ed) Computing power and legal reasoning. West Publishing Company, Minneapolis, pp 627–659 Peterson M, Waterman D (1985) Rule-based models of legal expertise. In: Walters C (ed) Computing power and legal reasoning. West Publishing Company, Minneapolis, pp 627–659
Zurück zum Zitat Platt JC (1999) Fast training of support vector machines using sequential minimal optimization. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in kernel methods. MIT Press, Cambridge, pp 185–208 Platt JC (1999) Fast training of support vector machines using sequential minimal optimization. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in kernel methods. MIT Press, Cambridge, pp 185–208
Zurück zum Zitat Ren Y, Fei H, Peng Q (2018) Detecting the scope of negation and speculation in biomedical texts by using recursive neural network. In: 2018 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 739–742 Ren Y, Fei H, Peng Q (2018) Detecting the scope of negation and speculation in biomedical texts by using recursive neural network. In: 2018 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 739–742
Zurück zum Zitat Rissland EL, Skalak DB (1989) Combining case-based and rule-based reasoning: a heuristic approach. In: 11th international joint conference on artificial intelligence, Detroit, pp 524–530 Rissland EL, Skalak DB (1989) Combining case-based and rule-based reasoning: a heuristic approach. In: 11th international joint conference on artificial intelligence, Detroit, pp 524–530
Zurück zum Zitat Sauro J, Dumas JS (2009) Comparison of three one-question, post-task usability questionnaires. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’09, pp 1599–1608 Sauro J, Dumas JS (2009) Comparison of three one-question, post-task usability questionnaires. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’09, pp 1599–1608
Zurück zum Zitat Shulayeva O, Siddharthan A, Wyner A (2017) Recognizing cited facts and principles in legal judgements. Artif Intell Law 25(1):107–126CrossRef Shulayeva O, Siddharthan A, Wyner A (2017) Recognizing cited facts and principles in legal judgements. Artif Intell Law 25(1):107–126CrossRef
Zurück zum Zitat Sulea O, Zampieri M, Vela M, van Genabith J (2017) Predicting the law area and decisions of French supreme court cases. In: RANLP. INCOMA Ltd, pp 716–722 Sulea O, Zampieri M, Vela M, van Genabith J (2017) Predicting the law area and decisions of French supreme court cases. In: RANLP. INCOMA Ltd, pp 716–722
Zurück zum Zitat Surdeanu M, Nallapati R, Gregory G, Walker J, Manning C (2011) Risk analysis for intellectual property litigation. In: Proceedings of the 13th international conference on artificial intelligence and law. ACM, Pittsburgh Surdeanu M, Nallapati R, Gregory G, Walker J, Manning C (2011) Risk analysis for intellectual property litigation. In: Proceedings of the 13th international conference on artificial intelligence and law. ACM, Pittsburgh
Zurück zum Zitat Westermann H, Walker VR, Ashley KD, Benyekhlef K (2019) Using factors to predict and analyze landlord-tenant decisions to increase access to justice. In: Proceedings of the 17th international conference on artificial intelligence and law, ICAIL’19. Association for Computing Machinery, New York, pp 133–142 Westermann H, Walker VR, Ashley KD, Benyekhlef K (2019) Using factors to predict and analyze landlord-tenant decisions to increase access to justice. In: Proceedings of the 17th international conference on artificial intelligence and law, ICAIL’19. Association for Computing Machinery, New York, pp 133–142
Zurück zum Zitat Wyner AZ, Peters W (2010) Lexical semantics and expert legal knowledge towards the identification of legal case factors. In: JURIX, frontiers in artificial intelligence and applications, vol 223. IOS Press, New York, pp 127–136 Wyner AZ, Peters W (2010) Lexical semantics and expert legal knowledge towards the identification of legal case factors. In: JURIX, frontiers in artificial intelligence and applications, vol 223. IOS Press, New York, pp 127–136
Zurück zum Zitat Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of NAACL-HLT, pp 1480–1489 Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of NAACL-HLT, pp 1480–1489
Zurück zum Zitat Yu H, Hatzivassiloglou V (2003) Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences. In: Proceedings of the 2003 conference on empirical methods in natural language processing, EMNLP’03. Association for Computational Linguistics, Stroudsburg, pp 129–136 Yu H, Hatzivassiloglou V (2003) Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences. In: Proceedings of the 2003 conference on empirical methods in natural language processing, EMNLP’03. Association for Computational Linguistics, Stroudsburg, pp 129–136
Metadaten
Titel
Scalable and explainable legal prediction
verfasst von
L. Karl Branting
Craig Pfeifer
Bradford Brown
Lisa Ferro
John Aberdeen
Brandy Weiss
Mark Pfaff
Bill Liao
Publikationsdatum
24.06.2020
Verlag
Springer Netherlands
Erschienen in
Artificial Intelligence and Law / Ausgabe 2/2021
Print ISSN: 0924-8463
Elektronische ISSN: 1572-8382
DOI
https://doi.org/10.1007/s10506-020-09273-1

Weitere Artikel der Ausgabe 2/2021

Artificial Intelligence and Law 2/2021 Zur Ausgabe

Premium Partner