Automatically classifying case texts and predicting outcomes

In 1995, U.S. case law comprised about 50 gigabytes of text and was growing by 2 gigabytes per year (Turtle 1995, pp. 6, 47).

“The appellate courts of the United States hand down about 500 decisions a day, so there is considerable pressure upon legal publishers to process case opinions in a timely fashion.” (Jackson et al. 2003).

A nearest-neighbor classifier has in memory “all the documents in the training set and their associated features. Later, when classifying a new document, D, the classifier first selects the k documents in the training set that are closest to D [using some distance metric like cosine similarity], then picks one or more categories to assign to D, based on the categories assigned to the selected k documents.” (Jackson and Moulinier 2007, p. 144).

In an IR program with a relevance-feedback module, given feedback from a user in the form of examples of retrieved documents that are relevant to the user’s query, the IR system retrieves additional documents similar to the examples.

409 S.W.2d 1 (Mo. 1966). This case was in the original CATO database. As in CATO, we focus only on the decision of the trade secret misappropriation claim regarding one defendant, Trieman. Other claims are ignored. In CATO, representing inconsistent results of trade secret claims against other defendants would require entering additional versions of the case.

IBP’s list of KO-Factors includes: F8 Competitive-Advantage (P) (i.e., defendant saved development time and money by using plaintiff's information), F17 Info-Independently Generated (D), F19 No-Security-Measures (D), F20 Info-Known-to-Competitors (D), F26 Deception (P), F27 Disclosure-In-Public-Forum (D).

The other Weak Factors include F1 Disclosure-in-Negotiations (D), and F16 Info-Reverse-Engineerable (D).

For other examples of the kind of explanations IBP generates to justify its predictions, see (Ashley and Brüninghaus 2006, p. 331).

Integrated Cash Management Services, Inc. v. Digital Transactions Inc., 732 F.Supp. 370 (S.D.N.Y. 1989).

Henry Hope X-Ray Products v. Marron Carrel, Inc., 674 F.2d 1336, 1339 (9th Cir. 1982).

E. I. duPont de Nemours v. American Potash & Chemical Corp., 200 A.2d 428, 535 (Del. Ch. 1964).

USM Corp. v. Marson Fastener Corp., 379 Mass. 90, 393 N.E.2d 895, 899 (1979).

For a description of CATO and additional examples of its squibs, see (Ashley 2000, 284–288; Ashley and Brüninghaus 2006).

In Table 1, the representation of the National Rejectors case as a training instance for the SMILE program differs slightly from the original CATO version, shown in the human column, input to the IBP program by the addition of F6 Security-Measures (D). There is an inconsistency between F6 Security-Measures (P) and F19 No-Security-Measures; courts sometimes focus in the same opinion on the security measures plaintiff took, justifying application of F6, and the sometimes many security measures that could have been taken but were not, justifying F19. In IBP, the inconsistency was resolved in favor of whichever the court seemed to emphasize more; for SMILE, the inconsistency was allowed to stand.

In generating the Factor classifiers, two other machine learning algorithms were tried, C4.5 (Quinlan 2004), and Naïve Bayes as implemented in Rainbow (McCallum 2004). The nearest-neighbor algorithm worked best.

The most commonly used text representation techniques in information retrieval include feature lists of words (i.e., bag-of-words), syntactic structure, frames/scripts, logic propositions, and network structures (Turtle 1995 p. 18).

445 N.E.2d 418 (Ill. App. 1 Dist. 1983).

The roles-replaced and propositional patterns text representations were introduced in (Brüninghaus and Ashley 2001, 47–48).

We developed automated information extraction techniques that use heuristics to perform the role substitution task for parties’ names and products (Brüninghaus and Ashley 2001). For the experiments reported here, however, we carried out all of the role replacements manually in order to make sure that the texts were as accurate as possible. Since the goal of our experiments here was to test whether role replacement improved Factor assignment, we believed that it was important to eliminate automated role replacements as a source of errors. If SMILE were to be used in larger-scaled applications, automatic role replacements would be necessary. Other commercial and research approaches exist for the task of performing the role replacements, which is a subset of the long-studied problems of named entity recognition and coreference (Jackson and Moulinier 2007, pp. 170–183).

In the parsing process, some identification of common phrases needs to take place. For instance, common constituents of trade secret cases described with multiword phrases such as “nondisclosure agreement” need to be recognized as individual entities (i.e., as a single head word like “nondisclosure_agreement”) for purposes of parsing and constructing ProPs. So far, we have handled that on an ad hoc basis, in effect manually substituting “nondisclosure_agreement” when a parse identifies “nondisclosure” as an adjective modifying agreement. Automating the common phrase recognition task should be possible, but we have not done it. See, (Kim and Wilbur 2000).

“ 29T Antitrust and Trade Regulation, 29TIV Trade Secrets and Proprietary Information, 29TIV(B) Actions, 29Tk429 Evidence, 29Tk432 k. Weight and Sufficiency of Evidence. … Evidence established that there were no actual trade secrets with respect to slug rejectors and electrical coin changers, in action based upon alleged misappropriation of trade secrets.”.

“ 212 Injunction …, 212II Subjects of Protection and Relief, 212II(B) Matters Relating to Property, 212k56 k. Disclosure or Use of Trade Secrets. … Although individual defendants, former employees of plaintiff, had improperly used plaintiff's materials and drawings in production of products to compete with plaintiff's products, where plaintiff had not considered information regarding its products to be trade secrets, no warning had been given against use of information, and key patent relating to plaintiff's devices had expired, plaintiff was not entitled to injunctive relief but was entitled to damages based upon profits which it lost by reason of competition of defendant corporation formed by individual defendants, during time that defendant corporation would not otherwise have been in production but for assistance obtained from use of plaintiff's drawings and materials.”

The vectors are in a multi-dimensional space with as many dimensions as there are different words in all of the sentences (Jackson and Moulinier 2007, pp. 30–33).

Jackson and Moulinier (2007, pp. 92–93) identifies a number of complexities in case opinions that bedevil information extraction: 1. The facts of the case may be intermingled with its procedural history. 2. Rulings in precedents are reported in much the same way as the ruling in the current case. 3. The opinions contain extensive quotations from other sources. 4. The opinions may contain extensive discussions of hypothetical, counter-factual, or qualified propositions. 5. The opinions often deal with many diverse points of law.

The F-Measure is the harmonic mean of the two rates, precision and recall. We used a version of the F-Measure that assigns equal weight to precision and recall. (Jackson and Moulinier 2007, p. 48).

We tested the results of our experiments for statistical significance in order to show whether the observed differences were caused by true differences between the representations and algorithms, and not merely by chance. Because our experiments were run as cross-validations, the commonly used T-test may not lead to reliable results (Dietterich 1996; Salzberg 1997). Since the cross-validation experiments compared more than two different parameter choices (specifically, three treatments are reported here, 1 machine learning algorithm X 3 representations) we followed the procedure recommended in (Dietterich 1996). We applied a non-parametric test, Friedman’s Test, to find whether there was a difference among the combinations of representations and algorithms. When this test showed significance, we used Wilcoxon’s Signed-Rank test to determine whether the difference between two variants was significant (e.g., between results obtained with ProPs and Roles-Replaced using Nearest-Neighbor.) Following convention, we say that results with probability p < 0.05 are statistically significant, and with p < 0.1 marginally significant. See (Cohen 1995a, b).

For a variety of reasons, it is difficult to compare F-measures across the legal text categorization work described supra in Section 2. Not only do the categorization tasks, classification concepts, and types of legal documents differ, but so do the relative importance of recall and precision in the particular application. For a different task, automatically categorizing opinion texts by WESTLAW topic categories, (Thompson 2001) reports F-measures with β = 2, indicating that twice as much weight is given to recall as precision. (The formula is F = ((β² + 1) × P × R)/(β² × P + R). In evaluating SMILE, we use β = 1, treating recall and precision as having equal weight.) For different categorization methods, he reports average F-measures (β = 2) across eighteen topics ranging from .495 to .637, and for individual topics using the two best categorization methods, the measures range from .253 to .860. In other work extracting the criminal offenses (based on a standardized list) and the legal principles applied from the text of legal opinions in criminal cases, the Salomon program achieved F-measures (β = 1) of .82 and .46, respectively (Uyttendaele et al. 1998). (Daniels and Rissland 1997) do not report precision and recall or F-measures for the SPIRE program, favoring a different metric, expected search length.) It should be remembered, however, that the concepts for categorizing cases in the above work were not specific enough to support a program’s reasoning about the cases as SMILE + IBP does.

National Rejectors, Inc. v. Trieman, 409 S.W.2d 1 (Sup. Ct. Mo., 1966).

Dynamics Research Corp. v. Analytic Sciences Corp. 400 N.E.2d 1274 (Mass.App.Ct., 1980).

“For current extraction technology to work, the information sought must be explicitly stated in the text. It cannot be merely implied by the text.” (Jackson and Moulinier 2007, p. 106).

These issues may interact with issues IBP’s model does address. For instance, if a defendant copied information fixed in a tangible medium of expression and covered by the subject matter of copyright, a trade secret claim may be preempted under s. 301 of the Copyright Act. A trade secret claim that involved an extra element of breach of confidence would not be preempted, but it could be if it involved only improper means. Although IBP’s model does not address preemption, some of the same Factors (e.g., regarding confidential relationship) would be relevant to the preemption analysis. One would need to modify the model to add preemption.

Currently, a legal IR system ranks the cases it retrieves according to statistical criteria that involve the frequencies of the query’s terms’ appearances in the retrieved cases and in the corpus.

Alternatively, a SMILE-type system might automatically classify cases drawn from legal IR systems and index them in a specialized database.

This AI & Law research on automatically processing case texts assumes that the opinion texts are reasonably complete and candid descriptions of courts’ decisions. SMILE + IBP’s focus on Factors for and against a court’s decision, for example, assumes that courts adequately disclose the facts in a case that favor a contradictory decision, a point about which legal scholars differ. See (Delgado and Stefancic 2007, fn. 100).