1 Introduction
-
To the best of our knowledge, this is the first study to data-mine large-scale court judgment documents (approx. 110,000) from the district to the supreme court level in Japan.
-
We propose a novel legal masked language prediction task to connect textual information (reason for citation) to legal codes and past court precedents.
-
We give both an extensive quantitative and qualitative analysis of major machine learning models and show that deep learning technology models lead to highly predictable outcomes. We also illustrate limitations and possible directions for future research.
-
Using the co-occurrence patterns in a court judgment, we propose a novel dynamic link prediction task that identifies the possible set of interactions within the law. This task is essential both as a technical exercise and a practical legal task.
-
We show that using a simple network model already leads to good performance. Still, a model that uses textual and network information leads to the best predictive performance.
-
We also provide a qualitative assessment of the learned embedding from the best performing model.
2 Japanese legal system and data set
2.1 Japanese legal system
2.2 Summary of the data set
Original text | Extracted citation | Normalized citation |
---|---|---|
これらの規定に違反しないことは当裁判所の判例(最高裁昭和22年(れ)第119号同23年3月12日大法廷判決・刑集2巻3号191頁)とするところであるから. | 最高裁昭和22年(れ)第119 号同 23 年 3 月 12 日大法廷判決・刑集 2 巻 3 号 191 頁 | 最高裁判所昭和 23 年 3 月 12 日判決 |
Past court precedents of the Supreme court has determined that these provisions are not violated (Supreme Court, Grand Bench judgment of March 12, 1948, 1947 (Re) No. 119, Keishu Vol.2, No.3, at 191). | Supreme Court, Grand Bench judgment of March 12, 1948, 1947 (Re) No. 119, Keishu Vol.2, No.3, at 191 | Supreme Court, Judgment of March 12, 1948 |
Category | Sentences | Labels |
---|---|---|
Civil articles | 600,558 | 8797 |
Criminal articles | 34,845 | 630 |
Civil court precedents | 47,369 | 1561 |
Criminal court precedents | 1009 | 76 |
3 Masked language prediction
3.1 Summary of the compared methods
3.1.1 Encoder
3.1.2 Decoder
3.2 Experimental settings
3.3 Results
3.3.1 Quantitative comparison
Model | Category | Accuracy | Accuracy | Accuracy | Accuracy | Accuracy |
---|---|---|---|---|---|---|
(\(m\)=0) | (\(m\)=1) | (\(m\)=2) | (\(m\)=3) | (\(m\)=4) | ||
onehot+XGB | Civil code | 57.84 | 63.56 | 66.09 | 67.67 | 68.75 |
onehot+XGB | Criminal code | 80.06 | 84.1 | 86.18 | 87.6 | 88.41 |
onehot+XGB | Civil case | 68.12 | 74.29 | 77.22 | 78.86 | 80.04 |
onehot+XGB | Criminal case | 68.32 | 72.61 | 75.25 | 76.24 | 77.89 |
USE+XGB | Civil code | 51.98 | 59.16 | 62.89 | 65.38 | 67.21 |
USE+XGB | Criminal code | 71.84 | 76.84 | 79.66 | 81.35 | 82.60 |
USE+XGB | Civil case | 61.54 | 69.05 | 72.56 | 74.59 | 76.08 |
USE+XGB | Criminal case | 65.02 | 70.63 | 74.59 | 76.24 | 77.89 |
T5 | Civil code | 70.32 | – | – | – | – |
T5 | Criminal code | 54.13 | – | – | - | – |
T5 | Civil case | 24.11 | – | – | - | – |
T5 | Criminal case | 8.37 | – | – | – | – |
LDA+XGB | Civil code | 52.24 | 58.46 | 61.43 | 63.27 | 64.56 |
LDA+XGB | Criminal code | 70.77 | 75.22 | 77.89 | 79.71 | 80.95 |
LDA+XGB | Civil case | 60.66 | 66.57 | 69.40 | 71.12 | 80.95 |
LDA+XGB | Criminal case | 36.30 | 39.27 | 40.26 | 41.91 | 43.89 |
doc2vec+XGB | Civil code | 0.09 | 0.11 | 0.18 | 0.33 | 0.83 |
doc2vec+XGB | Criminal code | 16.29 | 19.88 | 22.99 | 25.94 | 28.73 |
doc2vec+XGB | Civil case | 19.03 | 20.32 | 21.14 | 21.91 | 22.64 |
doc2vec+XGB | Criminal case | 27.06 | 33.33 | 35.97 | 37.95 | 39.93 |
Model | Category | \(F_\mu \) | \(P_\mu \) | \(R_\mu \) | \(F_M\) |
---|---|---|---|---|---|
onehot+XGB | Civil code | 7.57 | 7.18 | 8.54 | 3.9 |
onehot+XGB | Criminal code | 46.89 | 54.52 | 45.76 | 24.89 |
onehot+XGB | Civil case | 41.42 | 47.08 | 41.75 | 22.13 |
onehot+XGB | Criminal case | 49.04 | 48.44 | 56.41 | 26.06 |
USE+XGB | Civil code | 13.94 | 17.55 | 13.65 | 7.68 |
USE+XGB | Criminal code | 50.33 | 58.8 | 48.31 | 26.52 |
USE+XGB | Civil case | 41.73 | 50.84 | 39.94 | 22.37 |
USE+XGB | Criminal case | 44.27 | 43.94 | 52.38 | 23.89 |
T5 | Civil code | 22.74 | 25.15 | 23.68 | 12.2 |
T5 | Criminal code | 21.36 | 25.57 | 21.13 | 11.57 |
T5 | Civil case | 7.41 | 10.11 | 7.00 | 4.14 |
T5 | Criminal case | 1.38 | 1.8 | 1.55 | 0.83 |
doc2vec+XGB | Civil code | 0.00 | 0.00 | 0.03 | 0.00 |
doc2vec+XGB | Criminal code | 0.33 | 0.82 | 0.47 | 0.30 |
doc2vec+XGB | Civil case | 0.04 | 0.07 | 0.09 | 0.04 |
doc2vec+XGB | Criminal case | 3.23 | 2.73 | 4.17 | 1.65 |
LDA+XGB | Civil code | 4.87 | 4.43 | 5.69 | 2.49 |
LDA+XGB | Criminal code | 28.20 | 35.22 | 27.09 | 15.31 |
LDA+XGB | Civil case | 26.78 | 29.68 | 28.72 | 14.60 |
LDA+XGB | Criminal case | 5.01 | 4.43 | 6.33 | 2.61 |
3.4 Qualitative assessment
3.5 Discussion on T5
4 Legal link prediction
Rank | Law name | Page rank |
---|---|---|
1 | Penal Code 10 | 0.015 |
2 | Code of Criminal Procedure 396 | 0.014 |
3 | Code of Criminal Procedure 181 | 0.013 |
4 | Constitution 31 | 0.013 |
5 | Penal Code 60 | 0.013 |
6 | Penal Code 45 | 0.012 |
7 | Penal Code 54 | 0.011 |
8 | Code of Criminal Procedure 336 | 0.011 |
9 | Penal Code 47 | 0.011 |
10 | Penal Code 21 | 0.01 |
4.1 Models and evaluation metrics
-
Adamic-Adar is an index proposed in Adamic and Adar (2003). It evaluates the likelihood of a link based on common neighbors shared between nodes. Specifically, it is defined as \(A(x,y) = \Sigma _{u \in N(x) \cap N(y)} \frac{1}{ \log |N(u) |}\), where N(u) is the set of adjacent nodes of u.
-
The Jaccard coefficient is a coefficient similar to Adamic-Adar. The difference is in the normalization step, as specified in the following equation:
-
Preferential attachment is yet another basic link prediction score. It is defined by the following equation: \(|N(u) ||N(v) |\).
-
The Stochastic Block Model (SBM) (Holland et al. 1983) is a canonical latent block model that assumes that nodes are assigned to a block, and the interaction probabilities among the blocks fully determine the likelihood of a link. The limitation of the SBM is that theoretically, it is a model suited for no heavy-tailed distributions.
-
The degree-corrected stochastic block model (DCSBM) (Karrer and Newman 2011) is a variant of the SBM that incorporates degree heterogeneity. It is well known that from a model selection perspective, the DSCBM is often preferred over the SBM when the node degree distribution is heavy-tailed.
-
Node2vec is a method that embeds nodes to vectors using context information defined by random walks (Grover and Leskovec 2016).
-
Attri2vec is a method that adjusts for node attributes using node2vec to ensure structural similarity (Zhang 2019).
-
The graph convolutional network (GCN) is the basic graph neural network model where node embeddings are calculated via graph convolution (Kipf and Welling 2017).
4.2 Results
4.2.1 Quantitative comparison
Model | Text | Criminal | Criminal | Civil | Civil |
---|---|---|---|---|---|
feature | ROC | PR | ROC | PR | |
Adamic-Adar | No | 0.850 | 0.634 | 0.849 | 0.662 |
Resource allocation | No | 0.859 | 0.668 | 0.856 | 0.686 |
Jaccard coefficient | No | 0.743 | 0.310 | 0.761 | 0.391 |
Preferential attachment | No | 0.853 | 0.596 | 0.823 | 0.570 |
SBM | No | 0.880 | 0.678 | 0.852 | 0.645 |
DCSBM | No | 0.910 | 0.722 | 0.873 | 0.679 |
node2vec | No | 0.718 | 0.346 | 0.730 | 0.396 |
att2vec | Yes | 0.640 | 0.640 | 0.583 | 0.231 |
GCN | Yes | 0.910 | 0.728 | 0.901 | 0.716 |
4.2.2 Qualitative comparison
5 Conclusion
Cluster | Statutory laws and court precedents |
---|---|
1 | Public Offices Election Act; Local Autonomy Act; Constitution 42,43,44,45,46,47,54,59; |
House of Representatives Election District Election Council Establishment Act; | |
Court precedents | |
2 | Immigration Control and Refugee Recognition Act; Civil Code 752,809,877; Nationality |
Act 11; Court precedents | |
3 | Constitution 9,17,40,41,51,81,98; Civil Code 11,16,158,174,184,185,188,194,373,393,494, |
723; State Redress Act 6; Court precedents | |
4 | Local Civil Service Act; Local Autonomy Act; Local Government Finance Act; |
Basic Act on Education 10; Constitution 23; Court precedents | |
5 | Labor Standards Act; Labor Contract Act; Labor Union Act; Employment Insurance Act; |
Court precedents | |
6 | Pneumoconiosis Act; Labor Standards Act; Industrial Safety and Health Act; Industrial |
Accident Compensation Insurance Act; Ordinance on Prevention of Hazards Due | |
to Specified Chemical Substances; Court precedents | |
7 | Basic Resident Registration Act; Act on the Protection of Personal Information; |
Act on Access to Information Held by Administrative Organs; Court precedents | |
8 | Constitution; Child Welfare Act; National Pension Act; Employees’ Pension Insurance Act; |
Nationality Act; School Education Act; Court of Civil Procedure; Civil Code; Court precedents; | |
9 | National Public Service Act; Local Public Service Act; Local Public Enterprise Act; |
Act on the Organization and Operation of Local Educational Administration; | |
Local Autonomy Act; Constitution 39,94; Court precedents | |
10 | Patent Act; Utility Model Act; Design Act; Court precedents |
11 | Patent Act; Utility Model Act; Design Act; Order for Enforcement of the Act on |
Securing Quality, Efficacy and Safety of Products Including Pharmaceuticals and | |
Medical Devices; Court precedents | |
12 | Copyright Act; Act on Prohibition of Private Monopolization and Maintenance of |
Fair Trade; Code of Civil Procedure; Civil Code; Court precedents; | |
13 | Commercial Code; Civil Code; Court of Civil Procedure; Industrial Accident |
Compensation Insurance Act; Court precedents | |
14 | Medical Practitioners Act; Atomic Bomb Survivors’ Assistance Act; Civil Code; |
Court precedents | |
15 | Road Traffic Act; Act on Securing Compensation for Automobile Accidents; |
Civil Code 153,404,405,419,493,712,713,714,718,884; Code of Civil Procedure 117, | |
260; Civil Rehabilitation Act 87; Court precedents | |
16 | Waste Management and Public Cleansing Act; Administrative Case Litigation Act; |
Administrative Complaint Review Act; Health Insurance Act; Land Readjustment Act; | |
Court precedents | |
17 | Building Standards Act; Expropriation of Land Act; River Act; Basic Environment Act; |
City Planning Act; Court precedents | |
18 | Trademark Act; Unfair Competition Prevention Act; Court precedents |
19 | Financial Instruments and Exchange Act; Trust Act; Commercial Code; Civil Code; |
Companies Act; Banking Act; Court precedents | |
20 | Civil Code; Commercial Code; Financial Instruments and Exchange Act; Court precedents |
21 | Real Property Registration Order; Civil Code; Court precedents |
22 | Act on General Rules for National Taxes; Local Tax Act; Income Tax Act; Consumption |
Tax Act; Inheritance Tax Act; Civil Code; Court precedents | |
23 | Local Tax Act; Cropland Act 4,5; Court precedents |
24 | Money Lending Business Act; Interest Rate Restriction Act; Civil Code; Code of Civil |
Procedure; Court precedents | |
25 | Companies Act; Civil Provisional Remedies Act 7,13,23; Court of Civil Procedure 188,331; |
Court precedents | |
26 | Bankruptcy Act; Civil Rehabilitation Act; Act on General Rules for National Taxes; |
Commercial Code; Court precedents | |
27 | Act on Land and Building Leases; Act on Building Unit Ownership; Civil Code; |
Inheritance Tax Act; Civil Execution Act; Court precedents | |
28 | Income Tax Act 9,30,34;Court precedents |
29 | Act on General Rules for National Taxes; Income Tax Law; Corporation Tax Act; |
Consumption Tax Act; Act on Special Measures Concerning Taxation; Court precedents |
Cluster | Statutory laws and court precedents |
---|---|
1 | Constitution 1,3; Penal Code 1; Code of Criminal Procedure 256; Court precedents |
2 | Public Offices Election Act; Constitution 12; Police Act 2,26; Court precedents; |
3 | Constitution 15,16,19,41,73,98; National Public Service Act 19,102,108,110; |
Code of Criminal Procedure 39; Court precedents | |
4 | Firearms and Swords Control Act 31,32; Anti-Prostitution Act; Child Welfare Act; |
Act on Punishment of Organized Crimes and Control of Proceeds of Crime; | |
Narcotics and Psychotropics Control Act; Act on Punishment of Physical Violence; | |
Court precedents | |
5 | Juvenile Act; Offenders Rehabilitation Act 50,68,72; Child Welfare Act 5,25,27; |
Constitution 40; Criminal Compensation Act; Code of Criminal Procedures; | |
Penal Code 134,225; Court precedents | |
6 | Code of Criminal Procedure; Constitution 33; Court precedents |
7 | Code of Criminal Procedure; Constitution 7,9,11,13,14,18,20,21,31,32,35,36,37,38, |
39,65,76,82,97; Court precedents | |
8 | Labor Standards Act; Health Insurance Act 13; Court precedents |
9 | Penal Code 10,21,45,47,54,60; Code of Criminal Procedure 181,396 |
10 | Penal Code; Act on Punishment of Organized Crimes and Control 3,13; |
Firearms and Swords Control Act 3 | |
11 | Code of Criminal Procedures; Constitution 34,81;Court precedents |
12 | Code of Criminal Procedures; Measurement Act 17; Court precedents; |
13 | Code of Criminal Procedures; Child Welfare Act 7; Constitution 79; Road Traffic |
Act 52; Court precedents | |
14 | Dental Practitioners Act 28,31; Rabies Prevention Act 5,6; Medical Practitioners |
Act 17,31; Industrial Safety and Health Act 30; Penal Code; Court precedents | |
15 | Penal Codes; Act on Prohibition of Possession of Special Picking Tools, |
and Other Related Matters 4; Act on Punishment of Organized Crimes and | |
Control of Proceeds of Crime 2,4,13,16; Organ Transplant Act 2; Narcotics and | |
Psychotropics Control Act 1,65; Court precedents | |
16 | Act Concerning Special Provisions for the Narcotics; Court precedents |
17 | Road Traffic Act; Road Transport Vehicle Act; Court precedents |
18 | Code of Criminal Procedures 188,281,299,316; Attorney Act 23; Court precedents |
19 | Code of Criminal Procedures 191,195; Penal Code 193; Public Prosecutor’s Office |
Act 4,5; Act on Preventing Collisions at Sea 5; Court precedents | |
20 | Labor Standards Act 9,20,21,22; Industrial Safety and Health Act; Small and |
Medium Enterprises Basic Act 3; Medical Practitioners Act; Constitution 17; | |
Court precedents | |
21 | Act on Prohibition of Unauthorized Computer Access 3,8; Unfair Competition |
Prevention Act; National Public Service Act 100,109; Political Funds Control | |
Act 12,25; Court precedents | |
22 | Income Tax Act; Corporate Tax Act; Court precedents |
23 | Smaller Enterprise Retirement Allowance Mutual Aid Act; Public Accounting Act; |
24 | Public Accounting Act; Cabinet Order on Budgets, the Settlement of Accounts |
and Accounting; Local Autonomy Act; Court precedents | |
25 | Customs Act; Consumption Tax Act 64;Penal Code 18;Court precedents |
26 | Banking Act; Commercial Code; Court precedents |
27 | Waste Management and Public Cleansing Act; Tobacco Business Act 1,2; |
Basic Act for Establishing a Recycling-based Society 3,10; Constitution 22,25,94; | |
Court precedents | |
28 | Diet Act; National Government Organization Act; Act on the Organization and |
Operation of Local Educational Administration | |
29 | Foreign Exchange and Foreign Trade Act; Waste Management and Public |
Cleansing Act; Customs Act 67,111,113,117; Court precedents |