Sanitized Portfolio Report

Leakage-Aware Evaluation in Legal NLP

A paraphrased case study on extracting structured legal features from lengthy judgments while controlling target leakage and hindsight bias.

Domain: Legal Analytics Focus: NLP Evaluation Year: 2026 Status: Paraphrased Summary

Watch 16s walkthrough

Hero visual: public-safe view of judgment ingestion, feature extraction, legal embeddings, and case metrics.

Fast Scan

What this report proves

A recruiter-readable summary of the legal NLP signal, model-design choices, evidence, and privacy boundary.

Signal

Leakage-aware legal analytics pipeline that turns long judgments into structured, evaluation-ready case features.

Methods

IRAC-inspired extraction, temporal filtering, Legal-BERT embeddings, XGBoost baselines, Hierarchical Attention Network design, and JSON QA.

Evidence

Extraction architecture, HAN design, baseline paths, feature logic, label validation, and evaluation-risk diagrams.

Boundary

No raw legal excerpts, party names, academic identifiers, evaluator notes, or unredacted source pages are published.

PII-Safe Publication Note

This page is a rewritten public case study. It removes personal names, academic identifiers, emails, raw legal excerpts, evaluator details, and submission metadata. The report describes the methods and lessons learned without reproducing private coursework pages or unredacted source material.

Field Note

The most important lesson here was that a model can look impressive for the wrong reason. I became more skeptical of headline scores once I saw how easily judgment text can leak the answer. The better portfolio signal is the evaluation discipline: define what the model is allowed to know, then test against that boundary.

Executive Summary

The project studied how legal judgments can be converted into structured analytical features for litigation-risk modelling. A core challenge is that judgments already contain the final outcome, so a naive model can accidentally learn from hindsight cues instead of pre-decision facts. The portfolio version therefore emphasizes leakage control, temporal isolation, legal feature engineering, and careful evaluation rather than raw model scores.

The pipeline combines rule-based preprocessing, legal-domain NLP, IRAC-inspired extraction, row-level normalization, and benchmark models such as tree-based classifiers and legal text embeddings. The most important design principle is to separate facts available before a decision from language that reveals the decision itself.

Portfolio Signal Snapshot

This case study is positioned for legal-tech, NLP, risk analytics, and applied machine-learning review. It highlights data quality, target-leakage control, and evaluation design rather than presenting a model as legal advice.

Role Signal Legal NLP pipeline designer for litigation-risk analytics

Core Stack Legal-BERT, XGBoost, HAN, JSON QA, feature engineering

Engineering Depth Temporal isolation, counterclaim handling, leakage audits

Business Value Transforms long judgments into structured, reviewable risk signals

Legal NLP Legal-BERT XGBoost Hierarchical Attention Network Target Leakage Feature Engineering Model Evaluation LLM-as-Judge Data-Centric AI Risk Analytics

Problem Context

Legal documents are long, unevenly structured, and rhetorically complex. The same case can contain claims, counterclaims, facts, procedural history, legal reasoning, damages, and final orders. A useful analytics workflow must convert this material into consistent features without collapsing all text into one ambiguous document vector.

Document Length

Relevant facts may be distributed across many sections, while outcome language can appear near the end or inside judicial reasoning.

Target Leakage

If the model sees phrases that reveal judgment, liability, or relief granted, performance can be misleading.

Party Structure

Claims and counterclaims may point in opposite directions, so case-level labels need careful normalization.

Analytical Objectives

Extract structured metadata, party relationships, factual context, and outcome labels from lengthy legal texts.
Apply an IRAC-inspired structure so issues, rules, application, and conclusions are not treated as interchangeable text.
Reduce target leakage by removing or isolating language that would only be known after judgment.
Compare traditional machine-learning baselines with legal-domain embedding approaches.
Present findings honestly, with limitations around corpus size, annotation ambiguity, and real-world generalization.

Sanitized Extraction Flow

The original report contained a data extraction architecture diagram. This public version recreates the workflow at the method level while excluding raw judgment screenshots, case examples, and submission metadata.

Unstructured Judgment to ML-Ready Case Rows

Raw Judgment Long-form legal text from public judgment documents.

Stage 1 Extract pre-dispute facts, issues, rules, and case metadata.

Temporal Filter Remove or isolate hindsight language that reveals the final decision.

Stage 2 Separate cases, party roles, counterclaims, and normalized outcomes.

Stage 3 Audit JSON structure, label consistency, omitted entities, and leakage.

Human Review Correct ambiguous or non-compliant outputs before training.

Feature Store Prepare row-level and case-level records for modelling.

Evaluation Compare row-level discrimination with case-level business usefulness.

Curated Figure Gallery

The figures below preserve the original modelling evidence while using light-background recreations for the architecture diagrams that were too dense on dark slide canvases. Raw judgment screenshots, case pages, and report metadata are not published.

Light-background legal NLP extraction architecture diagram — **Extraction, audit, and label-validation pipeline.** Light-background recreation showing the staged path from raw PDF court judgments to ML-ready structured JSON.

Light-background LegalBERT and XGBoost model baseline diagram — **Baseline model comparison.** Documents the LegalBERT embedding flow and XGBoost ensemble baseline used before the custom HAN framing.

Original case-level aggregation table — **Case-level aggregation logic.** Shows how row-level risk predictions were rolled up differently for XGBoost, BERT classifier, and HAN outputs.

Light-background hierarchical attention network architecture diagram — **HAN architecture figure.** Light-background recreation of the report's main modelling idea: late fusion over fact, issue, rule, court, and judge representations.

Extracted model results table from judicial analytics report — **Model-results table.** Includes the original row-level and case-level evaluation table for XGBoost, BERT classifier, and HAN. It is framed as a coursework evaluation artifact, not as a production benchmark or audited legal-tech performance claim.

Short Demo Clip

This silent walkthrough focuses on the evaluation path: ingestion, IRAC extraction, temporal filtering, feature rows, leakage audit, baselines, and case-level metrics.

Muted Walkthrough

Judgment text to leakage-aware evaluation

The clip makes the senior-review point visible: metrics matter only after the information boundary and target-leakage risks are controlled.

Transcript

Judgment text is normalized, IRAC extraction and temporal filtering separate pre-decision facts from outcome language, feature rows are audited for leakage, and baseline models are compared before case-level metrics are framed as evaluation evidence.

Pipeline Design

1. Text Ingestion and Normalization

The first step standardizes document text so headings, paragraph breaks, dates, monetary references, and party labels can be processed consistently. This step reduces noise before feature extraction.

2. Case and Party Rows

Instead of treating a judgment as one flat object, the design can represent claims, parties, and counterclaims as structured rows. This helps prevent mixed signals when a case contains multiple directions of liability or relief.

3. IRAC-Inspired Extraction

The pipeline separates issue statements, legal rules, factual application, and conclusions. For modelling, pre-decision factual material is more appropriate than final judicial conclusions because the latter can reveal the label.

4. Temporal Isolation

Features should approximate what would have been knowable before the case outcome. This requires careful treatment of dates, procedural history, and explicit decision language.

5. Embeddings and Baselines

Legal-domain embeddings can capture semantic patterns in case facts. Traditional models such as boosted trees remain useful as baselines because they make feature importance and failure modes easier to inspect.

Model Architecture Notes

The source report compared a traditional baseline, a legal-domain transformer baseline, and a custom hierarchical architecture. The portfolio version keeps the model comparison without publishing raw screenshots or unsupported performance claims.

XGBoost Baseline Uses early fusion: structured text arrays are concatenated before vectorization. Efficient, but it can blur facts, issues, and legal rules into one flat representation.

LEGAL-BERT Baseline Uses legal-domain language representations, but still struggles with long judgments and flat input limits when narrative structure matters.

Hierarchical Attention Network Encodes legal text hierarchically, preserving a stronger separation between fact-level events and case-level representation.

Case-Level Aggregation Converts party-level predictions into lawsuit-level signals using aggregation rules aligned with practical legal-risk decisions.

Leakage Controls

Leakage control is the strongest part of the project from a portfolio perspective. The model should not be rewarded for reading the answer inside the judgment. The public writeup therefore foregrounds the safeguards rather than presenting performance numbers without context.

Remove or isolate sections that directly describe the final decision, relief awarded, or liability finding.
Normalize target labels symmetrically so claims and counterclaims are not accidentally coded in inconsistent directions.
Separate factual summaries from judicial conclusions where the document structure supports it.
Check whether high-performing features are actually proxies for the final outcome.
Evaluate at both row level and case level to avoid overstating performance on duplicated or related examples.

Technical Notes From Source Report

Temporal Isolation Facts are separated from trial-stage and outcome-stage language so the model does not learn from information unavailable before the dispute outcome.

Counterclaim Handling Main claims and counterclaims are represented separately so party roles and labels remain internally consistent.

LLM-as-Judge QA Automated review checks extraction accuracy, JSON validity, semantic consistency, and leakage risk before human intervention.

Metric Realignment Row-level ROC-AUC is useful for discrimination, while case-level recall and F1 better reflect the practical decision to pursue or avoid a case.

Decision Framework

The central modelling question was not simply which algorithm to use. It was how to preserve legal structure so the model learns from pre-dispute facts rather than accidental hints embedded in the final judgment.

Decision Area	Naive Option	Selected Approach	Reasoning
Prediction unit	One row per document	Party-level rows with case-level aggregation	Captures multi-party disputes and makes evaluation closer to real decision-making.
Feature source	Use the whole judgment text	Separate pre-dispute facts from outcome language	Controls hindsight bias and avoids inflated performance from leaked target signals.
Model baseline	Jump directly to transformer models	Compare XGBoost, Legal-BERT, and HAN-style representations	Creates a fair baseline ladder and exposes whether complexity adds practical value.
Quality assurance	Trust generated JSON as-is	Automated audits plus human review	Reduces entity omissions, malformed labels, and leakage-prone extraction errors.

Qualitative Modelling Priority Chart

Leakage Control Critical

Label Consistency Critical

Document Structure High

Model Complexity Selective

Interpretability High

What I optimized for Reliable training data and realistic evaluation before chasing higher-capacity models.

What I avoided Publishing raw legal excerpts or advertising model results that could be leakage-driven.

What I would harden next Section classifiers, annotation guidelines, temporal validation, and model cards for intended use.

Evaluation Approach

Extraction Quality

Check whether the pipeline correctly identifies parties, dates, claim direction, factual context, and outcome labels.

Model Comparison

Compare legal embeddings, bag-of-words style features, and structured variables against simple baselines.

Robustness

Inspect errors where the model relies on summary bias, procedural language, or outcome-adjacent wording.

Limitations

Legal outcomes are shaped by facts, evidence, procedure, and legal interpretation. A text model can support analysis but should not be framed as legal advice.
Judgment text is written after the outcome is known, so leakage control is inherently difficult.
Small or domain-specific corpora can make model performance unstable across case types.
Public reporting intentionally avoids raw excerpts and sensitive metadata, so the writeup focuses on method design rather than reproducing data samples.

Recommended Next Improvements

Build a transparent annotation guide for facts, issues, outcomes, and leakage-prone sections.
Add document-section classifiers to separate factual narrative from legal conclusion more consistently.
Run temporal validation where later cases are predicted from earlier training data.
Add model cards that describe intended use, prohibited use, evaluation scope, and residual risks.

Selected Sources From Original Report

The original bibliography covered legal judgment prediction, legal-domain encoders, weak supervision, hierarchical attention, and realistic evaluation risks. The portfolio version lists selected references most directly tied to the public case study.

Chalkidis, Androutsopoulos, and Aletras (2019), Neural Legal Judgment Prediction in English.

Chalkidis et al. (2020), LEGAL-BERT: The Muppets Straight Out of Law School.

Chalkidis et al. (2022), LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.

Katz (2012), Quantitative Legal Prediction.

Katz, Bommarito, and Blackman (2017), A General Approach for Predicting the Behavior of the Supreme Court of the United States.

Medvedeva, Vols, and Wieling (2020), Using Machine Learning to Predict Decisions of the European Court of Human Rights.

Ratner et al. (2017), Snorkel: Rapid Training Data Creation with Weak Supervision.

Yang et al. (2016), Hierarchical Attention Networks for Document Classification.

Recruiter Signal

This project demonstrates careful applied machine learning: feature extraction from messy documents, domain-specific NLP, leakage-aware evaluation, and honest communication of model limits. It is relevant for data science, NLP, risk analytics, and applied AI roles where correctness matters more than a flashy demo.

Legal NLP Legal-BERT XGBoost Feature Engineering Leakage Control Evaluation Design

Sanitized report prepared for portfolio publication. Raw academic materials and private data are not published.