The Calculated Law Exploring the limits of transformer models in legal reasoning

Abstract

This thesis delves into the capabilities and limitations of Large Language Models (LLMs) used in legal reasoning, examining how these models process legal information to solve complex legal problems. A techno-legal analysis was conducted, comparing the technologies of LLMs to jurisprudence theories, to identify challenges they face in legal problem-solving. Also, an empirical evaluation was performed, assessing whether LLMs could handle a basic course in the Swedish law program. The findings illustrate that LLMS excel in understanding legal language and are able to apply legal methodologies even in complex scenarios previously thought to be beyond the capabilities of AI. This includes more complex legal methods like analogies, and ethical considerations. However, they struggle with extensive problems requiring a step-by-step approach, where the problem needs to be broken down into smaller parts and addressed sequentially. Despite these limitations, the empirical test suggests that current LLMs could potentially pass a challenging law course. This raises concerns about academic integrity, as it becomes increasingly difficult to distinguish between student-written and AI-generated exam responses. The growing capability of LLMs in legal reasoning also calls into question their future role in the legal profession and their potential impact on the work of lawyers and judges. To further elaborate, the ability of LLMs to potentially pass law courses may create difficulties in upholding academic integrity, as it becomes challenging to distinguish between student-written and AI-generated exam responses. The increasing capability of LLMs in legal reasoning also raises questions about their future role in the legal profession and how they might impact the work of lawyers and judges. This implies a potential shift in the legal landscape where LLMs could play a more prominent role in legal research, analysis, and decision-making. However, it also necessitates careful consideration of ethical implications and the need for human oversight to ensure fairness, accountability, and responsible use of LLMs in the legal domain.