The sheer volume of legal contracts being generated and processed by businesses and legal organizations today, along with their increasing complexity, presents significant challenges. This project will address that gap by evaluating and comparing various legal LLMs, helping organizations identify the most effective tools for automating their contract review processes. Ultimately, this project aims to enhance the scalability and accuracy of contract management systems, contributing to better legal governance and operational efficiency at scale.
School
Computer Science and Engineering
Research Area
Legal Natural Language Processing (Legal NLP) | Natural language processing | Artificial intelligence
- Research environment
- Expected outcomes
- Supervisory team
- Reference material/links
The student will be mentored by the NLP research group at UNSW. The student will have access to compute credits via NCI GADI which will need to be used for the purpose of the project only.
Additional details about the project: Contracts often include complex language, known as "Legalese," which features intricate legal terminology, long and nested sentences or clauses, cross-references, jurisdictional differences, inconsistent formatting, and can sometimes span dozens or even hundreds of pages [1-3]. These factors make manual review an impractical solution. To address these challenges, automating legal contract classification (LCC) has become a promising solution. It can streamline the contract review process, making it more efficient and accessible, even for individuals who may not have the resources to consult legal experts [4]. While there are several legal-specific LLMs available, no comprehensive study currently exists to assess the relative effectiveness of these models for contract classification tasks. In this project, we will evaluate and compare various legal LLMs such as Legal-BERT [5], AdaptLLM [6], CaseLawBERT [7], and other, designed specifically for legal text analysis, with a focus on contract classification tasks.
- Well-documented code in Python
- Benchmark Report: Provide a comprehensive evaluation of various legal LLMs (e.g., Legal-BERT, AdaptLLM, CaseLawBERT, and others) in contract classification, highlighting their strengths and weaknesses. The report will need to be in a format that can be submitted to a research venue.
- Singh, Amrita, Preethu Rose Anish, Aparna Verma, Sivanthy Venkatesan, Logamurugan V, and Smita Ghaisas. "A data decomposition-based hierarchical classification method for multi-label classification of contractual obligations for the purpose of their governance." Scientific Reports 14, no. 1 (2024): 12755.
- Katrak, Malcolm. "The role of Language Prediction Models in contractual interpretation: The challenges and future prospects of GPT-3." Legal Analytics (2022): 47-62.
- ARIAI, FARID, and GIANLUCA DEMARTINI. "Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges." ACM Comput. Surv 1, no. 1 (2024).
- Guha, Neel, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters et al. "Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models." Advances in Neural Information Processing Systems 36 (2023): 44123-44279.
- Chalkidis, Ilias, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. "LEGAL-BERT: The Muppets straight out of Law School." In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2898-2904. 2020.
- Cheng, Daixuan, Shaohan Huang, and Furu Wei. "Adapting Large Language Models via Reading Comprehension." In The Twelfth International Conference on Learning Representations.
- Zheng, Lucia, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. "When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings." In Proceedings of the eighteenth international conference on artificial intelligence and law, pp. 159-168. 2021.