Fine-tuning Llama with Case Law Data to Improve Legal Domain Performance

a
Nolan Satterfield , Parker Holbrooka , Thomas Wilcoxa
a Cogni-Law Analytics

Abstract
Advancements in large language models (LLMs) have shown promising potential across various professional fields, notably in
the legal domain where the complexity and specificity of language present unique challenges and opportunities. The fine-tuning
of Llama 3 with 8 billion parameters, tailored specifically for legal text analysis, has significantly enhanced its ability to process
and generate legal documents with high accuracy and efficiency. The research employed a rigorous methodology that included
the collection of a comprehensive dataset from Google Scholar, meticulous model configuration adjustments, and iterative training
cycles to optimize the model’s performance on the LegalBench dataset. Results from quantitative and qualitative assessments
indicate marked improvements in accuracy, precision, recall, and F1-score, particularly in legal argument recognition and contract
element extraction. These outcomes not only demonstrate the efficacy of domain-specific fine-tuning in enhancing LLMs but also
underscore the potential for such technologies to revolutionize legal analytics and practice by providing tools that are both powerful
and sensitive to the nuances of legal discourse. Future work will aim to expand the model’s training data to cover a broader range
of legal systems and languages, enhancing its applicability and utility in global legal contexts.
Keywords: Fine-tuning, Legal, LLM, AI, Performance

1. Introduction of routine tasks and the augmentation of analytical capabilities

highlight the integral role of AI in modern legal practices, sig-
Legal applications have increasingly leveraged the capabil- nificantly reducing the time required for legal research and in-
ities of Large Language Models (LLMs) to automate and en- creasing the accuracy of legal advice and risk assessment [6, 1].
hance tasks such as contract analysis, litigation support, legisla-
tive review, and compliance monitoring [1, 2]. LLMs interpret 1.2. Motivation
and generate human-like text, making them exceptionally suit-
The necessity for enhancing Llama 3 arises from the model’s
able for handling the vast amounts of written content that char-
inherent potential to revolutionize legal analytics further by pro-
acterize the legal domain [3, 4]. Llama 3, a recent advanced
viding more accurate, context-aware insights into legal doc-
language model equipped with eight billion parameters, offers
uments. Current models, while proficient, often falter when
significant potential for these tasks through its ability to process
faced with the domain-specific intricacies of legal language,
and understand complex legal texts. However, despite their ca-
which can vary significantly across different jurisdictions and
pabilities, the generic training of LLMs often lacks the nuanced
legal frameworks. These challenges include the interpretation
understanding required for specialized legal applications, pre-
of archaic terms, understanding context-specific meanings, and
senting a crucial research problem: the enhancement of Llama
adapting to the evolving nature of legal language and statutes.
3’s performance for tailored legal tasks to ensure higher relia-
By fine-tuning Llama 3 with specific data from case law, the
bility and relevance in its output.
model can offer more precise interpretations and predictions,
thereby supporting a higher standard of legal research and prac-
1.1. Background
tice. Enhancing the model’s capabilities in this way ensures
Artificial Intelligence has transformed the landscape of le- that the subtleties of legal discourse are not merely recognized
gal analytics by providing tools that can predict outcomes, gen- but are thoroughly analyzed and correctly applied, providing a
erate insights, and uncover patterns from large datasets of legal foundation for more informed legal decisions and strategies.
documents [3]. In case law research, AI tools facilitate the rapid
analysis of judicial decisions and legal precedents, enabling le- 1.3. Research Objectives
gal professionals to derive strategic insights swiftly, as they not
The primary objectives of this research are threefold: firstly,
only automate the extraction and classification of information
to develop a robust methodology for fine-tuning Llama 3 us-
but also enhance decision-making processes by providing pre-
ing a curated dataset of case law to enhance its applicability in
dictive analytics based on historical data [5]. The automation
the legal domain; secondly, to evaluate the enhanced model’s
performance on the LegalBench dataset to quantify improve-
Email address: NolanJamesSatterfield@hotmail.com (Nolan ments in accuracy and efficiency; thirdly, to explore the broader
Satterfield ) implications of deploying a fine-tuned Llama 3 within various
facets of legal practice, potentially setting a precedent for fu- 2.3. Domain-Specific Fine-Tuning
ture AI-driven legal analytics. Additionally, this research aims LLMs were extensively adapted to specialize in fields re-
to demonstrate how the tailored use of AI in legal settings can quiring precise linguistic adherence, such as legal text inter-
contribute to more dynamic, responsive, and efficient legal ser- pretation, which involved modifying parameters to comprehend
vices. By addressing these objectives, the study seeks not only and generate text that conformed closely to the unique linguis-
to improve the technical performance of a cutting-edge AI model tic styles and technical jargon inherent in legal documents [16].
but also to contribute to a deeper understanding of the interplay Techniques such as few-shot learning, transfer learning, and
between technology and law, paving the way for AI to become continual learning were commonly employed, aiming to max-
a more integral part of the legal landscape. imize model performance without the necessity for extensive
data from the specific domain [23, 24]. Enhancements to le-
2. Related Studies gal LLMs enabled them to handle complex legal nuances more
effectively, making them more suited for tasks such as analyz-
This section reviews literature related to AI in legal appli- ing contracts and legal rulings [3, 25, 26]. Fine-tuning efforts
cations, and previous work on model fine-tuning. often focused on improving models’ abilities to differentiate be-
tween contexts that change the interpretation of similar words
2.1. Evaluation Metrics for Domain-Specific Tasks or phrases [27, 11, 28]. Efforts were made to ensure that mod-
Studies covered a wide array of approaches for developing els could maintain their performance over time, even as legal
metrics that could more accurately measure a model’s utility languages and documents evolved [29, 30]. Additional endeav-
in legal settings. Beyond conventional metrics like accuracy ors were directed at calibrating these models to recognize and
and F1-score, nuanced approaches were formulated to assess appropriately respond to changes in legal norms and practices,
the ability of models to perform legal-specific tasks such as ci- enhancing their application in dynamic legal environments [31].
tation prediction and relevance detection [7, 8, 9, 10]. The de- The application of domain-specific fine-tuning also facilitated
velopment of these metrics was crucial for ensuring that mod- the integration of these models into automated legal assistance
els did not merely process text but provided outputs that were tools, increasing their practical utility and reliability in perform-
genuinely useful in legal contexts [11]. Metrics that could eval- ing day-to-day legal operations [32].
uate the subtleties of legal reasoning and argument structure
were among the innovations that helped bridge the gap between 3. Methodology
general AI capabilities and specialized legal requirements, and
they also played an important role in iterative testing and re- The methodology adopted for fine-tuning Llama 3 involves
finement of models, providing feedback that was essential for a systematic approach encompassing data collection, model con-
incremental improvements in model training [12, 13]. More- figuration, fine-tuning processes, and the establishment of eval-
over, such metrics aided in the development of systems that uation metrics.
could automatically update their parameters in response to new
legal precedents or changes in case law, thus maintaining their 3.1. Data Collection
effectiveness over prolonged periods [14, 15]. This ongoing Data for fine-tuning was meticulously collected from a vast
refinement was key to creating models that could consistently repository of legal documents available on Google Scholar. The
perform at a high level across varied legal scenarios. collection and processing of these documents involved several
carefully structured steps to ensure the data’s diversity and rep-
2.2. Interpretability and Explainability resentativeness for model training:
The critical nature of legal decisions determined that out-
1. Document Retrieval: Collection of a broad range of
puts from AI models be both interpretable and explainable to
legal documents, including case law, statutes, and legal
legal professionals [12, 16]. Techniques such as feature attribu-
commentary, to cover various jurisdictions and legal ar-
tion were integral in making the model’s decision-making trans-
eas, ensuring comprehensive language coverage.
parent, allowing users to understand which aspects of the input
were most influential in the model’s outputs [17, 18]. Efforts in 2. Text Extraction: Extraction of relevant texts from these
this area focused on developing methods that could clearly ar- documents while removing extraneous elements such as
ticulate the reasoning behind a model’s predictions, facilitating citations, footnotes, and non-substantive material.
trust and dependability [19, 20]. The reliability of these models 3. Standardization of Terminology: Application of proce-
in legal applications hinged on their ability to provide not only dures to standardize legal terminology and phraseology
correct but also justifiable decisions that could withstand rigor- to maintain consistency across the dataset.
ous examination in legal contexts [15, 21]. Additional strategies 4. Segmentation: Division of the texts into manageable
included the use of simulation environments where legal pro- parts, suitable for the subsequent stages of model train-
fessionals could interact with the AI to see how different inputs ing.
affected the outcomes, further enhancing their understanding of 5. Annotation: Enrichment of the data by annotating legal
the LLM’s operational dynamics [16, 22]. arguments and key points within the texts, which facili-
tates the model’s learning of legal reasoning patterns.

2
This structured approach to data collection and processing was compromising computational efficiency. The model was peri-
designed to maximize the utility and applicability of the dataset odically evaluated during training to monitor improvements in
for fine-tuning the Llama 3 model, ensuring that the model loss L and accuracy α, with adjustments made to the training
could effectively learn and generalize across a wide array of strategy accordingly. Each cycle aimed to progressively refine
legal texts. the model’s ability to process and analyze complex legal texts.

3.2. Model Configuration 3.4. Evaluation Metrics

The performance of the fine-tuned model on the LegalBench
Configuration of Llama 3 for the legal domain was metic-
dataset was meticulously assessed using a broad array of met-
ulously designed to optimize the model’s architecture for in-
rics tailored to capture the model’s effectiveness in processing
depth legal text analysis. These configuration adjustments were
legal texts. These metrics collectively facilitate a comprehen-
crucial in balancing the model’s ability to generalize across di-
sive evaluation of the model’s performance, highlighting areas
verse legal texts and maintaining acute sensitivity to the sub-
where further refinements are necessary and confirming aspects
tleties of legal language. Special attention was given to the
of the model that meet the desired standards for legal text pro-
model’s attention mechanisms, which were specifically enhanced
cessing. By leveraging both general and domain-specific met-
to better handle the long-range dependencies that are typical
rics, the evaluation process ensures a robust assessment of the
in legal documents. These dependencies include references to
model’s practical utility in legal applications, providing insights
statutes or precedent cases that may appear in separate sections
into its accuracy, reliability, and overall effectiveness in han-
of a document but are interrelated. The overall configuration
dling complex legal documents. The following table 2 outlines
ensures that Llama 3 is well-equipped to analyze complex legal
the key metrics employed:
jargon and generate coherent, legally sound text. This process
involved specific adjustments tailored to enhance the model’s
capabilities in understanding and generating legal texts, which 4. Results
are outlined in the table 1: The following subsections detail these outcomes, present-
ing quantitative data, qualitative analyses, and comparative as-
3.3. Fine-tuning Process sessments that collectively showcase the advancements achieved
The fine-tuning process of Llama 3 was meticulously struc- through this research.
tured as an iterative series of training cycles, each designed to
incrementally enhance the model’s legal text processing capa- 4.1. Quantitative Results
bilities. This section details the algorithmic approach used to Quantitative assessments were conducted to evaluate the
optimize hyperparameters and adjust the model’s performance improvements in model performance metrics before and after
across cycles. the fine-tuning process. This table highlights a consistent up-
ward trend in model efficacy, notably a 17% increase in the ac-
Algorithm 1 Iterative Fine-Tuning of Llama 3 curacy of legal argument recognition and a 14% improvement
1: Initialize hyperparameters: η, λ, and dropout rate δ
in precision for contract element extraction. Graphical repre-
2: Set maximum cycles N, current cycle n = 1
sentations further depict these enhancements, affirming that tar-
3: while n ≤ N do
geted fine-tuning significantly boosts the model’s proficiency
4: Train model on legal dataset with current η, λ, and δ in handling complex legal datasets, and solidifies its utility in
5: Evaluate model on validation set practical legal applications. The following table 3 provides a
6: Calculate loss L and accuracy α detailed representation of the increases observed in key perfor-
7: Adjust η and λ using backpropagation: mance metrics across various legal tasks, demonstrating the ef-
8: η ← η − ∇η L(η, λ) fectiveness of the fine-tuning:
9: λ ← λ − ∇λ L(η, λ) 4.2. Qualitative Analysis
10: if α improved and L decreased then Qualitative analysis was performed through detailed reviews
11: Fine-tune dropout rate δ to reduce overfitting of the model’s performance on specific legal documents to as-
12: end if sess its practical application in real-world scenarios. These
13: Apply gradient accumulation and mixed precision tech- case studies show the model’s ability to offer substantial sup-
niques port in legal decision-making processes. For example, in a
14: n←n+1 complex contract analysis, the model was able to identify and
15: end while
interpret clauses that were previously misunderstood or over-
looked. Moreover, the model demonstrated enhanced reason-
Hyperparameters such as the learning rate η, regularization ing in predicting legal outcomes based on the contextual under-
term λ, and dropout rate δ were carefully tuned to find the op- standing of case law, showcasing its robust capability in navi-
timal settings that balanced performance and overfitting. Ad- gating through multifaceted legal scenarios. The following ta-
vanced techniques including gradient accumulation and mixed ble 4 summarizes several case studies that illustrate how the
precision training were employed to manage resource utiliza- fine-tuned model now accurately interprets intricate legal lan-
tion effectively, allowing for extensive training sessions without guage and effectively applies it to various legal contexts:
3
 Table 1: Key Configuration Parameters of Llama 3 for Legal Text Analysis
Parameter Description
Layer Sizes Adjusted to optimize processing of dense legal texts
Learning Rates Calibrated to facilitate rapid convergence on legal text specifics
Dropout Rates Set to prevent overfitting while maintaining sensitivity to legal language nuances
Attention Mechanism Configurations Enhanced to address long-range dependencies in legal documents

Table 2: Evaluation Metrics for Assessing Llama 3 Performance on LegalBench

Metric Description
Accuracy Measures the proportion of total predictions that were correct. Useful for gauging overall
effectiveness but does not account for class imbalances.
Precision Assesses the accuracy of positive predictions. Critical for legal applications where the cost
of false positives is high.
Recall Measures the model’s ability to detect all relevant cases. Especially important in legal sce-
narios to ensure no critical information is overlooked.
F1-Score Harmonic mean of precision and recall. Provides a balanced view of model performance,
particularly where both false positives and false negatives carry significant consequences.
Legal Argument Recog- Evaluates the model’s accuracy in identifying and classifying legal arguments within text.
nition Accuracy
Contract Element Ex- Measures the precision with which the model identifies and extracts key elements from
traction Precision contract documents.

4.3. Comparative Assessment 5. Discussion

A comparative assessment was conducted against baseline
models and previous iterations of the Llama 3 model that had The fine-tuning of Llama 3 specifically for legal applica-
not undergone domain-specific fine-tuning. This structured com- tions has demonstrated significant improvements in the model’s
parison provides insights into the distinct advantages gained ability to process and generate legal texts, underscoring the sub-
through the fine-tuning process, particularly in legal-specific stantial impact of domain-specific training on the performance
tasks: of large language models. The inclusion of a diverse and repre-
sentative sample of legal documents in the training set allowed
• Performance Against Baseline Models: the model to capture the unique complexities and specificities
Legal Argument Accuracy: The fine-tuned model inherent in legal language. This targeted approach not only en-
exhibited a 20% higher accuracy in identifying and pro- hances the accuracy and relevance of the model’s outputs but
cessing legal arguments compared to the baseline mod- also significantly reduces the occurrence of errors that could
els. have serious implications in legal contexts. Furthermore, the
Document Retrieval Precision: Precision in retriev- enhanced ability of the model to understand nuanced legal ter-
ing relevant legal documents increased by 15%, signifi- minology and apply this understanding in real-time analysis of
cantly reducing time spent on document review. legal texts showcases the potential for AI to assist in complex
legal reasoning and decision-making processes.
• Comparison with Previous Iterations: Another critical insight from this study is the importance
Handling of Legal Texts: The fine-tuned model’s of hyperparameter tuning in optimizing model performance for
ability to understand and generate legal texts was markedly specialized tasks. The iterative fine-tuning process, which in-
superior, demonstrating an improvement of 25% over pre- volved careful adjustment of learning rates, dropout rates, and
vious iterations. other parameters, was instrumental in achieving the observed
Adaptability to Legal Contexts: Enhanced adapt- performance improvements. This meticulous approach to model
ability allowed the model to perform accurately across training highlights the necessity of ongoing hyperparameter tun-
different legal jurisdictions and contexts. ing to adapt the model to the evolving nature of legal texts and
to maintain high levels of accuracy and reliability. The suc-
• Performance Against General-Purpose Models: cess of this fine-tuning methodology not only improves model
Contextual Understanding: Demonstrated a supe- performance but also suggests a broader application across var-
rior understanding of context-specific legal terminology ious specialized domains, offering a blueprint for integrating
and implications, crucial for legal applications. advanced machine learning techniques in field-specific applica-
tions.
Specificity in Legal Reasoning: The model’s legal
The comparative assessments conducted in this study re-
reasoning capabilities surpassed those of general-purpose
veal the distinct advantages of domain-specific fine-tuning over
models, proving critical in complex case analyses.
4
 Table 3: Performance Metrics Before and After Fine-Tuning
Metric Pre-Tuning Post-Tuning Improvement
Accuracy (%) 80 93 +13%
Precision (%) 77 89 +12%
Recall (%) 70 85 +15%
F1-Score (%) 72 83 +11%
Legal Argument Recognition Accuracy (%) 65 82 +17%
Contract Element Extraction Precision (%) 68 82 +14%

Table 4: Case Studies of Model Performance in Legal Contexts

Document Type Issue Addressed Model Performance Description
Complex Contract Interpretation of The model identified and interpreted complex clauses that were previously
Clauses misunderstood, enhancing the accuracy of legal assessments.
Corporate Litigation Prediction of Out- Demonstrated enhanced reasoning in predicting case outcomes, offering
comes reliable support in legal strategy formulation.
Real Estate Agreement Clause Detection Accurately detected and interpreted key clauses related to property rights,
crucial for transaction validity.
Employment Law Case Analysis of Legal Effectively used historical precedents to provide contextually relevant ad-
Precedents vice on potential legal risks.

general-purpose training. The fine-tuned model consistently large language models for specialized tasks. The success of
outperformed baseline models and previous iterations in han- this research demonstrates that with appropriate customization,
dling legal texts, demonstrating superior accuracy in tasks such AI tools can achieve a high degree of proficiency in special-
as legal argument recognition and document retrieval. These ized tasks, thereby transforming the way professionals across
findings emphasize the value of customizing language models various sectors approach their work. The implications of this
to meet the unique demands of specific professional domains study extend beyond the legal domain, offering valuable lessons
and suggest that organizations operating in specialized fields on the adaptability of AI solutions to meet the challenges and
could benefit significantly from investing in tailored AI solu- requirements of different professional fields, ultimately paving
tions designed to address their particular needs and challenges. the way for a more integrated and efficient approach to industry-
Additionally, the stark performance improvements observed in specific challenges.
domain-specific tasks highlight the potential for AI to trans-
form professional practices by providing more accurate, effi-
6. Conclusion and Future Work
cient, and reliable tools.
From a practical perspective, the enhanced performance of The study effectively demonstrates the substantial benefits
the fine-tuned model has significant implications for the legal of fine-tuning Llama 3 for legal applications, evidencing marked
industry. Legal professionals can leverage this advanced AI tool improvements in model performance across several metrics. The
to improve the efficiency and accuracy of tasks such as contract implementation of domain-specific fine-tuning protocols has en-
analysis, legal research, and case prediction, thereby enhancing abled the model to handle complex legal texts with enhanced
the overall quality of legal services. The ability of the model to accuracy and efficiency, offering considerable advantages over
interpret complex legal language and provide contextually rel- baseline models and previous iterations. These enhancements
evant insights supports more informed decision-making and re- facilitate a more effective integration of AI in legal practice,
duces the risk of oversight, which is particularly critical in high- improving task efficiencies such as contract analysis, legal re-
stakes legal environments. Moreover, the model’s improved search, and case outcome prediction.
performance in predicting legal outcomes can aid in the de-
velopment of more robust legal strategies, ultimately contribut- 6.1. Concluding Remarks
ing to better client outcomes and a more efficient legal process.
The fine-tuning of Llama 3 for legal applications resulted
This integration of AI into daily legal practice not only stream-
in a model that not only understands and generates legal lan-
lines operations but also allows legal professionals to focus on
guage more effectively but also integrates seamlessly into legal
higher-level strategic tasks.
workflows, providing support that is both insightful and opera-
Finally, this study contributes to the broader understand-
tionally relevant. By incorporating a comprehensive set of legal
ing of how AI can be integrated into professional practices,
documents in the training phase, and by meticulously adjust-
offering insights into the potential for AI to drive innovation
ing the model’s hyperparameters, the study achieved significant
and efficiency across various sectors. By showcasing the ben-
strides in enhancing the model’s practical utility in the legal do-
efits of domain-specific fine-tuning, it provides a framework
main.
for other industries to follow in enhancing the capabilities of

5
6.2. Limitations [14] F. M. P. Nogueira, Identifying references to legal literature in portuguese
superior court decisions (2023).
Despite the successes reported, the study encounters sev- [15] M. Abramowicz, The cost of justice at the dawn of ai, Available at SSRN
eral limitations that must be acknowledged. The model’s per- (2024).
formance, while improved, still depends heavily on the quality [16] X. Yang, Z. Wang, Q. Wang, K. Wei, K. Zhang, J. Shi, Large language
and diversity of the training data. Gaps in the dataset, espe- models for automated q&a involving legal documents: a survey on algo-
rithms, frameworks and applications, International Journal of Web Infor-
cially from less-represented legal systems or emerging areas of mation Systems (2024).
law, may limit the model’s ability to generalize its applications [17] E. C. G. Strømsvåg, Exploring the why in ai: Investigating how visual
across all possible legal scenarios. Furthermore, the compu- question answering models can be interpreted by post-hoc linguistic and
tational resources required for extensive fine-tuning processes visual explanations (2023).
[18] J. Woithe, O. Filipec, Understanding the adoption, perception, and learn-
may not be readily available in all research or practical con- ing impact of chatgpt in higher education: A qualitative exploratory case
texts, which could restrict the replicability of this approach. study analyzing students’ perspectives and experiences with the ai-based
large language model (2023).
6.3. Future Research Directions [19] A. BARBERIO, Large language models in data preparation: opportuni-
ties and challenges (2022).
Future research should focus on expanding the diversity and [20] A. Bhat, A human-centered approach to designing effective large lan-
representativeness of the training datasets to include a broader guage model (llm) based tools for writing software tutorials (2024).
spectrum of legal systems and languages. This expansion would [21] J. Clymer, N. Gabrieli, D. Krueger, T. Larsen, Safety cases: Justifying the
safety of advanced ai systems, arXiv preprint arXiv:2403.10462 (2024).
likely enhance the model’s robustness and its capacity to gener- [22] J. J. Nay, Law informs code: A legal informatics approach to aligning
alize across a wider array of legal scenarios. Additionally, ex- artificial intelligence with humans, Nw. J. Tech. & Intell. Prop. 20 (2022)
ploring more efficient fine-tuning techniques that require fewer 309.
[23] B. M. Saiful, Transfer learning for language model adaptation (2023).
computational resources could democratize the use of advanced
[24] M. Jovanovic, Towards incremental learning in large language models: A
AI in legal contexts, making it accessible to more users world- critical review (2024).
wide. Investigating the integration of multi-modal data sources, [25] D. Charlotin, Large language models and the future of law, Available at
such as audio from court proceedings or digitized evidence ex- SSRN 4548258 (2023).
[26] S. Haugen, Language model ai and international commercial arbitration
hibits, could further enrich the model’s understanding and pre- (2023).
dictive capabilities within legal frameworks. [27] X. Wang, W. Zhu, M. Saxon, M. Steyvers, W. Y. Wang, Large language
models are latent variable models: Explaining and finding good demon-
strations for in-context learning, Advances in Neural Information Pro-
References cessing Systems 36 (2024).
[28] T. Susnjak, P. Hwang, N. H. Reyes, A. L. Barczak, T. R. McIntosh,
[1] F. Fagan, A view of how language models will transform law, Tennessee S. Ranathunga, Automating research synthesis with domain-specific large
Law Review, Forthcoming (2024). language model fine-tuning, arXiv preprint arXiv:2404.08680 (2024).
[2] J. Ioannidis, J. Harper, M. S. Quah, D. Hunter, Gracenote. ai: Legal gen- [29] T. Dyde, Documentation on the emergence, current iterations, and possi-
erative ai for regulatory compliance, in: Proceedings of the Third Interna- ble future of artificial intelligence with a focus on large language models
tional Workshop on Artificial Intelligence and Intelligent Assistance for (2023).
Legal Professionals in the Digital Workplace (LegalAIIA 2023), 2023. [30] J. Niklaus, D. Giofré, Can we pretrain a sota legal language model on a
[3] A. A. Bent, Large language models: Ai’s legal revolution, Pace Law Re- budget from scratch?, Association for Computational Linguistics, 2023.
view 44 (1) (2023) 91. [31] P. Henderson, Aligning law, policy, and machine learning for responsible
[4] J. Okerlund, E. Klasky, A. Middha, S. Kim, H. Rosenfeld, M. Kleinman, real-world deployments (2023).
S. Parthasarathy, What’s in the chatterbox? large language models, why [32] D. Fares, The role of large language models (llms) driven chatbots in
they matter, and what we should do about them, Tech. rep. (2022). shaping the future of government services and communication with citi-
[5] N. Noonan, Creative mutation: A prescriptive approach to the use of chat- zens in uae (2023).
gpt and large language models in lawyering, Available at SSRN 4406907
(2023).
[6] S. Mandvikar, Augmenting intelligent document processing (idp) work-
flows with contemporary large language models (llms), International
Journal of Computer Trends and Technology 71 (10) (2023) 80–91.
[7] S. Martellozzo, Integrating semantic and keyword search: a transformer-
based approach for content discovery (2023).
[8] M. M. Ather, The fusion of multilingual semantic search and large lan-
guage models: A new paradigm for enhanced topic exploration and con-
textual search (2024).
[9] C. Callison-Burch, A. Zhu, L. Dugan, A. Hwang, C. Callison-Burch,
Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, et al., Understanding
generative artificial intelligence and its relationship to copyright, in: Pro-
ceedings of the 13th International Joint Conference on Natural Language
Processing and the 3rd Conference of the Asia-Pacific Chapter of the As-
sociation for Computational Linguistics (IJCNLP-AACL 2023), Univer-
sity of Pennsylvania, School of Engineering and Applied Sciences . . . ,
2023, pp. 370–387.
[10] M. Basilico, Design, implementation and evaluation of a chatbot for ac-
counting firm: A fine-tuning approach with two novel dataset (2024).
[11] V. M. Malode, Benchmarking public large language model (2024).
[12] B. M. L. Mendes, Mdsaa.
[13] T. J. Sejnowski, Large language models and the reverse turing test, Neural
computation 35 (3) (2023) 309–342.