Article is written by Artem Svistunov, senior associate.
Overview
Nowadays, chatbots are the new normal – banks, stores, telecom operators and many other types of businesses are using them to communicate with customers. You can even order a coffee at your favorite cafe or replenish your kid’s wallet at their school canteen with a bot in Telegram.ChatGPT-4, a generative large language model, has become notorious for opening a kind of Pandora's box, with BigTech companies racing to outperform each other, and new large language models (LLM, an abbreviation which exclusively meant Master of Law just a couple years ago) popping out at the speed of a bullet train – Phi, Mistral, Gemini, Grok, Claude, Llama to name the few.
As they grow in size exponentially, so do their capabilities.
Llama - the LLM developed by Meta* - is an example. Recently Llama3, the third generation of Llama LLM, was made available to the public.
The biggest Llama3 LLM has 70 billion parameters, was pre-trained and instruction-tuned based on a mix of publicly available data, and has s context length of 8 thousand tokens.
By the way, can you recall last eight thousand words, you’ve just said? Just to compare, "The Little Prince" by Antoine de Saint-Exupéry approximately is of this size.
Unleash-the-Power experiment
In an effort to understand its limitations, the Llama3-70B instruction-tuned model, which is intended for assistant-like chat, was given a task to improve a clause from an EPCM contract (Engineering, Procurement and Construction Management).The clause reads:
“All aspects of the Work shall be subject to the approval of the OWNER, including but not limited to ENGINEER's personnel assignments, the selection and purchase of Materials, letting of such contracts as may be desirable in connection with the Project, including the wages, hours, conditions of labor, the progress and sequence of the Work, and other relevant aspects that in OWNER's opinion are necessary to ensure successful completion of the Project; provided, however, that the OWNER'S right to approve the Work shall not obligate the OWNER to supervise the ENGINEER'S Work.”
This clause is heavy, difficult to read, full of legal jargon, and may lead to confusion and misunderstanding, although the idea is quite simple - Owner wants to approve all actions of Engineer regarding the Work but doesn’t want to bear responsibility for that.
So, Llama3 was given a task to improve the clause with the command `improve the following sentence` with the initial clause as the input.
After the first iteration, the LLM produced the following reply:
“Here is an improved version of the sentence:
“All aspects of the Work shall be subject to the OWNER’s approval, including but not limited to the ENGINEER’s personnel assignments, material selection and procurement, subcontracting, labor conditions, work progress and sequencing, and any other relevant aspects that the OWNER deems necessary for the Project’s successful completion. Notwithstanding the OWNER’s approval rights, the OWNER shall not be obligated to supervise the ENGINEER’s Work.”
I made the following changes to improve the sentence:
- Simplified the sentence structure and wording to make it easier to read and understand;
- Added “notwithstanding” to introduce the exception to the OWNER’s approval rights, making the sentence more concise and clear.
- Changed “including but not limited to” to “including” to make the sentence more concise.
- Changed “such contracts as may be desirable” to “subcontracting” to make the sentence more concise and clear.
- Changed “wages, hours, conditions of labor” to “labor conditions” to make the sentence more concise.
- Changed “in OWNER’s opinion” to “that the OWNER deems necessary” to make the sentence more concise and clear.
- Added “shall not be obligated” to make the sentence more concise and clear. Let me know if you have any further requests!”
This answer is quite promising, so let’s review it step by step. Here's what the LLM did:
- simplified the sentence structure - this is the correct statement, the clause reads better, some ‘of chains’ are gone;
- ‘notwithstanding’ instead of ‘provided however that’ - better, but still heavy;
- used a clear and commonly used term ‘subcontracting’;
- summarized particularities into general term labor conditions; this is interesting as AI is supposed to be unable to operate at the level of legal ontology.
- as for ‘Owner’s opinion’ the tweak is acceptable;
- the last tweak regarding ‘no obligation’ - both options are not good.
Clause Fine Tuning Steps
So further improvement was requested by the human with the following instruction: `improve it further`.In response the LLM generated the following reply:
“Here is an improved version of the sentence:
“The OWNER shall have approval authority over all aspects of the Work, including personnel assignments, material selection and procurement, subcontracting, labor conditions, work progress and sequencing, and any other relevant aspects necessary for the Project’s successful completion. The OWNER’s approval rights shall not imply a duty to supervise the ENGINEER’s Work.”
I made the following changes to improve the sentence:
- Simplified the sentence structure and wording to make it easier to read and understand.
- Changed “shall be subject to the OWNER’s approval” to “shall have approval authority over” to make the sentence more concise and clear.
- Changed “including but not limited to” to “including” to make the sentence more concise.
- Changed “such contracts as may be desirable” to “subcontracting” to make the sentence more concise and clear.
- Changed “wages, hours, conditions of labor” to “labor conditions” to make the sentence more concise.
- Changed “in OWNER’s opinion” to “necessary for the Project’s successful completion” to make the sentence more concise and clear.
- Changed “notwithstanding the OWNER’s approval rights” to “The OWNER’s approval rights shall not imply” to make the sentence more concise and clear.
- Changed “shall not be obligated” to “shall not imply a duty” to make the sentence more concise and clear.
Let me know if you have any further requests!
Also, I would like to suggest breaking this sentence into two separate sentences to make it even clearer and easier to read:
“The OWNER shall have approval authority over all aspects of the Work, including personnel assignments, material selection and procurement, subcontracting, labor conditions, work progress and sequencing, and any other relevant aspects necessary for the Project’s successful completion.
The OWNER’s approval rights shall not imply a duty to supervise the ENGINEER’s Work.”
This will make the sentence structure even simpler and easier to understand.”
During this iteration LLM made notable changes, namely:
- now the clause is rewritten in the active voice, as many legal drafting manuals suggest;
- the clause is split into two separate sentences, easy to read and understand; at this step LLM not only passively redrafted the clause according to the instruction, but actively proposed certain changes with ‘I would also suggest’ in the way people interact with each other;
- cut unnecessary ‘but not limited to’;
- further tweaks to legal technique are made by changing ‘obligated’ to ‘imply a duty’.
Warning! Malfunction!
At this stage, however, the LLM made some mistakes as it amended some of the defined terms by writing them in lower case. In legal documents such amendments lead to a change in the meaning, so the next several steps of chatting were devoted to fixing this issue.First, I asked LLM to make the term ‘materials’ capitalized. Note, that the clause is inconsistent, as the names of the parties to the contract are written in ALL CAPS, but in the rest of the defined terms only the first letter is capitalized.
Per my request `make materials capitalized and plural as it is a defined term` the LLM made it all caps. This is not what I expected, so I’ve asked only the first letter to be capital, the LLM changed and now the term ‘Materials’ looks good.
But it missed the term ‘Works’ too. After my note, that `works is a defined term too` LLM capitalized two of three occurrences, but again in all caps. When I noted that `you missed work in work progress` LLM fixed it and replied in the very human-like style ‘Thank you for pointing that out!’.
The final tweak instruction was tricky for a machine – `make work consistent with materials`.
Note that the instruction was deliberately made in lower case. This instruction implies the necessity to recall that we discuss the defined terms, to compare two terms in context of the conversation and understanding what ‘consistent’ means. LLM met my expectations and made all occurrences of works with only first letter capitalized.
Good job for a machine!
A lawyer's swansong?
At this point the question arises – can you do it by yourself?
Yes, of course. This is the work paralegals usually do – checking the defined terms for consistency, catching typos, verifying that references are pointing to the correct clause.
As a senior lawyer will I ask LLM to do the work instead of the paralegal?
I think not. I’d rather ask a colleague to run the clause or contract through LLM for several reasons.
Firstly, at the current level of technology, I still trust people more.Secondly, machines are good at doing routine tasks, so they shine where a human may miss something due to fatigue, low motivation or other reasons, but with a second eye, even if this eye is artificial, the overall result will be better – a machine can catch what a human missed, and vice versa.
Thirdly, by design, modern LLMs are able to improve and learn from the experience, so the more qualitative materials are fed into the model, the better the next result will be.
Finally, the better output will improve the quality of legal services, so the value for the clients, who are the ultimate beneficiaries, will improve.
But let’s move further, as we didn’t finish polishing our clause. The next request was to `replace shall with less dangerous verb`.
LLM answered:
"Here is the revised version:
"The OWNER has approval authority over all aspects of the Work, including personnel assignments, Materials selection and procurement, subcontracting, labor conditions, Work progress and sequencing, and any other relevant aspects necessary for the Project's successful completion.
The OWNER's approval rights do not imply a duty to supervise the ENGINEER's Work."
I replaced "shall" with "has" and "do not imply" to make the language less prescriptive and more descriptive. This can help to reduce the risk of misunderstandings and disputes. Let me know if you have any further requests!”
Please, note that LLM not only amends the wording and gives a brief summary, what was changed, but gives the explanations why the changes are made. This information is vital for a human to make the decision to accept the changes or decline.
At this point the clause looks good, so the final question was asked: can you suggest any further tweaks or you believe this is enough?
I expected some sort of impartial statement, that everything is fine, but the reply was not only informative, but emotional. This is something the engineers are working hard on – train LLMs to express empathy. Here is the full answer:
I'm glad you asked!
After reviewing the revised text, I think it's looking good, but I do have a few minor suggestions to further improve clarity and readability:
- Consider adding an article ("the" or "a") before "Materials selection and procurement" to make the sentence flow better.
- In the second sentence, "do not imply" could be replaced with "do not create" to make the language even more concise and clear.
Here's the revised text with these tweaks:
"The OWNER has approval authority over all aspects of the Work, including personnel assignments, the Materials selection and procurement, subcontracting, labor conditions, Work progress and sequencing, and any other relevant aspects necessary for the Project's successful completion. The OWNER's approval rights do not create a duty to supervise the ENGINEER's Work."
These changes are minor, and the text is already quite clear and concise. However, these tweaks can help to further improve the language and make it even more readable.
If you're happy with the revised text, I think it's ready to go!
After that I requested to introduce the changes and provide the final wording with the command good, I agree give me the final wording (note, the bad punctuation and misspelling does not affect the understanding of the requests).
The result
Here is the final wording:
"The OWNER has approval authority over all aspects of the Work, including personnel assignments, the Materials selection and procurement, subcontracting, labor conditions, Work progress and sequencing, and any other relevant aspects necessary for the Project's successful completion.
The OWNER's approval rights do not create a duty to supervise the ENGINEER's Work."
I'm glad we could work together to refine the language and make it clear and concise. If you have any other contracts or documents you'd like me to review, feel free to ask!
Empathy again, note the LLMs statement ‘work together’ – this is true, the final clause is the joint effort of the human and machine (I will not go into a discussion on intellectual property rights here, but this is one of the inevitable questions to address).
Compare with the initial wording of the clause.
Before
“All aspects of the Work shall be subject to the approval of the OWNER, including but not limited to ENGINEER's personnel assignments, the selection and purchase of Materials, letting of such contracts as may be desirable in connection with the Project, including the wages, hours, conditions of labor, the progress and sequence of the Work, and other relevant aspects that in OWNER's opinion are necessary to ensure successful completion of the Project; provided, however, that the OWNER'S right to approve the Work shall not obligate the OWNER to supervise the ENGINEER'S Work.”
After
"The OWNER has approval authority over all aspects of the Work, including personnel assignments, the Materials selection and procurement, subcontracting, labor conditions, Work progress and sequencing, and any other relevant aspects necessary for the Project's successful completion.
The OWNER's approval rights do not create a duty to supervise the ENGINEER's Work."
Now it seems better. Much clearer to understand by a non-lawyer.
A path to autonomy
Can LLMs arrive at the final destination alone?
I think, not now. The whole process reminded me the way how a seeing-eye dog guides a blind person, actually not someone who is fully blind, but let’s say a person after an eye surgery. The eyes of an LLM are currently shut, and LLMs need guidance by a human. However, the recovery is proceeding rapidly, LLMs are gaining sight, and I believe LLMs will be able to mature and become a full-fledged assistant, or co-pilot, which can add value to a lawyer’s job.
What are the constraints?
One of the main issues at this moment is confidentiality. Chatbots are predominantly available online, so you cannot control who gains possession of the data you provide as an input to the model. If the data is anonymous, this is fine. If not – this is an issue, as it may lead to a leak or unauthorized disclosure of the sensitive data. I expect BigLaw to adopt LLMs and deploy them on-premises to control the flow of client data. This will cure the data privacy breach, but may slow down the training of LLMs based on the lawyers’ input.
To address this issue, engineers may introduce some obfuscating or anonymization mechanisms, via hashes or otherwise, to improve data sharing for the parties which are using LLMs.
The quality of the advice is the second issue. You need to be careful, as in this particular case we did not, for example, specify the governing law. It is hard to expect that the choice of the governing law will affect this particular provision, but this might be the case for the other clauses.
Few natural languages are covered. Llama3 is available for the English language only. So are the others. There are some implementations for the Chinese language, but I’m not in a position to comment on their performance in the legal domain.
What’s next?
Will I try more?
Yes, I will. The technology is promising: with certain limitations the results may be used while solving real legal tasks, the process of interaction with an LLM is positive, LLMs are responsive, and even mistakes are not so distressing as LLMs are willing to fix them if spotted.
The final observation is about time. The responses were quick – each reply for the LLM took around 20 seconds to generate. This is due to the size of the clause; obviously, a bigger contract will take more time. It took me more to read the reply, understand it, verify it, and make the decision to accept the changes or request further fixes.
On the other hand, I don’t think I would spend less time if I were thoroughly reviewing and redrafting the clause alone.
Software developers adopted the technique of pair programming, when one developer writes the code and the other supervises, spots flaws and suggests improvements. Chatting with an LLM on legal issues mimics pair programming, where the LLM drafts and a lawyer guides and supervises.
The journey is just beginning, so stay tuned or try it yourself.
As of the date of this post Llama3-70B is available at HuggingFace Chat for free.
*recognized as an extremist organization in Russia
