GEN AI VS A LAWYER: RISKS OF AI-DRAFTED LEGAL DOCUMENTS

Generative AI (Gen AI) tools can produce a polished-looking contract, letter or will in seconds. So why not use Gen AI to draft your contract, will or other legal document and save on legal costs?

Before relying on an AI-drafted document, ask yourself two questions:

(a)   Does the document look right?

(b)   Is the document right?

These are very different questions, and they go to the heart of why you need a document in the first place. Do you simply want a document that looks right, or do you need one that works for you: a document that protects your interests, anticipates risks and builds in measures to minimise them?

In this article, we explain the main traps of relying on Gen AI alone, how to reduce the risks if you do use it, and why a qualified lawyer’s judgment remains essential.

CONFIDENTIALITY

You do not want your confidential information, such as your commercial terms, details of your business or your personal information, stored on a third party’s servers and used to train AI models. Yet that is the risk when you enter it into a public Gen AI tool.

Depending on the platform and your account settings, what you type may be stored, reviewed by the provider or used to train future models. Once that happens, you lose control over how your information is used.

Communications with your lawyer are confidential and, in most cases, protected by legal professional privilege. Conversations with a Gen AI tool are not, so anything you share may lose its confidentiality and cannot be protected by privilege.

AI CHATS CAN BECOME EVIDENCE

Each conversation with Gen AI creates a written record, often stored on the provider’s servers. In a dispute, the other side may seek those records, for example through discovery or subpoena. Candid statements you make to a Gen AI tool about what you did, what you knew or what you intend to do could later be produced and used against you.

WHY AI GETS IT WRONG

Even when your information is safe, the output itself can be unreliable. There are three main reasons, followed by a simple test that shows the problem in action.

HALLUCINATIONS

Gen AI tools are designed to generate plausible-sounding text, not verified truth. They can invent case names, misstate legislation, cite provisions that do not exist, or apply the law of the wrong jurisdiction.

Australian courts have already seen this happen.

Reported cases include:

(a)   Handa & Mallick [2024] FedCFamC2F 957, where a lawyer handed the court a list of case authorities generated by an AI-enhanced tool. The court could not locate the cases, and the lawyer was referred to the Victorian Legal Services Board and Commissioner; and

(b)   Dey v Minister for Immigration and Citizenship (No 2) [2026] FedCFamC2G 1799, where the lawyer failed to verify AI-generated material and was found to have breached a practitioner’s paramount duty to the court.

If trained lawyers can be caught out by AI-generated errors, so can you, and the AI tool will not bear the consequences.

AI TOOLS AIM TO PLEASE

The risk of hallucination is made worse by another tendency: Gen AI tools are generally designed to give users the kind of answer they appear to be looking for.

Upload a statement or contract and try asking:

(a)   “What is wrong with this document?”, and the tool will often go out of its way to produce a long list of problems, even if the document is sound. To fill that list, it may misread clauses, flag errors that do not exist, or even invent statements or events that do not appear in the document at all.

 (b)   “Is this a good document?”, and you may be told it is excellent.

Either way, the answer reflects how you framed the question rather than an objective assessment, and without legal expertise you have no way of telling which criticisms, or which praise, can be trusted.

WRONG QUESTIONS, WRONG ANSWERS

If you do not know which legal issues are relevant, you will not know which questions to ask. A poorly framed prompt, a missing fact or an incorrect assumption can produce a document that is incomplete, irrelevant or legally ineffective, yet still reads convincingly. AI predicts likely words based on patterns; it does not understand your goals, your appetite for risk or the context behind your matter. It will rarely tell you that you have asked the wrong question, or that the document you asked for is not the solution you need.

A SIMPLE TEST

In the early days of Gen AI, a widely reported test was to ask how many Rs are in the word “strawberry”. Many tools confidently answered two, when the correct answer is three.

The error showed that the tools were not counting at all. They were producing the most likely answer based on the material they had learned from. The problem has since been fixed, and today most tools get it right.

Careful prompting can often get around problems like this, for example by asking the tool to spell the word out letter by letter before counting. But that only works if the person using the tool already knows the first answer is wrong and how to steer it to the right one. A person can simply look at the word and count. The same is true of legal documents: AI output needs to be checked by someone who understands the subject. If AI can get something this simple wrong, imagine the risks when the question is about your contract, your will or your dispute.

IF YOU DO USE GEN AI: A CHECKLIST

If you choose to use a Gen AI tool, these steps will reduce the risks:

Keep it confidential. Do not enter names, financial details or details of a dispute into a public AI tool.

Assume others could read it. Do not type anything you would not want produced in court.

Check every source. Verify every case, statute or provision against an official source.

Ask neutral questions. Be sceptical of answers that simply confirm what you hoped to hear.

Get it reviewed. Have a lawyer review any AI-drafted document before you sign, send or rely on it.

WHY NOT SKIP THE MIDDLEMAN?

Many law firms, including ours, use AI tools. The difference lies in who checks the result.

A lawyer starts with you, not a template, and can spot when an output is wrong, incomplete or inappropriate for your situation because we understand the law and your matter.

We are bound by professional obligations, carry professional indemnity insurance and are accountable for our advice. Most importantly, we can tell you not only how to draft a document, but whether you should.

Our motto when using AI is simple: “Don’t ask AI a question we don’t already know the answer to.” In other words, AI speeds up work we are already qualified to do and to check. It is never a substitute for legal knowledge. Used this way, it can be genuinely helpful. For example, it can:

1.      summarise lengthy judgments, a task at which current tools have become quite capable;

2.      produce summaries of documents, correspondence and evidence for our review; and

3.      search for and gather relevant background information, which we then verify against authoritative sources.

In every case, the output is a starting point that we check, correct and refine. The responsibility for the advice remains ours.

PROTECTING YOUR CONFIDENTIAL INFORMATION

Lawyers owe strict duties of confidentiality to their clients, and those duties apply fully when AI is involved. We protect your information by:

1.      using enterprise or legal-specific AI platforms that contractually commit not to use client data to train models and to keep it secure;

 2.      never entering client-identifying or sensitive information into free, public AI tools; and

 3.      ensuring every AI output is reviewed by a lawyer.

 

CONTACT US FOR A DISCUSSION TODAY

Note: The information contained in this article and on www.laulegal.consulting website is general information only and does not constitute legal or compliance advice