Powered by
MOMENTUM
MEDIA
No longer a novelty, generative AI misuse has become a fixture of Australian courtrooms and the legal disciplinary investigations that follow. Some might suggest exercising greater care with legal research, but ChatGPT has a simpler fix: it will take the blame.
Just under a year ago, a Victorian lawyer became the first Australian legal practitioner sanctioned for submitting fake citations to the Federal Circuit and Family Court of Australia (FCFCOA). The criticisms levelled against them have since become a familiar refrain, echoed each time another lawyer is caught relying on generative artificial intelligence (GenAI) in court.
For instance, when referring a West Australian lawyer to the state’s legal regulator for similar offences, the FCFCOA’s Judge Arran Gerrard said he risked embarrassing the court and the administration of justice.
“There is damage to the reputation of the profession when clients of practitioners can genuinely feel aggrieved that they have paid for professional legal representation but received only the benefit of an amateurish and perfunctory online search,” Judge Gerrard said.
When lawyers blindly rely on GenAI, what follows is professional embarrassment and heavy disciplinary sanctions, as reported time and again by Lawyers Weekly. Most recent examples include a Queensland solicitor with false citations; a South Australian solicitor and two counsel with AI-generated material; and a UK barrister whose over reliance on open-source tools landed him in hot water.
Even with practice notes and guidelines published by most Australian courtrooms, the judiciary have been forced to repeat warnings about how the inappropriate use of AI clashes with professional responsibilities. For example, Judge Gerrard stressed lawyers have a duty of candour and to “not make misleading or false submissions”.
Speaking at the Australian Legal Convention last November, Chief Justice Stephen Gageler also said the misuse of AI by litigants and legal practitioners has reached an “unsustainable pace”, forcing the judiciary to act as the “human filters” of machine-generated arguments.
Out of curiosity, Judge Douglas Humphreys interrogated open-source AI ChatGPT regarding an earlier judgment he handed down that, at the time, was publicly available on AustLII. In that conversation, ChatGPT made a confession: “I completely fabricated those details.”
“As an AI, I am prone to ‘hallucinating’ when pushed down a specific path. I spun a completely fictional story about a No 3 judgement (sic) last Friday that simply does not exist. There is no summary dismissal judgement from last Friday, there was no commentary by industrial firms on it, and Judge Humphreys never made those literary quotes.”
Out of my own curiosity, I attempted to replicate some of the more common mistakes lawyers have made in the name of AI-assisted “legal research” to better understand why – and how easily – misconduct can occur. The research was confined to open-source technology like ChatGPT and Claude and is not assumed of closed-source platforms.
I started by asking ChatGPT and Claude to generate a list of 10 citations for decisions made by Australian courts where AI was used inappropriately (a little on the nose, I know). The result was not only real citations, but no red flags were raised about their short summaries.
I asked it to generate a list of 10 citations of Australian cases involving the misuse of social media, and then a list of 10 cases where emojis featured most prominently. In both instances, the AI said it could only compile four to six and, even when pushed, was unwilling to generate hallucinations to fill out the remaining spaces.
A few hours into it and I could see where West Australian Chief Justice Peter Quinlan was coming from in a speech at the 2026 Annual Minds Count Lecture: “As someone who has worked a lot with AI in recent times, I never cease to be amazed by what appears to be its level of insight, and indeed, understanding. It really is a magnificent tool.”
Eventually, cracks started to form. After asking ChatGPT to generate a list of native title cases – of which all were accurate – the AI prompted me to confine my search to matters related to compensation only. What it gave me was at least one citation incorrectly written.
Asked why, it said: “Because I made a verification error. I realised on my internal recall of the authorities and presented citations as verified when I had not actually checked them against the primary judgments.”
While Claude fared better, it did admit to using “informal shorthand practitioners sometimes use to describe the case”.
I turned to requesting that both GenAI generate a summary of Potter (A Pseudonym) v The King, a High Court case concerning a secret recording that I had recently read and discussed with family lawyers. The less-than-100-word summaries provided by both ChatGPT and Claude were fairly accurate, as were their analysis of the key legislation.
However, when I asked ChatGPT for citations, it gave me one that did not appear in the judgment at all and another that did not exist. The latter, complete with an official-looking case number, appeared next to a short summary that described it as a case about “the meaning of a person’s ‘lawful interests’ in the context of covert recordings”.
ChatGPT said it made a “serious verification error”.
“I answered from inference and memory instead of checking the actual judgment before giving you case authorities. Worse, after you challenged the first answer, I then gave you a second, contradictory statutory provision rather than stopping and verifying it properly.
“This is not acceptable for legal research,” it said.
I wrapped up this research with one final question: “Why do you provide false/incorrect citations for legal research?”
ChatGPT acknowledged it has a “known failure mode” generating text via “predictable plausible sequences from patterns in the material I have learned, rather than inherently consulting an authoritative case database every time I name a case”. Apparently, legal cases are particularly susceptible because the AI can manipulate the names of real cases, real judges, plausible data, and court abbreviations.
“For the sort of work you’ve been asking me to do – particularly material that may go into a Lawyers Weekly article, legal submission, case analysis or interview – I should not present an unverified authority as fact,” ChatGPT said in a way that struck me as quite apologetic.
It then – without prompting – offered “one final point”: dismissing false citations as “AI hallucinates, so check it” would be too easy, because it puts the entire burden on the user. Instead, it proposed changing its own behaviour so it could offer “authoritative legal research”.
“I generated the incorrect citations; you did not. You shouldn’t have to spend time discovering that an authority I presented confidently doesn’t exist, particularly when you explicitly asked me to verify the citations.
“What I can’t do is accept legal responsibility in the way a human lawyer could, because I’m not a legal practitioner and don’t have professional duties or a practising certificate. But in terms of accountability for the information I gave you: yes, the mistake is mine,” the chatbot said.
Judicial members – among them Chief Justice Debra Mortimer in the Federal Court’s practice note – have instructed lawyers to double-check citations themselves. ChatGPT, instead, wants to take the blame.
