ChatGPT accounts posing as law firms and attorneys banned by OpenAI

The leading American AI research and deployment company confirmed it banned a cluster of ChatGPT accounts tied to a covert operation, which it has dubbed “False Witness”.

The action follows OpenAI’s discovery that the group used its AI model to generate deceptive legal content, craft convincing fraudulent messages, build credible-looking online profiles, and ultimately steer victims into paying bogus fees.

In a report update published last month, OpenAI revealed that the activity appears to have “very likely” originated in Cambodia, echoing patterns seen in recent reports on sophisticated Chinese-led criminal networks operating in the region.

At the heart of the scheme, scammers exploited ChatGPT to create content for at least six fake law firms, each claiming to specialise in recovering victims’ lost funds.

OpenAI noted that “multiple” open-source red flags exposed the firms as fraudulent, including mismatched web domains and the routine funnelling of visitors to private messaging apps rather than legitimate office channels.

Adding to the deception, in some cases, the same fabricated attorney identities were recycled and repurposed across multiple sham firms.

VIEW ALL

The report detailed how the scammers relied on ChatGPT to support multiple stages of their fraudulent workflow, including crafting private messages designed to “emulate the tone and professionalism” of legitimate lawyers, often sent directly to targets via encrypted messaging platforms such as Telegram.

Prompts submitted to the model included requests to write in “American English” or “in the style of a lawyer”.

In these exchanges, OpenAI explained that the scammers falsely claimed they were operating under the supervision of the International Criminal Court and assured victims that no fees would be payable until lost funds were recovered.

In reality, the scheme followed a familiar pattern: victims were pressed to pay upfront, including a 15 per cent “service fee”, deposits to “activate” accounts, and additional consultation charges, despite assurances to the contrary.

Messages sent to victims, drafted using ChatGPT, also included detailed instructions for cryptocurrency transfers and requests for screenshots of transaction confirmations as proof of payment.

Some accounts went further, fabricating supporting documents such as fake attorney registration records, counterfeit bar association membership cards, and sham confidentiality agreements designed to discourage victims from seeking outside advice.

Assessing the full impact of the operation is difficult, with OpenAI warning that much of the available evidence is drawn from the “scammers’ own inputs” into its systems.

Even so, these inputs indicated that individual victims may have lost thousands of dollars, although the company stressed it could not independently verify the figures.

The company reiterated that its policies “strictly prohibit” using its tools to commit fraud or scams, emphasising its commitment to working with industry peers and authorities to monitor how AI is being exploited and to actively disrupt operations that abuse its services.

The FBI and a real law firm whose identity was impersonated have also issued public warnings, highlighting how the scam preys on vulnerable individuals – particularly elderly victims of prior fraud – exploiting their distress and urgency to recover lost funds.