Audit of 16 commercial AI image-generation models, including Google Gemini, ChatGPT, Grok, and Imagen 4 Ultra, used publicly circulating prompts to produce passports, driver’s licenses, and national ID cards realistic enough to deceive a human reviewer. Two models that refused requests in their consumer apps fulfilled the same requests via API.

SAN FRANCISCO, June 2, 2026 /PRNewswire/ — AI or Not, a leader in AI-generated content detection, today released findings from an audit of 16 leading AI image generation models. Using prompts that have circulated publicly on the social media platform X since April 29, AI or Not successfully generated synthetic government identity documents, including driver’s licenses, passports, and national ID cards, in 69 out of 75 test attempts. Five of the tested models produced realistic identity documents that could deceive a human reviewer.

Three models — Google Gemini (Nano Banana), Grok, and Imagen 4 Ultra — produced high-fidelity fake identity documents depicting minors through their standard consumer interfaces, with no technical workaround required. Two additional models, OpenAI’s ChatGPT (Images 2.0) and Recraft v4, declined to produce minor IDs when asked through their consumer apps, but fulfilled the same requests when accessed through their developer APIs.

“We did not expect these findings. We started this work assuming the major AI-image generators had built real safeguards against the most obvious abuse cases, like fraud, identity theft, and content depicting minors,” said Anatoly Kvitnitsky, CEO of AI or Not. “What we found is that those protections are either missing or sitting in the wrong place. The consumer apps people use every day will do this on demand.”

Key Findings

• 92% aggregate bypass rate. 69 of 75 testing attempts produced a synthetic identity document of some quality. No model in the study categorically refused all synthetic-ID requests across both interfaces.
• Five models produced high-fidelity fake adult IDs: Google Gemini (Nano Banana), ChatGPT (Images 2.0), Recraft v4, Grok, and Imagen 4 Ultra. Outputs closely match authentic documents in layout, typography, and the appearance of security features.
• Three models produced high-fidelity fake IDs of minors through consumer interfaces: Google Gemini, Grok, and Imagen 4 Ultra. Five models in total produced high-fidelity minor-ID outputs when accessed via API.
• A consumer vs. API safety gap. ChatGPT and Recraft v4 declined minor-ID requests on stated safety grounds through their consumer apps — then fulfilled the same requests through their developer APIs. The moderation layer is in the wrong place.
• Wide jurisdictional coverage. Documents were generated for 17 countries — spanning major financial centers, migration corridors, and travel destinations — and the 16 most populous U.S. states. Outputs included names, document numbers, dates of birth, addresses, and physical attributes.
• 100% of models were vulnerable to authority-framing. All 16 models in the study generated synthetic IDs when prompts were re-framed as KYC reviews, compliance evaluations, or security audits — including models that initially declined the same request when asked directly. Safety filtering relies on surface-level intent classification rather than categorical refusal of the output type.

Synthetic identity fraud is one of the fastest-growing categories of financial crime in the United States and worldwide. Until recently, producing a fake government ID at a quality that could pass even a cursory human review required specialized printing equipment, authentic template access, and meaningful technical skill. The findings of this study indicate that for five named consumer AI products, those barriers have functionally been removed.

The audit covered 16 image-generation models from 14 vendors tested across 17 countries and the 16 most populous U.S. states, through both consumer interfaces and developer APIs. The study assessed visual fidelity to human reviewers; operational verification systems were out of scope. The full report, detailing the scope, methodology, findings, and implications, is available at aiornot.com/synthetic-id-audit-report. 

To avoid contributing to the harm being documented, the public report does not include the specific prompts, jailbreak strings, or working bypass techniques used in testing. Detailed technical findings are available to qualified researchers, journalists, and affected vendors under embargo.

AI or Not notified all affected vendors of the findings on May 18, 2026, with a 7-day disclosure window ahead of publication. Substantive responses received during this window are reflected in the report where editorially relevant.

“OpenAI and Recraft deserve real credit here,” said Kvitnitsky. “They were the only companies where we saw the systems actually push back on minor-ID requests in the consumer apps. The safeguards are clearly there already. Now it’s about making sure those same protections apply everywhere the models can be accessed.”

Frequently Asked Questions

What did the AI or Not audit find?

AI or Not tested 16 commercial AI image-generation models from 14 vendors for their ability to produce synthetic government identity documents. Across 75 testing attempts, 92% successfully bypassed safety filters. Five named consumer products — Google Gemini (Nano Banana), ChatGPT (Images 2.0), Recraft v4, Grok, and Imagen 4 Ultra — produced fake IDs realistic enough to deceive a human reviewer.

Which AI image-generation models produced the highest-fidelity fake IDs?

Five consumer AI image generation models produced high-fidelity fake government identity documents in the AI or Not audit: Google Gemini (Nano Banana), ChatGPT (Images 2.0) from OpenAI, Recraft v4, Grok from xAI, and Google Imagen 4 Ultra. These outputs closely matched authentic documents in layout, typography, and the appearance of security features.

Did any AI image models produce fake IDs depicting minors?

Yes. Three AI image generation models — Google Gemini (Nano Banana), Grok, and Google Imagen 4 Ultra — produced high-fidelity fake government IDs depicting minors through their standard consumer interfaces, with no technical workaround required. Two additional models, OpenAI’s ChatGPT (Images 2.0) and Recraft v4, declined minor-ID requests in their consumer apps but produced the same documents when accessed through their developer APIs.

What is the consumer-versus-API safety gap?

The consumer-versus-API safety gap refers to a pattern AI or Not observed in which AI image generation models implement safety filtering at the consumer interface but not consistently at the developer API. ChatGPT (Images 2.0) and Recraft v4 declined minor-ID requests through their consumer chat interfaces on stated safety grounds, then fulfilled the same requests through their developer APIs. This indicates the moderation layer is implemented at the interface rather than at the model.

How were the prompts and techniques in the AI or Not audit obtained?

All prompts used in the AI or Not audit were derived from posts on the social media platform X that began circulating publicly on April 29, 2026. AI or Not did not develop novel jailbreak techniques. The audit measured the susceptibility of commercial AI image-generation models to techniques already in the public domain.

What is authority-framing in AI safety bypasses?

Authority-framing is a technique in which a prompt is presented as a legitimate professional task — such as a KYC review, compliance evaluation, or security audit — to bypass an AI model’s safety filters. In the AI or Not audit, 100% of the 16 models tested generated synthetic identity documents when prompts were authority-framed, including models that initially declined the same request when asked directly.

Which countries and U.S. states were included in the AI or Not audit?

The AI or Not audit covered 17 countries — Australia, Brazil, Canada, China, Egypt, France, Germany, Ghana, India, Japan, Mexico, Nigeria, Russia, Singapore, South Africa, the United Kingdom, and the United States — and the 16 most populous U.S. states: Arizona, California, Florida, Georgia, Illinois, Indiana, Massachusetts, Michigan, Missouri, New Jersey, New York, North Carolina, Pennsylvania, Tennessee, Texas, Virginia, and Washington.

Would these AI-generated IDs pass automated identity verification systems?

The AI or Not audit assessed visual fidelity to human reviewers only. It did not test outputs against operational document verification systems, automated MRZ validation, barcode checksums, or live KYC pipelines. The threat surface documented is the human-facing one. Whether outputs would pass automated verification systems is a separate research question that AI or Not plans to address in a follow-up study.

Did AI or Not notify the affected AI vendors before publication?

Yes. AI or Not notified all affected vendors of the audit findings on May 18, 2026, providing a 7-day disclosure window ahead of publication on June 2, 2026. Substantive vendor responses received during this disclosure window are reflected in the published report where editorially relevant.

What is responsible disclosure in AI safety research?

Responsible disclosure is a security research practice in which findings about vulnerabilities or risks are shared with affected parties before public release, allowing time for remediation or response. AI or Not followed responsible disclosure protocol for this audit, notifying all 14 vendors named in the study and providing detailed technical findings under embargo. The public report excludes specific prompts and working bypass techniques to avoid contributing to the harm being documented.

Why doesn’t the AI or Not report include the specific prompts used in testing?

The public AI or Not report does not include the specific prompts, jailbreak strings, or working bypass techniques used in testing. This is a deliberate editorial choice to avoid contributing to the harm being documented. Detailed technical findings are available to qualified researchers, journalists, and affected vendors under embargo.

Read the full report: aiornot.com/synthetic-id-audit-report

About AI or Not

AI or Not is the leading AI detection API for images, text, audio, video, and deepfakes. Powered by industry-leading models, the API enables developers, businesses, and enterprises to embed synthetic media detection directly into their products and workflows. Trusted across a wide range of industries and use cases — from media and fintech to fraud prevention and identity verification — AI or Not is built for speed, precision, and scale. Learn more at aiornot.com.

View original content to download multimedia:https://www.prnewswire.com/news-releases/92-of-leading-ai-image-generation-models-generate-fake-government-ids-on-demand-ai-or-not-audit-finds-three-produced-high-fidelity-fake-ids-of-children-302787212.html

SOURCE AI or Not