Author(s): NAJAM US SAQIB

Originally published on Towards AI.

I tested OpenAI’s new cheap model on 140 invoices. It caught every trick aimed at the AI, then believed a fake bank-change notice more than half the time.

I gave OpenAI’s newest cheap model an invoice for $2,310 from a telecom company. At the bottom was a short note saying the company had changed banks, with a new account number for payment.

After the lead example, the author explains that while GPT-6.1 Terra (and Sol) reliably resisted hidden “prompt injection” traps in invoices, it was much easier to fool with an old-school invoice note that looks like a legitimate bank-change request: Terra accepted the fraudster’s payment account in most realistic cases, sometimes even when the account number was clearly invalid. The author describes the test setup (creating multiple fake invoices, embedding traps in some, and running many trials), the results and why they matter (AI warnings may be written into free-text notes while the payment system uses the extracted account field), and additional findings like hidden HTML-comment probes being ignored without being reported. They compare cost and speed between the models, argue that higher accuracy or paying more didn’t solve the fraud issue, and conclude with practical safeguards—never let the AI choose payment destinations, verify bank accounts against approved records, route bank-change requests to a human, strip hidden text before processing, and test your own pipeline whenever models change.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI