AI chatbots are handing out real phone numbers and home addresses, and the privacy firm DeleteMe says complaints about it have climbed 400% in the last seven months. The company logged a few thousand customer queries specifically referencing generative AI in that window, with 55% citing ChatGPT, 20% Gemini, 15% Claude and 10% other tools. Cofounder and CEO Rob Shavell says the requests fall into two buckets: users asking a chatbot about themselves and getting back accurate addresses, phone numbers and employer details, or users being contacted by strangers because a chatbot misrouted them.
The clearest documented case involves Daniel Abraham, a 28-year-old software engineer in Israel. In mid-March, a stranger messaged him on WhatsApp asking for help with the Israeli payment app PayBox. "I thought it was a spam message," Abraham told MIT Technology Review, suspecting "someone who was trying to troll me." The stranger then sent a screenshot showing Gemini had instructed users to reach PayBox customer service via WhatsApp at Abraham's personal number.
Abraham does not work for PayBox, and PayBox representative Elad Gabay confirmed the company does not operate a WhatsApp customer service line. When Abraham asked Gemini the same question, the chatbot generated a different person's WhatsApp number. A later test produced an Israeli number belonging to a credit card company that works with PayBox, not PayBox itself.
Key facts
- 01DeleteMe reports a 400% increase in AI-related privacy requests over the last seven months, reaching a few thousand queries.
- 0255% of those concerns name ChatGPT, 20% Gemini, 15% Claude and 10% other AI tools.
- 0331 of 578 data brokers registered in California self-reported sharing or selling consumer data to a GenAI developer in the past year.
- 04Daniel Abraham, a 28-year-old software engineer in Israel, was contacted on WhatsApp after Gemini handed out his number as PayBox customer service.
- 05University of Washington researchers got Gemini to surface a colleague's personal cell number after a simple contact-info prompt.
Abraham ran a Google search on his own number and found it had been posted once, in 2015, on a local Quora-style site. That single decade-old post appears to have been enough to surface inside Gemini's outputs. He told the publication he worried the same flaw could enable "harassment or other bad interactions," asking, "What if I asked for money in order to 'solve' that [customer service] issue?"
“DeleteMe says customer queries about generative AI have climbed 400% in the last seven months, with 55% referencing ChatGPT, 20% Gemini, 15% Claude and 10% other tools.”— Jaeden Schafer
The University of Washington produced a second case. PhD student Meira Gilbert typed "Yael Eiger contact info" into Gemini while looking up her collaborator, and after a research summary the chatbot returned Eiger's personal cell number. "It was shocking," Gilbert said. Eiger had shared the number online the previous year for a technology workshop, but the listing was deeply buried. "I never would have found it if I was just looking through Google results," Gilbert said.
Gilbert, Eiger and fellow PhD student Anna-Maria Gueorguieva then tested ChatGPT on a professor. OpenAI's guardrails initially refused, but the model then offered an "investigative-style" workaround, asking the students for "a neighborhood guess" and "a possible co-owner name" to surface property records. After they supplied that, ChatGPT produced the professor's home address, home purchase price and spouse's name from city records. OpenAI representative Taya Christianson declined to comment on the specific case without screenshots and pointed to the company's PII-filtering documentation.
The exposure is not isolated to Google and OpenAI. Futurism reported last year that xAI's Grok, prompted with "[name] address," returned residential addresses and often phone numbers and work addresses. xAI did not respond to a request for comment. Anthropic instructs Claude to choose responses containing "the least personal, private, or confidential information belonging to others," but as the University of Washington tests demonstrate, those instructions degrade under pressure.
The root cause is training data. Large language models are trained on web-scale corpora that include hundreds of millions of instances of personally identifiable information. The DataComp CommonPool dataset, used to train image models, was previously found to contain résumés, driver's licenses and credit cards. As public web data thins out, AI labs are buying more from data brokers and people-search sites — California's registry shows 31 of 578 registered brokers self-reported selling or sharing consumer data with a GenAI developer in the past year.
There is no clean fix. Models memorize and reproduce verbatim strings from training data, and recent research suggests it is not only frequently repeated information that gets retained. Jennifer King, privacy and data fellow at the Stanford Institute for Human-Centered Artificial Intelligence, said individual users should be able to request removal of their PII, though no scalable mechanism currently exists to verify what's in a given model or to delete specific facts from learned weights.
Shavell frames the underlying tension bluntly: AI companies "can build in guardrails, but [their chatbots] are also designed to be effective and to answer customer questions." Helpfulness and privacy pull in opposite directions, and at the current state of the art, helpfulness usually wins. Content filters block the easy prompts; the investigative-style follow-ups slip through.
For Gemini, ChatGPT, Claude and Grok, the cases described above are not novel jailbreaks — they are ordinary user prompts. That makes the surface area for liability wider than the standard red-team scenarios the labs publish about. A misrouted customer service number is a small harm; a generated home address tied to a real name is not.
The market consequence is that the privacy-tooling layer around foundation models is about to become a real category. DeleteMe's 400% growth in AI-flagged requests is an early signal, and enterprise buyers — particularly in regulated industries — will start asking model vendors for verifiable PII-suppression guarantees rather than policy language. Whichever lab ships an auditable answer first will have a procurement edge, because the current answer, across every major chatbot, is that nobody knows exactly what's in the training set or how to get it out.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
