A corporate AI chatbot can leak confidential information without anyone hacking the chatbot -but a thoughtfully & skillfully built Enterprise AI chatbot will not.
An employee asks a normal question. The chatbot searches company files, CRM records or support tickets. It retrieves a document the employee was never supposed to see and includes part of it in the answer.
Technically, the chatbot worked.
From a security and compliance perspective, it failed.
This is becoming a serious enterprise problem. Verizon’s 2026 Data Breach Investigations Report analyzed more than 31,000 security incidents and over 22,000 confirmed breaches across 145 countries. It found that regular employee use of AI tools on corporate devices increased from 15% to 45% in one year. Of users accessing AI services, 67% were using non-corporate accounts. Source code was the most common type of information submitted to external generative AI tools in Verizon’s data loss prevention dataset.
The lesson for companies building custom AI chatbots is simple. Data privacy cannot be added after the chatbot works. It has to determine how the chatbot is designed.
The Numbers Show Why Enterprise AI Governance Can No Longer Be Optional
IBM’s 2025 Cost of a Data Breach research studied breaches at 600 organizations globally. Thirteen percent reported breaches involving AI models or applications, while another 8% did not know whether their AI systems had been compromised.
Among organizations that experienced an AI-related security incident, 97% lacked proper AI access controls. IBM also found that 63% of breached organizations either had no AI governance policy or were still developing one. Among organizations that did have governance policies, only 34% regularly audited for unsanctioned AI use.
The financial impact was also measurable. Organizations with high levels of shadow AI experienced breach costs averaging $670,000 more than organizations with little or no shadow AI. Incidents involving shadow AI exposed personally identifiable information in 65% of cases and intellectual property in 40%, compared with global averages of 53% and 33% respectively.
For US companies, the consequences can be particularly expensive. IBM reported that the average US data breach cost reached $10.22 million in 2025, compared with a global average of $4.44 million. Healthcare breaches averaged $7.42 million globally.
These figures are not an argument against corporate AI. They are an argument against deploying AI before deciding exactly what it can access.
Start With the Question Most AI Projects Skip: What Data Should the Chatbot Never See?
Many enterprise chatbot projects begin by asking which LLM to use, which vector database to deploy and whether to build with RAG.
The first question should be different.
What information should this chatbot never receive?
Consider an HR assistant. It might need an employee’s department, employment type and length of service to answer a benefits question. It does not need the employee’s bank details, Social Security number, medical records or complete HR file.
A sales assistant may need account history and approved pricing information. It may not need executive email threads, legal correspondence or confidential acquisition discussions.
This is data minimization in practice. The application retrieves only the information needed for the specific task rather than passing complete records to the model.
For organizations regulated by HIPAA, this principle has legal significance. HHS’s minimum necessary guidance generally requires regulated entities to make reasonable efforts to limit PHI uses, disclosures and requests to the minimum necessary for the intended purpose.
For an AI application, that principle should influence the retrieval architecture before information ever enters a model context.
Authentication Alone Does Not Protect Corporate Data
Logging employees into a chatbot with Microsoft Entra ID, Okta or another SSO provider is useful, but authentication only establishes who the user is.
The next question is what that person is allowed to retrieve.
Suppose a company indexes 500,000 SharePoint documents for a corporate knowledge chatbot. A sales representative, HR director and CFO all authenticate successfully.
They still should not receive identical search results.
If the CFO asks about a confidential financial forecast, the chatbot may legitimately retrieve it. If a sales representative asks a question that happens to match the same document semantically, the retrieval system must block it before the document reaches the LLM.
The safest pattern is to carry existing authorization into the AI retrieval layer. Document permissions, tenant boundaries, department restrictions, user groups and data classifications should influence retrieval before generation occurs.
A chatbot should not become a new way around permissions that already exist elsewhere in the company.
RAG Can Still Leak Data
Retrieval augmented generation is often described as a safer alternative to training a model on corporate data. It can certainly provide more control because information can be retrieved when needed instead of incorporated into model training.
It does not automatically create privacy.
If permissions are removed when documents enter the vector database, confidential information may become searchable by everyone.
If multiple customers share a vector store without strong tenant separation, one customer’s information could potentially appear in another customer’s retrieval results.
If sensitive metadata is embedded along with normal content, the retrieval system may expose information developers did not realize was searchable.
Permission information should therefore travel with the indexed content. Retrieval filters should verify user, tenant, department, classification and other relevant access rules before sending documents to the language model.
Prompt Injection Is a Data Security Problem, Not Just a Bad Prompt Problem
OWASP places prompt injection among the major security risks for LLM applications because instructions can enter a system through users or through content the system retrieves.
Consider a corporate chatbot that reads uploaded PDFs.
A malicious document could contain instructions attempting to make the model ignore its previous rules, retrieve additional information or execute an available tool.
The dangerous design is one where the model itself decides whether those actions are permitted.
Security decisions should be enforced outside the LLM. The application should determine which databases can be queried, which functions can be called, which records can be returned and which actions require additional authorization.
A system prompt that says “never reveal confidential information” is useful guidance for the model. It is not a replacement for access control.
NIST’s guidance for generative AI systems also recommends managing risks throughout the AI lifecycle, including development, deployment, monitoring and third-party dependencies rather than treating AI security as a one-time review.
The Vendor Chain Matters More Than Many Companies Realize
An enterprise chatbot rarely consists of one vendor.
A production system may involve a cloud provider, LLM provider, vector database, observability platform, authentication provider, document storage system, analytics service and development partner.
That matters because Verizon’s 2026 DBIR found third parties were involved in 48% of breaches, a 60% increase from the previous report.
Before sending corporate information to an AI provider, companies should establish whether submitted data is used for model training, how long prompts and responses are retained, whether zero-retention options exist, which subprocessors receive data, where information is processed, how deletion works and what happens to stored information when the contract ends.
The FTC has specifically warned AI providers that promises about privacy, confidentiality and whether customer data is used for model training must be honored.
Vendor terms are therefore part of the application’s security design, not simply paperwork for procurement.
HIPAA AI Chatbots Require More Than Encryption
HIPAA does not apply to every health-related chatbot. It applies to covered entities and business associates under the HIPAA Rules.
When a cloud provider creates, receives, maintains or transmits electronic protected health information on behalf of a covered entity or business associate, HHS states that the provider generally becomes a business associate and an appropriate Business Associate Agreement is required.
A particularly important HHS clarification is that encryption alone does not remove this responsibility. A cloud provider maintaining encrypted ePHI can still be a business associate even if it does not possess the encryption key.
For a healthcare AI chatbot, companies therefore need to examine the entire chain handling ePHI, including the application, model service, storage systems, logging systems and subcontractors.
HHS enforcement also shows the practical consequences of weak security controls. In April 2026, the Office for Civil Rights announced four HIPAA ransomware settlements involving incidents that collectively affected more than 427,000 people. HHS specifically emphasized risk analysis requirements and the need for administrative, physical and technical safeguards.
SOC 2 Does Not Automatically Make an AI Chatbot Compliant
SOC 2 is useful, but it is frequently misunderstood.
It is not a US privacy law and it is not an AI compliance certificate.
SOC 2 reports evaluate controls at service organizations using AICPA Trust Services Criteria related to security, availability, processing integrity, confidentiality and privacy.
A model provider having a SOC 2 report can provide valuable evidence during vendor assessment. It does not automatically make your chatbot compliant.
Your own organization still controls employee permissions, application architecture, identity management, logging, data retention, deployment, change management, incident response and integrations.
For enterprise buyers, asking “Do you have SOC 2?” is useful.
Asking “Which controls protect my data in this particular AI workflow?” is much better.
California Privacy Requirements Can Create Real Financial Exposure
Companies serving California users also need to consider the CCPA when applicable.
As of 2026, the CCPA generally applies to qualifying for-profit businesses doing business in California that meet at least one applicable threshold, including annual gross revenue above $26.625 million, buying, selling or sharing personal information of 100,000 or more consumers or households, or deriving at least 50% of annual revenue from selling or sharing personal information.
The current adjusted statutory damages range for certain qualifying data breaches is $107 to $799 per consumer per incident, or actual damages if greater. Administrative fines can reach $2,663 per violation and $7,988 for intentional violations and certain violations involving children under 16.
Those numbers become meaningful quickly when an AI application stores thousands or millions of conversations.
California enforcement is also active. In May 2026, California announced a $12.75 million privacy settlement with General Motors involving allegations around the sale of location and driving data. While the case was not an AI chatbot case, it demonstrates the financial importance of data minimization and controlling secondary uses of personal information.
Logging Can Become a Second Data Leak
Corporate AI systems need logs. Security teams need to know who accessed the system, what tool was called, what failed and what happened before an incident.
The problem begins when companies log every full prompt, every retrieved document and every response indefinitely.
At that point the monitoring platform can become a second database containing employee conversations, customer information, source code and proprietary documents.
A better design records the information necessary for security and troubleshooting while masking unnecessary sensitive values. Access to logs should be restricted, retention periods should be defined and high-risk environments should maintain audit trails showing who accessed sensitive resources.
What a Secure Enterprise AI Chatbot Should Actually Enforce?
A practical enterprise architecture should authenticate the user first, then apply authorization before retrieving corporate information. Retrieval should preserve source-system permissions and tenant isolation. Sensitive fields should be filtered or masked when they are not required. Only the necessary context should be sent to the LLM.
The model should operate with restricted tools and permissions rather than broad database access. Actions such as deleting information, modifying records, sending payments or changing account permissions should pass through deterministic authorization and may require human approval.
Outputs should also be checked where necessary for sensitive information. Logs should record enough information for incident investigation without storing unlimited copies of confidential prompts and documents.
Finally, security teams should regularly test the system using unauthorized retrieval attempts, prompt injection scenarios, cross-tenant requests and malicious documents.
The goal is not to make the LLM responsible for protecting corporate data.
The goal is to build an application where the LLM never receives data or permissions it should not have.
Final Thoughts
The biggest enterprise AI privacy mistake is treating the language model as the entire application.
The real system includes identity, permissions, APIs, databases, retrieval pipelines, vector storage, model providers, logs, cloud infrastructure and third-party vendors. A weakness anywhere in that chain can expose information even when the model itself performs exactly as designed.
The current numbers make the risk difficult to dismiss. Regular corporate AI use has tripled according to Verizon. IBM found that 97% of organizations suffering AI-related security incidents lacked proper AI access controls. Shadow AI increased average breach costs by $670,000 in IBM’s study, while the average US breach cost reached $10.22 million.
Companies should therefore build enterprise AI chatbots with the same discipline used for other systems handling valuable corporate data. That means least-privilege access, permission-aware retrieval, data minimization, controlled logging, vendor review, incident response and regulatory analysis before sensitive information begins flowing through the system.
At Supreme Technologies, we believe the useful question is not simply whether a company can connect its internal information to an LLM. Modern tools make that relatively easy.
The important question is whether the company can do it without accidentally giving the wrong employee, customer, model provider or attacker access to information they were never supposed to see.
References
- Verizon 2026 Data Breach Investigations Report
https://www.verizon.com/business/resources/reports/dbir/ - IBM 2025 Cost of a Data Breach Report
https://www.ibm.com/reports/data-breach - IBM Report on AI Model and Application Breaches
https://newsroom.ibm.com/2025-07-30-ibm-report-13-of-organizations-reported-breaches-of-ai-models-or-applications - NIST AI Risk Management Framework: Generative AI Profile
https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf - NIST Secure Software Development Practices for Generative AI
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218A.pdf - HHS Guidance on HIPAA and Cloud Computing
https://www.hhs.gov/hipaa/for-professionals/special–topics/health-information-technology/cloud-computing/index.html - HHS Guidance on Encrypted ePHI and Business Associates
https://www.hhs.gov/hipaa/for-professionals/faq/2076/if-a-csp-stores-only-encrypted-ephi-and-does-not-have-a-decryption-key-is-it-a-hipaa-business-associate/index.html - California Privacy Protection Agency CCPA FAQ
https://cppa.ca.gov/faq - California Privacy Protection Agency Updated CCPA Monetary Thresholds
https://cppa.ca.gov/regulations/cpi_adjustment.html - California Attorney General 2026 General Motors Privacy Settlement
https://oag.ca.gov/node/622991
