What information are you giving AI — and what happens when it goes wrong?

When business owners think about data breaches, they often picture a criminal breaking into a computer network. Many privacy incidents are less dramatic. A customer email is pasted into a public chatbot. A spreadsheet containing names and financial information is uploaded for analysis. A meeting recorder captures a confidential conversation. A browser extension is given access to a system containing customer records.

No attacker is involved. Nobody intended to cause harm. Information may still have left the business's control.

Start with the information, not the tool

The first question isn't whether the employee used ChatGPT, Copilot, Gemini or another product. The first question is: what information did they provide? This might include names and contact details, customer correspondence, employee records, financial information, identity documents, health information, photographs and recordings, details about a person's family or circumstances, complaints and legal matters, commercially confidential information, or passwords and security credentials.

Some information may be commercially sensitive without being personal information. That can still create a serious security, contractual or reputational problem. Where personal information is involved, privacy obligations may also apply.

Entering information into AI may be a disclosure

It is easy to think that information hasn't been shared because it was entered into software rather than sent to another person. That distinction may not hold. The OAIC says entering personal information into an AI system may constitute either a use of the information, if it remains within the organisation's effective control, or a disclosure, if it becomes accessible outside the organisation and is released from its effective control. That depends on the system, provider, contractual terms, account and settings.

For organisations covered by the Privacy Act, using or disclosing personal information must comply with the Australian Privacy Principles. The OAIC recommends, as a matter of best practice, that organisations do not enter personal information — and particularly sensitive information — into publicly available generative AI tools because of the significant and complex privacy risks. Read the OAIC's commercial AI guidance.

Removing someone's name may reduce the risk, but it doesn't automatically make information anonymous. A person may remain reasonably identifiable from their job, location, circumstances, transaction details or a combination of other information.

A privacy incident is not automatically a notifiable breach

Australia's Notifiable Data Breaches scheme does not require every error involving personal information to be reported publicly. An eligible data breach occurs when all three of the following apply:

1. Personal information has been lost, accessed without authorisation or disclosed without authorisation.
2. The incident is likely to result in serious harm to one or more individuals.
3. The organisation has not been able to prevent the likely risk of serious harm through remedial action.

If those conditions are met, a covered organisation must notify the affected individuals and the OAIC. See the OAIC's explanation of when a breach is reportable.

This means an accidental AI upload may be inappropriate use that should be corrected, a breach of an internal policy, a contractual or confidentiality issue, a security or privacy incident requiring investigation, or an eligible data breach requiring notification. The business should not make that decision based on instinct or on whether the employee meant any harm.

What does "serious harm" mean?

Serious harm can include identity theft, financial loss through fraud, physical harm, serious psychological harm, or serious reputational harm. Whether serious harm is likely depends on the circumstances.

Relevant factors may include the type and sensitivity of the information, whether several kinds of information were combined, whether the information was protected or encrypted, who may have obtained access, how the recipient could use it, whether it can be recovered or deleted, the vulnerability of the people affected, and the possible consequences of misuse.

Uploading a list of first names is different from uploading identity documents, bank details or information about a person experiencing financial hardship. The number of people affected matters, but a breach involving one person can still be serious.

The 30-day rule is an assessment limit

If a covered organisation suspects an eligible data breach, it must conduct a reasonable and prompt assessment. It must take all reasonable steps to complete that assessment within 30 calendar days of becoming aware of the grounds for suspicion.

The OAIC describes 30 days as a maximum, not a standard waiting period. It expects organisations to complete the assessment sooner wherever possible because the risk of harm can increase with time. Once the organisation has reasonable grounds to believe an eligible data breach occurred, it must promptly notify the OAIC and the individuals at risk of serious harm, unless an exception applies. Read the OAIC's Notifiable Data Breaches guidance.

This is one reason employees need a clear reporting process. The business cannot assess an incident it doesn't know about.

Which businesses are covered?

The Notifiable Data Breaches scheme applies to entities with relevant security obligations under the Privacy Act. This includes many businesses and not-for-profit organisations with annual turnover above $3 million. Some businesses below that threshold are also covered, including certain private health service providers, credit providers and credit reporting bodies, businesses that trade in personal information, and tax file number recipients.

Other obligations may arise under state or territory laws, industry rules, contracts or professional duties. A small business should not assume it is exempt without checking its particular circumstances.

What should happen after an AI privacy incident?

Employees should be instructed to report the incident immediately. They should not attempt to hide it, recreate it, contact affected customers independently or make assurances about what happened. The business response should include the following steps.

1. Contain the incident

Where possible: stop further use of the tool, remove its access to connected systems, preserve the account and relevant records, check whether uploaded information can be deleted, contact the provider through an appropriate support or security channel, and prevent the same information from being uploaded again. Containment should not destroy evidence needed to understand the incident.

2. Establish what happened

Record the tool and account used, the date and time, the prompts entered, files or information uploaded, whether the tool was connected to other systems, relevant account and retention settings, responses received from the provider, and people whose information may be affected. This information will be needed to assess the incident and seek advice.

3. Escalate internally

The incident should reach the person responsible for privacy or security immediately. Depending on its nature, the business may also need its IT or cyber security provider, privacy or legal adviser, insurance broker or insurer, relevant senior manager, or communications adviser. The policy should name these contacts before an incident occurs.

4. Assess the risk of harm

Determine whether personal information was involved, whether access or disclosure was unauthorised, who may have accessed the information, whether serious harm is likely, whether prompt remedial action can prevent that harm, and whether any contractual or regulatory notification requirements apply. Document both the assessment and the reasons for its conclusion.

5. Notify where required

If the incident meets the eligible data breach threshold, covered organisations must notify the OAIC and affected individuals. Other notification obligations may also apply — a contract, professional body, industry regulator or insurance policy may require notification even when the federal NDB threshold is not met. Specialist advice may be necessary. This article is not a substitute for a proper assessment of an actual incident.

Remedial action matters

An incident does not always proceed inevitably to notification. If the business acts quickly and successfully prevents the likely risk of serious harm, notification under the NDB scheme may not be required. For example, it may be possible to confirm that information was securely deleted before it could be accessed or used. Whether that is sufficient depends on reliable evidence and the circumstances.

Where insurance fits

Cyber insurance may assist with some costs arising from a privacy or security incident, but coverage varies considerably. A policy may provide access to incident-response specialists, forensic investigation, privacy and legal advice, notification services, credit or identity monitoring for affected people, public relations support, regulatory investigation costs, business interruption cover, and defence of certain third-party claims. It should not be assumed that every AI-related incident is covered.

Businesses should ask their broker or insurer:

• Does the policy cover employee-caused disclosures through AI tools?
• What about an incident involving a third-party AI provider?
• Are unapproved tools or personal accounts excluded?
• Is cover affected if the business has no AI usage policy?
• How quickly must the insurer be notified?
• Must the insurer approve legal, forensic or communications providers?
• Are regulatory investigations and notification costs included?
• Does the policy cover inaccurate AI output as well as data disclosure?
• What documentation or security controls does the insurer expect?

The Australian Cyber Security Centre recommends including insurance details and notification requirements in an organisation's incident-response plan. It also warns that cyber insurance is not a replacement for cyber security controls and cannot repair every consequence, including lost trust or compromised intellectual property. Read the ACSC's incident-response guidance.

Contacting the insurer should be part of the response plan, not something the business remembers after engaging several advisers and making public statements.

An AI policy is part of the control — not a legal shield

A policy cannot guarantee that an incident won't happen. It also doesn't automatically protect the business from liability. It can establish approved tools, permitted uses, prohibited information, required privacy and security settings, human review requirements, a clear internal reporting process, named responsibility, staff training, and evidence that the issue was actively managed.

A document nobody reads provides little protection. The policy must be explained using examples relevant to the business. Staff should know the difference between public, internal, confidential, personal and sensitive information. They should also know that promptly reporting a mistake is far better than attempting to conceal it. Technical controls may also be appropriate, particularly where the business handles health, financial, identity or other sensitive information.

Most workplace AI use will not cause a reportable data breach. But the ease of these tools can disguise what is happening to the information entered into them. Copying customer information into an AI chat window can feel informal even though it may involve disclosure to another organisation. Businesses need to make that consequence visible before an incident occurs.

An AI policy gives employees boundaries. Training helps them apply those boundaries. An incident-response process tells everyone what to do when something still goes wrong. None of those controls eliminates risk. Together, they give the business a much better chance of recognising and managing it.

This article provides general business information and is not legal, privacy or insurance advice. Privacy obligations and insurance coverage depend on the organisation, circumstances and applicable policy wording.