Shadow AI: what the Privacy Act expects when staff use generative AI tools

In most organisations, generative AI arrived through the side door. Staff discovered that a public chatbot could summarise a long email chain, draft a customer letter or tidy up meeting notes, and started using it. Some used personal accounts. Few asked whether the tool kept what they typed, who could see it, or where it was processed.

This is what is often called shadow AI: the use of AI tools that the organisation has not assessed or approved. It is not a new category of law. The Privacy Act 1988 and the Australian Privacy Principles (APPs) already apply to it, and in October 2024 the Office of the Australian Information Commissioner (OAIC) explained how.

Two pieces of OAIC guidance

On 21 October 2024 the OAIC published two guides. The first, on privacy and the use of commercially available AI products, is aimed at organisations that deploy AI tools built by others, including freely available products such as publicly accessible chatbots. It was updated on 17 January 2025. The second, on developing and training generative AI models, is aimed at developers but is useful context for anyone fine-tuning a model or supplying data to one.

Both are regulator guidance. They are not new legal obligations, but they set out how the OAIC interprets the existing APPs and how it is likely to approach a complaint or investigation. The central message of the first guide is straightforward: privacy obligations apply to personal information entered into an AI system and to output that contains personal information.

What the guidance says about public tools

The clearest statement in the guidance is a best-practice recommendation. The OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools, given the significant and complex privacy risks involved.

That recommendation goes to the heart of shadow AI. A staff member who pastes a customer complaint, a medical certificate or a performance review into a public chatbot may be disclosing personal information to a third party on terms the organisation has never read. Sensitive information, such as health information, attracts stricter rules under the Act, which is why the OAIC singles it out.

The APPs that do the work

APP 6: use and disclosure. Personal information entered into an AI system can only be used or disclosed for the primary purpose for which it was collected, unless an exception applies. The main exceptions are consent, or a secondary purpose the individual would reasonably expect and that is related to the primary purpose (directly related, for sensitive information). The guidance also asks organisations to check whether the provider’s terms allow it to access information entered into or generated by the system, and if so whether that can be switched off or controlled.

APP 10: accuracy. Generative AI can produce outputs that sound credible and are wrong. Organisations must take reasonable steps to ensure the personal information they use and disclose is accurate. The OAIC points to human oversight, treating outputs as probabilistic rather than fact, recording when information came from AI, and ongoing monitoring for errors and bias.

APP 1: transparency and governance. APP 1 requires reasonable steps to put in place practices, procedures and systems that ensure compliance, and a clearly expressed, up to date privacy policy. The guidance recommends internal AI policies that define permitted and prohibited uses, staff training, audits or monitoring of how tools are used, and privacy policies and notices that explain the organisation’s use of AI. From 10 December 2026, APP 1 will also require privacy policies to describe certain automated decisions that could significantly affect individuals.

APP 3 and APP 5: generated information. If an AI tool generates or infers personal information about an identifiable person, the OAIC treats that as a collection under APP 3. The collection must be reasonably necessary for the organisation’s functions or activities and made by lawful and fair means, and sensitive information generally requires consent. The OAIC is explicit that inferred, incorrect or artificially generated information, including hallucinations and deepfakes, can still be personal information. APP 5 notice obligations follow.

The guidance also flags security under APP 11 and cross-border disclosure under APP 8, since many tools are cloud hosted and may process data outside Australia.

Due diligence is not set and forget

The OAIC frames tool selection as due diligence. Before adopting a product, organisations should understand whether it has been tested for the intended use, what data it was trained on, its security posture, how it will be integrated, and whether inputs are accessible to the developer. The guidance includes two checklists: one for selecting an AI product and one for using it.

The OAIC is equally clear that due diligence should not be a one-off exercise. Models and their training data age, products change their terms and features, and staff find new uses. Review needs to continue for the life of the tool.

A practical governance approach

Banning AI outright tends to push use further out of sight. A more durable approach gives staff a sanctioned path and makes the rules easy to follow.

An approved tools list. Name the tools staff may use and the account type for each, such as an enterprise licence rather than a personal account. Make clear that anything not on the list is not approved for work information.

Acceptable use rules. Set out in plain language what information can go into each tool. A simple tiering works well: public information, internal information that is not personal, personal information, and sensitive information. Most public tools should sit at the first or second tier only, consistent with the OAIC’s best-practice position.

Training. Staff need to understand how the tools work, where they fail, how to check outputs, and why entering personal information matters. The OAIC lists training among its expected governance measures.

A register. Record each approved tool and use case, its owner, the categories of information involved, the supplier terms reviewed and the controls in place. A register makes review possible and gives the privacy officer a starting point when something goes wrong. It will also help with the automated decision-making disclosures that start in December 2026.

Supplier terms. Review contracts and settings for each tool: whether inputs are retained or used to train models, who at the provider can access them, where data is stored and processed, how incidents are notified and what happens at the end of the contract. The OAIC’s guidance asks organisations to carefully review the terms and settings that apply to their use of a product.

Where a proposed use involves personal information in a new way, the OAIC recommends a privacy by design approach and notes that a privacy impact assessment will help the organisation understand the privacy impacts and how to manage them.

Where to start

The first step is visibility. Ask teams which tools they use and for what. The answers usually show a small number of high-value uses that can be supported properly, and a handful of risky habits that can be stopped with clear rules and a better alternative. Governance that starts from how people actually work is far more likely to hold.

General information, not legal advice. Confirm how the Privacy Act and the OAIC’s guidance apply to your organisation with your legal adviser.