Shadow AI in Healthcare: What Your Clinical and Administrative Staff Are Using - And What You Don't Know About It

In December 2025, the Manage My Health patient portal was breached. Hackers used stolen patient credentials to access and copy documents from nearly 100,000 patient accounts - one of New Zealand's largest known privacy breaches. The Privacy Commissioner's Phase 1 inquiry, released in May 2026, found that both Manage My Health and Health NZ had breached Rule 5 of the Health Information Privacy Code. The failures were not primarily technical. They were governance failures: poor risk assessments, overreliance on vendor assurances, weak contractual controls, and the absence of privacy and security specialists in the project design process.

The Commissioner called it "a call to action for the health sector, and New Zealand organisations generally, to improve cybersecurity controls and governance."

Shadow AI isn't what caused the Manage My Health breach. But the governance failures the Commissioner identified - the same pattern of inadequate oversight, insufficient controls, and decisions made without specialist input - are precisely the conditions under which shadow AI creates its own category of risk in NZ healthcare organisations.

In Paubox's 2025 research, 85% of healthcare IT leaders suspected staff were using unauthorized AI tools, and only 26% had real visibility into that use. More than two-thirds had already found unsanctioned AI somewhere in their organisation.

The tools are already in your building. The question is whether anyone's governing them.

Why Healthcare Staff Are Turning to AI - With or Without Approval

Healthcare in New Zealand is operating under sustained pressure. Clinical staff are managing higher patient volumes with teams that haven't kept pace. Administrative workloads have grown alongside documentation requirements, compliance obligations, and reporting demands. Burnout is a documented reality across the sector.

Into that environment, AI tools have arrived - and healthcare staff have found them useful in ways that make intuitive sense. A GP who can produce a clinical note in half the time has more time for patients. A practice manager who can draft a patient communication in minutes rather than an hour has more capacity for the work that requires their specific judgement. An administrator who can summarise a referral letter has more bandwidth for the calls that need a human response.

In 2025, shadow AI surged across healthcare organisations, as staff across all aspects of care sought ways to improve efficiency amid persistent burnout, staffing shortages, and other factors.

More than 40% of medical workers and administrators said they were aware of colleagues using shadow AI tools, while nearly 20% reported using an unauthorised AI tool themselves. Those figures almost certainly understate the real position - the stigma of non-compliance means self-reported usage is lower than actual usage.

The problem isn't that healthcare staff want to use AI. The problem is that AI tools have entered healthcare workflows before governance frameworks have caught up - and in an environment where the information being processed is among the most sensitive that exists, that gap has specific consequences.

What Shadow AI Looks Like in a NZ Healthcare Setting

Shadow AI in healthcare doesn't look like a deliberate policy breach. It looks like this:

A GP using ChatGPT to draft a clinical note after a complex consultation, entering the patient's presenting symptoms, history, and clinical findings to generate a structured summary. The note is reviewed before it goes into the patient record. The patient's health information has already been entered into an external AI system with no data processing agreement with the practice.

A practice nurse using a consumer AI transcription tool to record and summarise a patient consultation. The transcript includes the patient's name, date of birth, symptoms, medications, and personal circumstances. It's stored on a US-based server. The practice has no visibility of this and no agreement governing that data.

An administrator using a public AI tool to draft a referral letter based on clinical notes they've summarised. They enter the patient's name, diagnosis, and relevant clinical history to help structure the letter. The draft is reviewed by the GP. The patient's health information is in an external system.

A practice manager using an AI writing tool to respond to a patient complaint. They paste the complaint and relevant background into the tool to help frame a response. The complaint contains the patient's account of their care, including sensitive personal and clinical details.

A clinical team using an AI meeting summariser added to Teams to capture notes from a multidisciplinary team meeting. The meeting discusses patient cases by name, including diagnoses, treatment plans, and clinical disagreements. The full transcript is stored in a system the organisation doesn't control.

A staff member using Copilot features embedded in Microsoft 365 that were activated through a recent update - without realising those features are now processing information from their inbox and documents, including patient correspondence.

In every case, the intent was efficiency. In every case, patient health information moved outside the organisation's governed environment without a deliberate decision having been made.

Where AI Is Genuinely Creating Value in Healthcare

The opportunity AI presents for healthcare is real. The goal is not to prevent AI adoption - it's to ensure that adoption happens in a way the organisation can stand behind.

Clinical documentation - AI-assisted note drafting and consultation summarisation are among the highest-value applications in clinical settings. The productivity gain is substantial and the benefit to clinician-patient time is direct. The critical requirement is that the AI tools used for this work have appropriate data processing agreements and operate within the organisation's governed environment.

Patient communication drafting - Drafting routine patient correspondence - appointment reminders, referral letters, discharge summaries, follow-up instructions - is a legitimate time-saving application. The correspondence still goes through clinical review before it's sent.

Administrative summarisation - Summarising referral letters, clinical histories, and correspondence for administrative purposes reduces handling time without necessarily requiring clinical judgement. Within a governed environment, this is low-risk and operationally valuable.

Microsoft Copilot within the organisation's environment - For healthcare organisations on Microsoft 365 Business Premium, Microsoft Copilot provides AI capability within the organisation's existing data governance structure. It doesn't send patient information to external AI models. It works within the organisation's existing permissions and access controls. For practices and organisations that haven't yet activated it properly, this is the governed alternative that already exists within their licence.

Research and clinical decision support - AI tools trained on clinical evidence and guidelines can support literature review, clinical protocol development, and evidence summarisation for internal use. The critical requirement is that these tools are used to inform clinical judgement, not replace it - and that the outputs are reviewed by a qualified clinician before being relied upon.

The consistent principle: AI that operates within the organisation's governed data environment is fundamentally different from AI that processes patient information through external systems. The patient outcome is similar. The data governance risk is not.

The Specific Risks for NZ Healthcare Organisations

The Manage My Health inquiry made explicit what the Privacy Commissioner has been signalling for some time: healthcare organisations in New Zealand face serious consequences when patient information isn't adequately protected. Shadow AI is a direct route to exactly that position.

Health Information Privacy Code 2020 - Rule 5 of the Health Information Privacy Code requires that healthcare organisations maintain reasonable security safeguards for the health information they hold. Using AI tools to process patient information without adequate data governance is a potential breach of that rule - the same rule that both Manage My Health and Health NZ were found to have breached. The Commissioner's inquiry makes clear that the standard of "reasonable safeguards" is being enforced, not just referenced.

Privacy Act 2020 obligations - Beyond the Health Information Privacy Code, the Privacy Act 2020 governs how all personal information is handled. Healthcare organisations hold some of the most sensitive personal information that exists. Notifiable privacy breach obligations apply when a breach is likely to cause serious harm - and patient health information being processed through an external AI system without consent or appropriate controls meets that threshold.

Patient consent - Patients in New Zealand have rights around how their health information is used and disclosed. Using AI tools to process patient information for purposes beyond direct care - particularly where that information leaves the healthcare organisation's environment - raises questions about whether the original consent for collection covers those uses.

Clinical risk from AI errors - AI tools are confident even when they're wrong. In a clinical documentation context, an AI-generated note that misrepresents a patient's symptoms, omits a relevant detail, or introduces an inaccuracy that isn't caught in review creates a patient safety risk that sits alongside the governance risk. The note becomes part of the patient record and may influence future clinical decisions.

Compliance notice exposure - The Privacy Commissioner's findings in the Manage My Health inquiry resulted in compliance notices - described as "the strongest tool" currently available. The Commissioner also recommended amending the Privacy Act to create direct liability for third-party providers who fail to maintain adequate security safeguards. For healthcare organisations whose staff are using third-party AI tools to process patient information, this regulatory direction is directly relevant.

The sector-wide signal - The Commissioner used the Manage My Health findings to call for structural reform across the entire NZ health sector - recommending a centralised verification process for health portals and stronger accountability frameworks. "With the waning public trust of health technology following three significant breaches in 2026 so far, this demands a sector-wide response if we want to confidently keep New Zealanders' health data safe." Healthcare organisations that aren't actively addressing governance gaps are operating against a backdrop of increasing regulatory attention.

The Visibility Problem

85% of healthcare IT leaders suspect staff are using unauthorised AI tools. Only 26% have real visibility into that use.

Between what leadership suspects and what it can actually see - is where shadow AI risk accumulates. The tools are in use. Patient information is being processed. The organisation doesn't know which tools, with which data, under which terms.

AI features have also been progressively embedded into software healthcare organisations already use - practice management systems, electronic health record platforms, Microsoft 365 - often activated by default through product updates without triggering any formal review. The result is AI processing happening across the organisation's environment without any governance decision having been made.

For smaller NZ healthcare organisations - GP practices, allied health providers, specialist clinics, community health organisations - the challenge is that the governance infrastructure that larger hospital systems have invested in doesn't typically exist at the practice level. The Privacy Act obligations are the same. The tools being used are the same. The visibility is often lower.

Visibility comes before policy. Understand what's actually happening - which AI tools are in use, what patient information is being processed through them, where the configuration gaps sit in the organisation's Microsoft 365 environment - before building policy around what you wish was happening.

What Good AI Governance Looks Like for Healthcare Organisations

Healthcare organisations that are approaching AI governance well share a consistent pattern: they start with visibility, not policy.

They find out what AI tools are actually in use - not by asking staff to declare tools they might assume aren't approved, but by gaining technical visibility into the applications connecting to their environment. They review their Microsoft 365 configuration to understand which AI features are active and whether data governance settings are appropriate for a healthcare context. They identify which tools have data processing agreements compatible with the Health Information Privacy Code and which don't.

From that foundation:

An approved tool list that distinguishes between tools appropriate for clinical information, tools appropriate for administrative information, and tools that are not appropriate for any patient information. Clear categories reduce the decision burden on staff who are making these choices under time pressure.

A simple, readable AI policy that addresses the specific scenarios healthcare staff encounter - note drafting, patient communications, referral summarisation - with clear guidance on what patient information can and can't be used with which tools. Practical enough that clinical staff under time pressure will actually follow it.

Staff education specific to healthcare - The Health Information Privacy Code, patient consent obligations, and the specific risks of AI errors in a clinical context are not addressed by generic AI awareness training. Healthcare staff need guidance that reflects the specific obligations and risks of their environment.

Microsoft Copilot as the governed alternative - For organisations on Microsoft 365 Business Premium, Copilot provides AI capability within the organisation's existing data governance structure - without patient information leaving the organisation's controlled environment. Positioning this as the available, approved alternative reduces the incentive to reach for public AI tools.

Clinical oversight of AI outputs - A clear standard that AI-generated clinical content - notes, summaries, documentation - requires review by a qualified clinician before it enters the patient record or is acted upon. Not an optional step, a required one.

Privacy and security specialist involvement - The Manage My Health inquiry found that the absence of privacy and security specialists in decision-making was a contributing factor in the breach. For healthcare organisations building AI governance, that specialist input - whether internal or through a provider - is part of what makes governance genuine rather than nominal.

Gaining Visibility Before Building Policy

If your organisation hasn't yet taken a structured approach to AI governance in a healthcare context, the starting point is understanding what's already happening.

Most NZ healthcare organisations that begin this process are surprised by the scope of what they find - AI tools in use across clinical and administrative teams, embedded AI features in software that were activated without any governance decision, and a gap between what leadership assumes is happening and what's actually occurring.

NSP's approach starts with exactly that visibility - mapping the current state of AI adoption across the organisation's Microsoft 365 environment, identifying shadow AI exposure, and reviewing configuration and governance posture against the specific obligations that apply in a healthcare context. For healthcare organisations, that exercise surfaces the Health Information Privacy Code, Privacy Act, and patient consent implications that generic AI governance frameworks don't account for.

For organisations ready to build a more comprehensive programme, NSP's Secure AI Accelerator provides a structured approach covering AI enablement, security, governance, and ongoing optimisation - with executive reporting that gives leadership a documented, evidenced picture of AI governance maturity. In a healthcare context, that documentation is also part of what the Privacy Commissioner is increasingly looking for as evidence of "reasonable safeguards."

The NZ healthcare sector is under more regulatory scrutiny around data governance than it has been at any point since the Privacy Act 2020 came into force. The Manage My Health inquiry has made the consequences of inadequate governance concrete and public. The healthcare organisations that respond to that signal - by understanding their current AI exposure and building governance that reflects it - are the ones that will be in a defensible position when the next question is asked.

If you're not sure how much AI is already in use across your organisation's Microsoft 365 environment, that's the question worth starting with. NSP can help you understand your current shadow AI exposure and governance readiness before you begin building policies or wider AI initiatives.

Talk to NSP about AI governance for your healthcare organisation →

Or call us: 0508 010 101

Frequently Asked Questions

What is shadow AI in healthcare? Shadow AI in healthcare refers to AI tools being used by clinical and administrative staff - GPs, nurses, practice managers, administrators - without formal approval or oversight from the organisation's leadership or IT function. This includes public AI tools used for clinical note drafting, patient communication, referral summarisation, and administrative tasks, as well as AI features embedded in existing software activated without a formal review. In healthcare, shadow AI is particularly significant because of the Health Information Privacy Code 2020, Privacy Act obligations, patient consent requirements, and the clinical safety risks of AI errors in patient documentation.

What are the risks of shadow AI for NZ healthcare organisations? The primary risks include patient health information being processed through external AI systems without adequate data governance - a potential breach of Rule 5 of the Health Information Privacy Code, the same rule at the centre of the Manage My Health Privacy Commissioner findings. Additional risks include Privacy Act notifiable breach obligations, patient consent questions, clinical safety risks from AI errors in patient documentation, and increasing regulatory scrutiny following the sector-wide response to NZ's 2026 healthcare data breaches.

Does the Manage My Health breach relate to shadow AI? Not directly - the Manage My Health breach involved stolen credentials used to access a patient portal. However, the governance failures identified in the Privacy Commissioner's inquiry - inadequate risk assessment, overreliance on vendor assurances, absence of privacy and security specialists - are the same conditions that allow shadow AI to create unmanaged risk in healthcare organisations. The inquiry's findings about what "reasonable safeguards" require are directly relevant to how healthcare organisations should approach AI governance.

What should NZ healthcare organisations do about shadow AI? Start with visibility - understand what AI tools are actually in use across clinical and administrative teams, including AI features embedded in existing practice management and Microsoft 365 software. From that foundation, develop a practical AI policy specific to healthcare contexts, build an approved tool list that distinguishes between tools appropriate for different types of information, and implement Microsoft Copilot as a governed AI alternative within the organisation's existing environment.

Is Microsoft Copilot safe for use with patient information? Microsoft Copilot, when properly configured within a Microsoft 365 Business Premium or higher environment, operates within the organisation's existing data governance structure and doesn't send information to external AI models. This makes it fundamentally different from public AI tools in terms of data governance risk for patient information. However, appropriate configuration, data governance settings, and staff training are still required - Copilot's safety depends on how the Microsoft 365 environment itself is configured, which is why a governance assessment is the recommended starting point.

Continue Reading

• Shadow AI in Legal Firms: What You Don't Know Is Happening
• Shadow AI in Accounting Firms: What You Don't Know Is Happening
• Shadow AI in Construction Firms: What You Don't Know Is Happening
• How Do I Know If My Business Has Been Breached?
• The Security Baseline Every NZ Business Needs Before Buying Cyber Insurance
• What Is Microsoft Entra ID and Why Does Every NZ Business Need to Understand It?
• NSP Secure AI Accelerator
• AI Governance - NSP
• Cybersecurity Assessments - NSP