Te Whatu Ora Advice about the Use of Artificial Intelligence in Healthcare

The use of artificial intelligence (AI) technology in New Zealand is still at a relatively early stage, reflected by the absence of any standard as yet promulgated by MCNZ.

In August 2023, Te Whatu Ora published Advice on the use of large language models and Generative AI in Healthcare. [1] Te Whatu Ora advised:

Te Whatu Ora DOES NOT ENDORSE the use of LLMs [Large Language Models] or Generative AI tools where non-public information is used to train the model or used within the context of the model.

... Te Whatu Ora employees and contractors:

Must NOT:

• Enter any personal, confidential or sensitive patient or organisational data into LLMs or Generative AI tools

• See these LLM or Generative AI tools for any clinical decision, or any personalised patient-related documentation, or for personalised advice to patients

If Generative AI tools or LLMs are used for any other purpose, Te Whatu Ora employees and contractors are fully responsible for checking the information generated.

The reason for this precautionary advice is that in the New Zealand healthcare context, available LLM and Generative AI tools have not been validated as safe and effective for use, nor have the risks and benefits of such tools been adequately evaluated. Such risks include breaches of privacy, inaccuracy of outputs such as in diagnosis or patient management, and lack of processing transparency. The privacy commissioner has also flagged potential security issues with AI tools leaking sensitive information.

While such tools may be safe to use in the future, for now Te Whatu Ora's advice needs to be followed.

Te Whatu Ora has established an advisory group called the national artificial intelligence and algorithm expert advisory group (NAIAEAG). If you are considering using ai in your medical practice, it is recommended you first contact NAIAEAG to register your plans for potential use and to obtain advice about appropriate processes, and whether the use of the tool you are considering is safe and appropriate (https://forms.office.com/r/5cemkdANdU).

It provides the following definitions: "Generative AI is a type of ai technology powered by very large machine learning models (algorithms) that are pre-trained on vast amounts of data to produce various types of content, including text, imagery, audio and synthetic data (computer code)". Examples include ChatGPT and medpalm2. "LLMs are a type of Generative AI that have been specifically designed to help generate text-based content. LLM deep learning algorithms are pre-trained on massively large data sets of written human language and textual data to summarise, generate and predict new text-based content.

Dated 11 December 2023.