In a perspective article published today in Nature Machine Intelligence, researchers at Rensselaer Polytechnic Institute (RPI) Drs. Ge Wang, Pingkun Yan, and Chang Niu presented “Medical Technology and AI (MeTAI)” in the metaverse that promises to develop new intelligent health care. This represents a multidisciplinary collaboration among academic and clinical researchers with University of Chicago, Johns Hopkins University, Stony Brook University, industrial leaders from GE Healthcare and Canon Medical Research, and regulatory experts at the FDA and Puente Solutions.

The metaverse has been a major focus of public interest over recent years. It is often misrepresented as merely an extension of computer games and social media. In essence, the metaverse integrates physical and virtual realities, enabling humans and their avatars to live and work in such a hybrid environment supported by innovative technologies.

The metaverse is the confluence of rapid and profound technical and sociological developments. It has the potential to encompass many disciplines of research and revolutionize many aspects of life. In addition to entertainment and social networking, the metaverse applications supported by MeTAI can enable advanced medical routines including interventional procedures with unprecedented experience.

In the article, researchers at RPI and their coauthors have described the MeTAI that can facilitate development, prototyping, evaluation, regulation, translation, and refinement of AI-based medical practice, especially medical imaging-guided diagnosis and therapy. The MeTAI ecosystem is exemplified with the four major health care applications:

• Virtual comparative scanning to find the best imaging technology in a specific situation (multiple imaging exams can be simulated to see which one gives the result that is the closest to the ground truth used in the simulation);
• Raw tomographic data sharing to allow controlled open access for research and diagnosis (raw tomographic data simulated on virtual scanners of digital twins can be publicly shared after de-identification with patient’s consent);
• Augmented regulatory science to extend virtual clinical trials in terms of scope and duration (which would be a much-advanced version of the virtual clinical trial);
• “Metaversed” medical intervention to perform interventional procedures aided by the metaverse (including remote robotic surgery, radiotherapy, and so on, guided by human and AI agents in the metaverse).

According to Dr. Ge Wang, professor of biomedical engineering at RPI, who is the first author of the paper, MeTAI in the metaverse needs to integrate the full capabilities of supporting technologies including high-speed internet involving 5G/6G, VR/AR/MR/XR, digital twins, haptics, holography, secure computation, and AI on a massive social and economic scale, enabling people to interact with each other and also their avatars, AI agents/algorithms, and medical devices/facilities. MeTAI in the metaverse is expected to improve health care quality, accessibility, cost-effectiveness, and patient satisfaction in novel ways.

“We envision the benefits of a medical metaverse is analogous to, but much more than, those provided by computer-aided design software in aerospace system engineering, where digital avatars of aircraft and spacecraft are thoroughly evaluated before instrumentation in the real world,” explained Dr. Wang. “We are confident that MeTAI will eventually become a reality and synergize various isolated medical systems into a unified health care infrastructure of real and virtual high dimensions. As metaverse technologies are constantly developed, novel aspects of the metaverse will keep emerging to redefine biomedicine and society.”