Augmented reality in surgery

Augmented reality images result from the superimposition of virtual and real anatomy of the patient (A-C and D-F). Augmented reality allows visualisation of intra-abdominal structures through virtual transparency of the abdominal wall (C) and recognition of important surgical anatomy through the retroperitoneal fat (F: adrenal gland and main adrenal vein seen through the periadrenal fat tissue). CT indicates computed tomography. Reproduced with permission (JAMA 2004;292:2214-5)

Many people have heard of virtual reality but fewer will have heard of “augmented reality”—an exciting technology that might bring huge benefits to the operating theatre.

In many ways augmented reality is analogous to virtual reality. It is a combination of virtual and graphic three dimensional images transmitted to a user. However, augmented reality differs from virtual reality in that it involves overlaying these graphic images onto real life, engaging the user in a semi-immersive, interactive, three dimensional environment. By superimposing virtual images, videos, or text onto real life, an experience can be heightened or even modified.

For many years the military has used this technology in areas such as aircraft navigation. Augmented reality also has applications in medicine and surgery, including in surgical training, preoperative planning, intraoperative imaging, and enhanced visualisation.

One of the driving forces that led to the development of augmented reality in surgery was the need for a headset to track the subtle movements of the surgeon. Previously, surgeons were limited to having constantly to alter their field of vision by looking away from the operating site and comparing this to a static graphic scan or representation. Augmented reality allows them to maintain a fixed field of vision on the surgical site while having graphic scans augmented over the top for reference or guidance.

Achieving augmented reality

This may sound great in theory, but how can augmented reality be used in the operating theatre? The technology must maintain a full view of a real time situation while superimposing virtual images. This can be achieved in many ways, but some of the first developments used displays mounted on the head. Video head mounted displays incorporate a headset screen connected to two cameras. These capture the real life image and then merge it with the pre-rendered virtual object while continually tracking and providing the computer with constantly updated data on the position and orientation of the patient. In contrast, optical head mounted displays incorporate a partially reflective visor that receives and reflects images allowing them to be augmented with the real world.

One aspect of augmented reality that is fundamental to surgical navigation is “registration.” This is critical because it involves taking a virtual image, such as a three dimensional image of a patient’s tumour, and transferring it so that it displays the route in which to resect the tumour. The result is a registered image that combines a graphic image overlaid on a real patient. Registration must be continually monitored and adjusted in relation to subtle movements throughout a procedure. Electronic markers or infrared light emitting diodes can be used to accurately map the patient’s orientation and anatomy and send this information back to the user.¹

The possibilities are endless

Because many surgical specialties use imaging technology there is much potential for augmented reality. One key way in which augmented reality might contribute is by providing three dimensional images rather than two dimensional scans. The technique has also been used in the superimposition of three dimensional tumour models on the breast, enabling a surgeon to perceive the position of a tumour through the skin.²

In orthopaedics, augmented reality could be used to guide and orient implantations. And using augmented reality to analyse and visualise a hip operation would make it possible to intraoperatively adjust the procedure to tailor it to a patient’s specific needs.

One of the main uses of augmented reality in maxillofacial surgery is its ability to visualise deep structures and allow minimally invasive operations in, for example, tumour surgery, temporomandibular joint repair, dental work, and prosthetic and cosmetic surgery.³ Points of incision on the mandibular bones could be preoperatively marked on an augmented image and then adjusted or modified before being overlaid on the real patient, allowing clear surgical procedures to be mapped.

Augmented reality could also help in research. Studies in cognitive neuroscience have used image guidance techniques to help to understand and map the detailed function of brain structures.⁴ Magnetic resonance imaging and computed tomography can help in cortical mapping and visualisation, but their resolution is limited. Further data can be derived from the use of augmented reality to represent these images as a three dimensional model.⁵

Neurosurgery leads the way

Neurosurgeons have to resect the smallest possible volumes of neurological tissue while avoiding the fatal removal of a vital functional area. Because of this extreme precision augmented reality can increase the efficiency, reliability, speed, and safety of neurosurgery.

One way is in preoperative planning and training. Neurosurgery requires such precise and delicate operating, and it is imperative that no mistakes are made. This can be difficult to ensure, however, if the surgeon has little experience. Augmented reality could enable a virtual patient to be constructed and a surgeon to preoperatively plan and practise the procedure allowing him or her to adapt and change their technique while being free to make mistakes or experiment with a new approach.

One of the key problems faced in neurosurgery today is that many functionally distinct subcortical structures are not differentiable on magnetic resonance scans. Augmented reality could counter this by superimposing a three dimensional anatomical image of a patient’s brain onto the real operating site to act as a reference and form an interactive environment. In conditions such as Parkinson’s disease or epilepsy this could help. Patients with Parkinson’s disease experience tremor, rigidity, and akinesia, but symptoms of extreme tremor can be reduced by creating a small, specific lesion in the thalamus (thalamotomy).^{6 7} This would normally be difficult to manage, however, with the help of augmented reality, functionally distinct structures within grey matter could be identified, which would help in thalamotomies and similar procedures.⁷

Counting the costs

Augmented reality technology is in its infancy, and there are still many risks, difficulties, and limitations to be assessed. Safety is of course paramount: a minor miscalculation could result in the patient’s death. Augmented reality relies upon an exact registration of virtual to real life, and misalignment cannot be allowed. It is important that the surgeon retains tight control and can monitor, evaluate, and adjust factors throughout a procedure that depends on augmented reality. Ergonomics and ease of use must also be considered. The headsets must not cause eyestrain, headaches, or nausea for the surgeon because this could compromise patient safety.

Augmented reality uses a variety of monitors, cameras, and complex computing equipment, which must be of a high quality but also affordable. The relatively high costs are likely to drop with increased demand and availability.

Into the future

Augmented reality could be implemented in routine surgical practice in the foreseeable future and complement and improve surgical procedures in many ways. Like all image guidance systems, though, it is not a substitute for surgical experience. Studies have compared the advantages of augmented reality with standard surgery, and clinical comparisons have been made between image guided and conventional meningioma surgery. The image guided group had fewer patients with complications and shorter operating times.⁸ Evidence for the clinical advantages of the technology is limited, and more studies need to quantitatively and qualitatively assess the advantages of augmented reality.

Future developments for augmented reality may encompass the application of augmentation to other senses as well. In particular, adding and removing sound might be useful. Auditory signals could warn if a surgeon begins to stray from the augmented resection line, and this could also help in preoperative planning and the training of surgeons.

The uses mentioned here are just a fraction of the possibilities that augmented reality holds. The potential that augmented reality has in surgery is limited only by the depth of our imaginations.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

1. Kawamata T, Iseki H, Shibasaki T, Hori T. Endoscopic augmented reality navigation system for endonasal transsphenoidal surgery to treat pituitary tumors. Neurosurgery 2002;50:1393-7.
2. Sato Y, Nakamoto M, Tamaki Y, Sasama T, Sakita I, Nakajima Y, et al. Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization. IEEE Trans Med Imaging 1998;17:681-93.
3. Shuhaiber JH. Augmented reality in surgery. Arch Surg 2004;139:170-4.
4. Kim JS, Singh V, Lee JK, Lerch J, Ad-Dab’bagh Y, MacDonald D, et al. Automated 3-D extraction and evaluation of the inner and outer cortical surfaces using a Laplacian map and partial volume effect classification. Neuroimage 2005;27:210-21.
5. Pandya A, Siadat MR, Auner G. Design, implementation and accuracy of a prototype for medical augmented reality. Comput Aided Surg 2005;10:23-35.
6. Kumar P, Clark M. Clinical medicine. 5th ed. London: WB Saunders, 2002.
7. St-Jean P, Sadikot AF, Collins L, Clonda D, Kasrai R, Evans AC, et al. Automated atlas integration and interactive three-dimensional visualization tools for planning and guidance in functional neurosurgery. IEEE Trans Med Imaging 1998;17:672-80.
8. Paleologos TS, Wadley JP, Kitchen ND, Thomas DG. Clinical utility and cost-effectiveness of interactive image-guided craniotomy: clinical comparison between conventional and image-guided meningioma surgery. Neurosurgery 2000;47:40-7.

       

 Printable version   Download PDF   E-mail this to a friend   Respond to this article