Jan 31 2018
In a series of procedures performed by a group at Imperial College London at St Mary’s Hospital, researchers have demonstrated for the first time how surgeons can employ Microsoft HoloLens headsets while carrying out reconstructive lower limb surgery on patients.
AR models reveal the patient's bones and blood vessels. (Image credit: Philip Pratt et al. Eur Radiol Exp, 2018; Microsoft HoloLens (c) Microsoft)
The HoloLens is a self-contained computer headset that plunges the wearer into ‘mixed reality’, enabling them to interact with ‘holograms’ – computer-generated objects made visible via the visor. In the UK, headsets are presently only obtainable by developers.
The Imperial team used the technology to overlay images of CT scans – including the position of key blood vessels and bones– onto each patient’s leg, in effect allowing the surgeon to ‘see through’ the limb during surgery.
According to the team trialing the technology, the method can help surgeons locate and reconnect important blood vessels during reconstructive surgery, which could enhance outcomes for patients.
We are one of the first groups in the world to use the HoloLens successfully in the operating theatre.
Through this initial series of patient cases we have shown that the technology is practical, and that it can provide a benefit to the surgical team. With the HoloLens, you look at the leg and essentially see inside of it. You see the bones, the course of the blood vessels, and can identify exactly where the targets are located.
Dr Philip Pratt, Lead Author
Repairing Damage
After severe trauma or a car accident, patients may have tissue damage or open wounds that require reconstructive surgery using fasciocutaneous flaps. These flaps of tissue, which are taken from other areas on the body and include the blood vessels and skin, are used to cover the wound and enable it to close and heal correctly.
A crucial step in the process is linking the blood vessels of the ‘new’ tissue with those at the site of the wound, so oxygenated blood can go onto the new tissue and keep it alive.
The typical approach for this element of reconstructive surgery has been the usage of a handheld scanner, which applies ultrasound to identify blood vessels under the skin by detecting the movement of blood pulsing via them, allowing the surgeon to estimate where the vessels are and their course through the tissue.
“Augmented reality offers a new way to find these blood vessels under the skin accurately and quickly by overlaying scan images onto the patient during the operation,” explained Dr Pratt.
Making the Model
In the procedures used to test the technology, five patients needing reconstructive surgery on their legs underwent CT scans to map the structure of the limb, including the position of bones and the course and location of blood vessels.
Images from the scans were then segmented into muscle, bone, fatty tissue, and blood vessels by Dr Dimitri Amiras, a consultant radiologist at Imperial College Healthcare NHS Trust (ICHNT), and loaded into intermediary software to develop 3D models of the leg.
These models were then loaded into specially designed software that renders the images for the HoloLens headset, which then overlays the model onto what the surgeon can see in the operating theater.
Clinical staff is able to control these AR images via hand gestures to make any fine modifications and suitably line up the model with surgical landmarks on the patient’s limbs, such as the ankle bone or knee joint.
Dr Amiras said: “St Mary’s Hospital is a major trauma centre, giving us the opportunity to try and improve the pre-operative planning for reconstructive flaps.”
"Over time, the scanning protocol has been optimised to give excellent images of the anatomy, however, at first we had to rely on rough measurements of anatomical landmarks taken from 3D CT reconstructions to guide surgery.”
“Now, using the HoloLens, we can identify where the blood vessels are in 3D space and use virtual 3D arrows to guide the surgeon.”
"Currently, data preparation is a time consuming process, but in the future much of this could be automated, with the consultant radiologist checking the accuracy of the model against the original scan.”
"I think this is a great example of what can be achieved in an Academic Health Science Centre.”
The Surgeon's View
Mr Jon Simmons, a plastic and reconstructive surgeon at ICHNT, led the team who performed the procedures using the HoloLens headset and AR models.
The cases varied from a 41-year-old man who had sustained leg injuries in a car accident, to an 85-year-old woman who had a compound fractured ankle.
The surgical teams described the HoloLens to be a robust tool in the theater, with the method being more reliable and less time-consuming than the ultrasound technique of locating blood vessels.
The application of AR technology in the operating theatre has some really exciting possibilities. It could help to simplify and improve the accuracy of some elements of reconstructive procedures.
While the technology can’t replace the skill and experience of the clinical team, it could potentially help to reduce the time a patient spends under anaesthetic and reduce the margin for error. We hope that it will allow us to provide more tailored surgical solutions for individual patients.
Mr Jon Simmons
The group emphasizes a few restrictions with the technology, which could include errors during the modeling stages as well as the potential for the overlaid model to be skewed.
Furthermore, the case studies thus far have been based on the leg, which has a number of plainly visible surgical ‘landmarks’, such as the knee or ankle.
Areas without these stiff landmarks, such as abdomen, may be more difficult with a more potential for movement of blood vessels.
However, the researchers are confident that, once refined, the method could be applied to other areas of reconstructive surgery necessitating tissue flaps, such as breast reconstruction following mastectomy. The subsequent steps include trialing the technology in a larger set of patients, with procedures performed by teams at many centers.
In future we hope to automate the process further. We can use software to improve the alignment and will attach markers to the patient when they have the scan, with the same markers present during the operation to use as additional points of reference.
There are a number of areas we would like to explore, and further improvements are needed, but the small case series has shown that for reconstructive surgery, this seems to be a valuable tool in the operating theatre.
Dr Philip Pratt, Lead Author