Keywords

computer vision, artificial neural networks, Baxter, intelligent systems, surgical tool detection

Research Abstract

Several hospitals face nurse staffing shortages for surgeries. This research focuses on building a system with Baxter robot capable of identifying surgical tools using computer vision and delivering them to the surgeon on demand. This would deal with the issue of nurse unavailability during simple surgical procedures. The key aspects of the project were: testing the accuracies of various Artificial Neural Networks (ANNs) in classifying surgical instruments, and programming Baxter to implement a surgical tool delivery system using magnets at the tip of its 7-DOF robotic arms. The methodology consisted of, first, implementing algorithms to enable Baxter to do pick and deliver tasks for surgical tools, and second, gathering Hu-Moments of various tools using the cameras on Baxter's arm, which were then used to train the ANNs. Tool detection accuracies of ANNs with hidden layer neuron number varying from 5 – 50 and learning rates varying from 0.005 - 0.1 were collected. Then, the tool identification and tool delivery system were merged together to create a turn by turn dynamic tool tracking and delivery system, which retrieved tools, based on the surgeons input, through a Leap Motion sensor. In addition to delivery, the system was modified to retrieve used tools back from the surgeon, using a computer vision based approach. The optimal ANN configuration consisted of an ensemble of various ANNs working together and achieved a detection accuracy of 93%. The average time taken for a mock abdominal incision surgery with the system is expected to be around 10 min and 30 sec.

Session Track

Health

Share

COinS
 
Aug 7th, 12:00 AM

Dynamic Surgical Tool Tracking and Delivery System using Baxter Robot

Several hospitals face nurse staffing shortages for surgeries. This research focuses on building a system with Baxter robot capable of identifying surgical tools using computer vision and delivering them to the surgeon on demand. This would deal with the issue of nurse unavailability during simple surgical procedures. The key aspects of the project were: testing the accuracies of various Artificial Neural Networks (ANNs) in classifying surgical instruments, and programming Baxter to implement a surgical tool delivery system using magnets at the tip of its 7-DOF robotic arms. The methodology consisted of, first, implementing algorithms to enable Baxter to do pick and deliver tasks for surgical tools, and second, gathering Hu-Moments of various tools using the cameras on Baxter's arm, which were then used to train the ANNs. Tool detection accuracies of ANNs with hidden layer neuron number varying from 5 – 50 and learning rates varying from 0.005 - 0.1 were collected. Then, the tool identification and tool delivery system were merged together to create a turn by turn dynamic tool tracking and delivery system, which retrieved tools, based on the surgeons input, through a Leap Motion sensor. In addition to delivery, the system was modified to retrieve used tools back from the surgeon, using a computer vision based approach. The optimal ANN configuration consisted of an ensemble of various ANNs working together and achieved a detection accuracy of 93%. The average time taken for a mock abdominal incision surgery with the system is expected to be around 10 min and 30 sec.