Electrical and Computer Engineering


Several hospitals face nurse staffing shortages for surgeries. This research focuses on building a system with a Baxter robot capable of identifying surgical tools using computer vision and delivering them to the surgeon on demand. This would deal with the issue of nurse unavailability during simple surgical procedures. The key aspects of the project were testing the accuracies of various artificial neural networks (ANNs) in classifying surgical instruments and programming Baxter to implement a surgical tool delivery system using magnets at the tip of its 7 degrees of freedom (DOF) robotic arms. The methodology consisted of first implementing algorithms to enable Baxter to do pick and deliver tasks for surgical tools, and second, gathering HuMoments of various tools using the cameras on Baxter's arm, which were then used to train the ANNs. Tool detection accuracies of ANNs with hidden layer neuron number varying from 5–50 and learning rates varying from 0.005‒0.1 were collected. Then, the tool identification and tool delivery system were merged together to create a turn-by-turn dynamic tool tracking and delivery system, which retrieved tools, based on the surgeons input, through a Leap Motion Controller. In addition to delivery, the system was modified to retrieve used tools from the surgeon, using a computer vision-based approach. The optimal ANN configuration consisted of an ensemble of various ANNs working together and achieved a detection accuracy of 93%. The average time taken for a mock abdominal incision surgery with the system is expected to be around 10 minutes and 30 seconds.