Vision-based robot gripping of surgical instruments for intra-operative assistance with deep learning methods

Abstract

This work constitutes an effort towards the development of a functional scrub nurse robot. Such a robot should be able to complete tasks normally executed by a human nurse during surgery. In this project, the tasks of instrument detection, localization, and gripping are addressed. A laboratory setup was prepared, including a table, surgical tissue, and a robot manipulator. A complete instrument set for wisdom teeth extraction was employed during the experiments. A stereo camera system was acquired to address the localization task and to provide raw data for the detection algorithm. Additionally, an electromagnet was chosen to be used as the robot’s gripper. For the detection task, the Mask R-CNN algorithm was selected. A new data augmentation approach was implemented, which proved to significantly improve the algorithm’s performance. Acceptable results for most of the instruments were obtained, while the performance for some of them was insufficient. Two different approaches were tested for the localization task. The results matched those of the detection algorithm, indicating that errors were produced due to a sub-optimal detection performance. While the electromagnet proved to be robust and capable of successfully gripping most of the instruments, some problems were identified. The main issue was the incapacity to release the instruments due to its residual magnetism. Hence, this device was considered unsuitable for future work. In general, the results were considered acceptable for an initial stage. Important improvements are required, especially regarding the performance of the detection algorithm. One of the main challenges for future work is the design of a suitable gripper, capable of lifting and manipulating the instruments.