An image processing method for changing endoscope direction based on pupil movement
© Cao et al. 2016
Received: 29 March 2015
Accepted: 11 January 2016
Published: 22 January 2016
Increased attention has been focused on laparoscopic surgery because of its minimal invasiveness and improved cosmetic properties. However, the procedure of laparoscopic surgery is considerably difficult for surgeons, thus paving the way for the introduction of robotic technology to reduce the surgeon’s burden. Thus, we have developed a single-port surgery assistive robot with a master–slave structure that has two surgical manipulators and a sheath manipulator for the alteration of endoscope direction. During the development of the surgical robotic system, achieving intuitive operation is very important. In this paper, we propose a new laparoscope manipulator control system based on the movement of the pupils to enhance intuitive operability. We achieve this using a webcam and an image processing method. After the pupil movement data are obtained, the master computer transforms these data into an output signal, and then the slave computer receives and uses that signal to drive the robot. The details of the system and the pupil detection procedure are explained. The aim of the present experiment is to verify the effectiveness of the image processing method applied to the alteration of endoscope direction control system. For this purpose, we need to determine an appropriate pupil motion activation threshold to begin the sheath manipulator’s movement. We used four kinds of activation threshold, measuring the time cost of a particular operation: to move the image of the endoscope to a specific target position. Moreover, we identified an appropriate activation threshold that can be used to determine whether the endoscope is moving.
KeywordsNon-rigid face tracking Single-port endoscopic surgery Master–slave structure Image processing Double-screw-drive mechanism Activation threshold
Laparoscopic surgery is a technique whereby a laparoscope and surgical instruments are inserted into the patient’s body through an artificial or natural body cavity, followed by the surgeon operating the instruments based on the monitor image captured by the laparoscope . Laparoscopic surgery has many advantages, such as shorter hospitalization times, lower physical burden on patients and cosmetic improvement  compared with open surgery. Although laparoscopic surgery has many advantages for patients as described above, the difficulty of performing the technique is so high that surgeons need to carry out long-term training at a special medical training center. Even if they have experienced professional training, they still suffer from high mental stress during operations, which may reduce their dexterity or judgment . One of the causes of such problems is that laparoscopic surgery requires another operator to hold the laparoscope for the surgeon. Therefore, the coordination between operators has a significant effect on the process and result of laparoscopic surgery .
To tackle the problems mentioned above, robotic technology is an effective solution. Naviot  provides surgeons with the possibility of solo surgery, and requires the surgeon to use one hand to hold a controller when they need to alter the direction of the laparoscope. The Da Vinci Surgical System  can change the control mode between the surgical manipulator and the endoscopic manipulator for the operator using a foot pedal. However, such methods do not allow the surgeon to alter the operative field while simultaneously manipulating tissue .
Other systems, like the automatic endoscope optimal positioning system (AESOP) from Computer Motion Inc.  and ViKY EP  use a voice control system to allow the operator to control the robotic endoscopic holder. However, rather than being helped by voice control, surgical time is actually often increased because of its slow response .
Alongside voice control, eye-tracking is a further intuitive hands-free method that can be used as an input signal for manipulating the laparoscope holder. As a practical example, Ubeda et al.  used the electrooculography signal from electrodes attached around the user’s eye to control a manipulator. In the video-oculography field, Noonan, et al.  used a stand-alone eye tracker to alter the laparoscope direction based on gaze.
Single port surgery (SPS) is one of form of laparoscopic surgery that requires only one incision port. SPS has attracted increasing attention from patients because of its cosmetic advantages . Although robots enhance the performance of standardized laparoscopic techniques , current systems, including those mentioned in the previous paragraphs, are not suitable for SPS because they require multiple incisions. Therefore, our laboratory has developed two prototypes of an SPS assistive robot system. Prototype 1  uses respective controllers for tool manipulators and laparoscope direction; in Prototype 2 , the control mode can be changed between tool manipulators and endoscope direction by pushing a foot pedal like in the Da Vinci Surgical System .
The purpose of this paper is to introduce a control method for the alteration of endoscope direction using pupil-tracking into the Prototype 2  system, which is achieved via image processing and the use of a webcam. The system will translate the obtained image data into an output signal. We propose a threshold for pupil movement distance: the sheath manipulator, which is used for the alteration of endoscope direction, activates when the user’s pupil movement distance exceeds a threshold value; the manipulator remains in a static state when the pupil movement distance is below the threshold. Therefore, an appropriate threshold for the output signal needs to be determined to judge whether the movement state of the sheath manipulator is dynamic or static. To determine this, we tested four threshold values one by one in a horizontal movement experiment. The experimental outcome variable was the completion time of moving the field of view to a specific target, so that the most appropriate threshold could be judged from the minimum completion time. The first part of this paper will describe how we obtain the pupil movement data. Then, the general system framework, including how the system integration is achieved, will be discussed. The second half of this paper describes an experiment to verify the effectiveness of the system and obtain a proper activation threshold value, which is important for the operability of pupil tracking.
Acquisition method of pupil movement
Applying non-rigid face tracking
Training image patches
Besides up–down movement and left–right movement, human eye movement also includes some unconscious movements such as blinking, saccade and tremble . The output signal caused by these movements is regarded as noise by the control system. Therefore, the moving average method was used to filter the noise signal. The average value was calculated from 10 samples and the window size of the filter was 400 (ms). As a result, the frequency of pupil tracking was 25 Hz.
P indicates the gain value and P∙Tcycle = 800.
As mentioned in the previous research, the frequency of image processing was 25 (Hz), and therefore the time delay was 40 (ms). A frequency equal to or greater than 25 (Hz) is regarded as real-time [24, 25]. Moreover, the response of the manipulator was less than 100 (ms) . Therefore, the overall delay was acceptable because it did not exceed 330 (ms), which was estimated as the maximum time delay compatible with the safe performance of surgical manipulations .
Before we design an experiment for determining the proper activation threshold value, we need to decide the range of D/t. If D/t is too big, the user has to rotate their eyes to move their gaze out of the screen; if D/t is too small, the sheath manipulator will activate even when the user keeps their eyes static. Therefore, three participants, engineering graduate students without glasses, tested the performance of the pupil-tracking method. During this test, we requested that the participants perform three kinds of eye motions: rotating the eyes to a maximum angle as much as possible, gazing at the edge of the monitor, and keeping the eyes static and gazing straight. Meanwhile, we recorded the variations in D and ΔX.
First of all, we made personal tracker models for each of the three participants as in the procedures mentioned in the previous sections. Then, we requested the participants to perform the three kinds of eye motions.
In the next section, we needed to identify an appropriate activation threshold that could be used to determine whether the sheath manipulator was moving or not. Using the above results, we confirmed that the activation threshold should be selected from a range between D/7 and D/25.
Purpose of the experiment
The aim of the present experiment is to verify the effectiveness of the image processing method applied to the sheath manipulator control system. Moreover, to judge whether the sheath manipulator is dynamic or static, an appropriate output signal threshold needs to be obtained. In this experiment, the activation thresholds in four conditions were evaluated using operation time.
Communication between master and slave
The master computer was installed with Windows XP. The pupil-tracking program we proposed was merged into the operating system of prototype 2 and was compiled with Visual Studio 2008. As shown in Fig. 19, the Infrared LED webcam mounted on the master side initially captures the images. After this, the pupil movement data are extracted from the captured images by the pupil-tracking program and are then converted into an output signal for controlling the servomotor. Subsequently, the output signal is sent to the slave via UDP. After receiving the output signal, the slave computer activates the servomotors to drive the double-screw-drive mechanisms , thus adjusting the direction of the endoscope.
The activation threshold was set for the control system.
The system was tested as to whether it altered direction with the participant’s pupil movement (Additional file 1).
- (c)Initialization: the image center of the endoscope was changed to match “point 0” (Fig. 22).
Each participant moved the image center of the endoscope from “point 0” to “point 3” four times, and from “point 0” to “point 4” four times. The time cost of every trial was recorded.
The threshold value was changed and the above flow from step a to step c was repeated.
The change order was D/14, D/18, D/22 and D/24.
For the present experiment, it was important that the system reflected the operator’s viewing intention so that the participants could confirm their location during the experiment. To judge whether the sheath manipulator was in the static state, each participant verbally confirmed when the operation was completed, i.e., when the image center of the endoscope had reached the target and stopped. Thus, the time measurement ended when the participant replied that the movement was complete.
Results and discussion
For the alteration of the endoscope direction operation system based on pupil position tracking, the shortest operation time was when the activation threshold value was equal to D/22. Similarly, the standard deviation values were smallest in both experiments when that threshold value was selected. Therefore, it is appropriate to use D/22 as an activation threshold for the operation system of the proposed SPS robot. In addition, we found that operating the sheath manipulator via pupil tracking can provide good stability and response when an appropriate threshold value is used. Also, the period between blinks was 6–8 s , which prevents the eye from fatiguing and from drying out. We observed an obvious difference for operation time and standard deviation in the D/22 condition in the two experiments. [29, 30] suggested that the center of monitor image should be aligned to the center of operative field. If the target points: “point 0”, “point 3” or “point” is projected on the center of monitor image, ΔX will not exceed the activation threshold value. Therefore, the operator can stop the desired target position. The limitation of this experiment is that the numbers of participants and conditions were both rather small. Therefore, there is limited evidence to support the optimal threshold value. Moreover, the accurate relationship between the motion of the manipulator and the movement of the pupils was not confirmed. After causal analysis, we found that the difference arose from the intertwining flexible shafts that are set between the manipulators and servomotors; such a situation affects the stability and speed of rotation. As a solution to this problem, these flexible shafts need to be sheathed in pipes or fixed on a fixation device so as to avoid intertwining. Furthermore, we hope to achieve better pupil tracking in the vertical direction by using a higher resolution webcam.
The proposed system is more intuitive than voice-control and pedals. The surgeons certainly need a hand-free strategy to manipulate the endoscope. Using webcam and image processing is a good approach because it does not require the surgeons to be attached to additional devices, which may increase their burden during an operation.
The proposed system is useful because the manipulator will stop when the center of the visual field aligns with the target. One of the fundamentals of manipulating the endoscope is aligning the center of the visual field with the center of operation.
The horizontal alternation of the visual field is greater than the vertical, but the vertical alternation is still indispensable.
A zoom function is indispensable for an endoscopic control system when the surgeon is performing a delicate operation. For example, the surgeon would ideally like to zoom in the visual field when they are peeling the tissue from around a vessel. Moreover, in a future system, it would be better to be able to adjust the rate of visual field alternation according to the magnification of the lens.
An emergency stop button is an indispensable part of the system, which is used to avoid a collision between the endoscope and tissue.
In this paper, a hands-free technique for controlling the alteration of endoscope direction using a pupil-tracking method via an image processing method was introduced. The novelty of the proposed method is its ability to achieve pupil tracking because the variation in distance of both intraocular angles and pupils could be obtained from the tracking trajectory. In this method, an appropriate output signal threshold needs to be obtained for judging whether the sheath manipulator, which alters the endoscope direction, is dynamic or static. An experiment was performed to verify the effectiveness of the image processing method applied to the sheath manipulator control system, and the activation threshold of the control system had to be determined and used for the horizontal direction movement of the sheath manipulator. We found an activation threshold value that fulfils stability and response simultaneously. This time, we only verified the horizontal direction because of the limitations of our method. At present, it is quite difficult for the sheath manipulator to make vertical movements, because the vertical movement range of the eye is much less than its horizontal movement range. To realize vertical direction movement, we need a higher resolution webcam to detect the relatively small vertical movement of the eyes. As surgeons pointed out, a zoom function is indispensable for endoscope manipulation. Thus, using only the pupil position parameter as shown in this experiment is not sufficient to achieve the zoom function. In future work, we aim to develop an algorithm that includes a large number of operating conditions to judge the movement state and achieve more types of movements. Also, we will improve the current control system based on the surgeons’ suggestions. Furthermore, we also plan to invite surgeons and medical trainees to be participants in the manipulation experiments.
automatic endoscope optimal positioning system
graphical user interface
user datagram protocol
YC derived the basic concept of the overall system, technically constructed the system and drafted the manuscript. All authors read and approved the final manuscript.
The authors sincerely thank the volunteers for participating in our experiments. The work was supported in part by a research grant from JSPS Global COE Program: Global Robot Academia, JSPS Grant-in-Aid for Scientific Research (A) No. 20339716, JSPS Grant-in-Aid for Scientific Research (S) No.25220005, JSPS Grant-in-Aid for Exploratory Research No. 15K12606 and the Program for Leading Graduate Schools, “Graduate Program for Embodiment Informatics” of the Ministry of Education, Culture, Sports, Science and Technology.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- ASCRS, Laparoscopic surgery—what is it?, Available from: http://www.fascrs.org/patients/treatments_and_screenings/laparoscopic_surgery/
- Horgan S, Vanuno D (2001) Robots in laparoscopic surgery. J Laparoendosc Adv Surg Tech A 11:415–419View ArticleGoogle Scholar
- Matern U, Koneczny S (2007) Safety, hazards and ergonomics in the operating room. Surg Endosc 21:1965–1969View ArticleGoogle Scholar
- Jihad K, George-Pascal H, Raj G, Mihir D, Monish A, Raymond R, Courtenay M, Inderbir G (2008) Single-Port laparoscopic surgery in urology: initial experience. Urology 71:3–6View ArticleGoogle Scholar
- Yasunaga T, Hashizume M, Kobayashi E, Tanoue K, Akahoshi T, Konishi K, Yamaguchi S, Kinjo N, Tomikawa M, Muragaki Y, Shimada M, Maehara Y, Dohi Y, Sakuma I, Miyamoto S (2003) Remote-controlled laparoscope manipulator system NaviotTM, for endoscopic surgery. Int Congress Ser 1256(2003):678–683View ArticleGoogle Scholar
- Intuitive Surgical Inc. [homepage on the Internet]. Available from: http://www.intuitivesurgical.com
- Meyer A, Oleynikov D (2013) Surgical robotics, Mastery of Endoscopic and Laparoscopic Surgery, Vol 2, 4th edn. p 62–71
- Kraft B, Jager C, Kraft K, Leibl B, Bittner R (2004) The AESOP robot system in laparoscopic surgery: increased risk or advantage for surgeon and patient? Surg Endosc 18:1216–1223View ArticleGoogle Scholar
- ViKY EP, available from: http://www.endocontrol-medical.com/viky_ep.php
- Kolvenbach R, Schwiez E, Wasillijew S, Miloud A, Puerschel A, Pinter L (2004) Total laparoscopically and robotically assisted aortic aneurysm surgery: a critical evaluation. J Vasc Surg 39:771–776View ArticleGoogle Scholar
- Ubeda A, Ianez E, Azorın J (2011) Wireless and portable EOG-based interface for assisting disabled people. IEEE/ASME Transact Mechatron 16(5):870–873View ArticleGoogle Scholar
- Noonan D, Mylonas G, Shang J, Payne C, Darzi A, Yang G (2010) Gaze contingent control for an articulated mechatronic laparoscope. Proceedings of the 2010 3rd IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, The University of Tokyo, Tokyo, Japan, 26–29 September 2010
- Autorino R, Cadeddu JA, Desai MM, Gettman M, Gill IS, Kavoussi LR, Lima E, Montorsi F, Richstone L, Stolzenburg JU, Kaouk JH (2010) Laparoendoscopic single-site and natural orifice transluminal endoscopic surgery in urology: a critical analysis of the literature. Eur Assoc Urol 59(1):26–45View ArticleGoogle Scholar
- Hubens G, Coveliers H, Balliu L, Ruppert M, Vaneerdeweq W (2003) A performance study comparing manual and robotically assisted laparoscopic surgery using the da Vinci system. Surg Endosc Other Intervent Tech 17(10):1595–1599View ArticleGoogle Scholar
- Kobayashi Y, Tomono Y, Sekiguchi Y, Watanabe H, Toyoda K, Konishi K, Tomikawa M, Ieiri S, Tanoue K, Hashizume M, Fujie GM (2010) The international journal of medical robotics and computer assisted surgery. Int J Med Robotics Comput Assist Surg 6:454–464View ArticleGoogle Scholar
- Kobayashi Y, Sekiguchi Y, Noguchi T, Liu Q, Oguri S, Toyoda K, Konishi K, Uemru M, Ieiri S, Tomikwa M, Ohdaira T, Hashizume M, Fujie GM (2015) The international journal of medical robotics and computer assisted surgery. Int J Med Robotics Comput Assist Surg 11:235–246View ArticleGoogle Scholar
- Rassweiler J, Gözen A, Frede T, Teber D (2011) Laparoscopy vs. robotics: ergonomics—does it matter? Robotics Genitourin Surg 2011:63–78View ArticleGoogle Scholar
- Ohno T, Mukawa N (2004) A free-head, simple calibration, gaze tracking system that enables gaze-based interaction. ETRA 2004, Eye Tracking Research and Applications Symposium, p 115–122
- Hill B (1996) Retina identification, Biometrics, p 123–141
- Liu Q, Kobayashi Y, Zhang B, Noguchi T, Takahashi Y, Nishio Y, Cao Y, Ieiri S, Toyoda K, Uemura M, Tomikawa M, Hashizume M, Fujie M G (2014) Development of a smart surgical robot with bended forceps for infant congenital esophageal atresia surgery. 2015 IEEE International Conference on Robotics and Automation, Hong Kong, p 2430–2435
- Emami S, Ievgen K, Saragih J (2012) Non-rigid face tracking, mastering openCV with practical computer vision projects. Chapter 6:189–233
- Gower J (1975) Generalized procrustes analysis. Phychometrika 40(1):33–51View ArticleMathSciNetMATHGoogle Scholar
- Robinson D (1964) The mechanics of human saccadic eye movement. J Pysiol 1964(174):245–264View ArticleGoogle Scholar
- Marchand E, Chaumette F (2002) Virtual visual servoing: a framework for real-time augmented reality. Comput Graph Forum 21(3):289–297View ArticleGoogle Scholar
- Daniilidis K, Krauss C, Hanse M, Sommer G (1998) Real-time tracking of moving objects with an active camera. Real-Time Imaging 4(1):3–20View ArticleGoogle Scholar
- Marescaux J, Leroy J, Gagner M, Rubino F, Mutter D, Vix M, Butner SE, Smith MK (2001) Transatlantic robot-assisted telesurgery. Nature 413:379–380View ArticleGoogle Scholar
- Rempel D, Willms K, Anshel J, Jaschinski W, Sheedy J (2007) The effects of visual display distance on eye accommodation, head posture, and vision and neck symptoms. Hum Factors J Hum Factors Ergon Soc 49(5):830–838View ArticleGoogle Scholar
- Barbato G, Ficca G, Muscettola G, Fichele M, Beatrice M, Rinaldi F (2000) Diurnal variation in spontaneous eye-blink rate. Psychiatry Res 93(2):145–151View ArticleGoogle Scholar
- Hashizume M (2005) Fundamental training for safe endoscopic surgery, Innovative Medical Technology, Graduate School of Medical Science Kyushu University, p 49 (in Japaeses)
- Donnez J (2007) Human assistance in laparoscopic surgery, Atlas of operative laparoscopy and hystereoscopy, 3rd edn, p 409