A method for using one finger to press three separate keys on a three-dimensional keyboard designed to be mounted on a mouse
© Suzuki et al. 2016
Received: 23 January 2015
Accepted: 11 December 2015
Published: 19 January 2016
Here we propose and verify the feasibility of a new keyboard structure for keyboard-mounted mouse. Reducing the distance that fingers move while typing within a three-dimensional (3-D) key arrangement that fits along the fingers is effective for keyboard-mounted mouse, provided that each finger can separately press three different keys. Our objective was to design and test a method for separately being able to press three keys with a single finger, and in the process, reduce finger-moving distance. We analyzed 3-D finger motion while participants typed on a standard keyboard, and used this data to develop a 3-D layout for keys that are arranged along the fingers. Analysis of 3-D finger motion while using our new keyboard showed that the distance fingers traveled was 74 % less than that when using a standard keyboard (p < 0.05). Moreover, this did not result in typing mistakes caused by interference between the keys and finger movements. After typing 30 characters 20 times, the average input error rate in the 20th trial was 18 %, while the average error rate across all trials for the standard keyboard was 9.5 %. We conclude that our proposed 3-D keyboard can be used accurately with one finger while reducing the distance fingers must move. However, input mistakes were caused by finger linkage motion. In the future, we must devise a character-input algorithm that eliminates such erroneous input. We must also include the mouse function and evaluate the operability of the device in tasks that require keyboard and mouse use.
In the current information-technology society, people spend a great deal of time using computers at work and at home. Keyboards and mice are widely used as the de facto standard for operating personal computers, and almost all people are accustomed to using them . Generally, keyboards are used for character input and mice are used to direct the cursor and make selections. However, because hands must be frequently moved between the two independent interfaces, typing efficiency and mouse operation are not as efficient as they could be. Thus, integrating the two interfaces, mounting mouse functions on the keyboard or mounting keyboard functions on the mouse, will be effective to reduce the movement between two interfaces and will improve the efficiency of typing and mouse operation. Developing an interface that combines keyboard and mouse function is therefore of keen interest.
TrackPoint is one of several interfaces that have integrated devices that mount mouse functions on the keyboard. The TrackPoint has a mouse trackpad that is placed in the center of the keyboard . By using the TrackPoint, mouse-cursor movement can be controlled by using the fingertips. Another keyboard can sense hand movement with a built-in infrared sensor, and some mouse functions are possible via finger motion as if the keyboard is a touchpad mouse . One keyboard can act as a touchpad mouse through touch sensors that are placed on the keyboard surface [4, 5]. These studies considered mouse functions such as cursor movement and scrolling, but studies that examine clicking and dragging functions have not yet been conducted. In order to mount mouse function onto the keyboard, part of the keyboard needs to be made into something like a touchpad mouse. However, mouse function via touchpad has been reported to take more time than that using a conventional mouse . Thus, improving efficiency of typing and mouse operation by integrating the devices, a more useful approach might be to mount the keyboard function onto a mouse that is operated by the hand and fingers.
Several attempts have been made to mount keyboard functions on a mouse. One is a multi-button mouse that can have several keys mounted on it . With this type of mouse, keys such as ‘Return’ or ‘Delete’ can be input using buttons on the side of the mouse. CombiMouse, which has the entire right half of a standard keyboard connected to a mouse, can be used similarly as a standard mouse [8, 9]. Character input speed of CombiMouse has been reported to be 70 % that of a standard keyboard. To improve typing and mouse operation efficiency, ideally the keys can be pressed with small finger movements while a hand is placed on the mouse. In contrast, CombiMouse requires the hand to be released from the mouse portion of the device when using the keyboard portion, and a suitable key structure for mounting on a mouse has not yet been considered. DataHand is a mouse equipped with a keyboard with a special concave key structure around the fingertip . Because the concave keys of DataHand can be pressed using small finger motions, the hand can stay on the mouse at all times, even when typing. However, character input speed with DataHand only reached 70 % that of the standard keyboard after practice for more than 13 h. Thus, learning the special key arrangement still presents some difficulties for character input. Based on these attempts of integrating a keyboard onto a mouse, an ideal interface should satisfy the following conditions: (1) the distance that fingers are required to move should be reduced and keys should be accessible while the hand is on the mouse and (2) learning the special key arrangement should be fairly easy. For this purpose, the best key arrangement is one that fits along the fingers and maintains the same direction of finger movement as is employed by the standard keyboard. This should eliminate difficulty learning a special key arrangement. Analyzing finger motion while typing on the standard keyboard is an effective way to determine the best arrangement based on these conditions.
Finger-motion analysis while typing on keyboards has been reported in several studies that measured finger-joint angle, joint-angle velocity, and joint-angle acceleration [11–13]. Although the displacement of finger-joint position trajectory is necessary to determine the key arrangement that requires the least finger movement, these data have rarely been reported.
We propose a keyboard-mounted mouse whose keys fit along the fingers, allowing minimal finger movement while the hand is placed on the mouse, and whose key arrangement is easy to learn. The conventional keyboard-mounted mouse is operationally satisfactory in terms of mouse function because the mouse is a simple interface that fits the hand nicely and whose only function is the control of cursor movement and selection via mouse click [8, 9]. However, the keyboard portion of the conventional keyboard-mounted mouse does not fit fingers effectively, and complex operations typically performed with a keyboard are difficult. Thus, slower character-input speed compared with the standard keyboard remains a problem for the conventional keyboard-mounted mouse [8–10]. One solution to this problem might be a keyboard design that fits along fingers, but studies of such a method have not been reported. If keys are to be fitted along fingers, multiple keys must be in close contact with a single finger. Because of this, a method for being able to separately press several different keys with a single finger must be part of the overall design. In this paper, we focus on designing the arrangement of keys that fit along fingers for a keyboard-mounted mouse. Our objective is to design and test a method for separately being able to press several keys with a single finger, and in the process, reduce finger-moving distance. To accomplish this, we tracked fingers while participants typed on a standard keyboard and used the data to derive a proposed key arrangement. We then developed a keyboard structure that incorporated this layout and tested our predictions. We used three-dimensional (3-D) finger-motion analysis to determine the distance that fingers traveled while performing a simple typing task, and recorded typing accuracy (Experiment 1). To determine how accuracy on the proposed keyboard compared with those of a standard keyboard, we computed error rates for participants who performed 20 trials of a typing exercise on each type of keyboard (Experiment 2).
Target keys on our keyboard-mounted mouse
We proposed a keyboard-mounted mouse that is used by the right hand because most people are right-handed and use a mouse with their dominant hands. Character keys are the most frequently used keys on a keyboard. When inputting character keys around the home position of the keyboard, people often “touch type” without actually looking at the keys. Normal typing divides the keyboard into a left and right side with the boundary being ‘Y’, ‘H’, and ‘N’ on the right and ‘T’, ‘G’, and ‘B’ on the left. However, ‘Y’, ‘H’, and ‘N’ could be typed with the left hand. Therefore, we focused on developing a keyboard for typing 12 characters around the home position on the right hand side: ‘U’, ‘J’, ‘M’, ‘I’, ‘K’, ‘,’, ‘O’, ‘L’, ‘.’, ‘P’, ‘;’, ‘/’. The 12 keys are arranged in the usual four columns, with a different finger assigned to the three keys in each column. Mouse clicks and scrolling are done by assigning the click keys and scroll keys (up and down) to some of the 12 keys. The thumb, which is not used for typing character keys on the standard keyboard, is used for operating the switch that toggles the device’s function between keyboard and mouse. The switch not only toggles between keyboard and mouse functions, but can also toggle the key map for the keys, similar to the ‘shift’ key on a standard keyboard. Thus, the 12 character keys can be made to represent the other keys on the right-hand side of a standard keyboard (Return, Delete, ‘-’, ‘=’, ‘[’, ‘]’, ‘\’, ‘’’, ‘←’, ‘→’, ‘↑’, ‘↓’) or the numeric keys (0–9).
Methods for using one finger to press three separate keys
A 3-D key arrangement that fits along the fingers may be effective for reducing the distance fingers move while typing, but because each finger joint touches different keys at the same time, typing accuracy might be reduced. However, with a careful key-arrangement design that considers the combinations and positions of finger-joint movements, a keyboard can be made such that three keys can be typed separately by each finger joint.
Moving some fingers individually is difficult because the finger muscles and corticospinal neurons are linked between fingers . Because finger extension causes more finger-linked movements than flexion [16, 17], using finger extension to separately type keys that are in close contact to fingers is difficult. Therefore, keys on our new keyboard are arranged such that they can be typed using flexion. By considering the finger motions depicted in Fig. 1b, we assumed that 3-D keys placed in the red areas depicted in Fig. 1c could be typed through flexion. We hypothesized that 3-D keys could be typed through flexion with limited finger movement and with as much accuracy as the standard keyboard if the keys in the red areas in Fig. 1c could sense initial finger motion and ignore subsequent interference by other joint movements. By analyzing finger motion, we should be able to determine the appropriate 3-D key positions for sensing the distinct and separate initial finger motions. We did so by determining which joint began moving first and how other joints subsequently moved during normal typing on a standard keyboard. Thus, we reasoned that placing the new keys in positions that sense the first movements of the joints, not interfering with tapping of the other keys, and matching the red areas in Fig. 1c, we would be able to develop a 3-D keyboard that can be used accurately through flexion but that does not require fingers to move as much as a standard keyboard.
Typing motion analysis
To minimize finger movement and to allow the three keys along each finger to be pressed separately, keys need to be arranged such that they are in close contact with the finger position that starts moving the earliest when typing, and such that extraneous joint movements do not interfere with pressing the target key. Thus, we analyzed typing motion on the standard keyboard. The objective of this experiment was to determine finger positions when typing on a standard keyboard, and then apply them to the 3-D arrangement.
Five healthy men (mean age 21.6 years; range 20–23 years; all right-handed) were asked to input character keys with the index finger, middle finger, ring finger, and little finger of the right hand. Informed consent was obtained from each subject before participation. The 12 characters were typed five times each, with each finger typing three different characters (60 character inputs for each participant). The keys ‘J’, ‘K’, ‘L’, and ‘;’ were set as the home position, and participants were asked to return their fingers to these four keys after inputting each character. We measured the 3-D trajectory of fingertips, distal interphalangeal (DIP) joints, proximal interphalangeal (PIP) joints, and metacarpophalangeal (MCP) joints during typing. Task instructions were as follows:away from the other
Place the palm on the palm rest and place the fingers naturally on the home-position keys.
Type the ‘U’, ‘J’, and ‘M’ keys five times each using the index finger.
Type the ‘I’, ‘K’, and ‘,’ keys five times each using the middle finger.
Type the ‘O’, ‘L’, and ‘.’ keys five times each using the ring finger.
Type the ‘P’ and ‘;’, and ‘/’ keys five times each using the little finger.
Results and discussion
Displacement start time of each finger part
y-axis start time (s)
z-axis start time (s)
y-axis start time (s)
z-axis start time (s)
y-axis start time (s)
z-axis start time (s)
Data from Fig. 4b also show that when typing a character in the bottom (lower) row, negative displacement in the y-axis is significantly early in the fingertip (df = 8, p < 0.001 vs. DIP joint y-axis, p < 0.001 vs. fingertip z-axis, p < 0.001 vs. DIP joint z-axis). Taking into account the red area in Fig. 1c so that the keys can be pressed through flexion, the 3-D key for lower-row characters should be placed at the fingertip in the negative y-axis (Fig. 5, right).
Figure 4c also indicates that when typing characters from the middle row, onset of negative movement in the z-axis occurs almost at the same time for the fingertip, DIP joint, and PIP joint, and t-tests show no significant differences. Thus, typing middle-row characters can be sensed at the same time by making the 3-D keys in close contact with the fingertip, DIP joint, and PIP joint. Taking into account the red area in Fig. 1c so that the keys can be pressed through flexion, and considering the proposed 3-D key placements for upper and lower keys, the middle 3-D key should be placed either in the same position as on a standard keyboard or at the middle phalanx (the part of the finger between the DIP and PIP joints) (Fig. 5, center).
We next used the data to verify that the proposed 3-D key positions can be pressed separately with a single finger. PIP joint movement used to press an upper key cannot interfere with other proposed keys because when the PIP joint moves to press an upper key, the fingertip moves in the positive direction in the y- and z-axes (Fig. 3), which is away from the other proposed keys. Fingertip movement to press a lower key might interfere with a middle key placed at the middle phalanx because when the fingertip moves to press the lower key, the DIP joint moves in the negative direction in the y-axis (Fig. 3). In contrast, fingertip movement to press a lower key cannot interfere with other keys if the middle key is placed in the same position as on a standard keyboard because fingertip movement is in a positive direction in the z-axis, which avoids pressing a middle key, and the PIP joint moves in a positive direction in the z-axis, which avoids pressing the upper key (Fig. 3).
Fingertip movement to press a middle key placed in the same position as on a standard keyboard or at the middle phalanx might interfere with other keys because the PIP joint and fingertip move in the negative direction in the z-axis (Fig. 3). However, because pressing a middle key requires only a small movement, by modifying the other key positions a little so that they cannot sense when the fingertip presses the middle key, fingertip movement should not interfere with the other keys.
Upper key: Vertically below the PIP joint of the proximal phalanx.
Middle key: Vertically below the fingertip (in the same position as on a standard keyboard).
Lower key: At the fingertip when the PIP joint is flexed.
3-D keyboard design
Character input algorithm
Users place their hand on the keys.
Each key position is adjusted so as to be in close contact with the fingers.
- 3.Users press each 3-D key five times, and the output of the force sensor is recorded (Fig. 12).
The maximum contact force Fmax N that was generated by unintentional touches is calculated.
Threshold is set to Fmax + 0.1 N (0.1 N is the force sensor specification for the maximum value of repeatability).
With correct calibration, only character codes from keys that are touched intentionally will be sent to the computer.
Experiment 1: Evaluation of the distance moved while typing and typing accuracy
The 3-D keyboard is designed to reduce the distance fingers must travel while typing, without reducing typing accuracy caused by inadvertently pressing undesired keys. The objective of this experiment was to evaluate whether the distance that fingers move when using the 3-D keyboard is less than that when using a standard keyboard, and whether the 3-D keyboard can be used as accurately. This was accomplished by measuring 3-D finger motion and the registered input when typing in a regular order.
We used the same equipment as was used in the analysis of typing motion. Participants were three healthy men (mean age 22.3 years; all right-handed). The experiment began after adjusting the keys to be in close contact with the fingers and performing the threshold calibration described above. The task was almost the same as in the typing-motion analysis experiment, except that each key was hit ten times. We measured the 3-D trajectory of the fingertips, DIP joints, PIP joints, and MCP joints during typing. The input characters were recorded in a text file. This experiment was carried out after obtaining informed consent from the subjects.
T-tests showed that when using the 3-D keyboard, the distance fingers traveled was reduced significantly in all keys compared with the standard keyboard (df = 4, p = 0.03 for upper-row keys, p = 0.007 for middle-row keys, p = 0.02 for lower-row keys). On average, distance was reduced 76 % for the upper-row keys, 58 % for the middle-row keys, and 88 % for the lower-row keys. When all keys were combined, the average reduction was 74 %. Participants did not make any typing errors in this experiment.
Experiment 2: Evaluation of typing accuracy when typing a series of characters
In Experiment 1, we found that typing with 3-D keys is accurate when typing in a regular order. However, this is very simple and differs from actual keyboard use. In Experiment 2, our objective was to compare typing accuracy between the 3-D and standard keyboards when typing a series of characters.
We used a 3-D keyboard and a standard keyboard in conjunction with a Windows XP computer running the software “Stamina Typing Tutor 2.5 ”, which can record error rate and typing speed.
In Experiment 1, we found that our 3-D keyboard can be used accurately, and that joint movement does not interfere with pressing the target key. Thus, our proposed method satisfies the condition that three keys can be pressed separately by a single finger despite close contact with all three keys.
In Experiment 2, two subjects were able to type 30 characters without any mistakes once in the 20 trials. This result shows that the 3-D keyboard can be used accurately even when typing a series of characters. While the initial trials contained more errors because subjects did not yet recognize the arrangement of keys, this type of mistake decreased as subjects got used to layout. By the last trial, the average error rate was only 18 %. Thus, 20 min of using the proposed keyboard reduced the error rate from 49 to 18 %.
This error rate was almost double that for the standard keyboard. This can be attributed to a software error in the proposed keyboard. Some erroneous inputs were generated by contact forces that exceeded the input threshold during the experiment. These errors seem to have been caused by finger-linkage motion that was different from that when calibrating the force threshold. The degree of finger linkage might have changed because finger movements changed as subjects became habituated to the 3-D keyboard. In the future, we should devise a method for updating the input-force threshold according to any adaptations to the 3-D keyboard made by the participants. Because the erroneous inputs were caused by finger linkage, further consideration will have to be given to pattern recognition methods that use a combination of force sensors to differentiate intentional input from unintentional input.
In Experiment 2, we also calculated typing speed. Average typing speed during the last trial on the proposed keyboard reached 92 % of that on the standard keyboard. Some participants reached maximum typing speeds on the 3-D keyboard that were faster than the average typing speed on the standard keyboard. Therefore, about 20 min of practice is all that is required to become accustomed to the 3-D keyboard and reach a typing speed equivalent to that of a standard keyboard. Changes in the software could improve the final speed and rate of improvement within the same amount of practice time.
Although we found some significant differences, the sample size was quite small. Therefore, we should conduct further experiments with larger sample sizes.
Here, we only focused on the keyboard portion of the keyboard-mounted mouse. In the future, we must equip the mouse function on the 3-D keyboard, and we must make the mechanism and algorism for switching the function between keyboard and mouse, or between different sets of keys. We also need experiments that evaluate the operability of our new device in tasks using both the keyboard and mouse.
Our objective was to design and test a 3-D keyboard design that allows each finger to control three separate keys, and in the process, reduce finger-moving distance. We analyzed 3-D finger motion while subjects typed on a standard keyboard, and used these data to develop a keyboard that accomplishes this goal. According to the results, the 3-D keys can be pressed separately without interference from other joint movements. Additionally, the distance that fingers moved when using the proposed keyboard was 74 % less than that when using the standard keyboard. Moreover, after using the proposed keyboard for about 20 min, the average input error rate was reduced to 18 %. We conclude that our proposed 3-D keyboard reduces the distance fingers move while typing and allows individual fingers to control three separate keys. In the future, we will devise a better character input algorithm that eliminates erroneous input caused by finger linkage. Further, we must add mouse functionality and the switching mechanism that toggles between keyboard and mouse functions. Then we will evaluate the operability of our proposed keyboard-mounted mouse when using both keyboard and mouse functions.
TS conceived the study, developed the device, carried out all experiments, analyzed the data, and drafted the manuscript. SM, YK and MF participated in the research design and sequence alignment. All authors read and approved the final manuscript.
This research was supported in part by the Outstanding Graduate COE Support Subsidy “Global Robot Academia (GRA)”, a Grant-in-Aid for Scientific Research (A) (26242061), and a Grant-in-Aid for Scientific Research (S) (25220005). All grants were issued by the Ministry of Education, Culture, Sports, Science and Technology in Japan.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Atkinson S, Woods V, Haslam RA, Buckle P (2004) Using non-keyboard input devices: interviews with users in the workplace. Intern J Ind Erg 33:571–579View ArticleGoogle Scholar
- Armbrüster C, Sutter C, Ziefle M (2007) Notebook input devices put to the age test: the usability of trackpoint and touchpad for middleaged adults. Ergonomics 50(3):426–445View ArticleGoogle Scholar
- Keskin C, Hilliges O, Izadi S, Helmes J (2014) Type–Hover–Swipe in 96 bytes: a motion sensing mechanical keyboard. ACM CHI conference on human factors in computing systems, p 1695–1704Google Scholar
- Tung YC, Cheng TY, Yu NH, Chen MY (2014) FlickBoard: enabling trackpad interaction with automatic mode switching on a capacitive-sensing keyboard. UIST’14 Adjunct proceedings of the adjunct publication of the 27th annual ACM symposium on user interface software and technology, p 107–108Google Scholar
- Habib I, Berggren N, Rehn E, Josefsson G, Kunz A, Fjeld M (2009) DGTS: integrated typing and pointing. Human-computer interaction—INTERACT, Volume 5727 of the series Lecture Notes in Computer Science, p 232–235Google Scholar
- Ulrich TA, Boring RL, Lew R (2015) Control board digital interface input devices—touchscreen, trackpad, or mouse. Resilience Week (RWS), p 1–6Google Scholar
- Scarlett D (2005) Ergonomic mice: comparison of performance and perceived exertion. Usability news from Software Usability Research Laboratory at Wichita State UniversityGoogle Scholar
- Slocum J, Thompson S, Chaparro B, Bohan M (2013) Evaluating the CombiMouse: a new input device for personal computers. Usability news from Software Usability Research Laboratory at Wichita State UniversityGoogle Scholar
- Slocum J, Thompson S, Chaparro B, Bohan M (2004) Examining first-time usage of the CombiMouse™. In: Proceedings of the human factors and ergonomics society 48th annual meetingGoogle Scholar
- Knight LW, Retter D (1989) DATAHAND: design, potential performance, and improvements in the computer keyboard and mouse. In: Proceedings of the human factors society 33rd annual meeting, p 450–454Google Scholar
- Baker Nancy A, Cham Rakie, Hale Erin, Cook James, Redfern Mark S (2007) Digit kinematics during typing with standard and ergonomic keyboard configurations. Intern J Ind Erg 37:345–355View ArticleGoogle Scholar
- Baker Nancy A, Cham Rakie, Cidboy Erin Hale, Cook James, Redfern Mark S (2007) Kinematics of the fingers and hands during computer keyboard use. Clin Biomech 22:34–43View ArticleGoogle Scholar
- Soechting John F, Flanders Martha (1997) Flexibility and repeatability of finger movements during typing-analysis of multiple degrees of freedom. J Comput Neurosci 4:29–46View ArticleGoogle Scholar
- Leijnse JNAL, Quesada PM, Spoor CW (2010) Kinematic evaluation of the finger’s interphalangeal joints coupling mechanism—variability, flexion–extension differences, triggers, locking swanneck deformities, anthropometric correlations. J Biomech 43:2381–2393View ArticleGoogle Scholar
- Aoki Tomoko, Francis Peter R, Kinoshita Hiroshi (2003) Differences in the abilities of individual fingers during the performance of fast repetitive tapping movements. Exp Brain Res 152(2):270–280View ArticleGoogle Scholar
- Yu WS, van Duinen H, Gandevia SC (2010) Limits to the control of the human thumb and fingers in flexion and extension. J Neurophysiol 103:278–289View ArticleGoogle Scholar
- Schieber MH (1991) Individuated finger movements of rhesus monkeys: a means of quantifying the independence of the digits. J Neurophysiol 65(6):1381–1391Google Scholar
- Kim JH, Aulck L, Bartha MC, Harper CA, Johnson PW (2014) Differences in typing forces, muscle activity, comfort, and typing performance among virtual, notebook, and desktop keyboards. Appl Ergon 45(6):1–8Google Scholar
- Stamina typing tutor by typingsoft. http://typingsoft.com/stamina.htm. Accessed 10 Jan 2015