Move-it: a paperclip-shaped sensing and actuating system for sticky notes

Abstract

We developed a paperclip-shaped motion-actuating system, named “Move-it”. With this system, users can add motions to sticky notes just by clipping it. This system is constructed by using clip-shaped devices that have sensors and actuators. The recognition system uses photo interrupter module as sensor. Each clip device can recognize which note it has been attached to by calculating how much infrared light is being reflected by the grayscale image printed on the flip side of each note. The actuation system is made of a coil-shaped shape memory alloy combined with a polyolefin sheet that is embedded to the clip device, and can provide a sufficient motion for actuating the note. By writing for example, “meeting 16:00” on the sticky note using a digital pen and our device, the user can get physical reminder without any complex setting. This paper presents the design and evaluation of the recognition and actuation system of our clip-shaped device.

Background

Introduction

Usually we use sticky notes as a reminder or prompt tag in the office or at home. Simply writing a message and sticking it allows us to attach to things or places information to them immediately. If this note can provide movement, an interactive information system based on one of the most familiar interface can be built. We present a system which can add active function to sticky note such as Post-it

BKF40012-xml0x.png○R\c:IMG. With this system, sticky note is not just an input interface but also an output interface to the user. Several types of sticky-note-based interaction which keeps the physical texture of paper can be provided with this system.

Move-it

We developed “Move-it” as a system designed for sticky-note-based interaction (Figure 1). We propose this actuating sticky note system as an interactive interface in the office and at home. With this system, user’s input and output interaction can all be accomplished on the sticky-note-itself.

Figure 1

Move-it.

For example, it becomes possible to bring controlled motion as a reminder to increase the reminder effect, or provide physical and quiet motion instead of annoying pop-up in a display. We believe this passive media, which many people is used to, can be an active information platform.

The system includes clip-shaped devices that are connected to a control PC via Arduino which is a kind of micro controller board. Each clip device has sensors and actuators embedded into it. Using infrared reflective sensors, each clip can recognize which sticky note it is clipped to, while a coiled shape memory alloy gives a controlled motion to the sticky note. Users can add motions to the note by clipping it.

Related works

Paper-based information processing

To integrate paper-based information and digital information, many challenges and inventions were proposed so far [1].

A piece of paper which has Radio Frequency IDentification (RFID) or printed circuit can carry various sorts of information as a tiny tag [2]–[4]. This approach to integration of paper-based information and digital information has the advantage of changeability of the digital information and tangibility of paper.

While at the same time, hand-writing recognition was one such approach. In the current era, high-precision hand-writing recognition and real-time writing assist is possible thanks to repeated upgrade of image processors and high-resolution image sensors [5].

For example, QUiCKiES [6] which is a combination of RFID and digital pen realized sticky-note-based real space tag system. The user can write and tag anything in real space. Then the user can search using graphical browser interface on a computer.

In addition, a small device called Anoto Digital Pen [7] can be used to send written strokes as digital information without any scanner or camera device. Furthermore, regular paper printed with the Anoto dot pattern in the background is enough for Anoto pen to operate. No special material and electric parts are needed to mount on the paper for this method. This is a great advantage for the purpose of sticky-note-based interaction because we can treat the sticky note like a ordinary one and throw away a waste sticky note. That is why we chose Anoto Pen as the input device of our paper-based interactive system.

Notification method

When dealings with paper-based interaction, the method of outputting information is also important. As a manner of notification of this interactive system, we decided the calm mechanical motion of sticky note for several reasons.

The main reason is the situation of assumed usage. Our focus of this research is paper-based interaction, especially establishing sticky-note-centered interaction system. The using situation we assumed is the same as the using scenario of ordinary sticky note, such as around office desk, or before a message board. In such a situation, alarm sounds sometimes disturb people around.

Furthermore, though it is difficult to distinguish a note which is generating the sound from a lot of sticky notes in front of a bulletin board, it is easy to recognize with adequate amount of change of visual stimulation, because human eyes have high spatial resolution than that of ears [8],[9]. Thereby, we decide to use visual stimulation as a manner of notification.

Of course a tiny LED can highlight a sticky note. However, a blight flickering light may disturb person in distance because decay by distance of light intensity is very low [10],[11]. Meanwhile, though an soft light can make visual stimulation, it is difficult to be noticed certainly because it is affected by environment light easily.

By contrast, the less distance between the moving object and the observer, the larger visual motion stimulation is perceived on the retina. That is, if a physical motion is selected as a output of this system, the notification does not disturb unrelated person but can be perceived sufficiently by a person nearby. Thus, we chose the calm mechanical motion of sticky note as a manner of notification of this interactive system.

Paper-based kinetic system

Furthermore, if there are no metallic materials attached to the note, the paper note itself is light and soft enough that even a tiny actuator can give it motion. Also, it is easy to throw away the note written mistake as flammable garbage. If the user want to change or cancel his/her reminder, it can be done by simply replacing the note.

Previously, in area of robotics and mechanics, there has been considerable research on artificial kinetic object. Recently, several actuation methods for paper are proposed, since this light and thin sheet is one of the most common material in the world.

Paper can not support heavy parts such as a motor, gear and battery due to its thinness and softness. Instead, it has the pliantness that even weak force can change its shape. Besides, Shape Memory Alloy (SMA) is one of the actuators often used to move paper.

SMA is a very light and small actuating element which generates contractile motion based on temperature. With SMA, many types of paper-based interactive work such as paper robot or prototyping toolkit have been proposed so far [12]–[14].

Applying this system and digital pen such as “Anoto pen”, a scheduling application which recognizes the written scheduled time with motion can be built. Just writing “meeting” and “16:00” on a note and clip it, the user can get reminder annotation at the right time.

Based on these researches and works, we developed a sticky-note-based interactive system by combining the input from digital pen and the output with motion which is made by SMA. This paper describes the construction and designing of this sticky-note-based interaction system. The contribution of this research is establishment of actuation method for notification by simple motion structure. We verified whether the generated motion provides a sufficient visual stimulation through the implementation of the paper-based interactive application.

Methods

System overview

Move-it consists of move-it-purpose notes, (clips which give motion to the paper), a digital pen for writing on the paper, and a control PC (Figure 2). To explain the sequence of this system and information flow, we use an application as an example.

Figure 2

System overview.

The move-it-purpose notes are specifically designed for each application. Boxes are printed on the note as input areas. The text, which the user writes in the input area using the digital pen, will be sent to the PC as a packet of strokes positions. Then, those strokes will be recorded as a data set of text and number such as “16:00, meeting” by using Microsoft’s handwritten-character-recognition SDK. After that, when the user attach the clip device to the note, the sensor embedded on the clip identifies which note is being clipped and the data set will be linked to the clip device. When the actuating data set is,(e.g at 16:00) the control PC actuates the SMA on the clip to create motion for the note. The detail of part is as follows:

(1) Input

We adopt Anoto Pen as the input device for the characters written on the note. Anoto Pen is a ball-point pen which has an infrared lamp, a small image sensor, a Bluetooth communication module, memory, processor, and a battery, in the body (Figure 3). Anoto dot pattern which is designed for Anoto Pen, consists of small black dots printed at intervals of about 0.3 mm, and the pattern arrangement of the dots can be used to uniquely identity the position of the paper. By identifying this dot pattern with the image sensor embedded at the tip of the pen, Anoto system can specify, record and transmit correctly what is written on a 60 million-square-kilometers-wide generable Anoto pattern.

Figure 3

Anoto Pen.

(2) Note recognition

For sticky note identification, an infrared interrupter is arranged to each clip. Infrared interrupter is an element which combines the InfraRed Emitting Device (IRED) and an InfraRed (IR) phototransistor in a small case (2 mm square, 1 mm thick) (Figure 4). It can detect the reflection of the IR light which is emitted from the IRED. Calculating how much the amount of IR is reflected and detected, we can use this device as a close-range sensor [15],[16].

Figure 4

Sensing system.

In this research, we use this device for sticky note identification by measuring the infrared reflection from a gray colored image printed on the flip side of the note, because the darker gray picture reflects the least IR. By detecting the IR reflection from the different gray colored image, the clip can grasp which note is clipped into.

(3) Processing

The input from Anoto pen is transmitted to the PC via Bluetooth as a packet of data. The data gets pattern analysis using handwritten-character-recognition SDK which Microsoft has published, and a dataset including written characters, note number, and input area is created. In addition, the control PC and clip-shaped devises are connected via Arduino board, and serial communications can be performed at anytime. The dataset is linked to the ID number of the clip device. Thus, the pieces of information such as written contents and ID number of the clip device are unified. If the dataset’s actuating condition is satisfied, the control PC switches on the actuator placed on the tip of the clip device.

(4) Clip actuation

We chose Bio-Metal helix(BMX) [17] for the actuator of Move-it. BMX is a coil-shaped SMA made of titanium and nickel. When an electric current is active and the temperature of the BMX reaches 70 degrees by the generated electric heat, it shrinks. BMX is mounted on a thin polyolefin sheet attached to the clip. When the BMX is shrunk by the current, the sheet of polyolefin is bent and when the current which was being sent is stopped, the polyolefin returns to the original shape quickly, due to the elasticity of the polyolefin sheet (Figure 5). Moreover, the time and the shrinkage can be controlled by changing the signal from the PC via Arduino.

Figure 5

Actuation method.

The following sections describe the compositions and implementations of the clip-shaped device in detail.

Implementation

Note recognition system

The requirement for designing the sensor part is to establish a reproducible method for identification of the note which is pinched by the clip device. Paper is an easily deformable material. The note can change the shape easily. However, it needs to be identified as the same note even if sensed by different devices.

The former prototype was developed based on wired paper-clip. The clip made by bent wire with some sensors and an actuator could hold paper note, recognize it, and move it. But the distance between the sensors and the note was unstable because of elasticity of the clip (Figure 6). Though the prototype could recognize some notes with proper thresholds, it was difficult to keep the stability of recognition. So we developed a clip-shaped device rather than paperclip-shape to stabilize the distance between the sensor and the paper.

Figure 6

1st prototype.

Design

A photo interrupter (SG105, Kodenshi corp.) is embedded on the tip of the clip device. To detect only the reflected IR from the note, the position of the IR phototransistor should be near from the note (Figure 7). If the note is closely-attached to the IR emitter and the sensor, the housing of the module interrupts the reflected IR and the sensor will have difficulty detecting it. We therefore set a 1 mm distance between the paper and the photo interrupter module by reference to the sensing position characteristics of data sheet of SG105 [18].

Figure 7

Clip device.

A resistance for anode terminal of the IRED is prepared to adjust the current supplied to 10 mA on the IRED and a 5.1 k Ω resistance is connected parallel to the IR phototransistor.

Actuation system

With the current method of mounting a Shape Memory Alloy on the paper directly, it will be difficult for the user to change or put away the notes, because of the lines, adhesive and cost. Additionally, the speed of getting back to the original shape depends on the elasticity of the paper.

Therefore, we developed the actuating technique using the SMA mounted polyolefin sheet for pushing and bending the paper. Because of the actuation mechanism, the note can be moved calmly, without making any gear noise and vibrating sound. Even in a quiet environment, the actuation and notification can be performed.

Design

We used BMX150 (Toki corp.), a kind of coil-shaped SMA, as an actuator of Move-it. The diameter of BMX is 0.15 mm and the coil diameter is 0.62 mm.

The reason why we used BMX as an actuator is due to the extraordinary shrinkage ratio. When the BMX is heated to about 70 Celsius degrees, it shrinks to a maximum of 50% in length. By contrast, the shrinkage ratio of a sister product, Bio-Metal Fiber (Toki corp.) which is a not coil-shaped, but rather straight line-shaped SMA can only shrink to 4% in length.

The electric power which actuates this BMX is controlled by the control PC and supplied through an Arduino board. We arranged DC 3.3 V, max 2 A power supply independently and put several FET for switching the current supply for each BMX using Arduino because the current that a piece of BMX requires for an actuation is about 100 mA though the current that an Arduino can supply is a maximum 50 mA.

The resistance of BMX can be changed with temperature dynamically. Resistance becomes lower as temperature is high. Therefore, if a current is flowing continuously, the temperature can get too high and eventually the circuit can become short. To avoid this, a 0.1 Ω of resistance is connected to the circuit of BMX. Simultaneously, this resistance takes on the role of overheat prevention.

Meanwhile, the BMX force required to get the paper back to the original shape is very weak relative to its pull force. Usually, in order to maintain the repeatability of repetitive actuation and quick motion, a spring or a leaf spring is used. In this implementation, we used a 0.2 mm thickness polyolefin sheet as a leaf spring for simplifying the actuating mechanism.

Results and discussion

Evaluation of the note recognition system

We printed different depth of gray colored image on a 4 cm square regular paper. The depth of the color of the gray square increases at 5% increments from 0% (“White”) to 100% (“Black”). Using the set of 21 sheets of single-side printed paper, we tested the sensor parts on the 3 clip devices. Each note got pinched by the device 4 times. Every time the note is pinched, the analog data was recorded and that paper was rotated 90 degrees. Additionally, the input values of the state when the device pinches nothing (“Default”) and when the clip was opened (“Open”) were also recorded.

The robustness of the recognition of sticky notes depends on the thresholds at which two different grayscale images can be discriminated, and the repeatability of sensor values. The graph shows the values under the different conditions (Figure 8). That means the small individual differences between each device make influences the value. The standard deviation of the value was 48.17 at a maximum. If the thresholds is set based on these values, the number of distinguishable grayscale pictures are limited. We calibrate each device using the values of “Open” and “White” in order to improve the standard deviation.

Figure 8

Graph1: Depth of the grayscale image and the analog value.

As a result of the recalibration, a second graph shows that the variation in the data has been reduced and stabilized (Figure 9). The maximum standard deviation was reduced to 17.07. Thus, one tiny photo interrupter can distinguish 13 levels including 11 levels of grayscale from white (0%) to black (100%), “Open”, and “Default” with considerable accuracy.

Figure 9

Graph2: Depth of the grayscale image and the calibrated value.

Evaluation of the actuation system

(1) Conditions

We verified whether the motion is enough for a visual notification from the viewpoint of human’s visual feature. The threshold value which human’s eye can distinguishes motion is called minimum motion threshold (MMT). That value depends on temporal frequency, spatial frequency, contrast, exposure time, and viewing field.

Since the assumption usage environment of Move-it is for attaching around PC or a bulletin board, the distance from the observer to the note is assumed to be 2 m at the maximum, in a bright-lit room. Additionally, the motion should be visually perceptible even in a peripheral vision. In human’s visual characteristics, the value of MMT is almost the same as the power of vision in 55 degrees measured retina eccentricity. The minimum motion distance threshold is considered 20’ with 55 degrees measured retina eccentricity and 1 second exposure time [19].

The size of the small note used by Move-it is 75 mm square-based on the size of a typical sticky note. The perimeter of the note is surrounded by black lines in order to maintain a certain intensity of contrast for the movement perception. Using this note and a clip-shaped Move-it device, the time length which the note takes to bend to 45 degrees is measured.

(2) Result

The measurement was held in a calm room. The temperature was 24 degrees Celsius. First, we measured the minimum time it takes to bend the paper at 45 degrees bend angle using the clip device. We observed that it takes 800 msec to bend the note to 45 degrees.

Next, we measured the shortest time that the heated BMX needs to be cooled for continuous repeating motion. After the 800 msec heating time, we set a definite period of time as cooling time for heated BMX and repeated that actuation for 10 times. If the cooling time was not enough, the BMX could not get back to the original length and the paper could not become flat. We set 3000 msec for the cooling period at first, and gradually decreased the cooling period to evaluate the minimum time for stability behavior.

As a result, we observed that it requires at leased 1200 msec for the repetitive actuation to be successful. That means it is able to actuate at 0.5 Hz to 45 bend angles with Move-it.

Within this motion parameters, the distance that the side of the note traveled was 11.7 mm in 800 msec. Converting the length to viewing angle of the observer from 2 m away, it was 20.1’. This distance exceeded the minimum motion distance threshold in a peripheral vision, and the motion is enough for notifying a person who is 2 m away. Moreover, if the distance from this note to the user is less than 2 m, or the user is observing the motion with their central visual field, it would be easier for the user to perceive (Figure 10).

Figure 10

Result: enough motion for annotation.

Discussion

Safety

In our implementation of Move-it, a shape memory alloy is heated to about 70 Celsius degrees at the maximum. At this temperature, the polyolefin sheet do not melt and paper do not burn. However, if the user touches the BMX directly and hold it for a few seconds, it may reach the temperature which may burn his or her skin.

To protect the user from touching BMX directory, first, the BMX is mounted between paper and a polyolefin sheet. It prevents also touching a user’s skin directly accidentally.

Moreover, since it can be detected whether the paper is pinched or not by the sensor built in the clip, the current flowing to the BMX can be stopped by the control PC when nothing is pinched or when paper is removed. This also prevents a user from touching the BMX in a heating state.

Limitation

The construction of current actuating system is very simple; it is just a plate and a wire, so it does not need any special mechanical grip or adhesive. It is easy to replace if there is something wrong in the actuating part.

However, it may be harder to actuate a larger paper. If the size of the paper is A4 size (210 mm × 297 mm), the motion may be slow and small that the observer can not perceive. Additionally, since each actuator requires about 100 mA for its actuation in current implementation, downsizing, unwiring, and power-saving technology is essential for the deployment from place to place in living space.

For this problem, improvement of the circuit, or different types of actuators such as electromagnetic actuator and membrane actuator could be the solution. As well, energy supplying system like wireless power transmission might be needed.

Similarly, regarding the sensing part, we showed that a simple tiny sensor could recognize 13 levels of grayscale. 13 is never a large number. However, we think that scanning a printed grayscale bar-code by multi sensors embedded in a clip device is one of the solutions of this limitation. We can generate more recognizable pattern by a combination of multi-sensor recognition or time-resolved grayscale recognition in the future.

Moreover, if the clip device has a image sensor which is the same as Anoto’s, it is needless to print grayscale image on the back side of the note, and the sensor can recognize a huge range of field with high accuracy. This simple implementation can also reduce the cost and risk. This is meaningful for a scenario where the user uses this system freely, not only around PC but also around refrigerator or a mirror.

Application

A feature of this system is that an interactive system where a user only interact with pen and paper can be built. To illustrate that, a part-time job information bulletin board is taken as an example application (Figure 11).

Figure 11

Application; a part-time job information bulletin board. Employers can add their profile using a sticky note and combining it with an active paper-clip. Once prospective employees are looking for a job, they simply fill-out another sticky note and get immediate feedback from the job board.

An employer takes a note and puts checks into the check boxes about the skills that the job needs, the age range, the available times, and writes occupational descriptions. The note is designed for a person who is looking for part-time staff and the contents the employer checked are sent immediately as a condition dataset.

When another user who is looking for a part-time job comes to the bulletin board, takes another note and puts checks into the check boxes which is suited to the user’s own competence, the notes which have appropriate information for the user will make a motion to let the user know where the suitable job is. Moreover, it is possible to show the degree of fitness by the pitch of the motion of the notes. This system helps person to find suitable information from the many information quickly.

Thereby, users can enjoy the benefit of the information processing capacity which a computer has, by the natural act such as writing and touching the paper. This system do not require the user to study any special operation method or technology in order to use it.

That is, with this system, it can be said that it is possible to merge analog abilities such as viewability, readability, and tangibility, with digital abilities such as variability, editability, and searchiblity.

Conclusion

In this paper, we present the idea of a sticky-note-based interactive system and described our implementation of a clip-shaped sensor-actuator device in detail.

It is shown by our sensor system that a small simple sensor can distinguish 10% differences in strength of the printed grayscale paper. In addition, with our actuating system, the motion of the paper can reach a level that is enough to be detected as a motion in peripheral visual field of human’s eye.

With a clip-shaped device which has a small actuator and a sensor, we can provide a motion as an active function to a piece of paper without any loss of the characteristics of the paper.