Concept proposal and experimental verification of a sidewalk supporting system utilizing a smartphone
© Muto et al. 2016
Received: 11 June 2015
Accepted: 17 February 2016
Published: 27 February 2016
In this report, we propose a system to be used as a basis for the efficient gathering of information on sidewalk usage and services for the purpose of building a system to support social infrastructures through which users can move safely and securely. We assume services for extracting and providing useful information to sidewalk users and local governments. Specifically, we propose a concept for a sidewalk supporting system that accumulates information from strollers, mobility scooters, bicycles, etc., through sidewalk users’ smartphones, sensors, and other such devices to provide valid information. Next, we show a proposed basic system configuration utilizing data collection technologies that have been developed in robots and walking appearance analysis with regard to responses to the diversity of sensors and the characteristics of smartphones. Furthermore, we introduce techniques for estimating sidewalk conditions using the subjectivity of sidewalk passers-by with regard to the acquisition of training data to analyse data effectively. As a specific application case, we evaluated the estimation of the gradient and unevenness of sidewalks by machine learning from the sensor data during the passage of strollers, and verified the effectiveness of this method.
KeywordsSidewalk supporting system Personal mobility Smartphone Sensor
Along with the spread of smartphones and cloud technology in recent years, an environment of collecting and accumulating records of individuals’ activities or so-called life logs through networks, has been prepared. Life logs include internet operation histories, application portfolios in place, and GPS information, that occur during daily activities. As an example of its application, a system has been proposed to estimate individuals’ preferences and life patterns from the accumulated data to provide a personalized information service . In addition, wearable devices  have generally been spread in a rapid manner, and a wide variety of services using sensors and network terminals owned by users have been considered.
To date, robotic rooms [3, 4] that assist users by managing a wide variety of sensor information installed indoors and studies that assume robot services in the street or in multiple dwellings [5–7] have been proposed as a framework for managing and utilizing a plurality of sensor information. Authors have built a network robot platform for sharing information with robots that deals with many types of sensor information with a plurality of guide systems [8–10]. Here we introduce abstracted information from sensors and robots in the form of 4W (who/where/when/what) user information or robots based on the XML format. Furthermore, while expanding on this, we have built a system to perform operations such as the physical transportation of goods in addition to the guidance of only providing information, and we have linked the information from a wide variety of sensors and robots [11, 12]. An approach has been adopted that streamlines the system by handling only the information which has been abstracted to a limited form. Through these methods, it is possible to accumulate information from multiple sensors designed and arranged for the purpose of measuring the specific information in the particular experimental environment and to capture it as an actual example of its application. It can also be said that these are examples where the data collector side built systems to collect data primarily with sensors whose specifications were defined in advance.
Among the wide variety of sensors and robots that are researched and developed, research and development of middleware and operating system from the viewpoint of software development include RT-Middleware  as a common platform that uses a network-distributed component technology. In RT-Middleware, devices such as servo motors, sensors, and cameras and the functional elements of robots such as various processing programs, are processed into RT components, which are then linked to develop a system. Moreover, in ROS , an environment is provided in which devices such as actuators and sensors and various processing programs are modularized or processed into components to develop software. They enhance the reusability of resources by processing various functional elements into software components to aim at efficient software development. Furthermore, in SensorML , where sensor information is managed by utilizing XML technology and focus is on the discovery of sensors and the acquisition of positional information, there are examples of utilization for the purpose of collecting information on the population and states of pollution by dividing the area into specific regions of widely distributed sensors.
In parallel with the evolution of demonstration experiments at laboratories or specific facilities and the preparation of system development environments as described above, services using sensors built in smartphones carried by users and information from connected sensors are thought to increase along with the rapid spread of smartphones in the future. If we look at such a point of view, in the demonstration of robotic room [3, 4] and multiple robot cooperation [5, 12], a particular device is determined in advance, it is also fixed function sharing the server and the client. Also the research and development of middleware [13–15] related development efficiency software is improved. But the overall system design is dependent on the developer. Here, the efficient and usable systems are considered to be important in the process from the early experiment stage to commercialization. It is considered to be important to change the flexible function sharing in consideration of the client’s capabilities and performance in the experimental stage. In addition, responses to the diversity of smartphone models and changes in their performances as well as the standpoint of dealing with the subjective evaluation values of users, which cannot be automatically acquired by sensors, are also important.
The authors have conducted studies on understanding the health status in those who suffer from rheumatoid arthritis as an application in the medical field using smartphone sensors, based on a platform [8–10] for liaising information to the robot as described above. The joint destruction of limbs caused by rheumatoid arthritis lasts for many years, and has become a social problem leading to severe motility disturbance and dysfunction in daily life activities . It is necessary to accurately grasp the daily health conditions of patients undergoing this treatment, and thus objective, quantitative, and temporal evaluation methods are sought. For these situations, the authors performed phased experiments to build a system for estimating the disease activity score 28 (DAS28) without using blood test values from the information on gait that was obtained from the acceleration information of patients’ smartphones. This was done while patients walked and self-reported information that they entered in their smartphones, verifying its effectiveness [17–19].
In this study, we propose technology for efficient information gathering as to situations involving sidewalks based on sensor information gathered from the smartphones of sidewalk-users. A sidewalk supporting system is used as the basis of the service for properly managing the collected data and results and extracting useful information such as that on traffic availability. This is then provided to sidewalk users, local governments, and others with the goal of expansion into daily life, including the aforementioned application to the medical field.
As conventional studies for detecting the states of road surfaces, a method of detection has been proposed which uses a smartphone’s acceleration sensor fixed to the vehicle against the differences in level that become roadway traffic obstacles . With regard to sidewalks, there is a test example of detection of surface damage using mobile robot equipped with laser scanner . And data specifications for organizing sidewalk information have been proposed , and appropriate information presentation has been demanded as a social need for people who find it difficult to pass along streets, such those who use wheelchairs or strollers. Technical points needed to build services systematically against such problems could include information gathering technologies through the carrier network from smartphone sensors with the aim of collecting sidewalk movement logs, which are less burdensome for users.
In this paper, we propose a concept for a system that accumulates information from strollers, mobility scooters, bicycles, etc. through sidewalk users’ smartphones, sensors and other such technology to provide valid information for the purpose of building a system to support social infrastructure where users can move safely and securely. Then we show a proposed basic system configuration utilizing data collection technologies in robotics and gait analysis for responding to the diversity of sensors and the characteristics of smartphones. By using this system, it can be handled physical data of a stroller, a bicycle and a wheelchair, the user’s movement as well as the user’s subjectivity data and vital data. Furthermore, we will introduce a method for estimating the conditions of sidewalks utilizing the subjectivity of sidewalks passers-by with regard to the acquisition of training data to analyze data effectively. As a specific application case, we evaluate the estimation of the gradient and the unevenness of sidewalks through machine learning from the sensor information and users’ subjective information during the passage of strollers to show the effectiveness of this technique.
We assume a method of efficient data collection utilizing users’ terminals by utilizing a mechanism for transmitting information on the sensors built into the smartphones (such as acceleration sensors, GPS, and attached sensors as the sensor information from bicycles, strollers, wheelchair, etc. that use sidewalks) into the cloud server system through the smartphones possessed by users. Based on the acceleration and GPS information, etc., it becomes possible to register information on whether users could move smoothly on the sidewalks they passed through or if they faced any difficulty in moving in situations with relatively significant acceleration changes.
What should be the focus of this report is the type of sensor and server connection function to move with the user mobility; and a system configuration with greater coverage is considered possible as a form to supplement this by linking to a system in which sensors are fixed, etc., as shown in Fig. 1(f). For example, it is desirable to collect information in sufficient temporal spatial density with a fixed sensor when measuring the sunshine and temperature on the sidewalk, etc., but it can be assumed to be difficult to install a device having the capability of network connection to the server in the sensor function at a sufficient density in terms of its costs. In this situation, the construction of a more efficient information collection system is considered by introducing the smartphone connecting to the server and the sensor having a radio function such as nearby WiFi and Bluetooth.
Basic configuration of the data collection system
In this chapter, the authors propose a sidewalk supporting system that utilizes the XML technology based on the network robot platform [8–10], and on the gait measurement system that is conducted from the acceleration information of a smartphone .
As a local server, we assume a computer having a network function disposed within a certain extent of equipment. The specific processing contents of the local server are considered to include processing that handles information for several clients, the execution of processing of information that the client cannot or preferably should not process due to the CPU power or batteries, and the execution of processing in cases where the network connection has not been made with the main server or where the environment of the network connection with the main server is bad. In addition, the execution of processing information, including the destruction of information that does not have to be left on the main server on a long-term basis, the destruction or anonymity of information that users do not desire to leave in the long term, and the calculation of statistical data. If there is no need for the processing described above, it is possible to configure the system with only the client and the main server.
As the main server, we assume a server system that is built in a cloud environment. Based on the capability of accumulating and managing data if they are those described with a predetermined XML technology, it facilitates their continued accumulation even if the client’s equipment type is changed. The construction of an application that is able to handle a wide range of data is anticipated through the installation of a management function that is available to an application user with the specified authority.
Client: a common function that turns the data transmission and reception functions with the server into the library and an application function requiring a high level of responsiveness such as real-time sensor feedback within the client.
Local server: an application function that requires several clients’ information.
Main server: a common function that accumulates and manages smartphones, sensors and application functions using such information.
The database in each stratum handles the basic information of sensor values themselves and the information on road surfaces, enabling the description of an application in accordance with the response speed and the computer power in each stratum. When considering the construction of several applications in the system configuration described above, it becomes possible to share the data management unit of the main server and make responses efficiently only by building application portions of each stratum. Furthermore, we consider that this system can make responses flexibly when there is a variation in measurement characteristics such as sampling time due to a change in terminal types, when the function as the application is improved by adding new information, or when the function deployment is modified among the client, the local server, and the main server.
Information description with XML
Information transmitted from a smartphone to a server, etc., is described in FDML (Field Data Mark-up Language) format . This description format is the XML format that has been developed to uniformly describe a variety of sensor information, which is a configuration allowing flexible responses even when an environment is generated where new sensor information can be acquired or when there is a need for transmitting the information that the users input into their smartphones. On the server side, the management unit for executing information management stores the information transmitted from the smartphone into the appropriate database with the assumption that it is information in the FDML format and manages the information in the database to be made available in response to requests by the applications.
By generating sensor data and information associated therewith in the smartphone as described above, it would be possible to handle the information from sensors built into smartphones and devices or sensors connected to WiFi and Bluetooth (e.g. control information for electric wheelchairs) in a unified manner. The XML format introduced here has past records of performances in handling data from the manufacturing equipment and sensors in factories, the information from sensors in an environment utilized by the guidance service robot, and the information from various types of smartphone sensors from the earlier days to date; and thus it is considered able to respond flexibly to the future development of these devices and the sophistication of the personal mobility.
Because it can handle users’ input values to the smartphone applications similarly, subjective information, such as users’ comments on how difficult their movement is and the information that users can understand at a glance when they see it, can be collected efficiently, even if it cannot be automatically determined with the sensor. It is difficult to impart all of such users’ subjective data to the sensor data. But the advancement of the automated analysis can be expected with the techniques such as machine learning by collecting a set of users’ subjective data and sensor data to be a certain amount of training data. It can be handled physical data of a stroller, a bicycle and a wheelchair, the user’s movement as well as the user’s subjectivity data and vital data synthetically by introducing FDML.
Examples of sensor data and applications
Estimation experiment of road surface conditions of the sidewalk using the training data of users’ subjective view.
Here, we discuss the method of estimating sidewalk conditions using the sidewalk users’ subjectivity and results of the verification experiment as the basic experimental study of the information collection system for sidewalk road conditions. We assume that it is intended to improve the convenience for sidewalk passers by analyzing the information that can be collected easily through the passage on the sidewalk without conducting special research activities on each sidewalk and providing information on hindrances. As a specific example, we show the results of evaluating the estimation by machine learning for the gradient and unevenness of the sidewalk from the sensor data during the passage of a stroller.
[Shape of road surface] flatness/slope/composite gradient.
[Property of road surface] smoothness/unevenness.
The gradient is comprised of the upstream/downstream gradient (longitudinal gradient) and the left/right gradient (cross grade) with respect to the traveling direction. The composite gradient refers to the shape of the road surface that includes the cross grade addition to the longitudinal gradient. There is a report that such a gradient becomes a major factor in increasing stroller users’ body burdens and senses of danger . The composite gradient was determined to be the label of the road surface shape in addition to that of the flatness and slope, and considered to be an example of a place recognized subjectively to be difficult to pass through, as it is difficult to be measured by objective criteria. Furthermore, it was expected that problems to the label estimation model would be clarified by adding the composite gradient.
In the data collection, sensor data were acquired by passing through the sidewalks with a gradient or unevenness located around the NTT Yokosuka Research and Development Center, while a smartphone was fixed to the stroller. In order to collect information on what impedes passage through sidewalks, such as gradients and unevenness, data were acquired and recorded from built-in sensors in regard to GPS, angles (between the gravity direction and the smartphone), and angular velocity at a regular interval during the passage through the sidewalk. It was assumed that the angle represents the slope and that the angular velocity reflects the road surface properties such as unevenness. The data acquisition interval was determined to be approximately 1 s for GPS and 20 ms for others. Additionally, the duration of sensor data acquisition is approximately 45 min. We used the “Caldia Auto 4-Cass Egg Shock HB” manufactured by Combi Corporation for the stroller and ARROWS NX F-01F (OS Android 4.2) by NTT DoCoMo for smartphones. The smartphone was fixed to the pipe-shaped frame so as to lay to the left and right rear wheels of the stroller using fixing equipment manufactured by National Products, Inc.: RAM MOUNTS X-Grip Holder.
[Road surface shape] flatness/slope/composite gradient.
[Road surface property] smoothness/unevenness.
Results and discussion
Conditions of experiments
Road surface shape
Mean and variance of pitch angle (1 s = 50 points)
Mean and variance of roll angle (1 s = 50 points)
GPS: 1 s
Others: 20 ms
Road surface property
Mean and variance of pitch angular velocity (1 s = 50 points)
Mean and variance of roll angular velocity (1 s = 50 points)
Evaluation result for road surface shape
Evaluation result for road surface property
The figure in each column in Tables 2, 3 represents the number of data whose respective labels were estimated for the sensor data considered to be unlabeled. Italics numbers represent the number of data whose estimated results are correct. In addition, the recall ratio represents the percentage of the data correctly estimated for labels. The precision factor represents the ratio of correct answers for the estimated results.
Looking at the results, angle sensors and angular velocity sensors were effective in determining road surface shapes and properties, respectively. However, the relevance factor of the flatness for the composite gradient: 83.4 %, and the recall ratio of the composite gradient: 88 %, are lower values as compared to others in the estimated results for the road surface shapes. The estimation of composite gradients is likely to be less sufficient with only angle sensors. In addition, composite gradient points involve the difficulty or reluctance to press for the travelling of strollers as uneven points. In order to improve estimation accuracy, we plan to proceed with the efficient collection of sensor data and user-subjective training data based on the proposed system, considering that if such subjective experiences in the difficulties in passage affect the acceleration and the angular velocity, there is a possibility to improve the estimation accuracy of the composite gradients using their feature amounts.
In this report, we proposed a system to be the basis for efficient information gathering on sidewalk situations and services for extracting useful information and providing it to sidewalk users and local governments for the purpose of building a system to support social infrastructures through which users can move safety and securely. Specifically, we proposed a concept for a sidewalk supporting system that accumulates information from strollers, mobility scooters, bicycles, etc. through sidewalk users’ smartphones, sensors, and other such devices to provide valid information. Next, we showed a proposed basic system configuration utilizing data collection technologies that have been developed in robots and walking appearance analysis with regard to responses to the diversity of sensors and the characteristics of smartphones. Furthermore, we introduced techniques for estimating sidewalk conditions using sidewalk passers’ subjectivity with regard to the acquisition of training data to analyse data effectively. As a specific case of application, we evaluated the estimation of the gradient and unevenness of sidewalks by machine learning from the sensor data during the passage of strollers, verifying the effectiveness of this technique.
Disease Activity Score 28
Field Data Mark-up Language
Robot Operating System
Support Vector Machine
eXtensible Markup Language
SM proposed the concept of the sidewalk supporting system, designed the information system, analyzed experimental results and drafted the manuscript. YN helped to design the information system, built an experimental system and did the verification experiments. TI and HI built an experimental system and did the verification experiments. MO analyzed experimental results and helped to draft the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Tezuka H, Ito K, Murayama T, Seko S, Nishino M, Muto S, Abe M (2011) Restaurant recommendation service using lifelogs. NTT Tech Rev 9(1):1–6Google Scholar
- Mann S (1997) Wearable computing: a first step toward personal imaging. Computer 30(2):25–32View ArticleGoogle Scholar
- Sato T, Harada T, Mori T (2004) Environment-type robot system “robotic room” featured by behavior media, behavior contents and behavior adaptation. IEEE/ASME Trans Mechatron 9(3):529–534. doi:10.1109/tmech.2004.834650 View ArticleGoogle Scholar
- Noguchi H, Mori T, Sato T (2006) RDF sensor description for heterogeneous sensors in human behavior monitoring environment. In: Proc. of the 2006 IEEE Int. Conf. on robotics and automation, pp 4369–4371, 15–19 May 2006
- Sanfeliu C (2009) URUS Project: Communication systems. In: Proc. of workshop on network robot systems at 2009 IEEE/RSJ Int. Conf. on Intelligence robots and systems (IROS09-WS), 12:31, 10–15 Oct 2009
- Kim H, Cho Y.J, Oh S.R (2007) Implementation and application of URC server framework. In: Proc of workshop on Network Robot System at 2007 IEEE Int. Conf. on Robotics and Automation (ICRA07-WS), No.SF-2-5, 23:27, 10–14 April 2007
- Dustbot (2006) Dustbot - Networked and cooperating robots for urban hygiene http://www.dustbot.org/. Accessed 08 May 2015
- Tezuka H, Katafuchi N, Nakamura Y, Machino T, Nanjo Y, Iwaki S, Shimokura K (2006) Robot platform architecture for information sharing and collaboration among multiple networked robots. J Robot Mechatron 18(3):325–332Google Scholar
- Nakamura Y, Machino T, Motegi M, Iwata Y, Miyamoto T, Iwaki S, Muto S, Shimokura K (2008) Framework and service allocation for network robot platform and execution of dependent services. Robotics and Autonomous Systems Journal 56(831):842Google Scholar
- Muto S, Shimokura K, Nakamura Y, Tezuka H, Abe M (2010) Feasibility study of platform-based network robot systems through field experiments. IEICE Trans Inform Syst D J93-D(10), 2240:2256 (in Japanese)
- Nakamura Y, Muto S, Motegi M (2012) Mizukawa M (2012) Proposal of active information supplement platform based on 4w information. IEEE/SICE Int. Symp Sys Integr (SII) 951(956):16–18. doi:10.1109/SII.2012.6427284 Google Scholar
- Nakamura Y, Muto S, Maeda Y, Motegi M, Takashima Y (2014) Proposal of framework based on 4W1H and properties of robots and objects for development of physical service system. J Robot Mechatron 26(6):758–771Google Scholar
- Ando N, Suehiro T, Kitagaki K, Kotoku T, Woo-Keun Y (2005) RT-middleware: distributed component middleware for RT (robot technology). In: Proc. of IEEE/RSJ Int. Conf. on intelligent robots and systems (IROS2005), pp 3933–3938, 2–6 Aug. 2005
- Quigley M, Gerkey B, Conley K, Faust J, Foote T, Leibs J, Berger E, Wheeler R, Ng A (2009) ROS: an open-source robot operating system, In: Proc. of open-source software workshop of the Int. Conf. on robotics and automation (ICRA), 12–17 May 2009
- Open Geospatial Consortium (2007) SensorML. http://www.ogcnetwork.net/SensorML. Accessed 08 May 2015
- Murphy D (1996) Lycra working splint for the rheumatoid arthritic hand with MCP ulnar deviation. Aust J Rural Health 4(4):217–220View ArticleGoogle Scholar
- Nishiguchi S, Yamada M, Nagai K, Mori S, Kajiwara Y, Sonoda T, Yoshimura K, Yoshitomi H, Ito H, Okamoto K, Ito T, Muto S, Ishihara T, Aoyama T (2012) Reliability and validity of gait analysis by android-based smartphone. Telemed e-Health 18(4):292–296View ArticleGoogle Scholar
- Yamada M, Aoyama T, Mori S, Nishiguchi S, Okamoto K, Ito T, Muto S, Ishihara T, Yoshitomi H, Ito H (2012) Objective assessment of abnormal gait in patients with rheumatoid arthritis using a smartphone. Rheumatol Int 32(12):3869–3874View ArticleGoogle Scholar
- Muto S, Nakamura Y, Ito T, Ishihara T, Shinohara A, Nishiguchi S, Yamada M, Aoyama T, Okamoto K, Yoshitomi H, Furu M, Ito H (2014) Examinations for the extensibility of gait outline monitoring applications that use information gathering system for sensors in mobile terminal. Transactions of the JSME 80(819):324. doi:10.1299/transjsme.2014dr0324 (in Japanese) Google Scholar
- Perttunen M, Mazhelis O, Cong F, Kauppila M, Leppänen T, Kantola J, Collin J, Pirttikangas S, Haverinen J, Ristaniemi T (2011) Riekki J (2011) Distributed road surface condition monitoring using mobile phones. Proc Ubiquitous Intell Comput 64(78):2–4Google Scholar
- YamadaT, Ito T, Ohya A (2013) Detection of road surface damage using mobile robot equipped with 2d laser scanner. In: Proc. of the 2013 IEEE/SICE International Symposium on System Integration, 364:369, 15–17 Dec. 2013, doi: 10.1109/SII.2013.6776679
- Ministry of Land, Infrastructure, Transport and Tourism (2010) Walking space network data maintenance specification plan. http://www.mlit.go.jp/common/000124059.pdf. Accessed 08 May 2015 (in Japanese)
- Ministry of Land, Infrastructure, Transport and Tourism (2010) Research report about the state of the environmental improvement that everyone can rear children surely. http://www.mlit.go.jp/common/000116746.pdf. Accessed 08 May 2015 (in Japanese)
- Kodoka K, Sinzaki A Hatoko M (2003) Extracting problems from existing barrier-free transportation policy in terms of parents with babies and infants. In: Proc. of infrastructure planning and management, JSCE, 28, VIII(287), 6–8 Jun. 2003 (in Japanese)
- Ogami H, Saito S, Okura M, Muraki S (2005) Investigation of Physical Strain and Danger in Sidewalk in Stroller Users. In: Proc. of the 3rd annual meeting of the Japanese Society for Wellbeing Science and Assistive Technology, 1P2-C4, 8–9 Dec 2005 (in Japanese)