Skip to main content
ROBOMECH Journal
Download PDF
  • Development Report
  • Open access
  • Published:

Computer vision-based visualization and quantification of body skeletal movements for investigation of traditional skills: the production of Kizumi winnowing baskets

ROBOMECH Journal volume 11, Article number: 12 (2024) Cite this article

Abstract

This paper explores the application of computer vision and mathematical modeling to analyze the intricate movements involved in weaving a traditional farming tool, the winnowing basket. By utilizing OpenPose algorithms, the study simplifies and visualizes the craftsmen's motions, particularly focusing on wrist movements. Video data of craftsmen in Chiba, Japan, creating Kizumi (place name) winnowing baskets is used as the basis for analysis. The extracted information is used to generate 2D motion trajectories of the wrist, allowing a comparison between beginners who watched parsed videos and those who watched the original videos in terms of skill acquisition and learning time. By visualizing human body behavior and combining statistical results, this study demonstrates the potential of artificial intelligence techniques such as computer vision for observing repetitive human movement and inheriting traditional skills.

Introduction

Winnowing basket is a traditional farming tool that is widely used in Asia and around the world. It is made of thin bamboo strips and wisteria, and has a shape similar to a dustpan [1] (Fig. 1). The technique used for producing Kizumi winnowing basket in Chiba Prefecture, Japan, has been designated as a national intangible folk cultural properties [2]. The delicate production process creates exquisite baskets possessing both high artistic and practical value. Most current producers and users of winnowing baskets are elderly craftsmen who are highly skilled at their craft [3]. In order to inherit traditional skill and explore properties of the various body movements involved in the production and use of winnowing baskets, this study applies computer vision visualize and quantitatively analyze the associated body movement habits and movement details. A quantitative comparative test is then conducted to determine the effectiveness of the approach.

Fig. 1
figure 1

Kizumi winnowing basket

Full size image

Because it is easier to use existing computer vision technologies than to create new ones, existing tools were used to simplify and visualize some of the key movements in the production and use of Kizumi winnowing baskets. Algorithms and data of the body recognition (Dense Pose [4], RADiCAL Motion AI [5] and OpenPose [6]) were attempted in preparatory experiments, evaluated based on the following aspects: algorithms, databases, human body models, parsing speed, and readability and debuggability of the code (Table 1). The human body model of Dense Pose is not conducive to observing the key points of the skeleton, RAdiCAl Motion AI, as a package, its parsing speed and parsing accuracy are poorer than others. OpenPose, a real-time system that can detect 130 key points, including human bodies, hands and faces, from images in a C++ library using OpenCV and Caffe, which are capable of detecting and multi-threading multiple key points in real-time, was used in the present study.

Table 1 The brief comparison of the three techniques of OpenPose, Dense Pose and RADiCAL Motion AI
Full size table

It was found that the image processing and mathematical modeling of the results of the AI predictions described in this paper are effective in providing instruction to beginners while seeming to avoid some errors or misjudgments caused by oral communication and watching the original video only. It is hoped that the practical AI approach taken in this research can be used to explore how the human body responds when producing and learning to use other traditional tools.

Visualization of hand movement during the production of winnowing basket

Research purpose

The body movement habits and subtle body movements associated with many of the traditional techniques employed by craftsmen are difficult to communicate, as they often cannot be clearly expressed in words and vary from person to person [10]. However, there are still some common or similar body movement details (sometimes referred to as tricks of the trade) that can be identified. In this section, OpenPose is used to analyze existing video data and to extract the hand key point movements of a craftsman using a Kidachi [3, 11] (a specialty tool used in the production process) during the Itami weaving process [12] (an important process for producing the Kizumi winnowing basket, using the tool of Kidachi to fix the horizontal material of the winnowing basket to the vertical material) in order to make the production process easier to observe and understand, and thus facilitate the learning of those intending to become the next generation of craftsmen and inheritors of the technique. It is expected that with this approach, the technique can be learned in less time and the production process will be more interesting, which will be helpful in spreading the traditional technique.

Existing inheritance methods

The technique for producing Kizumi winnowing baskets (Fig. 2) continues to be passed down from the older generation [13]. As an aid, the Tokyo National Research Institute for Cultural Properties [14] (an national Institute for Cultural Properties Research in Japan) has produced a video [15] and several handbooks featuring the production process, allowing successors to acquire a comprehensive understanding of the technique.

Fig. 2
figure 2

Body skeleton analysis using OpenPose

Full size image

Field visits

In Kizumi, Sosa, Chiba Prefecture, the Kizumi Winnowing Making Preservation Association offers a winnowing basket-making course once each month in the Inheritance Classroom for those wishing to learn the technique. From July to November 2018, we attended the course several times, and made a video of the craftsman’s production technique, and experienced the production and use of the baskets. Based on our visits, we identified a number of key points and problems associated with teaching and learning the winnowing basket-making technique:

  1. (1)

    From the collection and processing of materials to the forming of the winnowing basket, all steps are done manually. The flexible use of hands and implements is the key to making a Kizumi winnowing basket.

  2. (2)

    The remote location of the heritage classroom is not conducive to the spread of the winnowing basket-making technique. In addition, the heritage classroom is relatively cramped and not suitable for the operation of a large equipment.

  3. (3)

    Most of the handicraftsmen are elderly, and there are few opportunities to inherit the production technique. Some do not want their bodies marked for use in optical motion capture.

  4. (4)

    It is difficult to teach the production process solely through words or by demonstration due to the many detailed actions, bodily sensations and individual work habits of the handicraftsman.

  5. (5)

    Some of the craftsmen acknowledged that it is very difficult for a beginner to master the production technique given the complexity and uniqueness of the process, especially the process of separating and weaving the two layers of the winnowing basket with a Kidachi, which are the key and most difficult points in learning the technique (Fig. 7a).

Research objects and multiple methods for parsing the teaching video

We selected for key analysis one case showing the difficulties of winnowing basket-making. During the process of weaving, a Kidachi (Fig. 3) is used to separate the two layers of materials of the basket. According to the comments of the handicraftsmen, the hand movement, especially wrist movement, in this process is both critical and difficult. It is extremely difficult to clearly observe the movement of the wrists and fingers by viewing the video from the Tokyo National Research Institute for Cultural Properties. Therefore, we considered several methods to solve these problems, including:

  1. (1)

    Optical Motion Capture [16] (Fig. 4): In addition to the relatively high cost, it takes a substantial amount of time to set up and commission Motion Capture, and a large space is required. A marker needs to be set around the wrist of the handicraftsmen in order to map the basket weaving.

  2. (2)

    Although 2D animation of the human body can be converted by applications such as Dense Pose and RADiCAL Motion AI into 3D animation using video data to generate 3D motion for human body surface mapping, motion analysis without key points of the human skeleton and with low accuracy.

  3. (3)

    OpenPose, no body surface mapping, motion analysis with key points of the human skeleton (which was ultimately selected to overcome the shortcomings of the AI human recognition applications noted above, although in this study, OpenPose performs only a 2D analysis of the hand.

Fig. 3
figure 3

3D structures at the tip of the tool of Kidachi

Full size image
Fig. 4
figure 4

Motion capture equipment set-up site

Full size image

Preparation of experiments on field visits

Based on our field visit experiences, we analyzed the video produced by the Tokyo National Research Institute for Cultural Properties to try to determine the movement trajectory and bodily sensations of the handicraftsmen when making a winnowing basket. However, we found that the video materials which could be parsed by OpenPose were insufficient [15]. As a consequence, we decided to use digital cameras for photography inside the Inheritance Classroom for our research. The arrangement of the photographic site is shown below (Figs. 5, 6), The two cameras used in the experiment were the Nikon D3100 and Nikon D5500, and the video was recorded with the same resolution of 1920*1080 pixel.

Fig. 5
figure 5

Craftsman working on site

Full size image
Fig. 6
figure 6

Arrangement of photographic site

Full size image

Analytical environment and precision description of OpenPose

There are a number of different versions of OpenPose, with different debugging methods, configuration environments, and levels of analytical precision (The Hithub Website of the Openpose [17] can be referred to for details). In this study, the following version was used in order to better capture the operator’s hand movements:

  1. a.

    OpenPose-master

  2. b.

    OpenPose-1.3.0-win64-gpu-binaries

  3. c.

    Visual studio 2015 professional

  4. d.

    Python 2.7.0

The detection sensitivity of OpenPose is inversely proportional to detection speed [6]. In this paper, the render threshold (detection sensitivity value) is set to be 0.03 (the range of this threshold is between 0–1, and the default value is 0.02), when the confidence level of the detected key point is larger than this threshold, the parsed video will render the key point. After several attempts, we found that even at a low threshold of 0.03, the confidence level of the detected key points (greater than 75) can be guaranteed, and the smoothness of the video parsing can also be ensured (the key point confidence level in OpenPose is obtained by processing the input images through Convolutional Neural Networks (CNNs) [7]). In this experiment, a Json file [18, 19] containing the coordinates of the skeletal key points and their confidence scores (ranging from 0–100) is automatically generated for each frame of parsed rendering results, and we consider a score of 75 to be very high because pose estimation in medical images, such as evaluating a patient's movement or rehabilitation, requires a confidence score of only 80 or more [20].

Analytical process and analysis of results

First, we edited the video clips on the use of the Kidachi in Itami weaving, and then used OpenPose (with the setting and de-bugging done) to identify all the joints of the body, except for the face, in order to produce a video of the whole body skeleton movement. Next, we used the heatmap (Fig. 7b, c) to separately identify the wrist parts and locate the key point. Finally, we used After Effect [21] to draw the movement trajectory of the wrist key points (white lines) and create an analysis video. The completed video (Fig. 7) includes:

  1. (1)

    the key points of the body skeleton and fingers,

  2. (2)

    the motion trajectory of the wrist,

  3. (3)

    the motion trajectory of the end of the Kidachi.

Fig. 7
figure 7

Skeletal movement analysis of the body and wrist movement trajectory of the craftsman

Full size image

At the same time by listening to the opinions of the craftsmen, we reached the following conclusions:

  1. (1)

    During the Itami weaving process, the weaving materials for the basket are on a fixed plane; the craftsman sits in the middle of the plane.

  2. (2)

    The Kidachi has a special three-sided structure (Fig. 3). Although a slight rotation or tilt can change its function, this will not damage the materials or injure the operator’s hands. For this reason, the implement can be used for separating the two layers of the winnowing basket materials and fixing the position of one layer of these materials. The Kidachi, which functions as an extension of the hands, can be regarded as a rigid body joined to the hands, making the motion trajectories of the craftsman’s wrists those of the Kidachi.

  3. (3)

    In the process of Itami weaving, the left hand plays the role of fixing the materials, and the shape of the fingers of the left and right hands is basically unchanged.

An experiment to compare the performance of learners watching the original video and those watching the teaching video

In order to verify the effectiveness of watching videos parsed by OpenPose for teaching purposes, a comparison experiment was conducted. The details of the experiment are described below:

  1. (1)

    Subjects: A group of 14 studies students (5 males and 9 females, with an average age of 25) who were beginners in winnowing basket production were randomly divided into two groups of 7: Group A (Member: A1-A7, 3 males and 4 females) and Group B (Member: B1-B7, 3 males and 4 females).

  2. (2)

    Experimental materials and tools: The materials and tools included wisteria bark, bamboo strips, a Kidachi, water (to keep hands and materials wet during production), misting, cameras for recording the video, teaching video, and a sheet (Fig. 8).

  3. (3)

    Experimental procedure: Group A watched only the original video, Group B watched the teaching video showing the wrist motion trajectory involved in using the Kidachi in the Itami weaving process. The subjects were asked to stack three sheets of wisteria bark between the bamboo strips. Only the analytical video featuring the wrist motion trajectory was used in the experiment in order to control the experimental variable. The working processes of the subjects were recorded on video taken from a distance of one meter, measured diagonally from the front of the camera. The body skeleton, especially the wrist key point, was analyzed with OpenPose, the video analysis results of OpenPose were then input into Adobe After Effects and the trajectory tracking function of Adobe After Effect was used to generate and output the teaching video with the wrist key point motion trajectory (Fig. 7d).A comparison of the forms of the woven products completed by the two groups was then made. Following the experiment, a questionnaire on the production process was administered to the 14 subjects.

  4. (4)

    Experimental results: According to the results of the analysis video by OpenPose and the motion trajectory generated by Adobe After Effects, the wrist movements of the subjects in Group B were smoother than those of the subjects in Group A. The Group A subjects had an intermittent action or curved wrist trajectory and significant difficulty in manipulating the materials during the operation; in contrast, the group B subjects had a circular wrist trajectory and a large range of arm movements during the operation (Fig. 9a, b).

Fig. 8
figure 8

Student field experiments and set-up

Full size image
Fig. 9
figure 9

Student experiments in Group A (Left) and Group B (Right)

Full size image

The subjects in Group A spent a longer mean time to master the technique than those in Group B. When making the middle layer of material, the mean time spent in the experiment of group A is 63.29 seconds, and the mean time spent in the experiment of group B is 42.67 seconds. The P value for significant difference between the two groups is 0.048 (Fig. 10, Table 2).

Fig. 10
figure 10

The time spent by all experimenters in group A and group B to make the middle layer of material (Mean, Standard deviation and P-value)

Full size image
Table 2 Average time and significant difference, etc.
Full size table

According to the products produced by the two groups, the woven baskets produced by group A subjects were rough and untidy, with loose connections between the materials. On the other hand, the Group B woven products were neat, with a relatively straight arrangement of the concave and convex surfaces (Fig. 9c, d).

Based on the results of the questionnaire, most subjects in Group B found the weaving process interesting (Fig. 11). Although only a part of the winnowing basket was made, Group B subjects wanted to complete and use their baskets. Some subjects in Group A thought it difficult to master the technique of Itami weaving with the Kidachi and found it especially challenging to understand the wrist movement (Figs. 12, 13). In general, Group A subjects, who had only viewed the original video, thought the operation time-consuming and easy to abandon.

Fig. 11
figure 11

The results of the interview questionnaire I

Full size image
Fig. 12
figure 12

The results of the interview questionnaire II

Full size image
Fig. 13
figure 13

The results of the interview questionnaire III

Full size image

Conclusion

In this paper, techniques such as computer vision, image processing, and statistics are used in conjunction. OpenPose is used to extract body skeletal movements of a handicraftsman during the process of manipulating a Kidachi, a difficult-to-use specialty tool for Itami weaving, the wrist trajectory was also traced using after effects in order to create a 2D analytical video to visualize the motion of the craftsman’s wrist key points. Through a comparative experiment in which beginners watched either the original video or the analytical video that was created, it was shown that AI can be effectively used to enhance teaching efficiency and learner enjoyment when winnowing basket production is being taught. Furthermore, the approach enables beginners to grasp the basic technique more accurately and more quickly, providing a supplement for the key, difficult points that the handicraftsmen are often unable to express in words.

Simplifying and deforming the production and use of an item with human movement through computer vision techniques, the limitations of human vision can be extended, so that important details (e.g. the trajectory and rhythm of the human motion) can be uncovered and simplified models of the operational process can be used to explore the tacit knowledge of human movement and the human dynamics nature of human behavior when using tools. In particular, the approach and viewpoint have significant application potential for teaching and researching traditional technology, and hopes to predict the efficiency of human work to a certain extent in the future.

Our study is not without limitations:

  1. (1)

    Regarding the 2D databases and algorithms of human skeletal movement used in the paper, although they are the most widely used and trusted, there may still be prediction errors in certain details of body movement when applied to the unique skills involved in producing and using winnowing baskets.

  2. (2)

    The number of samples and experimenters in our experiment is still small, and we hope to recruit more experimenters to participate in the experiment to make the experimental data more reliable.

  3. (3)

    In the future, we hope to combine 3D human skeleton analysis technology (such as 3d-pose-baline algorithm [22]) to analyze and track the movements of more parts of the human body when people use and make tools, making the evaluation of this project more comprehensive and extensive.

Data availability

The data supporting the findings of this study are available upon request from the corresponding author.

References

  1. Yang P, Wang S, Kubo M (2022) Extraction of morphological elements composing winnowing basket and 3d reproduction. J Sci Des 69(1):35–44. https://www.jstage.jst.go.jp/article/jssdj/69/1/69_1_35/_article/-char/ja/

    Article  Google Scholar 

  2. Techniques for making Kizumi Fuji winnowing basket. Database of national cultural properties from Japan Agency for Cultural Affairs, Accessed: 26 Jun 2019. https://bunka.nii.ac.jp/heritages/detail/137367 (Japanese)

  3. Imaishi M, Odawara N, Kurogouchi T, Ishii R (2017) The tool of Kidachi, creating Kizumi winnowing basket, Kizumi, Sosa City, Chiba Prefecture, p. 7. Tokyo National Research Institute for Cultural Properties, Tokyo

  4. MaRıza A, Natalia N, Iasonas K (2018) DensePose: dense human pose estimation in the wild. Computer Science, Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.1802.00434

  5. RADiCAL: A Real-time 3D Human Virtualisation technology: RADiCAL Motion Capture. Accessed: 13 Sep 2019. https://radicalmotion.com/

  6. Cao Z, Hidalgo G, Simon T, Wei S, Sheikh Y (2021) Openpose:realtime multiperson 2d pose estimation using part affinity fields. IEEE Trans Pattern Anal Mach Intell 43:172–186. https://doi.org/10.1109/TPAMI.2019.2929257

    Article  Google Scholar 

  7. Keiron OS, Ryan N (2015) An introduction to convolutional neural networks (CNN). Comput Sci Neural Evol Comput. https://doi.org/10.48550/arXiv.1511.08458

    Article  Google Scholar 

  8. Thang V, Hyunjun J, Trung XP, Chang DY (2019) Cascade RPN: delving into high-quality region proposal network with adaptive convolution. Comput Sci Comput Vis Pattern Recogn. https://doi.org/10.48550/arXiv.1511.08458

    Article  Google Scholar 

  9. Ionescu C, Papava D, Olaru V, Sminchisescu C (2014) Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2013.248

    Article  Google Scholar 

  10. Kamei N (2016) Opening up the intangible cultural heritage—transmission of attractions and power from outside. Tokyo National Research Institute for Cultural Properties, Tokyo

  11. KokuiH (2015) The tool of Kidachi. Winnowing basket production techniques for restoring from recorded videos. pp. 34–35. Fukushima Prefecture Cultural Promotion Foundation, Fukushima

  12. Education CPB. Techniques for making Kizumi winnowing basket. Itami weaving. Accessed 3 Jun 2023. https://www.pref.chiba.lg.jp/kyouiku/bunkazai/bunkazai/n321-006.html (Japanese)

  13. Imaishi M, Odawara N, Kurogouchi T, Ishii R (2017) Creating Kizumi winnowing basket, Kizumi, Sosa City, Chiba Prefecture, p. 1. Tokyo National Research Institute for Cultural Properties, Tokyo

  14. Tokyo National Research Institute for Cultural. An national Institute for Cultural Properties Research in Japan, Accessed: 3 Jun 2023. https://www.tobunken.go.jp/index_j.html (Japanese)

  15. Cultural Properties, T.N.R.I.: Creating Kizumi winnowing basket, Kizumi, Sosa City, Chiba Prefecture. Accessed: 3 Sep 2021. https://www.youtube.com/watch?v=kyLr-tgElaA&t=878s (Japanese)

  16. Tsuchiya A, Chen Y, Takagi Y, Ueda A (2019) Utilizing motion capture for eliciting information from the expert. Bull JSSD. https://www.jstage.jst.go.jp/article/jssd/66/0/66_490/_article/-char/ja/

    Article  Google Scholar 

  17. Hidalgo, G., Cao, Z., Simon, T., Wei, S., Joo, H., Sheikh, Y.: OpenPose Library, (2017). Accessed: 30 Oct 2018. https://github.com/CMU-Perceptual-Computing-Lab/openpose

  18. JSON—Introduction. A text format for storing and transporting data, standing for JavaScript Object Notation, from W3schools, Accessed 2 Jun 2023. https://www.w3schools.com/js/js_json_intro.asp

  19. JSON. JavaScript Object Representation, an open standard file format for sharing data, which uses human-readable text to store and transfer data. Accessed: 1 Apr 2023. https://www.json.org/json-ja.html

  20. Jan S, Kendra M, Cherry A, Connor OP, Rachel DR, Michael FV, Ryan TR (2021) Applications of pose estimation in human health and performance across the lifespan. Sensors (Basel). 21(21):7315. https://doi.org/10.3390/s21217315

    Article  Google Scholar 

  21. Adobe After Effects. A graphics and video processing software developed by Adobe, Inc., Accessed: 1 Apr 2023. https://www.adobe.com/jp/products/aftereffects.html

  22. Martinez J, Hossain R, Javier Romero J, Little J (2017) A simple yet effective baseline for 3d human pose estimation. Computer Science Computer Vision and Pattern Recognition ICCV 17, 2659–2668. arXiv:1705.03098

Download references

Acknowledgements

We first want to express my gratitude to the handicraftsmen of the Inheritance Classroom in Chiba for their guidance and opinions and for allowing me to join the Inheritance Classroom, conduct site visits, and carry out the experiments used in this study. We would also like to thank the Tokyo National Research Institute for Cultural Properties for providing the video materials, which greatly enriched the arguments presented in this paper. Finally, we would like to thank the students of Chiba University who participated in the experiments.

Funding

This work was funded by the grant-in-aid for scientific research-basic research (C) (Research Project/Area Number 21K12541).

Author information

Authors and Affiliations

  1. Graduate School of Science and Engineering, Chiba University, 1-33 Yayoicho, Inage, Chiba, Chiba, 2638522, Japan

    Peng Yang

  2. Institute for Solid State Physics, Tokyo University, 5-1-5 Kashiwanoha, Kashiwa, Chiba, 2778581, Japan

    Yuka Furukawa

  3. Tokyo National Research Institute for Cultural Properties, 13-43 Ueno Park, Taito, Tokyo, 1108713, Japan

    Migiwa Imaishi

  4. Design Research Institute, Chiba University, 1-33 Yayoicho, Inage, Chiba, Chiba, 2638522, Japan

    Mitsunori Kubo & Akira Ueda

Authors
  1. Peng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuka Furukawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Migiwa Imaishi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mitsunori Kubo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Akira Ueda
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the final version of the manuscript and approved it for publication.

Corresponding author

Correspondence to Peng Yang.

Ethics declarations

Ethics approval and consent to participate

The participants in the experiment of this work permitted the publication and use of images relating to them in this paper.

Competing interests

There is no competing interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, P., Furukawa, Y., Imaishi, M. et al. Computer vision-based visualization and quantification of body skeletal movements for investigation of traditional skills: the production of Kizumi winnowing baskets. Robomech J 11, 12 (2024). https://doi.org/10.1186/s40648-024-00280-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40648-024-00280-4

Keywords