Motion Laboratory

In recent years, Augmented Reality (AR) smartphone applications have become widespread, and it has become easier to develop AR content.In this study, we developed a system for creating choreography and movement paths of CG characters using AR markers to support creating CG content using motion data.The system allows users to interactively manipulate AR markers to assign choreography to CG characters and edit their movement paths while that choreographic animation of CG characters is overlaid and displayed.For an intuitive editing movement path, three AR markers are used as the start point, end point, and control point of a Bezier curve. The created choreography and movement path can be saved in the AR markers and shared with other systems.

Participants in a darts competition have few places to practice and get advice. Therefore, in this study, we developed a system using machine learning to predict in-the-counter attempts of throws from videos showing darts advice. The user enters the video of the throw and the frames from start to end. This system determines the direction thrown and outputs advice. Here, the system uses Openpose to calculate elbow angle and thrown speed from the joint points of the elbow, shoulder, and wrist. I created a learning model using similar feature quantities from videos of professional players in advance. The system determines which user's throws will fly to the top, center, or bottom of the target, and it outputs the advice associated with the judgment result. I used five videos each of throws to the areas above, at the center of, and below the target, and the system was about 94% accurate overall. Furthermore, the high accuracy of the system's evaluation results and the usage of the elbow's angle during a throw in the book system, as well as the throwing speed, provide the characteristic amounts. It was found that the relevant factors were the angle and speed of the elbow during a throw.

In this research, I developed a system to assist people who are not familiar with dance to create choreographies by converting some of the choreography created by the user's tablet operation into equivalent movements. The system suggests similar motions for the short basic motions created in choreographies. Similar motions are recommended in the order of decreasing Euclidean distance between the features extracted from the basic motion. In this work, for the basic movements of Ryukyuan dance, features were calculated in advance based on the spread of joint positions throughout the time series in the three axes of motion and the plane of motion of a 3DCG character. In addition, in order to perform a search that takes into account the direction in which the character is moving, the user can select from among the candidates those features used to calculate the Euclidean distance and the actual motion that the user is replacing. In order to verify the usefulness of the system, an evaluation experiment was conducted with nine students from the University of Tokyo to have them actually experience the system. As a result, all respondents indicated that the system was useful for assisting them in creating choreographies, thus confirming the system's effectiveness.

In recent years, there are many studies to verify the sense of body ownership and the sense of agency due to the spread of VR devices. The purpose of this study is to verify the effects of delayed vision in a VR space on the sense ownership and the sense of agency. For that purpose, I developed a system in which the arm of the character model to be manipulated moves with a delay from the actual movement. This system uses an HMD and two controllers to manipulate a character model from a first-person perspective in a VR space. I changed the delay time and verified the delay time when the sense of ownership and the sense of agency couldn't be obtained. By implementing IK, this system can freely move the upper body of the character model in the VR space with only HMD and two controllers. By setting the delay time, the arm of the character model will move later than the real arm by the set time. In order to reduce discomfort in operation and increase immersion, I made it possible to match the length of the user's arm with that of the character model.In order to verify the effect of the difference between visual information and motor information in VR space on self-sense, we conducted an evaluation experiment with 10 people under five delay conditions of 100ms, 200ms, 300ms, 400ms, and 700ms. As a result, it was found that the sense of ownership and the sense of agency began to decline when there was a 300ms gap between visual information and motor information, and they ceased to occur when there was a 400ms gap. It was shown that it is necessary to suppress the synchronization lag within at least 300ms in order to obtain a sense of ownership and a sense of agency in the VR space.

Recently, the estimation of human emotions has become important due to the rapid spread of artificial intelligence technology.Therefore, in this study, we develop a system to determine emotions from joints whose body poses are estimated in real time by OpenPose. In addition, we evaluate the accuracy of machine learning to verify the usefulness of the developed system for judging emotion.In this system, the shoulder angle and elbow angle are calculated as features based on OpenPose posture estimation from the body posture images prepared in advance. Based on the data learned from the calculated feature values, a neural network with supervised learning is used to determine the emotion from the pose in real time. The character model is made to play a specific animation according to the determined emotion.To improve the judgment of emotion from a pose in this system, the number of intermediate nodes and the number of training cycles were changed to determine the values with the best discrimination accuracy. The discrimination accuracy was 97.2% when the number of learning times was 3000 and the number of intermediate nodes was 16.These results confirm the usefulness of machine learning for emotion judgment in the developed system.

In this study, I developed a practice system for tennis strokes by pitch type with the aim of mastering the flat, topspin, and slice swings. The user wears an HMD, holds a general-purpose racket with a VIVE tracker attached, and swings at a flying ball in the VR space. The system judges the pitch type of the user's swing and provides feedback in real time. The angle of impact between the ball and racket and the trajectory of the ball before impact are important in hitting between the different pitch types. The system calculates these two features and judges the user's swing as one of the three types of pitches by using machine learning. The system also provides feedback in the form of textual advice and a CG display of the racket's position and posture at the time of the collision. To evaluate the effectiveness of this system, we asked 16 people (5 experienced tennis players and 11 beginners) to actually experience the system and conducted a questionnaire. As a result, about 90% of the respondents answered that the system was useful for practicing different types of pitches, confirming that the system can be used as a support for learning to swing at different types of pitches.

In recent years, there has been increasing development of systems that use motion-capture input of body movements and information technology to enhance the understanding of exhibits in museums and other facilities.In this study, I developed an interaction system with Interlocking control of multiple CG characters to promote understanding of the illustrations on the sides of the Sarira Casket. The system uses Azure Kinect to recognize a user's motion and IK to connect hands among multiple dancers in 3D space, allowing the dancers'arms to move with the user's movements. To make the experience more interactive, IK was set on the dancer's entire body, and IK targets on the ankles were set on the ground so that when the user sits down, the dancer is also linked to the user and the entire body can be manipulated. In addition, to perform the wave, the user's hand movements are transmitted to the hands of the four dancers at regular intervals with a delay. I evaluated the usefulness of this system by conducting an evaluation experiment with 14 people. As a result, about 80% of those who experienced the system answered that it was effective in helping them understand the pictures painted on the sides of the Sarira Casket, confirming the usefulness of the system.

Turns are the basic behavior on snowboards. For beginners, we examined the feature values for proficiency determination.In this study, a machine learning model was created to determine proficiency and a feature values was examined from the accuracy of the judgment. The system obtained joint points using OpenPose from images where experienced and beginners are snowboarding turns and calculated four feature values from the acquired joint points. I chose two from the four feature values I calculated and made a six machine learning model with a scale for learning the selected feature values. I entered a snowboard turn image in the machine learning model I created, and the system determined the image. learning model created. I entered 5 images of the experienced and beginner for all machine learning models and compared the judgment accuracy. Results, the judgment accuracy of machine learning models is most likely created using the angle of the waist and the angle of the legs and boards, and is considered to be optimal as the feature values used to determine the proficiency of snowboard turns.

Lacrosse is a minor sport in Japan, and there is little technical information or research on lacrosse. In this study, I developed CG contents for the purpose of learning the ideal passing motion in lacrosse for beginners. The ideal passing motion for lacrosse was measured using an optical motion capture system. The created CG contents display three factors from the measured motion data: animation of the passing motion, feature values, and locus of the racket. These CG contents facilitate the user's understanding of the ideal passing form and how to move the racket.To achieve this aim, the CG contents calculate the direction of the racket surface as a feature value. The result is displayed as an arrow in 3D space. In addition, the trajectory of the racket in the passing motion is displayed. To represent the important factor of timing in the pass motion, in salient frames, the racket's locus is rendered in a different color. A total of 14 people (7 inexperienced and 7 experienced) experienced this content and were asked to complete a questionnaire. The results confirm that these CG contents can help users understand the ideal path behavior, since 80% of the inexperienced respondents were able to understand the ideal passing form.

The purpose of this research is to reproduce traditional performance arts and festival performances using CG animation. In this research, I propose a description method to facilitate the use of motion data and the creation of data for content creation, and a motion archive was constructed for Ryukyu dance. This work introduces data processing to handle the movements necessary to describe Japanese dance animation in a generic way, the structure of the human body for dancing with props, and the structure of CG models of traditional Japanese costumes. The description method was used in a choreography creation application and a festival exhibition system. The hierarchical structure of the CG model was described in order to reproduce a performance in which multiple performers dance while riding on special vehicles akin to parade floats, which is often seen in festivals. To confirm the usefulness of the motion archive, a choreography-creation application was developed to recreate the choreography of Ryukyu dance by combining basic movements. In order to reproduce the characteristic hand motion of Ryukyu dance, the arm and finger movements could be replaced by the appropriate human body movements. In addition, to reproduce the motion of dancing with props, finger motions were replaced by the human finger motion made while holding a prop. In order to verify the extent to which classical dance could be reproduced by creating choreographies with the constructed motion archive and the proposed application, I reproduced the choreographies for the female and male dances appearing in different forms of Ryukyu dance. As a result, the choreographies for female dance were reproduced at an 81% correct rate, while those for male dance were correctly reproduced at 58%. A festival exhibition system was developed to reproduce past festivals in which Ryukyu dances were performed, using motion data and CG, and thus it was possible to restore a performance of "Ryusyu-no-Gi" (dragon boat race), a festival performance historically staged in Okinawa. Since the system was designed to display a festival in a museum, CG is superimposed on a diorama of a pond so that the festival can be faithfully restored and appreciated. An evaluation experiment was conducted to verify whether the application is effective in supporting exhibitions of festivals. This capability was in fact verified because the application could successfully provide an understanding of Ryusyu-no-Gi.

In this study, I developed the base systems for 1) choreography creation support using machine learning and 2) dance movement creation by extending body movements using a VR device. This work's aim is to leverage motion archives accumulated in the past as a way to support the creation of dance choreographies.In constructing the infrastructure system to support choreographic creation by machine learning of dance motion data, I developed a system for generating learning data and a system for real-time pose judgment. The latter system was tested using learning models for difficult/easy pose discrimination and hip-hop dance/daily movement pose discrimination. The correctness rate of the model on the test data was over 80%.In the dance movement creation system using a VR device, a system was developed to control the body of a virtual dancer in a virtual space in real time based on input from the VR device. The movements of the virtual dancer do not reflect all of the user's natural movements, however, since the generated movements differ from those of the user due to changes in the body part to which the input movements are applied, as well as through processing and amplifying the movements. The processing and amplification of movements were based on the choreographic creation methods used to create contemporary dance choreographies, and six methods were implemented. Three choreographers applied the system to explore the possibilities of creating dance movements in a virtual space. The results show that all three choreographers were positive about the usefulness of the system for their creative activities, but there were differences in the methods they judged useful. One of the choreographers actually used the proposed system in a stage performance.

• Takumu Matsushita, Asako Soga, Prototype System of Dance Movement Creation by VR Experience of Augmented Human Body, Proc. of NICOGRAPH International 2022, p.104 (Online), Jun. 2022