Home

Dancing Humanoid Robot
 Studying Creativity, Dance, and the Human Response to Music

PIs: Dr. Youngmoo E Kim & Dr. Paul Oh
Students: Rob Ellenberg & David Grunberg



Introduction


Motivation

    I am making a robot dance in order to study the relationship between music and dance creativity. It is not currently known exactly how humans decide which sequences of dance moves are appropriate for different songs. Training a humanoid robot to dance in response to music, though, could give some insight into this problem. When the robot understands both dance movements and dance styles, it will be able to generate its own sequences of moves based on a song and the style rules. At this point, generative features might arise that could lead to insight into how humans determine what a 'good dance' looks like.

    Objectives

      I have two immediate objectives. The first is to make a fully robust beat tracker that will be able to identify the beats for a significant body of audio. This information is used by the robot in order to synchronize its gestures to the music. The second is to enable gestures that build on each other. These will allow the robot to perform smoothly connected sequences of gestures, instead of having to return to its base position after every motion.

      Longer term goals include preparing a similarly sized robot with a more powerful processor, which will allow the program to run more quickly and more flexibly, and programming rules for dance styles into the robot.

      In the long term, we will be transfering this algorithm to the life-sized HUBO robot. We hope to them make the HUBO dance almost as a human would. This will more closely mimic humans so that any generative features are also more likely to apply to people. It will also have the convenient side effect of being a platform for rapid testing of dance choreography.



    Video

    Click to Play



    Algorithm


    • First, gestures are selected for the robot. These are either chosen by the user with a Graphical User Interface or by the robot.
    • The chosen gestures are then analyzed and sorted in time so the robot knows when to begin each one. There are 30 possible gestures so far.
    • Once the music begins, the robot analyzes it and determines where the beats are.
    • When a beat approaches, the robot begins a set of gestures. The gestures reach their apex exactly on the beat. This continues until all gestures are performed.

    GUI

      We created a Graphical User Interface to aid with rapid demonstration of dance sequences. Users can easily select which motions the left arm, right arm, and legs perform. These options are placed into a score that allows for quick examination of the dance sequence. Once the user has finished making his or her gesture selections, music is selected and the robot is told to perform the gestures to the music. This allows anyone to have the robot act out their own dance sequences, making it possible to quickly iterate choreography sequences. Alternately, there is a button for the robot to produce its own gestures. Currently it chooses them randomly, but once dance style rules are implemented it will wait for the music to begin and then choose logical gestures based on what it hears.

    Gestures

      We have thirty separate arm and leg movements in the robot's gesture vocabulary. Each movement can be 'held' in position for as long as the user wants before being returned to its neutral position. The robot cannot yet connect gestures.

      Because gestures take varying amounts of time to complete, the robot must be able to begin some gestures before others in order to stay synchronized. For example, if it must raise one arm, tap one knee, and step forward before the next beat, it must know to order the gestures so that they all finish at the same time. This is accomplished with a large table that lists how long each gesture takes to complete. After gestures are selected, each set of simultaneous gestures is sorted in time using this table. Gestures can then be begun at the correct time.

    Robot platform

      The current robot platform for this project is the RoboNova-1, pictured above. This robot is useful to us for several reasons:

    • The RoboNova kit can be configured as a humanoid, so it can mimic human gestures.
    • Each limb allows a fine degree of control, enabling precise fine-tuning of gestures.
    • The programming environment is straightforward.
    • The robots provides several sensors for feedback.

    The RoboNova has one main disadvantage in that its processor is underpowered. Its update rate is relatively slow at about 5 Hz, which leads to imprecisely timed gestures. It also cannot send much feedback to the offboard computer because so much of the processor's power is needed to select the requested gesture from its library that little remains for anything else.

    To solve this problem, we are switching soon to the Bioloid robot. This platform has a much faster processor. This will enable more accurate gesture timing and more sensor feedback in real time.

    Eventually, we will upgrade to the HUBO, similar to the Bioloid except much larger and with a much more powerful onboard computer. The processor onboard the robot will be able to run the entire algorithm, eliminating the need for an offboard computer. The larger size will also enable closer mimicing of human dance moves since the robot's size and motions will be more comparable to humans than the scaled down models are.

    Beat tracker

    • The first frame of audio is passed through a low pass filter. This removes most frequencies. The lowest ones, which usually include any drumbeats, are kept.
    • Distortion is reduced by smoothing the frame with a Hanning window.
    • The audio data is reduced to about 1/8 of its original size. This is still enough data for accurate beat determination, but the reduction in frame size reduces processing time.
    • The tempo is extracted from the audio. This is done by passing the frame through several filters that have different periods. Filters with periods matching the period of the audio will result in large values. The filter with the largest value is chosen, and the tempo is calculated from the period of that filter.
    • The beat position is extracted from the selected filter by seeing at which point in time the filter gives the largest values.
    • The beat information is saved and the next frame is sent through the system.

      • This system requires relatively little computation, and can be accomplished in real time.

    Results


    Current state

    • The system detects beats in music in real time, and the robot is able to perform movements synchronously as music is played.
    • The RoboNova can perform 30 gestures.
    • The developed GUI provides the capability to rapidly explore humanoid motion and choreograph complex strings of gestures.

    Future Work

    • Enhance the beat tracker to make it more robust. Eventually, make it predictive.
    • Continue the investigation of potential emergent behavior using the robot.
    • Examine how to best incorporate sensor feedback from the robot.
    • Further explore the requirements and constraints of different dance 'styles,' and then teach these to the robot.
    • Upgrade the robot platform to the Bioloid and then the HUBO.

    Conferences

      This work has been presented at the following conferences:

    • Association for the Advancement of Artifical Intelligence 2008 (Chicago).
    • International Symposium of Music Information Retrieval 2008 (Philadelphia).


    Rob Ellenberg (DASL): rwe24g@gmail.com
    David Grunberg (MET-Lab): dkg34@drexel.edu