My current work is with the SignCom Project, which combines traditional signed language linguistic data with motion capture data (mocap data) in order to:
- perform quantitative linguistic analyses on signed language data, and
- drive a signing avatar system that moves more fluidly than existing systems.
Signed Language Analysis
Because mocap data describes the 3-dimensional placement of a signer’s body, we can derive some statistical models of signed language motion norms. For example, the image below addresses a phonological rule described by Scott K. Liddell and Robert E. Johnson. They claim that subsequent repetitions of a repeated sign will be made smaller than previous repetitions.
The graph below shows the hand velocity during two different repeated signs, distinguished by the line style – solid versus dashed. In Sign 1, for example, the first repetition is made quite large and is the shown by the tall, solid-line arc in the graph; the second and third repetitions are signed smaller, shown in the two smaller solid-line arcs.
Signing Avatar System
Below is a sample of our signing avatar. Here, she uses signs that were originally spread out across our recording session. The system is able to collect desired signs, invert them if desired, and create new transitions that make the signer feel natural. Here, the signer describes making a cocktail for a friend.