ellis3D | rotoCap | MFA Thesis

RotoCap: Off-the-Shelf Motion-Capture

2010-02-16T11:47:00.000-08:00

RotoCap and Microcontroller Applications

2009-11-28T11:28:00.000-08:00

Good news. My thesis paper has been initially accepted by my committee. This has freed me to continue research in the fascinating world of human-computer interaction. The Arduino is an open-source microcontroller from Italy. It's cheap, well documented, and has numerous online communities each with their specific niches (diy-drones, music-production, art-installations). Above is my tinkering which will soon be connected to rotoCap to utilize its color-handling/vision functions.

How Color-Tracking Enhances Sony's PS3 Motion Controller

2009-09-15T15:31:00.001-07:00

Sony's motion interface uses not only accelerometers like the Wii, but also uses the color emitted from the sphere to maintain tracking of different controllers. Since the sphere is a known dimension, Z-depth can also be extracted.

Face Camera Rig

2009-08-04T15:03:00.001-07:00

rotoCap Successfully Controls a Character

2009-08-03T23:27:00.000-07:00

This is a test movie of my MFA thesis project, rotoCap. The person is repeating "Hello world" in different ways to drive the controllers of the 3D character ("Andy" by John Doublestein) only using three markers. The performance was shot on a scratch-built face-camera rig made from:

discarded webcam
used plastic 3D glasses
steel bailing wire
heat shrink wire wrap

Tracking fast movement and optimized centroid detection

2009-05-14T11:56:00.000-07:00

I have updated the search method to handle motion blur and find centroids faster with greater accuracy. In this video I show how easy it is to setup targeting and simple key cleanup. The cleanup portion of the video will soon be automated and will be an option for the user to implement.

Arduino-based Motion Capture System

2009-04-14T12:12:00.000-07:00

Another mocap project based on the Arduino microcontroller, from: http://blog.makezine.com/archive/2009/04/wireless_inertial_data_glove_using.html

Wireless Inertial Arduino Data Glove v0.3 from Noah Zerkin on Vimeo.

This is a quick vid of one of my later prototype glove units. The software is a proof-of-concept exercise. Regardless, you should be able to get the gist of what I'm trying to do. The thing that makes this special is how insanely inexpensive it was to build. The ultimate aim is to create a low-cost modular full-body interaction capture system for use in gaming, AR, and creative applications. The first commercial app I'm targeting for integration is Maya. After that, we'll look at integration with a game engine. I'm not sure what a glove/arm kit will cost, but it should be under $250 (Perhaps well under... we'll see... there are costs besides parts). Mass-production units would cost considerably less.

Making rotoCap Data Useful

2009-03-25T21:39:00.000-07:00

In order to be adaptable to each user's situation I have come up with a scheme to re-interpret data coming from rotoCap into a usable format. Motion data such as this is converted to keyable attributes which could be used to drive the user's own character rigs or anything possible via the connectionEditor. All this is done by essentially drawing the slider around the drone.

The image above shows the boundary box of the sphere's motion-extremes across the image surface. The dark triangles were some previous diagnostic graphics, ignore them. Scaling the green box redefines the parameter range of the drone (sphere) outputs by clamping the X and Y attributes. Additional number fields will be added to the drone's GUI section for North, South, East, West. Each field will define what the extreme value is for that direction or to ignore that parameter or clamp it to infinity. The user should be able to just plug in the extreme values from the control rig, blank fields are ignored:

yourFaceRig_eyeControlSlider.minY = 0.0 = drone.South
yourFaceRig_eyeControlSlider.maxY = 1.0 = drone.North

Example: The user would like the drone's Y-axis movement to control the blink slider for an eye rig. When the sphere rises, the eye gets wider; completely open equals an attribute slider value of 1.0. When the sphere lowers, the eye closes till 0.0 is reached. By translating and transforming the box, the drone's movement is filtered to the desired animation curve equivalent of amplitude and frequency in a visual and intuitive manner.

Simulation
User Defined Drone Attributes

Target Name = eyeControlSlider
North = 1.0
South = 0.0
West = SKIP //ignore
East = SKIP //ignore

Output = 0.85

.85 would equal a slightly closed eye, or 85% open. The eye is parented to a null-dummy node which acts as a buffer between rotoCap and the user's rig just in case the user needs to remove portions or all of rotoCap from a scene which actually is the ultimate goal. rotoCap should work behind the curtains and leave as little impact to a scene beyond passing attributes in the form of a bare-bones node network.

rotoCap Progress (Updated)

2009-03-02T08:29:00.000-08:00

The past weekend has yielded many successes. After finally fixing a bug that wouldn't advance the frame properly I've fully re-engineered the core tracking engine and even ran some benchmark tests which will be posted shortly (updated, see below). The movie above shows the raw output, single pass with "Simplify Curves" command in from the Graph Editor, and a final hand-tweaked version. The raw output creates lots of key frames that aren't all necessary so by keeping the important key frames, this makes it much easier for the final manual cleanup due to the reduced key set.

Test Results & Observations
Environment

50 JPG image sequence
1 Drone sampled (Pink)
timerX MEL function for timing

Parameters

Tolerance: (0.0 to 1.0) range of RGB value rotoCap accepts as close enough to desired color.
Step: (0.0 to 1.0) distance to next sampled UV coordinate
# of Keys: how many keyframes rotoCap created for a drone
Time Elapsed: (seconds) starts when "Launch Drones" is pressed, ends after last drone is tested

Test 1: Initial Test

Tolerance .2
Step .05
# of Keys 7
Time Elapsed = 101.47

Observations:

Almost all of the keys occurred at the beginning while the target color was stationary. Tracking was not detecting motion over time. I suspected the step size was too large and rotoCap just couldn't sample enough UV's to find the drone.

Test 2: Smaller Step Size/Centering Off

Tolerance .2
Step .025
# of Keys 16
Time Elapsed = 1217.69

Observations:

Reducing the step turned out to fix the sparse key issue. rotoCap still seemed to favor the edge of the color mass instead of centered which makes sense since this was run with self-centering turned off.

Test 3: Centering On

Tolerance .2
Step .025
# of Keys 35 (simplified: 17)
Time Elapsed = 71.86

Observations:
The time difference is substantial due to the centering sub-process passing the centered UV coordinates to the next cycle of the colorSeek process. The key generation also went up significantly. Running a simplify curves on the XY curves in the Graph Editor essentially deletes the unnecessary stationary keyframes. I'm very happy how well the script is working now.

Alternate Search Modes

2009-02-28T12:51:00.000-08:00

While working on rotoCap today I thought a nice feature would be allowing the user to manually launch rotoCap on a per-frame basis instead of a run of the whole image sequence. Additionally I am adding a flag to the keying sub-process whether or not to overwrite existing keyframes on a drone. If existing keyframes are found then the engine can use them as initial search coordinates, saving time by not having to start at the origin of a UV plane during the initial UV survey.

Homogenous RGB Mass Self-Centering

2009-02-20T09:49:00.000-08:00

The colorSeek engine of rotoCap has been given some new methods for searching and evaluating masses of targeted RGB ranges. Instead of basing tracker position on a single pixel test, the new method treats a positive RGB test as the origin for a localized survey of the surrounding pixels. If it detects the pixels immediately surrounding it are within the RGB range it will expand the survey (patch) till it finds the edges of the color mass.

The engine then calculates the center of the mass by simply offsetting the target tracker half the size of the mass. This theoretically will eliminate any false positives due to the quality of the source image due to sensor noise and blurs due to fast movement. The centered coordinates are then passed on to the next frame's processing cycle to use as an origin for the new search.

Break for Debugging

2009-02-16T13:35:00.001-08:00

While not moving as quickly as I had hoped, the ideas I've been integrating have been encouraging. Though like any grand scheme, there comes a time to pause and make sure future development is on a stable foundation. For the past weekend I've been going back and fixing little bugs so I don't have to worry about them later. Though not as satisfying, it is essential to maintain a level of coding discipline.

Integrating Path Prediction

2009-01-19T15:21:00.000-08:00

The core tracking engine has been successfully installed into the main program. It remains virtually unchanged since its creation but there are some adjustments to the search-style I'm working out currently. When the engine spots a target RBG value it uses that "hit" as the initial center for successive UV searches. Limiting the search area was the first attempt at speeding up processing and now predicting where the target is in the given area is the next step.

Noting Newton's First Law of Motion, a target will tend to continue along if not a straight line, a near approximation. The engine will make a list of coordinates to look at first in order of probability (the opposite direction being the last). This will require re-writing the engine but every engine needs an overhaul eventually. I believe the time I spend on this portion will be worth it in terms of process efficiency.

rotoCap GUI & File Handling Demo Video

2008-12-15T17:07:00.000-08:00

Drone Generation Subprocess "makeDrone"

2008-12-03T13:05:00.000-08:00

I'm calling the visual representations of each color track drones. These, like worker bees and pollen, will track the RGB elements across the UV space. Currently the makeDrone subprocess creates:

visible geometry
RGB attributes from a colorEditor call
unique object name
colorSeek inclusion toggle (include in batch color tracking or not)
RGB swatch via canvas command
target-rig attribute to control (what does this drone control)
"edit RGB" and delete options

Possible future features:

per-drone area constraints
initial search pattern
per-frame census (how many frames does drone find target RGB)

Over 100!

2008-11-10T09:13:00.000-08:00

My method of file numbering is when I get a portion of the script running it gets saved as a separate file. This past weekend, the bugs squashed broke 100 (as of this writing at 103).

The GUI is being refined into a sleeker interface, but still with the consideration of an Adobe-like familiarity. Currently I'm working on a timeline-independent playback system for an animated texture. While scrubbing and stepping frames was relatively easy, the animation of the texture at a constant framerate is a little trickier. I've been using the timerX function to wait a certain amount of time then step the frame. It works but I'm trying to make it run consistently on different machines. The amount of times the function checks the elapsed time varies depending on the speed of the machine. I don't think this is a big issue, but I'm trying to make rotoCap as consistent and stable as I'd rather have all the processing power for the color-tracking engine than the GUI.

Simple Shape Tracking

2008-10-16T10:19:00.000-07:00

Andy Wilson has developed an elegant control method based on the shape of the hand. By comparing the background shapes to newly created shapes (formed by the hand), he is able to track the movement allowing for mouse-less interactions. (Source: http://procrastineering.blogspot.com/2008/10/andy-wilson.html)

Video-Based vs Laser-Based Tracking of Dynamic Environments

2008-09-25T10:23:00.000-07:00

In both fields of motion-capture and surface-capture (scanning) the need to discriminate unique points in space is paramount to the accuracy of the capture system. When I was first conceiving rotoCap I tried to figure out this problem by looking what I had to work with and not having ready access to the specialized cameras of a Vicon system or laser scanner I turned to consumer video cameras. I conceived I could lessen the amount of computer load by not having the program calculate trajectories but color. An infrared machine-vision camera is suited to higher resolutions and frame rates, how could the lowly camcorder compete? The answer is color.

Most machine vision cameras used in mocap and 3d scanning are grayscale in which their output is turned into a binary image. A binary image is simply a high-contrast image that is only black and white with no gradient. This makes the image easier to process for the software since it only deals with white dots on a black background, but the other calculations it uses to discriminate between two white dots which come near to each other can be complex and costly to the performance. By adding in the color value, the need to track each dot is made easier by giving each its own RGB value.

Why bring this up? The DARPA Grand Challenge is a great source for inspiration because of the real-world applications of scanning/mocap. The Stanford University team's Standley uses a hybrid system of laser scanning and a video overlay allowing it to use color to tell the difference between the road color and everything else. This is a great confirmation that we're both on the right track when it comes to using color over binary alone.

Sequence Viewer and Shader Progress (UPDATE)

2008-09-14T10:33:00.001-07:00

rotoCap is really coming along as more pieces of code unlock multiple areas of development. The past few weeks have seen a streamlining of file management, GUI revisions for better interactivity, and it's now ready to start processing images.

Since the core tracking engine relies on an image plane to calculate movement, rotoCap needs to:

generate a shader with an animated texture
disconnect the animated texture from timeline dependency
connect animated texture to rotoCap control
apply the shader to an image plane
scale image plane the same aspect ratio as source image file
create a camera to lock onto the image plane perpendicularly
use the new camera's viewport as part of the rotoCap GUI
create playback widget for different framerates

I considered embedding the images into the GUI, but thought the user should be able to pan, zoom into each image using the familiar Maya orthographic camera navigation. A "Fit Image" button would also be added to reset any camera translation.

Currently, the shader generator works great the first time the user loads up the image sequence into rotoCap but changing the image directory results in errors as the script tries to create new shaders connected to previous file nodes. The tricky part at this time is how to query if a shader exists since the shaderNode command does not have an -exists flag to test. To get around this I'm loading up all the shaders into an array and deleting the rotoCap-created shader network based on wildcard name-space search.

UPDATE
The shader handling is fixed now. The bug was due to old nodes conflicting with the newly generated nodes. Each time the user changes the image sequence directory a process is called to delete all nodes generated by rotoCap from previous executions. The nodes are filtered by a gmatch wildcard name-space execution, then deleted.

3D Scanning for Animation

2008-08-20T09:00:00.000-07:00

After attending SIGGRAPH this year and talking with Dr. Paul Debevec about the subject I am convinced 3D scanning will progress more and more into the realm of motion-capture where the fields will merge within 5 years. With Image Metrics' release of Emily, a 3D face tracked onto an actor's head, we see the current state of the synergy. So far, the use of 3D scan data in animation has been to capture key poses, this still leaves simulations to drive the actual motion. By capturing the full performance in 3D, the best of both worlds can be had by being able to use the animation uncut and or pulling key poses off choice portions.

Are we there yet? Not exactly. It has been pointed out that Emily was still animated by humans using the scan data to target a face rig. But, it is a good running start for the industry's jump over the valley.

File Handler Works!

2008-08-18T13:11:00.000-07:00

This past weekend has been very productive. The script is now automatically picking out the file extension and using that to filter out the other files which the user doesn't want (e.g. thumbs.db, OSX files, etc.). This makes it faster for the user since they simply choose the first file of their image sequence and the script takes care of the rest.

The next step will be to create the image viewer. I see some trouble coming since Maya doesn't have a resize for the image command. One work around will be to load up the files as an animated texture and do screenshots of each frame with a perpendicular camera through the saveImage command. The thumbnailed images will be used for the GUI window while the full-resolution images will be used by the script for the actual processing. The image dimensions also need to be queried so the height and width ratio can be applied correctly to the NURBS plane. After doing some research it seems querying the outSize of a node will work.

How to fix getFileList

2008-08-09T11:13:00.000-07:00

When you use the MEL command getFileList, make sure there is a trailing "/" on the folder string you pass to it. Otherwise the command returns the folder string you passed to it initially.

string $imageDir = (dirname (`fileDialog`) + "/"); // Stroke must be added
string $fileList[] = `getFileList -folder $imageDir`;

This was a frustrating bug and hopefully this will show up in search results when someone else has this issue.

File Handler Progress!

2008-08-05T17:06:00.000-07:00

After some frustrating times with fileBrowser.mel, I went back to using the fileDialog command to pass a path to the image sequence file handler so I'll only have to call up array elements. Theoretically this should save some processing cycles and makes it a little more cross-platform friendly.

Radiohead & 3D Scanning

2008-07-14T16:02:00.000-07:00

Thom York underwent a scanning session to capture his performance of House of Cards. The video was created using two different scanning methods, laser and structured light. It's encouraging to see fine artists using technology instead of abhorring it. In an exciting move, Radiohead released the raw data from the session as a CSV sequence which I was able to process into a polygonal mesh to reveal what Mr. York looked like during the capture session. There is some surface warping due to the deliberate distortion of the process by shooting through a piece of glass or Lucite, but the results were interesting.

The Story So Far...

2008-07-13T15:29:00.000-07:00

The core image processing script is functional, now it's time to get it functional for the user. I've decided that the Pane Layout will be the best to hold the different program elements because of the resizable areas. In the future I may make the image area it's own window but only if it doesn't impact the performance of the script by handling the cross-window chatter.

The trickiest part so far has been the file-handling. MEL has about 4 ways to handle file dialogs and each seems to have their strengths. For now I only need to worry about storing directory paths.

FIRST!

2008-07-13T14:30:00.000-07:00

Master of Fine Arts Thesis Project : Creating an Optical Motion-Capture System from "off-the-shelf" Hardware and Software.

What is Motion-Capture? Motion-capture (commonly known as "mocap") is a method of recording the movements of a real-world performance, usually a human. The movements are then applied to a computer model which mimics the actions of the original performance. The performer usually wears a special suit which has markers or sensors which are used to turn the movements into a numeric representation, easily understood by a computer.

Who uses Mocap? Both the entertainment industry and research entities use mocap. Many of today's video games and movies use mocap to create performances where realistic movement is desired. Researchers use mocap to study the movement of the body in detail. Since the performance is captured as data, it is infinitely repeatable and allows for greater scrutiny with less variables.

Why write your own? Traditional drawbacks to a commercial system are its prohibitive cost and complexity, thus limiting motion capture to primarily larger studios and research entities. These barriers are on the verge of falling away due to the dropping cost of near-professional electronics and new methods of motion data capture by said electronics. Independent animators and small studios will be able to utilize motion capture as they have with other technologies which previously were unavailable to them such as 3D animation software itself and high-end workstations.

How does it work? The program is actually a script that runs within a 3D package called Maya. The user will record the performance using off-the-shelf video cameras and bring it into the script (which I've dubbed rotoCap). The video is analyzed frame by frame to synchronize the multiple data streams and to locate the markers placed on the performer at the time of the capture session. The markers are isolated in 3D space and used to drive the performance of a computer model.