Time Varying Surface Appearance
In this project, we are interested in acquiring, modeling and rendering time-varying surface appearance. Traditional computer graphics and computer vision algorithms generally assume that the appearances of surfaces remain constant over time. However, there are a number of natural processes that cause surface appearance to vary dramatically, such as the burning of wood, the wetting and drying of rocks and fabrics, the decay of fruit skins, and the corrosion and rusting of steel and copper. We have captured the first time-varying surface appearance database (with 26 samples), which includes a variety of natural processes -- burning, drying, decay and corrosion. We have developed a novel Space-Time Appearance Factorization (STAF) model, which factors space- and time-varying appearance effects. The STAF model includes an overall temporal appearance variation characteristic function, two spatially varying textures corresponding to the initial and final frames, and two spatially varying textures corresponding to the rates and offsets at each point on the material that determine the evolution of the appearance over time. Our experiments demonstrate that the STAF model can accurately capture appearance variations due to a wide variety of natural processes. Moreover, it enables a number of novel rendering applications, such as the transfer of a time-varying effect to a new material, the control of the evolution of a time-varying process, and the extrapolation of a process beyond what is captured by the acquired data. This project is done in collaboration with Ravi Ramamoorthi and Peter Belhumeur at Columbia University, and Chien-i Tu and Wojciech Matusik at MERL.

Publications

"Time-varying Surface Appearance: Acquisition, Modeling, and Rendering,"
J. Gu, C. Tu, R. Ramamoorthi, P. Belhumeur, W. Matusik and S. K. Nayar,
ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),
Jul, 2006.
[PDF] [bib] [©]

Pictures

  Database of Time-Varying Surface Appearance and Factorization:
This image shows the 26 samples acquried in the database and the time-varying process that each sample is subjected to.
  Space-Time Appearance and Factorization (STAF):
This image shows the STAF representation of a drying wood sample. The top panel compares the reconstruction of a sample using STAF to the original sample for a chosen lighting and viewing direction. STAF model can also be used for time normalization, wherein the overall appearance changes maintained but the spatial drying patterns are removed. The panel below shows the estimated temporal characterisitic curves for both diffuse and specular parameters. We also show visualizations of the spatial diffuse textures A, D, R and O, along with the normalized initial frame A*f(0)+D and final frame A*f(1)+D. Please see the paper for more details.
  STAF Results for More Samples:
This image shows STAF estimation results for 5 samples in the database, each of which has 3 spatial locations marked. The middle row shows the time-varying curves p(x,y,t) (for the red diffuse component) for spatial locations A, B and C. The curves are quite different for the three different points. In the bottom row, we align these time-varying curves using STAF model. The data closely matches the temporal characteristic curve f(t') computed from all the points on the sample.
  Examples of Rendering:
This image shows some of our rendering results (please see videos as well). STAF model enables us to manipulate the acquired data. We can create a time-varying texture with larger extent by applying standard 2D texture synthesis to the initial and final frames. Within limits, we have extrapolate the time domain of the original acquired data. We can control the rate and offset terms according to physical laws or user-derfined rules. We also can transfer a time-varying effect to a new material.

Videos

  SIGGRAPH 2006 Video:
This video shows the complete pipeline we have used for our work on time-varying surface appearance, including acqusition, modeling, and rendering. (With narration)
.
[ .avi | 45 MB ] [ .mov | 64 MB ]
  Some Samples in the Database:
This video shows the acquired data for 6 samples in our database for a single lighting and view direction. Our captured data includes a large number of lighting and viewing directions. (With narration)
.
[ .avi | 20MB ]
  Rendering:
This video shows 5 rendering examples including texture synthesis, time-extrapolation, control, and transfer. (With narration)
.
[ .avi | 9MB ]
  More Synthesis Results:
This video shows texture syntheis examples for the drying rock and the drying wood samples.
.
[ .avi | 4MB ]

Slides

SIGGRAPH 2006 presentation     With videos (zip file)

Databases

  STAF: Time Varying Appearance Database & Source Code
The database includes 26 samples, covering a range of phenomena
including burning, corrosion, drying on smooth and rough surfaces, and
decay. We also include the source code used to process the captured data
and estimate the STAF model. For each sample there are three parts of
data to download: 6 of the 1280 measurements of the raw data at each
time step, the fitted BRDF parameters, and the estimated STAF results.
The total size of the database is around 3 GB. Due to its enormous size
(about 1,000 Gb), we are not able to release all of the 1280
measurements of the raw data. If you really need it, please send an email
to staf@cs.columbia.edu

Related Projects

Bidirectional Texture Function

Time-Varying BRDF

Oren-Nayar Reflectance Model