Relighting Framework

Sebastian Enrique [ ], Ravi Ramamoorthi [ ]


Relighting Framework is a framework for synthetic and real scenes image-based relighting with a fixed viewpoint but with any point / area lights and environment lighting conditions.

The method consists of a large preprocessing stage, in which sets of images with different lighting conditions are provided, and two passes of PCA (Principal Component Analysis) compress data and generate the final matrices for each color component. Those matrices are then used in realtime to calcuate the correct image appeareance given the novel lighting conditions.

The Relighting Framework is based on the Lighting Sensitive Display by Shree Nayar, Peter Belhumeur, and Terry Boult, and extends it to use environment maps.


Image Adquisition

Images can be synthetic or real photographs. The viewpoint should be fixed. The target object should be in the center of an imaginary cube. Each face of the cube should be divided in a grid of f x f elements. So, those 6 x f x f positions should be used to place one distant point light and take -render- 6 x f x f pictures -images-, i.e., one image for each light position.


Preprocessing Stage

First PCA
Consider the collection of n images of a scene over varying lighting. The images may either be synthetic or real. Each image Ii is an image of the scene illuminated by a single distant point light source. Divide the image up into m square blocks each containing p pixels. Let Iij denote the jth block in the ith image. For each block in the scene we compute a low-dimensional approximation. We create a p x n matrix Ij as a collection of image blocks in which the ith column of Ij is formed by p pixels from the jth block. Using Singular Value Decomposition (SVD), we find a rank b approximation to Ij as

Ij ~= Ej Sj CjT = Ej Lj

Where Lj is an b x n matrix which we call the block lighting coefficients.

Let the lighting coefficient matrix L be formed by stacking all of the m Lj block matrices. Likewise, let the collection of image bases for all blocks be denoted by E. The collection of submatrices within E and L contain all the information needed to approximate the collection of images corresponding to the n lighting directions.

Second PCA
Second stage exploits the fact that there is much coherence in the image blocks. Similar to the previous stage, we apply SVD to the lighting coefficient matrix L and find a rank q approximation to L as

L ~= U V

Where U is an (m x b) x q column-orthogonal matrix which we call the lighting coefficient bases, V is a q x n matrix which we call the compressed coefficient matrix and q denotes the number of linear bases kept to approximate L.


Realtime Stage

We represent the illumination as the s vector field, of dimension n x 1. Each element in s is the contribution of the corresponding original distant point light image.

To render an image of the scene as if it were illuminated by s we appeal to the previously calculated matrices E, U, and V. First, we compute a compressed coefficient vector as the product V s. Next, we compute a lighting coefficient vector as the product U V s. Note that the vector U V s has dimensions (m x b) x 1 and that the subvector lj given by the rows b x (j - 1) + 1 through b x j of U V s contain the coefficients necessary for rendering the jth block. Thus, the subvectors can be unstacked and used to render the jth block of the displayed images as the product Ej lj. This process is repeated for all m blocks.

Point lights are mapped directly to one element of the s vector.

Area lights are mapped to more than one element of the s vector. If the area light with intensity k occupies e elements, the contribution on each element is k / e.

High-Dynamic Range Cube Map texels are interpolated to avoid problems when the resolution of each face of the cubemap is different from the quantity of images taken from each face of the object. n texels are taken according to the corresponding directions and mapped onto s. Texels contributions are weighted by the solid angle.


We converted Michelangelo's David data from spherical coordinates to our needed cube spatial coordinates. David was processed with the following parameters:

Images : 6 x 16 x 16 = 1536
Image Resolution : 480 x 640 pixels
Image Resolution: 30 x 40 blocks
Block Resolution: 16 x 16 pixels
Rank 1st Stage SVD: 15
Rank 2nd Stage SVD: 100

Although the quality for environment maps is good enough, artifacts can be seen in the shadows using point lights. Greater rank in the first SVD should avoid this problem, but more experimentation is needed. Using a rank 10 in the first SVD, blocks discontinuities are noticeable.

We processed images from Nicole face rendered -with shadow mapping- using RBF from the data acquired by Todd Zickler of Yale. Nicole was processed with the following parameters:

Images : 6 x 16 x 16 = 1536
Image Resolution : 512 x 512 pixels
Image Resolution: 32 x 32 blocks
Block Resolution: 16 x 16 pixels
Rank 1st Stage SVD: 10
Rank 2nd Stage SVD: 200


Preprocess Stage [ ] [ readme2.txt ] [ relighting.m ]
Includes Matlab source function for preprocessing and output files and input images for Nicole (both shadowed and unshadowed) and David.
Compressed in zip format at ~260MB.
RCafe Presentation [ relightingv02.ppt ]
Powerpoint presentation used in November 2003 in a RCafe talk to show the work in progress at that moment.
Realtime Binary [ ] [ readme.txt ]
Includes executable file for realtime relighting, 10 HDR environment maps, data sets for Nicole (both shadowed and unshadowed) and David, and batch files to run every possible combination with included datasets.
Compressed in zip format at ~240MB.
Source Code [ ]
Includes Visual C++ 6.0 project for realtime relighting and Matlab source function for preprocessing stage.
Compressed in zip format at ~50KB.

Lighting Sensitive Display [ pdf ]
Nayar, S., Belhumeur, P., Boult, T. 2001. In Technical Sketch, Siggraph 2001, page 218.
Paul Debevec [ ]
Light probes (HDR Cubic Environment Maps) were taken from Paul Debevec's website.

1 : Restricted site.
Page Created: Feb 4, 2004 - Last Update: Feb 18, 2004 - Created by: Sebastian Enrique []
Computer Science Department Homepage Columbia University Homepage