Face Swapping: Automatically Replacing Faces in Photographs

Advances in digital photography have made it possible to capture large collections of high-resolution images and share them on the internet. While the size and availability of these collections is leading to many exciting new applications, it is also creating new problems. One of the most important of these problems is privacy. Online systems such as Google Street View allow users to interactively navigate through panoramic images of public places created using thousands of photographs. We believe that an attractive solution to the privacy problem is to remove the identities of people in photographs by automatically replacing their faces with ones from a collection of stock images. Automatic face replacement has other compelling applications as well. For example, people commonly have large personal collections of photos on their computers. These collections often contain many photos of the same person(s) taken with different expressions, and under various poses and lighting conditions. One can use such collections to create novel images by replacing faces in one image with more appealing faces of the same person from other images. For group shots, the burst mode available in most cameras can be used to take several images at a time. With an automatic face replacement approach, one could create a single composite image with, for example, everyone smiling and with both eyes open.

In this project, we present a complete system for automatic face replacement in images. Our system uses a large library of face images created automatically by downloading images from the internet, extracting faces using face detection software, and aligning each extracted face to a common coordinate system. This library is constructed off-line, once, and can be efficiently accessed during face replacement. Our replacement algorithm has three main stages. First, given an input image, we detect all faces that are present, align them to the coordinate system used by our face library, and select candidate face images from our face library that are similar to the input face in appearance and pose. Second, we adjust the pose, lighting, and color of the candidate face images to match the appearance of those in the input image, and seamlessly blend in the results. Third, we rank the blended candidate replacements by computing a match distance over the overlap region. Our approach requires no 3D model, is fully automatic, and generates highly plausible results across a wide range of skin tones, lighting conditions, and viewpoints. We show how our approach can be used for a variety of applications including face de-identification and the creation of appealing group photographs from a set of images.

Publications

"Face Swapping: Automatically Replacing Faces in Photographs,"
D. Bitouk, N. Kumar, S. Dhillon, P.N. Belhumeur and S. K. Nayar,
ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),
Aug, 2008.
[PDF] [bib] [©]

Images

  Color and Lighting Adjustment:

In this example, we demonstrate the importance of our color and lighting adjustment algorithm. We replace (a) the face in the input photograph with (b) the face selected from the library. Replacement results (c) without and (d) with recoloring and relighting. Notice the significantly improved realism in the final result.

     
  Face Replacement Results:

Here, we show several examples of the results obtained using our system. Each example shows, in order from left to right, the input face image, a candidate face, and the replacement result. Note the realism of the results, despite differences in pose, lighting and facial appearance. Each row contains (from left to right) the original photograph, a candidate face selected from the library, and the replacement result produced automatically using our algorithm. The age and gender mismatches in (c) and (d) could be avoided by enforcing consistency across those attributes (which our system does not currently do).

     
  Face De-Identification:

To preserve privacy in online collections of photos, one can use our system to automatically replace each face in an input image with the top-ranked candidate taken from a collection of stock photographs. We show the result of automatically replacing the input faces (top) with the top-ranked candidate from the face library to obtain the de-identified results (bottom). No user intervention was used to produce this result.

     
  Face Switching:

As a special case of face de-identification (or for use as a special effect), we can limit the system to use candidates only within the same image, resulting in the switching of faces. Here, we show the result of switching Elvis Presley and Richard Nixon's faces (left) with each other to obtain the de-identified output (right).

     
  Burst Mode Replacement:

When taking group photographs, it is often difficult to get a "perfect" picture -- where, for example, everyone is smiling, with eyes open, and looking at the camera. From a set of images taken using the "burst" mode of a camera (left panel), a composite image is created in which everyone is smiling and has their eyes open (right panel). The candidate faces for each child are constrained by the relative positions of the faces in all images, and thus no face recognition is required. While in this case the best replacement face for each child was selected manually (outlined in blue), blink and smile detection could be applied to select them automatically.

     
  Limitations:

Here, we show several examples of limitations of our algorithm itself.Input and replacement candidate faces are shown on the top, replacement results in the middle, and a detailed inset of the problem area on the bottom. The lack of eyeglasses in (a) and the occluding finger in (b) cause visual artifacts in the results. In (c), the extreme pose of the face results in it being blended into the background. These problems could be solved by dynamically selecting optimal replacement regions. (d) shows a relighting failure case, caused by forcing a replacement between images with very different lighting (skipping our lighting selection step).

     

Video

If you are having trouble viewing these .wmv videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  SIGGRAPH 2008 Video:

This video introduces the complete system for automatic face replacement in images, summarizes the face replacement algorithm, and demonstrates several applications of our method, including face de-identification, personalized face replacement, and the creation of appealing group photographs from a set of images. (With narration)

Youtube Video

     

Slides

SIGGRAPH 2008 presentation     With videos (zip file)

Speech-Enabled Avatars

Optimal Illumination for Video Relighting

Appearance Matching

FaceTracer: A Search Engine for Large Collections of Images with Faces