{
"title": "Face Swapping: Automatically Replacing Faces in Photographs",
"keywords": ["face swapping", "large image collections", "face replacement", "deidentification", "privacy protection"],
"date": "Fall 2007 - Fall 2008",
"desc": "Using large collections of images to automatically replace faces in photos",
"longdesc": "<p>Advances in digital photography have made it possible to
capture large collections of high-resolution images and share them on the
internet. While the size and availability of these collections is leading to
many exciting new applications, it is also creating new problems. One of the
most important of these problems is privacy. Online systems such as <a
href=\"http://maps.google.com/help/maps/streetview\">Google Street View</a>
allow users to interactively navigate through panoramic images of public places
created using thousands of photographs. We believe that an attractive solution
to the privacy problem is to remove the identities of people in photographs by
automatically replacing their faces with ones from a collection of stock
images. Automatic face replacement has other compelling applications as well.
For example, people commonly have large personal collections of photos on their
computers. These collections often contain many photos of the same person(s)
taken with different expressions, and under various poses and lighting
conditions. One can use such collections to create novel images by replacing
faces in one image with more appealing faces of the same person from other
images.  For group shots, the burst mode available in most cameras can be used
to take several images at a time. With an automatic face replacement approach,
one could create a single composite image with, for example, everyone smiling
and with both eyes open.</p>

<p>In this project, we present a complete system for automatic face replacement
in images. Our system uses a large library of face images created automatically
by downloading images from the internet, extracting faces using face detection
software, and aligning each extracted face to a common coordinate system. This
library is constructed off-line, once, and can be efficiently accessed during
face replacement. Our replacement algorithm has three main stages.  First,
given an input image, we detect all faces that are present, align them to the
coordinate system used by our face library, and select candidate face images
from our face library that are similar to the input face in appearance and
pose. Second, we adjust the pose, lighting, and color of the candidate face
images to match the appearance of those in the input image, and seamlessly
blend in the results. Third, we rank the blended candidate replacements by
computing a match distance over the overlap region. Our approach requires no 3D
model, is fully automatic, and generates highly plausible results across a wide
range of skin tones, lighting conditions, and viewpoints. We show how our
approach can be used for a variety of applications including face
de-identification and the creation of appealing group photographs from a set of
images. </p>",

"publications": [{"id": "facereplace_siggraph2008"}],

"thumbfmt": "%(base)s_thumb.%(ext)s",

"images": [{
    "url": "images/relighting.png",
    "name": "Color and Lighting Adjustment",
    "desc": "In this example, we demonstrate the importance of our color and
    lighting adjustment algorithm. We replace (a) the face in the input
    photograph with (b) the face selected from the library.  Replacement
    results (c) without and (d) with recoloring and relighting.  Notice the
    significantly improved realism in the final result."
}, {
    "url": "images/some_results.png",
    "name": "Face Replacement Results",
    "desc": "Here, we show several examples of the results obtained using our
    system. Each example shows, in order from left to right, the input face
    image, a candidate face, and the replacement result. Note the realism of
    the results, despite differences in pose, lighting and facial appearance.
    Each row contains (from left to right) the original photograph, a candidate
    face selected from the library, and the replacement result produced
    automatically using our algorithm.  The age and gender mismatches in (c)
    and (d) could be avoided by enforcing consistency across those attributes
    (which our system does not currently do)."
}, {
    "url": "images/de-ident.png",
    "name": "Face De-Identification",
    "desc": "To preserve privacy in online collections of photos, one can use
    our system to automatically replace each face in an input image with the
    top-ranked candidate taken from a collection of stock photographs. We show
    the result of automatically replacing the input faces (top) with the
    top-ranked candidate from the face library to obtain the de-identified
    results (bottom).  No user intervention was used to produce this result."
}, {
    "url": "images/elvis.png",
    "name": "Face Switching",
    "desc": "As a special case of face de-identification (or for use as a
    special effect), we can limit the system to use candidates only within the
    same image, resulting in the switching of faces.  Here, we show the result
    of switching Elvis Presley and Richard Nixon's faces (left) with each other
    to obtain the de-identified output (right)."
}, {
    "url": "images/burstmode2.png",
    "name": "Burst Mode Replacement",
    "desc": "When taking group photographs, it is often difficult to get a
    \"perfect\" picture -- where, for example, everyone is smiling, with eyes
    open, and looking at the camera. From a set of images taken using the
    \"burst\" mode of a camera (left panel), a composite image is created in
    which everyone is smiling and has their eyes open (right panel). The
    candidate faces for each child are constrained by the relative positions of
    the faces in all images, and thus no face recognition is required. While in
    this case the best replacement face for each child was selected manually
    (outlined in blue), blink and smile detection could be applied to select
    them automatically."
}, {
    "url": "images/limitations.png",
    "name": "Limitations",
    "desc": "Here, we show several examples of limitations of our algorithm
    itself.Input and replacement candidate faces are shown on the top,
    replacement results in the middle, and a detailed inset of the problem area
    on the bottom. The lack of eyeglasses in (a) and the occluding finger in
    (b) cause visual artifacts in the results.  In (c), the extreme pose of the
    face results in it being blended into the background. These problems could
    be solved by dynamically selecting optimal replacement regions. (d) shows a
    relighting failure case, caused by forcing a replacement between images
    with very different lighting (skipping our lighting selection step)."
}],

"videos": [{
    "url": "videos/SIGGRAPH2008.wmv",
    "thumb": "images/SIGGRAPH2008Thumb.png",
    "name": "SIGGRAPH 2008 Video",
    "desc": "This video introduces the complete system for automatic face
    replacement in images, summarizes the face replacement algorithm, and
    demonstrates several applications of our method, including face
    de-identification, personalized face replacement, and the creation of
    appealing group photographs from a set of images (With narration).",
    "youtube": "rRQQNJFV7Gw"
}],

"slides": [{
    "url": "face_replace_siggraph_2008.ppt", "name": "SIGGRAPH 2008 Presentation"
}, {
    "url": "face_replace_slides.zip", "name": "SIGGRAPH 2008 Presentation (with videos)"
}],

"relateds": [{"url": "../facesearch"}]

}