{
"title": "What is a Good Nearest Neighbors Algorithm <br>for Finding Similar Patches in Images?",
"keywords": ["nearest neighbor search", "vantage point tree", "kd-tree", "epsilon search", "image patches"],
"date": "Spring 2007 - Fall 2008",
"desc": "Efficient methods for finding similar image patches and analysis of their performance",
"longdesc": "<p>Finding similar patches in images is a critical, yet expensive, step in a
wide variety of computer vision tasks, including object and shape recognition,
texture synthesis, and image denoising.  For an image with <b>N</b> patches, an
exhaustive search by scanning through the entire image for all <b>N</b> patches
takes O(<b>N^2</b>) time, which can take several minutes or even hours.
However, by treating each image patch as a point in a high-dimensional space,
we can use a Nearest Neighbors (NN) algorithm to compute the exact same results
in a fraction of the time. Although a large number of general-purpose NN
algorithms have been developed over the years, applying them directly to image
search will not achieve the best results -- images have distinct structure and
unique properties that can be taken advantage of, to greatly improve search
performance.</p>

<p>In this work, we optimize many NN approaches specifically for finding
similar patches in images and experimentally evaluate them to compare their
performance.  The following are the main contributions of our work:</p>

<ul>
<li>We describe several NN methods, some of which have not been widely
used in computer vision. We restrict ourselves to exact approaches which
guarantee that all nearest neighbors are found -- approximate methods are not discussed.</li>

<li>We use several techniques to significantly improve the efficiencies of the
different NN algorithms (in some cases exploiting the inherent properties of
images). These include precomputing or using look-up tables for computing
distances, as well as using priority queues for certain types of searches.</li>

<li>We evaluate the off-line (construction) and on-line (search) performances
of the different methods using a set of real images. We study the behaviors of
the algorithms with respect to various parameters, including patch size, image
size, number of nearest neighbors, search range, and number of input images.
We find that the <b>vantage point tree</b>, which has not been widely used in
vision, has the best overall performance.</li>
</ul>",

"publications": [{"id": "nn_eccv2008"}],

"thumbfmt": "%(base)s_thumb.%(ext)s",

"images": [{
    "url": "images/sample_search.png",
    "name": "Finding Similar Patches",
    "desc": "The closest neighbors of the patches outlined in red are shown in
    blue.  The matching eyes in (a) took 2125 ms to compute using an exhaustive
    search, but only 1.25 ms (1700X faster) using our image-optimized
    implementation of the <i>vp</i>-tree, one of the methods discussed in this
    paper.  Similarly, finding the nosetips in (b) took 1375 ms using brute
    force, but only 3.43 ms (401X faster) using the <i>vp</i>-tree."
}, {
    "url": "images/nn.png",
    "name": "Illustration of Various Nearest Neighbors Methods",
    "desc": "This figure illustrates the partitioning of a 2D point set using
    different types of nearest neighbor trees, all with a maximum leaf size of
    1 and a branching factor of 2. Line thickness denotes partition order
    (thicker lines were partitioned first). Note the very different structures
    created by the methods, which result in very different search speeds."
}, {
    "url": "images/dataset.png",
    "name": "The Image Dataset used for Evaluation",
    "desc": "This image shows thumbnails of the 15 images used to evaluate the
    various nearest neighbors algorithms.  They were chosen to represent a wide
    variety of image types, e.g., indoor, outdoor, portraits, etc. Note that
    even with this small number of test images, there are still millions of
    patches to search through."
}, {
    "url": "images/comprehensive.png",
    "name": "Comprehensive Results",
    "desc": "This table shows comprehensive results for a variety of tree
    types. The splits are given as branching factors for ball trees,
    <i>k</i>-means, <i>k</i>d-trees, and <i>vp</i>-trees (<i>k</i>), and as bin
    sizes for PCA trees and <i>vp</i>-trees (<i>d</i>). Construction costs are
    given as average number of distance calculations per patch. Search speeds
    are given as improvement over brute-force search. Optimal values are shown
    in <b>bold</b>."
}, {
    "url": "images/bargraph.png",
    "name": "Summary of Construction and Search Performance",
    "desc": "Summaries of the (a) construction costs and (b) search performance
    for different types of NN trees. The construction costs are given as
    average number of distance computations per input patch, plotted on a
    logarithmic scale. The search speeds are given as ratio of time taken for
    brute force scan versus time taken using the given tree type. All numbers
    are for best parameter settings.  Note that although ball trees and
    <i>vp</i>-trees have similarly good search performance, the latter has a
    much smaller construction cost."
}, {
    "url": "images/perf.png",
    "name": "Performance Along Various Dimensions",
    "desc": "Search performance of the <i>vp</i>-tree plotted as a function of
    (a) search distances in log-log, for (top) <i>e</i>-NN and (bottom)
    <i>k</i>-NN searches; (b) patch size in semi-log; and (c) number of random
    input images (top) and video frames (bottom). See figure for details."
}, {
    "url": "images/summary.png",
    "name": "Summary of Results",
    "desc": "This table show a summary of the results for each tree type. The
    <i>vp</i>-tree performs well in all respects."
}],

"posters": [{
    "url":  "nnsearch_eccv_2008_poster.ppt",
    "name": "ECCV 2008 poster"
}],

"relateds": [{"url": "http://www.cs.columbia.edu/CAVE/projects/search/", "name": "Nearest Neighbor Search in High Dimensions"}]

}