NSF IIS Project on
"Tagging and Browsing Videos According to the Preferences of Differing Affinity Groups"
John R. Kender
Columbia University



We investigate the new task of determining which textual tags are preferred by different affinity groups for news and related videos. We use this knowledge to assign new group-specific tags to other videos of the same event that have arisen elsewhere. We map visual and multilingual textual features into a joint latent space using reliable visual cues, and determine tag relationships through various canonical correlation analyses (CCA) variants. For human-interest international events such as epidemics and transportation disasters, we detect country-specific tags from US, Chinese, European, South American, and other countries' news coverage.

We catalog statistically significant cross-group differences in multimedia creation and tagging, and explore variants of Deep CCA, finding them better suited to capturing those preferences in a three view space (one common video dimension, two culturally-determined tag dimensions). We investigate how these non-linear methods can be extended to the videos of multiple affinity groups, including more subtle shadings such as US compared to UK or even Canada. As different groups are differentially sensitive to particular images, we investigate the day-to-day spreading influence of visual memes across countries through a novel application of the PageRank algorithm.

We demonstrate and evaluate a novel cross-group multimedia browser that accesses online webpage archives of international events from two different countries. It visualizes these results with country-specific information on separate timelines, but with cross-country images and tags straddling both. This system provides an exploratory, zoomable differential view of clips and text, and graphs their development over time. We demonstrate that this browser expands and improves the effectiveness of video retrieval

For further information: jrk atsign cs.columbia.edu


This material is based upon work supported by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Last update: Jun 14, 2018