Carnegie Mellon University
Browse

Automatic Multimedia Cross-modal Correlation Discovery

Download (192.11 kB)
journal contribution
posted on 1975-01-01, 00:00 authored by Jia-Yu Pan, HyungJeong Yang, Pinar Duygulu, Christos Faloutsos
Given an image (or video clip, or audio song), how do we automatically assign keywords to it? The general problem is to find correlations across the media in a collection of multimedia objects like video clips, with colors, and/or motion, and/or audio, and/or text scripts. We propose a novel, graph-based approach, "MMG", to discover such cross-modal correlations.Our "MMG" method requires no tuning, no clustering, no user-determined constants; it can be applied to any multimedia collection, as long as we have a similarity function for each medium; and it scales linearly with the database size. We report auto-captioning experiments on the "standard" Corel image database of 680 MB, where it outperforms domain specific, fine-tuned methods by up to 10 percentage points in captioning accuracy (50% relative improvement).

History

Publisher Statement

All Rights Reserved

Date

1975-01-01

Usage metrics

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC