file.pdf (610.82 kB)
Download file

Empirical Risk Minimization for Probabilistic Grammars: Sample Complexity and Hardness of Learning

Download (610.82 kB)
journal contribution
posted on 01.09.2012, 00:00 by Shay B. Cohen, Noah A. Smith

Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures. They are used ubiquitously in computational linguistics. We present a framework, reminiscent of structural risk minimization, for empirical risk minimization of probabilistic grammars using the log-loss. We derive sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting. By making assumptions about the underlying distribution that are appropriate for natural language scenarios, we are able to derive distribution-dependent sample complexity bounds for probabilistic grammars. We also give simple algorithms for carrying out empirical risk minimization using this framework in both the supervised and unsupervised settings. In the unsupervised case, we show that the problem of minimizing empirical risk is NP-hard. We therefore suggest an approximate algorithm, similar to expectation-maximization, to minimize the empirical risk.

History

Publisher Statement

© 2012 Association for Computational Linguistics The version of record is available online at http://dx.doi.org/10.1162/COLI_a_00092

Date

01/09/2012

Usage metrics

Exports