Linguistics 251: Probabilistic Methods in Linguistics (Fall 2012)

This course is about probabilistic approaches to language knowledge, acquisition, and use. Today, studying language from a probabilistic perspective requires mastery of the fundamentals of probability and statistics, as well as familiarity with more recent developments in probabilistic modeling. In this course we'll move quickly through basic probability theory, then cover fundamental ideas in statistics–parameter estimation and hypothesis testing. We'll then cover a fundamental class of probabilistic models–the linear model–which as a side effect will familiarize you with the most widely used tools in statistics: linear regression, analysis of variance (ANOVA), and generalized linear models (including logistic regression). We'll cover these topics using both frequentist methods (what you need to use in order to write publishable data analyses) and Bayesian methods (which are becoming increasingly popular in all sorts of settings, especially in cognitive modeling of language). We'll then move on to the more advanced topic of hierarchical (a.k.a. multilevel or mixed-effects) modeling, and perhaps even a bit of probabilistic grammars if we have a chance.

The course will involve a hands-on approach to data and modeling, and we'll be using the open-source R programming language (and a bit of JAGS, which interfaces nicely with R, for Bayesian modeling). You'll learn the basics of data visualization and statistical analysis in R, and the class will involve periodic programming practica to ensure that your R programming questions are adequately addressed. Transcripts of programming practica will also be put up online. I encourage you to download R hereas soon as you can, get it running on your own computer, and go through the R tutorial found in Chapter 1 of Harald Baayen's book, or this hands-on introduction to R. You can also download JAGS here.

The course assumes no expertise in linguistics, quantitative methods, or programming, but background in one or more of these areas will be useful. We'll start from elementary probability theory and build up briskly. We will make a fair amount of use of high school algebra, and also a bit of calculus and liner algebra; there's an appendix in the book that provides you with what you need for the latter two.

The main reading material will be draft chapters of a textbook-in-progress, Probabilistic Models in the Study of Language, that I am writing. These draft chapters can be found here. There are also a number of other reference texts that may be of use in the course, including:

• Harald Baayen's book: Analyzing Linguistic Data. A Practical Introduction to Statistics. Cambridge University Press. Available online here
• Shravan Vasishth's book draft: The foundations of statistics: A simulation-based approach (free download)
• Keith Johnson's book on quantitative methods in linguistics ($40 on Amazon; no longer available as a free download)
• David MacKay's Information Theory, Inference, and Learning Algorithms – a great text available freely online
• Manning & Schuetze's Foundations of Statistical Natural Language Processing, available online through UC Libraries here.
• Jurafsky & Martin's Speech and Language Processing
• Christopher Bishop's Pattern Recognition and Machine Learning
• Gelman, Carlin, Stern, and Rubin's Bayesian Data Analysis
• John Rice's Mathematical Statistics and Data Analysis – a good general book for introductory statistics (mostly classical).
• Brian Roark and Richard Sproat's Computational Approaches to Morphology and Syntax for the Probabilistic Grammar Formalisms section

Finally, we may supplement these with additional readings, both from statistics/NLP texts and pertinent linguistics articles.

If you are taking the course for credit, there are four things expected of you:

1. Regular attendance in class.
2. Doing the assigned readings and coming ready to discuss them in class.
3. Doing several homework assignments to be assigned throughout the quarter. Email submission of the homework assignments is encouraged, but please send it to lign251-homework@ling.ucsd.edu instead of to me directly. If you send it to me directly I may lose track of it.

   You can find some guidelines on writing good homework assignments here. The source file to this PDF is here.

4. A final project which will involve computational modeling and/or data analysis in some area relevant to the course. Final project guidelines are here.

There is a mailing list for this class, lign251-l@mailman.ucsd.edu. Please make sure you're subscribed to the mailing list by filling out the form at https://mailman.ucsd.edu/mailman/listinfo/lign251-l! We'll use it to communicate with each other.

For this class I'll be maintaining an FAQ. Read the FAQ here.

I also run the R-lang mailing list. I suggest that you subscribe to it; it's a low-traffic list and is a good clearinghouse for technical and conceptual issues that arise in the statistical analysis of language data.

In addition, the searchable R mailing lists are likely to be useful.