PP279: RESEARCH DESIGN AND DATA COLLECTION

FOR PUBLIC POLICY ANALYSIS

Professor Robert MacCoun

Goldman School of Public Policy, 2607 Hearst Ave.

642-7518, maccoun@ berkeley.edu

Spring 2008: Tuesdays and Thursdays, 2-3:30, 105 GSPP (the new bldg.)

(Office Hours: right after class or by appt.)

 

The online version of this syllabus is at http://ist-socrates.berkeley.edu/~maccoun/PP279_s08.html 
Please see the online version for the most up-to-date version; I will announce any revisions in class. 

COURSE DESCRIPTION

Empirical arguments and counterarguments play a central role in policy debates, thus public policy analysis requires a sophisticated understanding of a variety of types and sources of data. Quantitative analysis courses teach you how to analyze data; this course will introduce you to strategies of data collection and principles for critically evaluating data collected by others. Topics include measurement reliability and validity, questionnaire design, sampling, experimental and quasi-experimental program evaluation designs, qualitative research methods, and the politics of data in public policy.

 

READINGS

DeVellis, R. F. (2003).  Scale Development: Theory and Applications.  Sage.

Groves, Fowler, Couper, Lepkowski, Singer, and Tourangeau (2004).  Survey Methodology.  Wiley.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Houghton Mifflin.  [If you are interested, my review of this book appears here: http://socrates.berkeley.edu/~maccoun/JPAM_2003_BookRev.pdf.]

Many of the additional web-based readings are marked with * to indicate that they are optional.

If a link is bad, email me (maccoun@berkeley.edu) and I’ll try to fix it ASAP.  But every reading (except my handouts) is available on the web – that’s where I found them! – and you should be able to hunt them down yourself (e.g., using Google Scholar (http://scholar.google.com/).  The ANNUAL REVIEW essays are available online but only from a UC computer account because we have a site license.

 

ASSIGNMENTS (see due dates in Schedule at end of syllabus)

  • Two draft research proposals:  Each worth 40%. Like most research proposals, these will be developed in small groups of 2-3 people (size will depend on class enrollment). Teams will give in-class briefings on the second proposals at the end of the term.

1) Survey proposal (http://socrates.berkeley.edu/~maccoun/PP279_proposal1.html)

2) Program evaluation proposal/group briefing:  http://socrates.berkeley.edu/~maccoun/PP279_proposal2.html

READINGS - Schedule appears on last page

NOTE: The readings are NOT in a reader; they are online at http://socrates.berkeley.edu/~maccoun/PP279.html

The Philosophy of Science and the Politics of Data (First Look)

MacCoun, R. (1998). Biases in the interpretation and use of research results, Annual Review of Psychology, 49, 259-287. http://socrates.berkeley.edu/~maccoun/MacCoun_AnnualReview98.pdf

MacCoun, R. J. (2001). American distortion of Dutch drug statistics. Society, 38, 23-26.  [Note that there was a "distortion" introduced in typesetting: Second-to-last sentence should read "Accuracy won't invariably breed consensus," rather than "Accuracy will invariably breed consensus."]
http://socrates.berkeley.edu/~maccoun/Society2001_DistortionDutchDrugStats.pdf
 

Describing the World: Surveys and Other Measures

Asking Questions

Chapters 7, 8, and 9 of Survey Methodology text

 

OPTIONAL READINGS:

 

*Krosnick, Jon A. (1999). Survey research. Annual Review of Psychology, 50, 537-567. http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.50.1.537 

 

*Schaeffer, Nora Cate, & Presser, Stanley (2003).  The science of asking questions.  Annual Review of Sociology Aug 2003, Vol. 29: 65-88.

http://socrates.berkeley.edu/~maccoun/PP279_SchaefferPresser2004.pdf

 

Reliability and Validity (basic psychometrics)

First, skim the entire Scale Development book to get a general feel for the topic.  Then take a second pass through, reading Chapters 2 through 6 more carefully.  These chapters are directly relevant to the homework and to the Proposal 1 assignment.

 

Rob’s memo on coefficient alpha: 

http://socrates.berkeley.edu/~maccoun/CoefAlpha.pdf

 

Rob’s memo on how low reliability weakens the ability to detect relationships (e.g., program effects)

http://socrates.berkeley.edu/~maccoun/PredictiveValidity.pdf

 

 

OPTIONAL READINGS (can be found using Google Scholar):

 

*Schmitt, Neal, Uses and abuses of coefficient alpha. Psychological Assessment. 1996 Dec Vol 8(4) 350-353.
http://socrates.berkeley.edu/~maccoun/PP279_Schmitt.pdf

 

*Neisser, U., et al. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.

http://socrates.berkeley.edu/~maccoun/PP279_Neisser1.pdf

and his replies to critics:

http://socrates.berkeley.edu/~maccoun/PP279_Neisser2.html

 

*Flynn, James R., Searching for justice: The discovery of IQ gains over time. American Psychologist. 1999 Jan Vol 54(1) 5-20.

http://socrates.berkeley.edu/~maccoun/PP279_Flynn.pdf

 

*Kuncel, N. R., Hezlett, S. A., Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin. 127(1) 162-181.
http://socrates.berkeley.edu/~maccoun/PP279_GRE.pdf

 

*Lubinski, D. (2000). Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points." Annual Review of Psychology, 51, 405-444.   

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.51.1.405

 

Survey Sampling

Chapters 1 through 6 of Survey Sampling. 

Rob’s memo on computing sample sizes.  http://socrates.berkeley.edu/~maccoun/pp279_samplesize.pdf

OPTIONAL READINGS:

 

*Magnani, R., Sabin, K., Saidel, T., & Heckathorn, D. (2005). Review of sampling hard-to-reach and hidden populations for HIV surveillance.  AIDS 2005, 19, S67-S72.  [This has a good discussion of a sophisticated version of snowball sampling]

http://socrates.berkeley.edu/~maccoun/PP279_Magnani.pdf

 

*Birnbaum, Michael H. (2004).  Human research and data collection via the internet.  Annual Review of Psychology, 55, 803-832.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.55.090902.141601

 

* Marshall, G.N., Burnam, M. A., Koegel, P., Sullivan, G., & Benjamin B. (1996).  Objective Life Circumstances and Life Satisfaction: Results from the Course of Homelessness Study.  Journal of Health and Social Behavior, Vol. 37, No. 1. (Mar., 1996), pp. 44-58.  [Note: This paper has a very interesting sampling strategy for a difficult to sample population; it also has a nice example of multiple-indicator measurement of latent constructs.  And two of the authors attended my wedding!]

http://socrates.berkeley.edu/~maccoun/PP279_Marshall.pdf

 

Inferring Cause and Effect: Experimental and Quasi-Experimental Design

 

Dealing with Threats to Internal Validity

 

Shadish, Cook, & Campbell text, Chapter 1, pp. 53-62 of Chapter 2, and Chapter 8.

 

OPTIONAL READINGS: 

 

* Sander Greenland and Hal Morgenstern (2001). Confounding in health research. Annual Review of Public Health, 22, 189-212.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.22.1.189

 

* A. G. Barnett, J. C. Van der Pols, & A. J. Dobson (2005).  Regression to the mean: What it is and how to deal with it.  Int. J. Epidemiology, 34, 215-220. 

http://socrates.berkeley.edu/~maccoun/PP279_Barnett.pdf

 

Quasi-Experiments

Shadish, Cook, & Campbell text, Chapters 4-7

OPTIONAL READINGS:

*Mark W. Lipsey and David S. Cordray (2000). Evaluation Methods for Social Intervention.  Annual Review of Psychology, 51, 345-375. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.51.1.345

*Joseph P. Newhouse and Mark McClellan (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34.   http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.19.1.17

 

Dealing with Threats to Statistical Conclusion Validity

REQUIRED READINGS:

 

Rosnow, R. L., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284. http://ist-socrates.berkeley.edu/~maccoun/PP279_Rosnow.pdf


Cohen, Jacob  The earth is round (p < .05). American Psychologist. 1994 Dec Vol 49(12) 997-1003

http://ist-socrates.berkeley.edu/~maccoun/PP279_Cohen1.pdf

 

Cohen, J. (1992b). A power primer. Psychological Bulletin, 112, 155-159.

http://ist-socrates.berkeley.edu/~maccoun/PP279_Cohen2.pdf

 

OPTIONAL READINGS: 

 

* Lenth R. (2001).  Some practical guidelines for effective sample size determination.  The American Statistician, 55, 187-193.

http://socrates.berkeley.edu/~maccoun/PP279_Lenth.pdf  [NOTE: LENTH AND COHEN DISAGREE ABOUT SOME ISSUES; WE CAN DISCUSS THE POINTS OF DISAGREEMENT IN CLASS]

* Christopher Winship and Stephen L. Morgan (1999).  The estimation of causal effects from observational data Annu. Rev. Sociol., 25, 659-706.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.25.1.659

* Roderick J. Little and Donald B. Rubin (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.21.1.121

Here are some useful web-based power calculators.  I recommend you verify your results using more than one calculator.  I also recommend that you create a table and solve for N under a variety of assumptions about alpha and effect size.

 

http://www.dssresearch.com/toolkit/spcalc/power.asp

 

http://home.clara.net/sisa/power.htm

 

http://www.stat.uiowa.edu/~rlenth/Power/index.html

 

http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/

 

http://statpages.org/

 

Dealing with Threats to External Validity

Shadish, Cook, & Campbell text, Chapters 13 and 14

 

 

OPTIONAL READINGS:

 

* Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181. 

http://ist-socrates.berkeley.edu/~maccoun/PP279_Schmidt.pdf

 

 * Hunter, John E.; Schmidt, Frank L., Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, & Law. 1996 Jun Vol 2(2) 324-347.

http://ist-socrates.berkeley.edu/~maccoun/PP279_Hunter.pdf


* R. Rosenthal and M. R. DiMatteo (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.52.1.59

 

 

Qualitative Methods

 

REQUIRED:

 

Shadish, W. R. (1995).  Philosophy of science and the quantitative-qualitative debates: Thirteen common errors.  Evaluation and Program Planning, 18, 63-75.

http://socrates.berkeley.edu/~maccoun/PP279_Shadish.pdf

 

 

OPTIONAL:

* Morgan, G., & Smircich, L. (1980).  The case for qualitative research. The Academy of Management Review, 5, 491-500.

http://socrates.berkeley.edu/~maccoun/PP279_Morgan.pdf

 

* David L. Morgan (1996). Focus groups. Annual Review of Sociology, 22, 129-152.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.22.1.129

 

PP279 SCHEDULE – SPRING 2008 -- REVISED

Week

Day

Date

Topic

 

1

Tues

Jan 22

Course overview

 

 

Thurs

Jan 24

Philosophy vs. politics

 

2

Tues

Jan 29

Asking questions

 

 

Thurs

Jan 31

Asking questions

 

3

Tues

Feb 5

Asking questions

 

 

Thurs

Feb 7

Intro to psychometrics

 

4

Tues

Feb 12

Measurement reliability

 

 

Thurs

Feb 14

Measurement validity

 

5

Tues

Feb 19

Measurement validity

1 page PR#1 preposal due

 

Thurs

Feb 21

Survey sampling

 

6

Tues

Feb 26

Survey sampling

HW due Tu Feb 26, hard copy, bring to class

 

Thurs

Feb 28

Special populations

 

7

Tues

Mar 4

NO CLASS

 

 

Thurs

Mar 6

Special populations

 

8

Tues

Mar 11

Threats to internal validity

 

 

Thurs

Mar 13

Threats to internal validity

 

9

Tues

Mar 18

Experimentation

 

 

Thurs

Mar 20

Experimentation

PR#1 due Fri Mar 21, 5pm via email attachment

10

Tues

Mar 25

SPRING BREAK

 

 

Thurs

Mar 27

SPRING BREAK

 

11

Tues

Apr 1

Quasi-experimentation

 

 

Thurs

Apr 3

Quasi-experimentation

1-page PR#2 preposal due

12

Tues

Apr 8

Stat conclusion validity

 

 

Thurs

Apr 10

Stat conclusion validity

 

13

Tues

Apr 15

Stat conclusion validity

 

 

Thurs

Apr 17

Threats to external validity

 

14

Tues

Apr 22

Threats to external validity

 

 

Thurs

Apr 24

Qualitative methods

 

15

Tues

Apr 29

Group briefings

 

 

Thurs

May 1

Group briefings

 

16

Tues

May 6

Group briefings

 

 

Thurs

May 8

Group briefings (if needed)

 

FinalsWk

 

 

 

PR#2 due Mon May 12, 5pm, via email attachment

 

 

 

 

This schedule may shift a bit depending on how class discussions go, but the order of topics will not change.


Last revised on 3/11/08