Course Description:  The course offers a general introduction to psychometric methods primarily emphasizing classical test theory, test construction and validation, and test use.  The emphasis lies with developing a firm understanding of basic psychometric concepts.  This course lays a foundation for more advanced courses in specific topics introduced here. The course understands psychometrics and testing as applying broadly, not just to paper and pencil tests but also to performance assessments, behavioral observations, measured variables in experiments and quasi-experiments, surveys, and other forms of behavioral data collection. However, much of the material will emphasize measurement involving multiple indicators of a common construct.

Course Objectives: The course assumes a foundation in basic statistics and a healthy curiosity but little more. The more you put into the course, the more you will get out of the course. The course design reflects the following objectives.
1. Students will gain a basic understanding of the foundations of test theory that will prepare them to pursue more advanced topics (e.g., item response theory, structural equation modeling).
2. Students will gain the background and confidence to critically read technical manuals and other documentation in conjunction with use of published tests.
3. Students will gain facility with conceptual tools for thinking through issues of validity and reliability as applied to all measures from dependent variables in experiments to large scale testing programs.
4. Students will gain a level of comfort with algebraic representations of test scores and the use of these to think through applied problems related to test use and interpretation.
5. Students will gain an increased sensitivity to the fallibility of educational and psychological tests and the limits to their use and interpretation.
6. Students will gain exposure to the use of statistical software for conducting psychometric analyses and some experience with such analyses.

Text Book:
Draft chapters by Professor Deborah Bandalos will be provided for reading.  These are provided for your personal use in the course only.  Do not share them with anyone outside the class.  This condition was stipulated by the publisher in order to make it possible for us to use the draft chapters this term.  Please return the kindness of the author by not sharing the draft chapters.

I will illustrate psychometric concepts using a variety of software packages.  Familiarity with the software is not a course requirement. However, learning psychometrics simply by reading about it is akin to learning to swim, ski, or play a musical instrument simply by reading about it. Actual practice is a much more effective method. Whether you use a simple calculator, a spreadsheet, or advanced statistical software, it is a good habit to play around with the material by constructing concrete examples and taking a try-and-see attitude toward the material. If something seems puzzling, make up an example and try it out. If something seems counter-intuitive to you, try to construct a counterexample. The more concrete you make psychometrics, the more comfortable you will feel with the material, the better you will understand it, and the more skills you will develop that you can apply outside of the class. None of the this is required for the course, but it will make it more fun, more interesting, and more valuable at a practical level.

Homework:  Turn in your chapter review electronically before the start of class each week.  Chapter reviews do not apply the chapters from the Standards.  For each chapter not from the Standards, turn in a text or rich text (rtf) document with the following three questions followed by your answers for the chapter in question.  Copy and paste the questions into our document and type an 'X' over the number for your rating.

1. What one part of the chapter would you most like to see improved and why?
2. What one part of the chapter would you most like to see kept as it is and why?
3. Rate the chapter on the following scale and briefly explain your rating, citing examples if applicable.
(1) No new information for me
(2)
(3) Too easy
(4)
(5) Just right
(6)
(7) Too hard
(8)
(9) Completely incomprehensible

Course Project:
Week 3: Project Topic
Post your choice of test population and 2 constructs to Blackboard. Include all three pieces of information in the subject line. Do not choose something that has already been taken. State your choice with sufficient specificity that others can still choose distinct but related options.  The test population constitutes the population of test takers for whom you intend the test.  The two constructs should be unidimensional quantities across which you can expect variability in the test population.

Week 7: Project Part I
Begin a proposal for the validation of a test of your construction following the format below. Use the following lettered headings to clearly mark each section of your proposal.  Double space the proposal and use APA format and style. However, printing on both sides of the page is fine.  Turn in a paper copy at the start of class.

A. Title page including your name and affiliation.
B. Abstract (180 words max).  (Only include Part I material.)
C. Purpose of the test (250 words max). Describe the intended use of the test. Describe the intended users and the intended test taking population. Explain what the test would contribute over and above existing tests. Describe the theoretical rationale behind the test.
D. Test Blueprint (500 words max).
  1. Define the constructs to be assessed by the test. Your test should include at least two constructs and at least six items per construct. Describe the relationship(s) between the constructs, conceptually and statistically.
  2. Specify the format of the items and response options.
  3. Specify the content of the items. If a scale on your test includes more than one kind of item, specify the number of items of each type.
  4. Specify the acceptable range of item statistics (mean or proportion correct, standard deviation) for each item and test statistics (mean, standard deviation, reliability) for each subscore.

Week 11: Project Part II
Turn in a paper copy at the start of class.

Add the following to your proposal. 
A. Expand the abstract to include parts E and F.
E. Draft test. Provide a draft version of the test including instructions and a full set of items that conform to parts 1-3 of the test blueprint.
F. Proposed validation plan (1000 words max).
  1. Outline your validity argument for each scale by stating the desired interpretation and listing supporting assumptions (250 words max).  Mark the assumptions that you consider essential to the interpretation of the test scores and then rank them in terms of degree of support.  Identify the five least well supported essential assumptions.
  2. For each of the five assumptions identified in section F1, describe a validity study to test that assumption (750 words max).  Keep each study individually focused on one source of evidence, but try to design a set of studies that collectively provide a range of different types of validity evidence.  Clearly identify the subsection for each study using subheadings.

Week 15: Project Part III
Convert your paper to portable document format (PDF), and turn in both a hard copy and a PDF file. (If you cannot save directly to PDF, download a free PDF print driver such as Cute PDF.)

Add the following to your proposal.
A. Expand the abstract to cover the entire proposal.
G. Factor model (250 words max).
  1. Present a 2-factor model appropriate to our test (present this in theory, this task does require conducting a factor analysis of data). Choose plausible values for the item parameters (loadings, unique variances, intercepts, factor correlation[s]).  Interpret the model in terms of which items assess which constructs and how well.
  2. Use the factor model to simulate data as if it came from your test (use any software you like, spreadsheet or statistical software).  Simulate at least 100 cases.
  3. If necessary, tweak your factor model values until your items satisfy part 4 of the test blueprint.
H. Pilot study (500 words max). Report this as if you had completed an empirical pilot study, but use the simulated data from part G. (Because your subscales measure different things, you will want to analyze them separately. However, if a total score has meaning for your test, it may also give you something interesting to write about if you analyze the scale as a whole as well.)
  1. Conduct an item analysis of the data set. Report the item statistics (means, standard deviations, item total correlations and regression R-square values). Describe how differences between item statistics relate to differences between item parameters in the factor model.
  2. Report the scale statistics (means, standard deviations). Compare and contrast the scale statistics for each scale.
  3. Report both alpha and lambda 2 reliability estimates. Describe the alphas-if-item-deleted.
  4. Relate the results from parts H1 to H3 to the test blueprint. Provide an overall evaluation of the functioning of the draft test based on these results.
J. Appendices: Include the spreadsheet or syntax used for your simulation as Appendix A, and the item anlysis output as Appendix B.

Proposals will be graded using the following rubric.

Completeness (50% of grade, 13 points total)
A&B = 1 point
C = 1 point
D = 2 points. Each of four sections = .5 points.
E = 1 point.
F = 3 points.
G = 1 point.
H = 2 points. Each of four sections = .5 points.
J = 2 points.

Overall quality dimensions (50% of grade, 40 points total)
Clarity of presentation (1 - 10)
Technical accuracy of reporting (1 - 10)
Depth with which issues are presented within allowed space (1 - 10)
Overall conceptualization and design of proposed test and test development (1 - 10)
1-5 = unsatisfactory.
6 = minimally satisfactory.
7 = some significant weaknesses.
8 = generally good with a few weak points.
9 = overall very well done.
10 = outstanding effort.

Grading:  The course project is worth another 70% of your grade. The topic is worth 10% and each of the three parts is worth 20%.  That leaves 30% for the homework assignments.  Letter grades will be assigned as indicated below.