Course Description:  The course offers a general introduction to psychometric methods primarily emphasizing classical test theory, test construction and validation, and test use.  The emphasis lies with developing a firm understanding of basic psychometric concepts.  This course lays a foundation for more advanced courses in specific topics introduced here. The course understands psychometircs and testing as applying broadly, not just to paper and pencil tests but also to performance assessments, behavioral observations, measured variables in experiments and quasi-experiments, surveys, and other forms of behavioral data collection. However, much of the material will emphasise measurement involving multiple indicators of a common construct.

Course Objectives: The course assumes a foundation in basic statistics and a healthy curiosity but little more. The more you put into the course, the more you will get out of the course. The course design relfects the following objectives.
1. Students will gain a basic understanding of the foundations of test theory that will prepare them to pursue more advanced topics (e.g., item response theory, structural equation modeling).
2. Students will gain the background and confidence to critically read technical manuals and other documentation in conjuction with use of published tests.
3. Students will gain facility with conceptual tools for thinking through issues of validity and reliability as applied to all measures from dependent variables in experiments to large scale testing programs.
4. Students will gain a level of comfort with algebraic representations of test scores and the use of these to think through applied problems related to test use and interpretation.
5. Students will gain an increased sensitivity to the fallibility of educational and psychological tests and the limits to their use and interpretation.
6. Students will gain exposure to the use of statistical software for conducting psychometric analyses and some experience with such analyses.
7. Those students who choose to avail themselves of it will optionally leave the course with additional experience using various software packages for psychometric data analysis.

I will illustrate psychometric concepts using a variety of software packages.  Familiarity with the software is not a course requirement. However, learning psychometrics simply by reading about it is akin to learning to swim, ski, or play a musical instrument simply by reading about it. Actual practice is a much more effective method. Whether you use a simple calculator, a spreadsheet, or advanced statistical software, it is a good habit to play around with the material by constructing concrete examples and taking a try-and-see attitude toward the material. If something seems puzzling, make up an example and try it out. If something seems counter-intuitive to you, try to construct a counterexample. The more concrete you make psychometrics, the more comfortable you will feel with the material, the better you will understand it, and the more skills you will develop that you can apply outside of the class. None of the this is required for the course, but it will make it more fun, more interesting, and more practical.

Examinations:  The examinations will not be cumulative but later material will always presuppose a familiarity with prior material.  You are allowed one two-sided 8.5 x 11 inch hand-written page of notes and a calculator to be used during each examination.  Examinations will emphasize your ability to reason using psychometric principles studied in the course.  Although examinations will not emphasize computations, they will require some computation.  Exercises in the text book offer the best test preparation.

Homework:  Come prepared to turn in homework assignments at the beginning of class.  Given that the homework comes due before the corresponding lecture (and the fact that you can look up the answers if you get stuck), I will grade more on completeness than accuracy.  The assignments primarily serve the purpose of allowing you to test your understanding of the reading before the lecture and thus better recognize where you have questions about the material.

Course Project: Write a proposal for the validation of a test of your construction following the format below. Double space the proposal and use APA format and style. However, printing on both sides of the page is allowed to save paper.

A. Title page including your name and affiliation.
B. Abstract (180 words max).
C. Purpose of the test (250 words max). Describe the intended use of the test. Describe the intended users and the intended test taking population. Explain what the test would contribute over and above existing tests. Describe the theoretical rationale behind the test.
D. Test Blueprint (500 words max).
  1. Define the constructs to be assessed by the test. Your test should include at least two constructs and at least six items per construct. Describe the relationship(s) between the constructs, conceptually and statistically.
  2. Specify the format of the items and response options.
  3. Specify the content of the items. If a scale on your test includes more than one kind of item, specify the number of items of each type.
  4. Specify the acceptable range of item statistics (mean or proportion correct, standard deviation) for each item and test statistics (mean, standard deviation, reliability) for each subscore.
E. Draft test. Provide a draft version of the test including instructions and a full set of items that conform to parts 1-3 of the test blueprint.
F. Proposed validation plan (750 words max). Describe five validation studies for your test (one paragraph each). Design one study for each of the five main sources of test validity evidence listed in the Standards (content, response processes, internal structure, relationships with other variables, and consequences of test use). Explain the rationale behind the intended interpretation of the test and how each study tests an assumption of that rationale (validity argument).
G. Factor model (250 words max). Download the SPSS code for simulating item response data from an assumed factor model. Enter plausible values for the item parameters (loadings, error variances, intercepts, facor correlation[s]). If necessary, tweak your values until your items satisfy part 4 of the test blueprint. Report the final set of values in a table. In the text, describe your general interpreation of the resulting factor model. Include your interpretation of each factor, a description of which items assess which factors, and a general description of the strengths and weaknesses of the item set.
H. Pilot study (500 words max). Report this as if you had completed an empirical pilot study, but use the simulated data from part G.
  1. Conduct an item analysis of the data set. Report the item statistics (means, standard deviations, item total correlations and regression R-square values). Describe how differences between item statistics relate to differences between item parameters in the factor model.
  2. Report the scale statistics (means, standard deviations). Compare and constrast the scale statistics for each scale.
  3. Report both alpha and lambda 2 reliability estimates. Describe the alphas-if-item-deleted.
  4. Relate the results from parts H1 to H3 to the test blueprint. Provide an overall evaluatoin of the functioning of the draft test based on these results.
J. Appendices: Include the SPSS syntax used for your simulation as Appendix A, and the SPSS output as Appendix B.

Convert your paper to portable document format (PDF), and turn in both a hard copy and a PDF file. (If you cannot save directly to PDF, download a free PDF print driver such as Cute PDF.) Note the due dates on the course schedule. Proposals will be grading using the folloiwng rubric.

Completeness (50% of grade, 13 points total)
A&B = 1 point
C = 1 point
D = 2 points. Each of four sections = .5 points.
E = 1 point.
F = 3 points.
G = 1 point.
H = 2 points. Each of four sections = .5 points.
J = 2 points.

Overall quality dimensions (50% of grade, 40 ponts total)
Clarity of presentation (1 - 10)
Technical accuracy of reporting (1 - 10)
Depth with which issues are presented within allowed space (1 - 10)
Overall conceptualization and design of proposed test and test development (1 - 10)
1-5 = unsatisfactory.
6 = minimally satisfactory.
7 = some significant weaknesses.
8 = generally good with a few weak points.
9 = overall very well done.
10 = outstanding effort.

Grading:  Each of the two examinations is worth 25% of your total grade.  The course project is worth another 30%. That leaves 20% for the homework assignments.  Letter grades will be assigned as indicated below.