Course Description: The course offers a general
introduction to psychometric methods primarily emphasizing
classical test theory, test construction and validation, and test
use. The emphasis lies with developing a firm understanding
of basic psychometric concepts. This course lays a
foundation for more advanced courses in specific topics introduced
here. The course understands psychometircs and testing as applying
broadly, not just to paper and pencil tests but also to
performance assessments, behavioral observations, measured
variables in experiments and quasi-experiments, surveys, and other
forms of behavioral data collection. However, much of the material
will emphasize measurement involving multiple indicators of a
common construct.
Course Objectives: The
course assumes a foundation in basic statistics and a healthy
curiosity but little more. The more you put into the course, the
more you will get out of the course. The course design reflects
the following objectives.
1. Students will gain a basic understanding of the foundations of
test theory that will prepare them to pursue more advanced topics
(e.g., item response theory, structural equation modeling).
2. Students will gain the background and confidence to critically
read technical manuals and other documentation in conjunction with
use of published tests.
3. Students will gain facility with conceptual tools for thinking
through issues of validity and reliability as applied to all
measures from dependent variables in experiments to large scale
testing programs.
4. Students will gain a level of comfort with algebraic
representations of test scores and the use of these to think
through applied problems related to test use and interpretation.
5. Students will gain an increased sensitivity to the fallibility
of educational and psychological tests and the limits to their use
and interpretation.
6. Students will gain exposure to the use of statistical software
for conducting psychometric analyses and some experience with such
analyses.
7. Those students who choose to avail themselves of it will
optionally leave the course with additional experience using
various software packages for psychometric data analysis.
I will illustrate psychometric concepts using a variety of
software packages. Familiarity with the software is not a
course requirement. However, learning psychometrics simply by
reading about it is akin to learning to swim, ski, or play a
musical instrument simply by reading about it. Actual practice is
a much more effective method. Whether you use a simple calculator,
a spreadsheet, or advanced statistical software, it is a good
habit to play around with the material by constructing concrete
examples and taking a try-and-see attitude toward the material. If
something seems puzzling, make up an example and try it out. If
something seems counter-intuitive to you, try to construct a
counterexample. The more concrete you make psychometrics, the more
comfortable you will feel with the material, the better you will
understand it, and the more skills you will develop that you can
apply outside of the class. None of the this is required for the
course, but it will make it more fun, more interesting, and more
valuable at a practical level.
Examinations: The examinations
will not be cumulative but later material will always presuppose a
familiarity with prior material. You are allowed one
two-sided 8.5 x 11 inch hand-written page of notes and a
calculator to be used during each examination. Examinations
will emphasize your ability to reason using psychometric
principles studied in the course. Although examinations will
not emphasize computations, they will require some
computation. Exercises in the text book offer the best test
preparation.
Homework: Come prepared to turn in
homework assignments at the beginning of class. Given that
the homework comes due before the corresponding lecture (and the
fact that you can look up the answers if you get stuck), I will
grade more on completeness than accuracy. The assignments
primarily serve the purpose of allowing you to test your
understanding of the reading before the lecture and thus better
recognize where you have questions about the material.
Course
Project:
Week 4: Post your choice of test and 2 constructs to Blackboard.
Include all three pieces of information in the subject line. Do
not choose something that has already been taken. State your
choice with sufficient specificity that others can still choose
distinct but related options.
Week 13: Write a proposal for the validation of a test of your
construction following the format below. Double space the proposal
and use APA format and style. However, printing on both sides of
the page is fine.
A. Title page including your name and affiliation.
B. Abstract (180 words max).
C. Purpose of the test (250 words max). Describe the intended use
of the test. Describe the intended users and the intended test
taking population. Explain what the test would contribute over and
above existing tests. Describe the theoretical rationale behind
the test.
D. Test Blueprint (500 words max).
1. Define the constructs to be assessed by the test. Your
test should include at least two constructs and at least six items
per construct. Describe the relationship(s) between the
constructs, conceptually and statistically.
2. Specify the format of the items and response options.
3. Specify the content of the items. If a scale on your
test includes more than one kind of item, specify the number of
items of each type.
4. Specify the acceptable range of item statistics (mean or
proportion correct, standard deviation) for each item and test
statistics (mean, standard deviation, reliability) for each
subscore.
E. Draft test. Provide a draft version of the test including
instructions and a full set of items that conform to parts 1-3 of
the test blueprint.
F. Proposed validation plan (750 words max). Describe five
validation studies for your test (one paragraph each). Design one
study for each of the five main sources of test validity evidence
listed in the Standards (content, response processes, internal
structure, relationships with other variables, and consequences of
test use). Explain the rationale behind the intended
interpretation of the test and how each study tests an assumption
of that rationale (validity argument).
G. Factor model (250 words max). Download the SPSS code for
simulating item response data from an assumed factor model. Enter
plausible values for the item parameters (loadings, error
variances, intercepts, factor correlation[s]). If necessary, tweak
your values until your items satisfy part 4 of the test blueprint.
Report the final set of values in a table. In the text, describe
your general interpretation of the resulting factor model. Include
your interpretation of each factor, a description of which items
assess which factors, and a general description of the strengths
and weaknesses of the item set.
H. Pilot study (500 words max). Report this as if you had
completed an empirical pilot study, but use the simulated data
from part G. (Because your subscales measure different things, you
will want to analyze them separately. However, if a total score
has meaning for your test, it may also give you something
interesting to write about if you analyze the scale as a whole as
well.)
1. Conduct an item analysis of the data set. Report the
item statistics (means, standard deviations, item total
correlations and regression R-square values). Describe how
differences between item statistics relate to differences between
item parameters in the factor model.
2. Report the scale statistics (means, standard
deviations). Compare and contrast the scale statistics for each
scale.
3. Report both alpha and lambda 2 reliability estimates.
Describe the alphas-if-item-deleted.
4. Relate the results from parts H1 to H3 to the test
blueprint. Provide an overall evaluation of the functioning of the
draft test based on these results.
J. Appendices: Include the SPSS syntax used for your simulation as
Appendix A, and the SPSS output as Appendix B.
Convert your paper to portable document format (PDF), and turn in
both a hard copy and a PDF file. (If you cannot save directly to
PDF, download a free PDF print driver such as Cute PDF.) Note the due dates
on the course schedule. Proposals will be grading using the
following rubric.
Completeness (50% of grade, 13 points total)
A&B = 1 point
C = 1 point
D = 2 points. Each of four sections = .5 points.
E = 1 point.
F = 3 points.
G = 1 point.
H = 2 points. Each of four sections = .5 points.
J = 2 points.
Overall quality dimensions (50% of grade, 40 points total)
Clarity of presentation (1 - 10)
Technical accuracy of reporting (1 - 10)
Depth with which issues are presented within allowed space (1 -
10)
Overall conceptualization and design of proposed test and test
development (1 - 10)
1-5 = unsatisfactory.
6 = minimally satisfactory.
7 = some significant weaknesses.
8 = generally good with a few weak points.
9 = overall very well done.
10 = outstanding effort.
Grading: Each of the two examinations
is worth 25% of your total grade. The course
project is worth another 30%. That leaves 20% for the homework assignments. Letter grades will be
assigned as indicated below.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|