|
|
|
|
|
|
|
Site Map |
Time: Wednesday 6:30-8:30 PM
Room: TBA, The course maybe assigned a room but
will be taught online using Zoom. We have the option of
using the room if a need arises but I have no current plans for
using the room.
CRN: 52065
Office Hours: Please contact me by
email. (I have listed my offices below but do not expect
to make much use of them during the fall term.)
Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784
Office: Room 10.63.11, 524 W59
Street. By appointment: GSUC Room
3204.02.
Address: Psychology Department, John Jay College of
Criminal Justice, 524 W 59th Street, New York, NY 10019 USA
Please do not leave anything for me in campus mailboxes.
Please email digital documents.
Course Description: This course will examine
approaches to evaluation and methods to evaluate the
effectiveness of programs and projects providing educational
services. Topics will include how to plan an evaluation, methods
of collecting data, design and testing issues, data analysis,
and the politics and use of evaluations. Techniques will be
drawn from Anthropology, Economics, Psychology, Sociology and
Statistics. (Note: Evaluation has matured since this course
description was written and the influences of the social and
behavioral science disciplines listed are now primarily
indirect, filtered through a substantial literature specific to
evaluation. Evaluation as a transdiscipline will be considered.
Both effectiveness and efficacy will be considered. Material
likely to overlap introductory research methods courses will not
be emphasized.)
Course Objectives:
1. Expose students to the basic theory and methods of program
evaluation.
2. Expose students to professional standards for program
evaluation.
3. Provide practice applying theory, methods, and standards to
practical evaluation problems.
4. Provide practice with various forms of writing important to
program evaluation.
5. Provide a strong foundation for further study of program
evaluation either through additional course work or through self
study.
This course assumes that students are already familiar with
basics of research design as might be covered in an introductory
research design course.
This course is equivalent to PSYCH U80103, Program
Evaluation.
Examinations: There are no
examinations in this course. A series of assignments (four memos
and a short paper) take the place of take-home examinations. It
is important that you keep up with the reading in order to make
this examination-free approach work.
Memo 1: Stakeholders and
their concerns.
Review the web page for the CUNY Preparatory Transitional
High School Program (CUNY Prep, http://cunyprep.org/sites/distance-learning/)
and the 2008 evaluation report (https://www1.nyc.gov/assets/opportunity/pdf/cprep_prr.pdf).
Consider how the CUNY Prep program impacts four distinct
stakeholder groups. In the first section, briefly discuss each
of the four stakeholder groups and what you envision as their
primary stake in the activities of the program.
In a second section, propose a cost-effective methodology
($500 or less) for (a) identifying any additional primary
concerns among these stakeholder groups, (b) identifying
additional stakeholder groups beyond the four you considered,
and (c) identifying the primary concerns of any additional
stakeholder groups.
In the third and final section, discuss the stake that the
general public has in the activities of the program and compare
and contrast it with the specific concerns of the four
stakeholder groups identified above.
Memo 2: Theory of Change.
Choose a possible unintended outcome of the CUNY Prep
program. For the purposes of this memo, consider any
outcome not discussed in the 2008 report an unintended
outcome. Write a memo outlining a proposed theory of
change for the unintended outcome. Construct an explanation of
the unintended outcome that traces the outcome back to program
activities. Do this at a level most useful for the
planning of an evaluation. Provide a causal path diagram
depicting your theory of change. Select three elements of
your theory of change as most important for the evaluation, and
propose evaluation questions that correspond to them.
Propose one evaluation question for each of the three
aspects. Formulate your questions to be clear and precise,
amenable to empirical evaluation, and stated at an appropriate
level of abstraction to make them useful. Relate your
questions to your causal model and theory of change.
Clearly distinguish program activities (possibly represented in
the model) from evaluation activities (never represented in the
model). Note that "how much" or "to what extent" questions
typically prove more useful than "whether or not" or "yes or no"
questions (unless the later involves comparisons between
amounts).
Pick an
unassigned entry in the Evaluation Thesaurus. Follow up that
entry with a related entry (listed at the end), and continue
this process until you have read five entries. If you are not
satisfied with the first five entries, continue your travels
through the thesaurus until you strike upon a series of five
entries about which you want to write. Write 500-1000 words
based on your five-entry trek through the thesaurus. Do not
summarize the entries any more than necessary to set up your
observations or conclusions about them. Instead, describe
how the entries interrelate with one another and how the
implications of the five entries for the practice of program
evaluation are connected. You might also discuss how the
material sheds light on things you have read in other courses,
or your understanding of material from this course. Critical
evaluation of the material offers another option. Whatever you
include, aim to demonstrate that you have thought about the
material. Keep mere summary of the material from the
entries to a minimum (at most, 25% of your travelogue).
Memo 3: Evaluation
Milestones.
Write a memo proposing three performance measures for each
of two program outcomes. Both outcomes should be impacts
of program activities on program participants. The first
outcome should involve students perceptions or attitudes.
The second outcome should come from the "Outputs" box from one
of the two logic models included in the report. For each
of the two outcomes, (a) propose three outcome measures, (b)
propose specific program milestone (a.k.a., yardstick) for each
measure, and (c) briefly describe the method of data collection
required by each measure. (Check to confirm that you have six
separate outcome measures and provide three distinct pieces of
information about each of these six outcomes.) Note: Take
care not to confuse measures with milestones. Each
milestone depends upon a measure, but the same measure can
support many different milestones. Do not stop at
specifying a measure, but be sure to also specify the
accompanying milestone. (Be sure to read the Evaluation
Milestones document before attempting this assignment.)
Write a memo devoting one paragraph to each of the five
main sets of program evaluation standards. Discuss the 2008 CUNY
Prep Program Review report cited above from the perspective of
each of the five sets of standards. Highlight strengths
and weaknesses of the report from the perspective of each set of
standards.
Evaluation Proposal:
Respond to the following request for proposals.
Request For Proposals
Educational Game Evaluation
Fall 2021
IV. Evaluation Criteria
Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design: A summary
judgment of the degree to which the proposal fulfills initiative
objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision
making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.
[End of RFP]
Evaluation Project Report:
Complete the project described in your proposal, incorporating
any feedback from the proposal. Write up your report using
the following format.
A. Title page.
List proposal title, author's name, and affiliation on
the title page. Begin page numbering at 1 on the title
page.
B. Summary.
1. Summary of research report. (250 words)
Be sure to summarize all sections of the report including
results and recommendations. As with the proposal, the
summary should provide a condensed statement of the content of
the report. It should not simply list topics covered in
the report or describe the report.
2. Summary of changes. (125 words)
Summarize any changes in the evaluation design, questions or
method from the proposal. If there were no changes, simply
state that.
C. Introduction (500 words)
Present the context of the evaluation. Describe the
purpose of the evaluation and close with the specific evaluation
questions.
D. Method (750 words)
Summarize the methodology used in the evaluation. Coding
instructions, coding sheets, rating scales, and other materials
can be included as an appendix and do not count toward the 750
words. The primary focus of this section is on what data
you collected, but it may also describe other aspects of method
such as evaluation criteria or evaluation standards.
E. Results (1500 words)
Present the results for each evaluation question. Tables
and figures do not count as part of the word count, but should
be included directly inside the text (not at the end of the
document).
F. Recommendations (1000 words)
State each recommendation in a sentence followed by text that
elaborates and clarifies the recommendations and text that
provides a rationale for the recommendation. You may find
it helpful to identify each of these two subtopics with a
separate subheading beneath each recommendation.
G. Reference list
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|