|
|
|
|
|
|
|
Site Map |
Time: Wednesday 6:30-8:30 PM
Room: GSUC 5417
Office Hours: Tuesdays and Thursdays
4:15-5:15pm
Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784
Office: Room 10.63.11, 524 W59
Street. By appointment: GSUC Room
3204.02.
Address: Psychology Department, John Jay College of
Criminal Justice, 524 W 59th Street, New York, NY 10019 USA
Course Description: This course will examine
approaches to evaluation and methods to evaluate the
effectiveness of programs and projects providing educational
services. Topics will include how to plan an evaluation, methods
of collecting data, design and testing issues, data analysis,
and the politics and use of evaluations. Techniques will be
drawn from Anthropology, Economics, Psychology, Sociology and
Statistics. (Note: Evaluation has matured a bit since this
course description was written and the influences of the social
and behavioral science disciplines listed are now primarily
indirect, filtered through a substantial literature specific to
evaluation. Evaluation as a transdiscipline will be considered.
Both effectiveness and efficacy will be considered. Material
likely to overlap introductory research methods courses will not
be emphasized.)
Course Objectives:
1. Expose students to the basic theory and methods of program
evaluation.
2. Expose students to professional standards for program
evaluation.
3. Provided practice applying theory, methods, and standards to
practical evaluation problems.
4. Provide practice with various forms of writing important to
program evaluation.
5. Provide a strong foundation for further study of program
evaluation either through additional course work or through self
study.
This course is equivalent to PSYCH U80103, Program
Evaluation.
Course Flow: Familiarize yourself with the
reading material before the corresponding class. Classes
will summarize and clarify the reading. In general, I
would rather answer your questions than lecture. Come to class
prepared for active discussion of the reading material.
Examinations: There are no
examinations in this course. A series of assignments (four memos
and a short paper) take the place of take-home examinations. It
is important that you keep up with the reading in order to make
this examination-free approach work.
Memo 1: Stakeholders and
their concerns.
Review the web page for the Mina Rees library (http://library.gc.cuny.edu/)
and the 2005 CUNY LibQual survey report (http://libraries.cuny.edu/LibQualFinalReport.pdf),
focusing on the material most directly related to the Mina Rees
library.
Consider
how
the
library impacts (a) Graduate Center students, (b) Graduate
Center faculty, (c) Library users from other CUNY campuses, and
(d) Mina Rees librarians and other staff members. In the first
section, briefly discuss each of these stakeholder groups and
what you envision as their primary concerns with the activities
of the library.
In a second section, propose a cost-effective methodology
($500 or less) for (a) identifying any additional primary
concerns among these stakeholder groups, (b) identifying
additional stakeholder groups not listed above, and (c)
identifying the primary concerns of any additional stakeholder
groups.
In the third and final section, discuss the stake that the
general public has in the activities of the Mina Rees library
and compare and contrast it with the specific concerns of the
four stakeholder groups identified above.
Memo 2: Evaluation
Yardsticks.
Write a memo proposing three performance measures for each
of two library outcomes. Both outcomes should be impacts
of library activities on library users. The first outcome
should involve students and the second should involve
faculty. For each of the two outcomes, (a) propose three
outcome measures, (b) propose specific program yardstick for
each measure, and (c) briefly describe the method of data
collection required by each measure. (Note: You should have six
separate outcome measures and provide three distinct pieces of
information about each of these six outcomes.)
Evaluation Thesaurus
Travelogue.
Pick an unassigned entry in the Evaluation Thesaurus.
Follow up that entry with a related entry, and continue this
process until you have read five entries. If you are not
satisfied with the first five entries, continue your travels
through the thesaurus until you strike upon a series of five
entries about which you want to write. Write 500-1000 words
based on your five-entry trek through the thesaurus. Do not
summarize the entries any more than necessary to set up your
observations or conclusions about them. Instead, describe
how the entries interrelate with one another and how the
implications of the five entries for the practice of program
evaluation are connected. You might also discuss how the
material sheds light on things you have read in other courses,
or your understanding of material from this course. Critical
evaluation of the material offers another option. Whatever you
include, aim to demonstrate that you have thought about the
material. Keep mere summary of the material to a minimum.
Memo 3: Program Model.
Write a memo outlining a proposed program model for a specific
activity of the Mina Rees library. Describe inputs, program
activities, outputs, and outcomes at a level most useful for the
planning of an evaluation. Provide a diagram of your program
model. Clearly distinguish direct outputs of library
activities from the outcomes that they facilitate. Select
three elements of your program model as most important for the
evaluation, and propose evaluation questions that correspond to
them.
Memo 4: Evaluation
Standards.
Write a memo devoting one paragraph to each of the five main
sets of program evaluation standards. Discuss the 2005 CUNY
LibQual report cited above from the perspective of each of the
five sets of standards. Highlight strengths and weaknesses
of the report from the perspective of each set of standards.
Evaluation Proposal:
Respond to the following request for proposals.
Request For Proposals
Program Evaluation Research Text Book Evaluation
Fall 2013
IV. Evaluation Criteria
Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design: A summary
judgment of the degree to which the proposal fulfills initiative
objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision
making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.
[End of RFP]
Evaluation Project Report:
Complete the project described in your proposal, incorporating
any feedback from the proposal. Write up your report using
the following format.
A. Title page.
List proposal title, author's name, and affiliation on
the title page. Begin page numbering at 1 on the title
page.
B. Summary.
1. Summary of research report. (250 words)
Be sure to summarize all sections of the report including
results and recommendations. Like an abstract, the summary
should provide a condensed statement of the content of the
report. It should not simply list topics covered in the
report or describe the report.
2. Summary of changes. (125 words)
Summarize any changes in the evaluation design, questions or
method from the proposal. If there were no changes, simply
state that.
C. Introduction (500 words)
Present the context of the evaluation. Describe the
purpose of the evaluation and close with the specific evaluation
questions.
D. Method (750 words)
Summarize the methodology used in the evaluation.
Coding instructions, coding sheets, rating scales, and other
materials can be included as an appendix and do not count toward
the 750 words. The primary focus of this section is on
what data you collected, but it may also describe other aspects
of method such as evaluation criteria or evaluation standards.
E. Results (1500 words)
Present summary data comparing each of the four texts to
your performance standards and to one another based on any
evaluation measures used in your evaluation study. Also
present the results of data analyses for each evaluation
question. Tables and figures do not count as part of the
word count, but should be included directly inside the text (not
at the end of the document).
F. Recommendations (1000 words)
State each recommendation in a sentence followed by text that
elaborates and clarifies the recommendations and text that
provides a rationale for the recommendation.
G. Reference list
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
W 8/28 |
Course Overview, What is program evaluation
research and how does it differ from behavioral science
research? |
|
W 9/4 |
Class
does not meet. No classes scheduled. |
|
W 9/11 |
The purposes of program evaluation. Weiss (W) Chapters 1-2. |
|
W 9/18 |
Evaluation as a transdiscipline Scriven (S) Introduction: The Nature of Evaluation, "Logic of Evaluation". |
|
W 9/25 |
Understanding the program and planing the
evaluation. W3-4, S "Process Evaluation," AEA Guiding Principles. |
Memo 1: Stakeholders & Concerns |
W 10/2 |
Roles of the evaluator, program measures. W5-6 |
|
W 10/9 |
Collecting
data, Designing the evaluation. (Go over evaluation
review assignment) W7-8, Chapter Summaries for W9-10. (Note 1) |
Memo 2: Evaluation Milestones |
Week
7: W 10/16 |
Class does not meet
due to American Evaluation Association conference. Class will be online (asynchronous) using Blackboard. Qualitative methods, interpreting data. W11-12 |
|
W 10/23 |
Dissemination
& Evaluation integrity. W13-14, S "Ethics" & "Ethics in Evaluation." |
Evaluation Proposal (due at start of class) |
W 10/30 |
Standards and Best
Practices for Evaluation Discuss Evaluation Review. Standards Introduction & Applying the Standards (YSHC). Morris, Chapter 1 (M1), S "Key Evaluation Checklist" |
|
W 11/6 |
Propriety Standards YSHC:Propriety, M3, S "Conflict of Interest" |
Thesaurus Travelogue |
W 11/13 |
Utility Standards YSHC:Utility, M6-7 |
|
W 11/20 |
Feasibility Standards YSHC: Feasibility & Evaluation Accountability, M2 |
Memo 3: Program Model |
W
11/27 |
Class does not
meet. Classes follow a Friday schedule |
|
W 12/4 |
Accuracy Standards YSHC: Accuracy, M4-5, 8 |
Memo 4: Standards |
W 12/11 |
Empowerment Evaluation Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007 |
|
W 12/18 |
Come to class
prepared to discuss your report.
|
Evaluation Report |