Time:  Wednesday 6:30-8:30 PM
Room:  GSUC 5417

Office Hours:  Wednesday 5:15-6:15pm

Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784
Office: Room 10.63.11, 524 W59 Street.  By appointment: GSUC Room 3204.02.
Address: Psychology Department, John Jay College of Criminal Justice, 524 W 59th Street, New York, NY 10019 USA

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured a bit since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provide practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course is equivalent to PSYCH U80103, Program Evaluation.

Examinations:  There are no examinations in this course. A series of assignments (four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Memo 1: Stakeholders and their concerns.

Review the web page for the CUNY Preparatory Transitional High School Program (CUNY Prep, http://www.cunyprep.org/cp.html) and the 2008 evaluation report (http://www.nyc.gov/html/ceo/downloads/pdf/cprep_prr.pdf). Consider how the CUNY Prep program impacts four distinct stakeholder groups. In the first section, briefly discuss each of the four stakeholder groups and what you envision as their primary stake in the activities of the program.

In a second section, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups beyond the four you considered, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final section, discuss the stake that the general public has in the activities of the program and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Memo 2: Evaluation Yardsticks.

Write a memo proposing three performance measures for each of two program outcomes.  Both outcomes should be impacts of program activities on program participants.  The first outcome should involve students perceptions or attitudes.  The second outcome should come from the "Outputs" box from one of the two logic models included in the report.  For each of the two outcomes, (a) propose three outcome measures, (b) propose specific program yardstick for each measure, and (c) briefly describe the method of data collection required by each measure. (Note: You should have six separate outcome measures and provide three distinct pieces of information about each of these six outcomes.)  Note: Take care not to confuse measures with yardsticks.  Each yardstick depends upon a measure, but the same measure can support many different yardsticks.  Do not stop at specifying a measure, but be sure to also specify the accompanying yardstick.

Evaluation Thesaurus Travelogue.

Pick an unassigned entry in the Evaluation Thesaurus. Follow up that entry with a related entry, and continue this process until you have read five entries. If you are not satisfied with the first five entries, continue your travels through the thesaurus until you strike upon a series of five entries about which you want to write. Write 500-1000 words based on your five-entry trek through the thesaurus. Do not summarize the entries any more than necessary to set up your observations or conclusions about them.  Instead, describe how the entries interrelate with one another and how the implications of the five entries for the practice of program evaluation are connected. You might also discuss how the material sheds light on things you have read in other courses, or your understanding of material from this course. Critical evaluation of the material offers another option. Whatever you include, aim to demonstrate that you have thought about the material.  Keep mere summary of the material to a minimum.

Memo 3: Theory of Change.
Choose a possible unintended outcome of the CUNY Prep program.  For the purposes of this memo, consider any outcome not discussed in the 2008 report an unintended outcome.  Write a memo outlining a proposed theory of change for the unintended outcome. Construct an explanation of the unintended outcome that traces the outcome back to program activities.  Do this at a level most useful for the planning of an evaluation. Provide a causal path diagram depicting your theory of change.  Select three elements of your theory of change as most important for the evaluation, and propose evaluation questions that correspond to them.  Propose one evaluation question for each of the three aspects.  Formulate your questions to be clear and precise, amenable to empirical evaluation, and stated at an appropriate level of abstraction to make them useful.  Relate your questions to your causal model and theory of change.  Clearly distinguish program activities (possibly represented in the model) from evaluation activities (never represented in the model).  Note that "how much" or "to what extent" questions typically prove more useful than "whether or not" or "yes or no" questions (unless the later involves comparisons between amounts).

Memo 4: Evaluation Standards.
Write a memo devoting one paragraph to each of the five main sets of program evaluation standards. Discuss the 2008 CUNY Prep Program Review report cited above from the perspective of each of the five sets of standards.  Highlight strengths and weaknesses of the report from the perspective of each set of standards.

Evaluation Proposal: Respond to the following request for proposals.

Request For Proposals
Program Evaluation Research Text Book Evaluation
Fall 2013

IV. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.

[End of RFP]

Evaluation Project Report:
Complete the project described in your proposal, incorporating any feedback from the proposal.  Write up your report using the following format.

A. Title page.
List proposal title, author's name, and affiliation on the title page.  Begin page numbering at 1 on the title page.

B. Summary.

1. Summary of research report. (250 words)
Be sure to summarize all sections of the report including results and recommendations.  Like an abstract, the summary should provide a condensed statement of the content of the report.  It should not simply list topics covered in the report or describe the report.

2. Summary of changes.  (125 words)
Summarize any changes in the evaluation design, questions or method from the proposal.  If there were no changes, simply state that.

C. Introduction (500 words)

Present the context of the evaluation.  Describe the purpose of the evaluation and close with the specific evaluation questions.

D. Method (750 words)

Summarize the methodology used in the evaluation.  Coding instructions, coding sheets, rating scales, and other materials can be included as an appendix and do not count toward the 750 words.  The primary focus of this section is on what data you collected, but it may also describe other aspects of method such as evaluation criteria or evaluation standards.

E. Results (1500 words)

Present the results for each evaluation question.  Tables and figures do not count as part of the word count, but should be included directly inside the text (not at the end of the document).

F. Recommendations (1000 words)
State each recommendation in a sentence followed by text that elaborates and clarifies the recommendations and text that provides a rationale for the recommendation.  You may find it helpful to identify each of these two subtopics with a separate subheading beneath each recommendation.

G. Reference list