Time:  Wednesday 6:30-8:30 PM
Room:  3306, Graduate Center, 365 Fifth Avenue:
Section:  01
CRN:  48879
Cross-listing:  PSYC 80100-07 (53067)

Office Hours:  Please contact me by email.  I can answer most questions by email.  However, we can also schedule a Zoom meeting or office meeting by appointment.  I will only be at the Graduate Center on days that class meets.  It is best to make an appointment to avoid two students dropping in at the same time.

Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com (Best way to reach me.)
212-237-8784 (Please do not leave voicemail, I no longer receive it as email.)
Office: Room 10.63.11, 524 W59 Street.  By appointment: GSUC Room 3204.02.
Address: Psychology Department, John Jay College of Criminal Justice, 524 W 59th Street, New York, NY 10019 USA
If you leave something in my mailbox, please alert me by email.

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provide practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course assumes that students are already familiar with basics of research design as might be covered in an introductory research design course.

This course is equivalent to PSYCH U80103, Program Evaluation.

Text Books:

This used to be assigned reading but I have dropped it due to the current cost of the book.  If you have library access or a used copy, I still recommend taking a look at it. It is full of practical and thoughtful advice on program evaluation.

Examinations:  There are no examinations in this course. A series of assignments (two posters, four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Memo 1: Stakeholders and their concerns.

Review the web page for the CUNY Preparatory Transitional High School Program (CUNY Prep, http://cunyprep.org/sites/distance-learning/) and the 2008 evaluation report (https://www1.nyc.gov/assets/opportunity/pdf/cprep_prr.pdf). Consider how the CUNY Prep program impacts four distinct stakeholder groups. In the first paragraph, briefly enumerate your four stakeholder groups and what you envision as their primary stake in the activities of the program.

In a second paragraph, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups beyond the four you considered, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final paragraph, discuss the stake that the general public has in the activities of the program and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Choose a possible unintended outcome of the CUNY Prep program.  For the purposes of this memo, consider any outcome not discussed in the 2008 report an unintended outcome.  Write a memo outlining a proposed theory of change for the unintended outcome. Construct an explanation of the unintended outcome that traces the outcome back to program activities.  Do this at a level most useful for the planning of an evaluation. Provide a causal path diagram depicting your theory of change (see Quick Start Guide to Path Diagrams.pdf).  Select three causal effects from your theory of change as most important for the evaluation, and propose evaluation questions that correspond to them.  Propose one evaluation question for each of the three causal effects.  Formulate your questions to be clear and precise, amenable to empirical evaluation, and stated at an appropriate level of abstraction to make them useful.  Relate your questions to your causal model and theory of change.  Clearly distinguish program activities (possibly represented in the model) from evaluation activities (never represented in the model).  Note that "how much" or "to what extent" questions typically prove more useful than "whether or not" or "yes or no" questions (unless the later involves comparisons between amounts).

Write a memo devoting one paragraph to each of the five main sets of program evaluation standards. Discuss the 2008 CUNY Prep Program Review report cited above from the perspective of each of the five sets of standards.  Highlight strengths and weaknesses of the report from the perspective of each set of standards (as best possible with the information available).

Evaluation Proposal: Respond to the following request for proposals.  Create a self-contained, self-explanatory deck of presentation slides with one slide for each topic listed in the request for proposals below.  Use the slide titles in the request for proposals on your slides (Slides 1-11).  Do not combine sections or break them into multiple slides.

Bear in mind that while the request for proposals is embedded in a fictionalized context (New Argleton Schools) your evaluation is very much real.  You will need to be able to complete the evaluation with available resources (e.g., no funding) within the time frame of the course (see syllabus schedule for due dates).  So, design your evaluation accordingly.  Be realistic about what you can accomplish in a semester project.  Remember that this is an evaluation study not a research study:  You are evaluating the game, not hypotheses about it.  When a game does not meet objectives, it is the game that has to change, not the objectives.

You can use whatever software you like to create your slides but, once finished, please save them in a pdf file (Portable Document Format) and turn in the pdf file only.  Use a font size of at least 24 for all text including the title.  (This font size should allow a maximum of about 14 lines per slide but try to be as concise as possible.)  Feel free to divide the slide horizontally if and when useful.

Request For Proposals
Educational Game Evaluation
Fall 2025

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.

[End of RFP]

Evaluation Project Report Poster:
Complete the project described in your proposal, incorporating any feedback from the proposal.  Create your evaluation project report poster using the following format.  Include each heading in your poster (B through G) and do not combine headings.

Format your poster to be 36 inches high (3 feet) and 48 inches wide (4 feet).  Use Times New Roman font.  Follow the font size guidelines from the UCLA library reference document at this URL: https://guides.library.ucla.edu/c.php?g=223540&p=1480858.  The document offers other helpful guidelines as well.  Many people find presentation software useful for creating posters.  Save and submit your poster as a file in pdf format.

A. Header
List proposal title, author's name, and affiliation across the top of the poster.

B. Executive Summary (maximum 200 words).

1. Summary of research report.
Be sure to summarize all sections of the report including results and recommendations.  As with the proposal, the summary should provide a condensed statement of the content of the report.  It should not simply list topics covered in the report or describe the report.

2. Summary of changes.
Summarize any changes in the evaluation design, questions or method from the proposal.  If there were no changes, simply state that.

C. Introduction
Present the context of the evaluation.  Describe the purpose of the evaluation and close with the specific evaluation questions.  Phrase the questions to include evaluation milestones.  (This section may be briefer than the corresponding section of the proposal.)

D. Method
Summarize the methodology used in the evaluation.  The primary focus of this section is on what data you collected, but it may also describe other aspects of method such as evaluation criteria or evaluation approach.

E. Results
Present the results for each evaluation question.  Consider tables and figures to present material more concisely.  (The APA Publication Manual offers helpful guidance regarding when to use text, tables or figures.)

F. Recommendations
State each recommendation in a sentence followed by text that elaborates and clarifies the recommendations and text that provides a rationale for the recommendation.  You may find it helpful to identify each of these two subtopics with a separate subheading beneath each recommendation (headings do not need to follow APA format or be on a separate line).  Include at least four recommendations based on your findings.

G. Reference list
Do not let this take up too much space.  Focus on key references.