Keith Markus' Urban Sprawl

  EDPSY 73100 GC
  Program Evaluation Research
Course Information
Blackboard Page
American Evaluation Association
Site Map

Fall 2021

Time:  Wednesday 6:30-8:30 PM
Room:  TBA, The course maybe assigned a room but will be taught online using Zoom.  We have the option of using the room if a need arises but I have no current plans for using the room.
CRN: 52065

Office Hours:  Please contact me by email.  (I have listed my offices below but do not expect to make much use of them during the fall term.)

Contact Information:
Dr. Keith A. Markus
Office: Room 10.63.11, 524 W59 Street.  By appointment: GSUC Room 3204.02.
Address: Psychology Department, John Jay College of Criminal Justice, 524 W 59th Street, New York, NY 10019 USA
Please do not leave anything for me in campus mailboxes.  Please email digital documents.

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provide practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course assumes that students are already familiar with basics of research design as might be covered in an introductory research design course.

This course is equivalent to PSYCH U80103, Program Evaluation.

Text Books:
    Chen, H. T. (2015)  Practical program evaluation:  Theory-driven evaluation and the integrated evaluation perspective (2nd ed.).  Thousand Oaks, CA:  Sage.

    Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications.

    Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications.

    Morris, M. (2008). Evaluation ethics for best practice: Cases and commentaries. New York: Guilford.

Additional Reading:

   American Evaluation Association (2004). Guiding principles for evaluators.

   Fetterman, D. & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28, 179-198.

   Miller, R. L. & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319.

   Smith, N. L. (2007).  Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28, 169-178.

Course Flow:  Familiarize yourself with the reading material before the corresponding class.  Classes will summarize and clarify the reading.  In general, I would rather answer your questions than lecture. Come to class prepared for active discussion of the reading material.

Examinations:  There are no examinations in this course. A series of assignments (four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Evaluation Memos: Each memo should contain at least 500 words but not exceed 1000 words. Strive for clear and concise writing for an audience that may or may not have training in evaluation. Length is an evaluation criterion for the memos, so keep them within the prescribed range. Conciseness is an accomplishment that takes time and effort. Give yourself time to draft, review and revise each memo.  Remember: "All writing is re-writing."

Memo 1: Stakeholders and their concerns.

Review the web page for the CUNY Preparatory Transitional High School Program (CUNY Prep, and the 2008 evaluation report ( Consider how the CUNY Prep program impacts four distinct stakeholder groups. In the first section, briefly discuss each of the four stakeholder groups and what you envision as their primary stake in the activities of the program.

In a second section, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups beyond the four you considered, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final section, discuss the stake that the general public has in the activities of the program and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Memo 2: Theory of Change.
Choose a possible unintended outcome of the CUNY Prep program.  For the purposes of this memo, consider any outcome not discussed in the 2008 report an unintended outcome.  Write a memo outlining a proposed theory of change for the unintended outcome. Construct an explanation of the unintended outcome that traces the outcome back to program activities.  Do this at a level most useful for the planning of an evaluation. Provide a causal path diagram depicting your theory of change.  Select three elements of your theory of change as most important for the evaluation, and propose evaluation questions that correspond to them.  Propose one evaluation question for each of the three aspects.  Formulate your questions to be clear and precise, amenable to empirical evaluation, and stated at an appropriate level of abstraction to make them useful.  Relate your questions to your causal model and theory of change.  Clearly distinguish program activities (possibly represented in the model) from evaluation activities (never represented in the model).  Note that "how much" or "to what extent" questions typically prove more useful than "whether or not" or "yes or no" questions (unless the later involves comparisons between amounts).

Evaluation Thesaurus Travelogue.

Pick an unassigned entry in the Evaluation Thesaurus. Follow up that entry with a related entry (listed at the end), and continue this process until you have read five entries. If you are not satisfied with the first five entries, continue your travels through the thesaurus until you strike upon a series of five entries about which you want to write. Write 500-1000 words based on your five-entry trek through the thesaurus. Do not summarize the entries any more than necessary to set up your observations or conclusions about them.  Instead, describe how the entries interrelate with one another and how the implications of the five entries for the practice of program evaluation are connected. You might also discuss how the material sheds light on things you have read in other courses, or your understanding of material from this course. Critical evaluation of the material offers another option. Whatever you include, aim to demonstrate that you have thought about the material.  Keep mere summary of the material from the entries to a minimum (at most, 25% of your travelogue).

Memo 3: Evaluation Milestones.

Write a memo proposing three performance measures for each of two program outcomes.  Both outcomes should be impacts of program activities on program participants.  The first outcome should involve students perceptions or attitudes.  The second outcome should come from the "Outputs" box from one of the two logic models included in the report.  For each of the two outcomes, (a) propose three outcome measures, (b) propose specific program milestone (a.k.a., yardstick) for each measure, and (c) briefly describe the method of data collection required by each measure. (Check to confirm that you have six separate outcome measures and provide three distinct pieces of information about each of these six outcomes.)  Note: Take care not to confuse measures with milestones.  Each milestone depends upon a measure, but the same measure can support many different milestones.  Do not stop at specifying a measure, but be sure to also specify the accompanying milestone.  (Be sure to read the Evaluation Milestones document before attempting this assignment.)

Memo 4: Evaluation Standards.

Write a memo devoting one paragraph to each of the five main sets of program evaluation standards. Discuss the 2008 CUNY Prep Program Review report cited above from the perspective of each of the five sets of standards.  Highlight strengths and weaknesses of the report from the perspective of each set of standards.

Mini-papers:  The course requires two mini-papers.  The first is an evaluation proposal and the second is an evaluation report.  The second paper builds on the first.

Evaluation Proposal: Respond to the following request for proposals.

Request For Proposals
Educational Game Evaluation
Fall 2021

I. Introduction
The web contains a wide variety of free educational games.  The New Argleton, New York, school district is considering installing a selection of such games in computer labs in its elementary schools serving grades K-6.  The goal is to select games that will effectively reinforce (but not replace) instruction aligned with the Common Core curriculum standards.  The district is seeking evaluations of educational games to help guide their choices of which games to install.

II. Eligibility
Applicants must be currently enrolled in EDPSY U73100 during the Fall of 2021.

III. Proposal Requirements
Proposals must contain each of the following elements and must not exceed indicated page lengths per section.  Proposals can be single spaced, and there is no need for page breaks between sections except after the title page.  All word counts refer to maximum word counts.  Please use Times New Roman size 12 or equivalent.

A. Title Page
List proposal title, author's name, and affiliation on the title page.  Begin page numbering at 1 on the title page.

B. Summary (200 words)
Summarize the proposal, including all elements from C to H below.  Convey the content of the proposal in condensed form.  Do not list topics covered without summarizing the content under the topics.  The summary should be a capsule statement of the proposal, not a description of it from an external perspective.

C. Evaluation Topic and Program Theory (500 words maximum)
Identify the game to be evaluated including its URL.  List at least three learning objectives for the game.  Present a program theory including inputs, program activities, outputs and direct outcomes.  You may optionally include more distal outcomes.  Describe the various aspects of the program theory.  If your proposal is based on a particular educational theory or theory of learning, describe that here.  If your proposal draws upon existing literature, discuss that here.  (For the purposes of this RFP, the term 'program' refers to the creation of computer labs with educational games at the elementary schools.)

D. Logic Model (1 page figure)
Include a logic model that illustrates your program theory.  You may refer to this in Part C.

E. Evaluation Questions (250 words)
State your evaluation questions.  Include at least four questions related to the game described in Part C.  For each question, provide a brief rationale for why you chose this question and how the answer will contribute to program implementation.  (Choose feasible questions in light of the time frame and available resources.)

F. Evaluation Methods (500 words)
For each evaluation question listed in Part E, describe the methodology that will be used to answer the question.  Provide as clear and precise as description as possible of the methods that will be used to evaluate the game for each evaluation question, including any coding or rating methods used in the research.  Avoid overlap between your description of your methodology in this part and your description of measures in Part G.

G. Evaluation Measures and Performance Objectives (500 words)
Describe evaluation measures used in the evaluation.  Include verbatim rating scales or content coding instructions.  If necessary, summarize coding instructions here and include the full instructions as an appendix.  If necessary, rating scales can also be included as an appendix.

For each evaluation measure, provide a performance objective (i.e., milestone) that represents what you would consider minimally acceptable performance for a game on that dimension.

H. Guiding principles (500 words)
Relate your proposed evaluation to the Guiding Principles and explain how your evaluation responds to these principles.

I. Time budget (1 page table)
Your resources for this evaluation project are hours of work rather than dollars.  Provide a budget allocating work hours to various evaluation activities.  Include a total and any appropriate category subtotals.

J. Budget justification (250 words)
Briefly justify your budget allocation.

K. Time line (1 page table)
Provide a weekly time line for completion of the project.

L. Reference list
Provide references for any literature cited in the proposal.  Use APA format (7th edition).

M. Appendices
See Part G instructions for possible appendices.

IV. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.

[End of RFP]

Evaluation Project Report:
Complete the project described in your proposal, incorporating any feedback from the proposal.  Write up your report using the following format.

A. Title page.
List proposal title, author's name, and affiliation on the title page.  Begin page numbering at 1 on the title page.

B. Summary.

1. Summary of research report. (250 words)
Be sure to summarize all sections of the report including results and recommendations.  As with the proposal, the summary should provide a condensed statement of the content of the report.  It should not simply list topics covered in the report or describe the report.

2. Summary of changes.  (125 words)
Summarize any changes in the evaluation design, questions or method from the proposal.  If there were no changes, simply state that.

C. Introduction (500 words)
Present the context of the evaluation.  Describe the purpose of the evaluation and close with the specific evaluation questions.

D. Method (750 words)
Summarize the methodology used in the evaluation.  Coding instructions, coding sheets, rating scales, and other materials can be included as an appendix and do not count toward the 750 words.  The primary focus of this section is on what data you collected, but it may also describe other aspects of method such as evaluation criteria or evaluation standards.

E. Results (1500 words)
Present the results for each evaluation question.  Tables and figures do not count as part of the word count, but should be included directly inside the text (not at the end of the document).

F. Recommendations (1000 words)
State each recommendation in a sentence followed by text that elaborates and clarifies the recommendations and text that provides a rationale for the recommendation.  You may find it helpful to identify each of these two subtopics with a separate subheading beneath each recommendation.

G. Reference list

H. Appendices (optional)
See Part D above.

Evaluation criteria:
1. Overall clarity.
2. Overall completeness and informativeness.
3. Results are presented in a useful manner.
4. Recommendations are useful and well justified.

Grading:  Mini papers count 50% (25% each) and memos and travelogue count 50% (10% each).


Letter Grade
Percent Grade

Academic Honesty:    
The Graduate Center of The City University of New York is committed to the highest standards of academic honesty. Acts of academic dishonesty include—but are not limited to—plagiarism, (in drafts, outlines, and examinations, as well as final papers), cheating, bribery, academic fraud, sabotage of research materials, the sale of academic papers, and the falsification of records. An individual who engages in these or related activities or who knowingly aids another who engages in them is acting in an academically dishonest manner and will be subject to disciplinary action in accordance with the bylaws and procedures of The Graduate Center and the Board of Trustees of The City University of New York.  

Each member of the academic community is expected to give full, fair, and formal credit to any and all sources that have contributed to the formulation of ideas, methods, interpretations, and findings. The absence of such formal credit is an affirmation representing that the work is fully the writer’s. The term “sources” includes, but is not limited to, published or unpublished materials, lectures and lecture notes, computer programs, mathematical and other symbolic formulations, course papers, examinations, theses, dissertations, and comments offered in class or informal discussions, and includes electronic media. The representation that such work of another person is the writer’s own is plagiarism.

Care must be taken to document the source of any ideas or arguments. If the actual words of a source are used, they must appear within quotation marks. In cases that are unclear, the writer must take due care to avoid plagiarism.

The source should be cited whenever:
(a) a text is quoted verbatim
(b) data gathered by another are presented in diagrams or tables
(c) the results of a study done by another are used
(d) the work or intellectual effort of another is paraphrased by the writer

    Because the intent to deceive is not a necessary element in plagiarism, careful note taking and record keeping are essential in order to avoid unintentional plagiarism.

    For additional information, please consult “Avoiding and Detecting Plagiarism,” available in the Office of the Vice President for Student Affairs, the Provost’s Office, or at
(From The Graduate Center Student Handbook 05-06, pp. 36-37)

Reading Assignments Due
Assignments Due
(due at start of class)
Week 1:
W 8/25
Course Overview, What is program evaluation research?  Does it differ from behavioral science research?  If so, how?
Week 2:
W 9/1
Program evaluation fundamentals and approaches.
Chen (C) Chapters 1-2.
Week 3:
W 9/22
(Classes do not meet 9/8 or 9/15)
Evaluation as a transdiscipline
Scriven (S) Introduction: The Nature of Evaluation, "Logic of Evaluation".

Week 4:
W 9/29

Program Theory and Program Scope.
C3-4, AEA Guiding Principles.
Memo 1:
Stakeholders & Concerns
Week 5:
W 10/6
Action Plans and Early Process Evaluation.
S "Process Evaluation"

Week 6
W 10/13
Implementation Fidelity and Program Monitoring (Go over evaluation proposal assignment)
C7-8, Chapter Summaries for W9-10.
Memo 2:
Theory of Change
Week 7:
W 10/20
Outcome Evaluation.
, "Evaluation Milestones" pdf handout

Week 8:
W 10/27
Holistic Effectuality Evaluation and Theory Driven Evaluation.
Evaluation Proposal

Week 9:
W 11/3
Standards and Best Practices for Evaluation
Standards Introduction & Applying the Standards (YSHC). Morris, Chapter 1 (M1), S "Key Evaluation Checklist", "Ethics" & "Ethics in Evaluation."

Thesaurus Travelogue
Week 10:
W 11/10
Propriety Standards
YSHC:Propriety, M3,
S "Conflict of Interest"

Week 11:
W 11/17
Utility Standards
YSHC:Utility, M6-7
Memo 3:
Evaluation milestones
Week 12:
W 11/24

(Plan accordingly)
Feasibility Standards
YSHC: Feasibility & Evaluation Accountability, M2

Week 13:
W 12/1
Accuracy Standards 
YSHC: Accuracy, M4-5, 8
Memo 4: Standards
Week 14:
W 12/8
Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007

Finals Week:
W 12/15
Come to class prepared to discuss your report.
Evaluation Report

Source for Academic Calendar:

Created 18 August 2008
Updated 10 August 2021