Keith Markus' Urban Sprawl

  EPSY 73100 GC
  Program Evaluation Research
Course Information
Links
Syllabus
Blackboard Page
Assignments
American Evaluation Association
Schedule
 Site Map

Syllabus
Fall 2023

Time:  Wednesday 6:30-8:30 PM
Room:  6495, Graduate Center, 365 Fifth Avenue:
CRN: 41003

Office Hours:  Please contact me by email.  I can answer most questions by email.  However, we can also schedule a Zoom meeting or office meeting by appointment.

Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784 (I will receive email sooner than voice mail.)
Office: Room 10.63.11, 524 W59 Street.  By appointment: GSUC Room 3204.02.
Address: Psychology Department, John Jay College of Criminal Justice, 524 W 59th Street, New York, NY 10019 USA
If you leave something in my mailbox, please alert me by email.

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provide practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course assumes that students are already familiar with basics of research design as might be covered in an introductory research design course.

This course is equivalent to PSYCH U80103, Program Evaluation.

Text Books:
    Chen, H. T. (2015)  Practical program evaluation:  Theory-driven evaluation and the integrated evaluation perspective (2nd ed.).  Thousand Oaks, CA:  Sage.

    Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications.

    Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications.

    Morris, M. (2008). Evaluation ethics for best practice: Cases and commentaries. New York: Guilford.

Additional Reading:
   American Evaluation Association (2004). Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles.

   Fetterman, D. & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28, 179-198.

   Miller, R. L. & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319.

   Smith, N. L. (2007).  Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28, 169-178.

Course Flow:  Familiarize yourself with the reading material before the corresponding class.  Classes will summarize and clarify the reading.  In general, I would rather answer your questions than lecture. Come to class prepared for active discussion of the reading material.

Examinations:  There are no examinations in this course. A series of assignments (two posters, four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Evaluation Memos: Each memo should contain at least 500 words but not exceed 750 words. Strive for clear and concise writing for an audience that may or may not have training in evaluation. Length is an evaluation criterion for the memos, so keep them within the prescribed range. Conciseness is an accomplishment that takes time and effort. Give yourself time to draft, review and revise each memo.  Remember: "All writing is re-writing."

Memo 1: Stakeholders and their concerns.

Review the web page for the CUNY Preparatory Transitional High School Program (CUNY Prep, http://cunyprep.org/sites/distance-learning/) and the 2008 evaluation report (https://www1.nyc.gov/assets/opportunity/pdf/cprep_prr.pdf). Consider how the CUNY Prep program impacts four distinct stakeholder groups. In the first paragraph, briefly enumerate your four stakeholder groups and what you envision as their primary stake in the activities of the program.

In a second paragraph, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups beyond the four you considered, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final paragraph, discuss the stake that the general public has in the activities of the program and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Memo 2: Evaluation Milestones.

Write a memo proposing three performance measures for each of two program outcomes for the CUNY Prep program.  Both outcomes should be impacts of program activities on program participants.  The first outcome should involve students perceptions or attitudes.  The second outcome should come from the "Outputs" box from one of the two logic models included in the report.  For each of the two outcomes, (a) propose two outcome measures, (b) propose specific program milestone (a.k.a., yardstick) for each measure, and (c) briefly describe the method of data collection required by each measure. (Check to confirm that you have four separate outcome measures and provide three distinct pieces of information about each of these four outcome measures.)  Note: Take care not to confuse measures with milestones.  Each milestone depends upon a measure, but the same measure can support many different milestones.  Do not stop at specifying a measure, but be sure to also specify the accompanying milestone.  (Be sure to read the Evaluation Milestones document before attempting this assignment.)

Evaluation Thesaurus Travelogue.

Pick an unassigned entry in the Evaluation Thesaurus. Follow up that entry with a related entry (listed at the end), and continue this process until you have read three entries. If you are not satisfied with the three entries, continue your travels through the thesaurus until you strike upon a series of three entries about which you want to write. Write 500-750 words based on your three-entry trek through the thesaurus. Do not summarize the entries any more than necessary to set up your observations or conclusions about them.  Instead, describe how the entries interrelate with one another and how the implications of the three entries for the practice of program evaluation are connected. You might also discuss how the material sheds light on things you have read in other courses, or your understanding of material from this course. Critical evaluation of the material offers another option. Whatever you include, aim to demonstrate that you have thought about the material.  Keep mere summary of the material from the entries to a minimum (at most, 25% of your travelogue).


Memo 3: Theory of Change.

 Choose a possible unintended outcome of the CUNY Prep program.  For the purposes of this memo, consider any outcome not discussed in the 2008 report an unintended outcome.  Write a memo outlining a proposed theory of change for the unintended outcome. Construct an explanation of the unintended outcome that traces the outcome back to program activities.  Do this at a level most useful for the planning of an evaluation. Provide a causal path diagram depicting your theory of change (see Quick Start Guide to Path Diagrams.pdf).  Select three causal effects from your theory of change as most important for the evaluation, and propose evaluation questions that correspond to them.  Propose one evaluation question for each of the three causal effects.  Formulate your questions to be clear and precise, amenable to empirical evaluation, and stated at an appropriate level of abstraction to make them useful.  Relate your questions to your causal model and theory of change.  Clearly distinguish program activities (possibly represented in the model) from evaluation activities (never represented in the model).  Note that "how much" or "to what extent" questions typically prove more useful than "whether or not" or "yes or no" questions (unless the later involves comparisons between amounts).

Memo 4: Evaluation Standards.

Write a memo devoting one paragraph to each of the five main sets of program evaluation standards. Discuss the 2008 CUNY Prep Program Review report cited above from the perspective of each of the five sets of standards.  Highlight strengths and weaknesses of the report from the perspective of each set of standards.

Posters:  The course requires two posters.  The first is an evaluation proposal and the second is an evaluation report.  The second paper builds on the first.  Format your poster to be 36 inches high (3 feet) and 48 inches wide (4 feet).  Use Times New Roman font.  Follow the font size guidelines from the UCLA library reference document at this URL: https://guides.library.ucla.edu/c.php?g=223540&p=1480858.  The document offers other helpful guidelines as well.  Many people find presentation software useful for creating posters.  Save and submit your poster as a file in Portable Document Format (pdf).

Evaluation Proposal: Respond to the following request for proposals.  Include each title in the request for proposals in your poster.  Do not combine sections.  Bear in mind that while the request for proposals in embedded in a fictionalized context (New Argleton Schools) your evaluation is very much real.  You will need to be able to complete the evaluation with available resources (e.g., no funding) within the time frame of the course (see syllabus schedule for due dates).  So, design your evaluation accordingly.  Be realistic about what you can accomplish in a semester project.  Remember that this is an evaluation study not a research study:  You are evaluating the game, not hypotheses about it.  When a game does not meet objectives, it is the game that has to change, not the objectives.

Request For Proposals
Educational Game Evaluation
Fall 2023

I. Introduction
The web contains a wide variety of free educational games.  The New Argleton, New York, school district is considering installing a selection of such games in computer labs in its elementary schools serving grades K-6.  The goal is to select games that will effectively reinforce (but not replace) instruction aligned with the Common Core curriculum standards.  The district is seeking evaluations of educational games to help guide their choices of which games to install.

 II. Eligibility
Applicants must be currently enrolled in EPSY U73100 during the Fall of 2023.

III. Proposal Requirements
Proposals must contain each of the following elements and must fit on a 3'×4' poster.

A. Header
List proposal title, author's name, and affiliation on the header at the top of the poster.

B. Executive Summary (maximum 200 words)
Summarize the proposal, including all elements from C to H below.  Convey the content of the proposal in condensed form.  Do not list topics covered without summarizing the content under the topics.  The summary should be a capsule statement of the proposal, not a description of it from an external perspective and not a list of topics with no content.

C. Evaluation Topic and Program Theory
Identify the game to be evaluated including its URL.  List at least three learning objectives for the game.  Present a program theory including inputs, program activities, outputs and direct outcomes.  You may optionally include more distal outcomes.  Describe the various aspects of the program theory.  If your proposal is based on a particular educational theory or theory of learning, describe that here.  If your proposal draws upon existing literature, discuss that here using a few key references.  (For the purposes of this RFP, the term 'program' refers to the creation of computer labs with educational games at the elementary schools.)

D. Logic Model (1 figure)
Include a logic model that illustrates your program theory.  You may refer to this in Part C.

E. Evaluation Questions
State your evaluation questions.  Include at least four questions related to the game described in Part C.  For each question, provide a brief rationale for why you chose this question and how the answer will contribute to program implementation.  (Choose feasible questions in light of the time frame and available resources.)  It may be helpful to relate the questions to your performance milestones in part G, parts E, F and G should all work together in an integrated fashion.

F. Evaluation Methods
For each evaluation question listed in Part E, describe the methodology that will be used to answer the question.  Provide as clear and precise as description as possible of the methods that will be used to evaluate the game for each evaluation question, including any coding or rating methods used in the research.  Avoid overlap between your description of your methodology in this part and your description of measures in Part G.

G. Evaluation Measures and Performance Objectives
Describe evaluation measures used in the evaluation.  Include verbatim rating scales or content coding instructions.  If your measure or coding is too expansive to fit, give concrete examples and summarize the rest as informatively as possible.

For each evaluation measure, provide a performance objective (i.e., milestone) that represents what you would consider minimally acceptable performance for a game on that dimension.

H. Guiding principles
Relate your proposed evaluation to the Guiding Principles and explain how your evaluation responds to these principles.

I. Time budget (1 table)
Your resources for this evaluation project are hours of work rather than dollars.  Provide a budget allocating work hours to various evaluation activities.  Include a total and any appropriate category subtotals.

J. Budget justification
Briefly justify your budget allocation in words.

K. Time line (text or 1 table)
Provide a weekly time line for completion of the project.

L. Reference list
Provide references for any literature cited in the proposal.  Use APA format (7th edition).

IV. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.

[End of RFP]

Evaluation Project Report Poster:
Complete the project described in your proposal, incorporating any feedback from the proposal.  Create your evaluation project report poster using the following format.  Include each heading in your poster and do not combine headings.  Follow the same formatting guidelines (e.g., font and font size, page size) as for the previous poster.

A. Header
List proposal title, author's name, and affiliation on the title page.

B. Executive Summary (maximum 200 words).

1. Summary of research report.
Be sure to summarize all sections of the report including results and recommendations.  As with the proposal, the summary should provide a condensed statement of the content of the report.  It should not simply list topics covered in the report or describe the report.

2. Summary of changes.
Summarize any changes in the evaluation design, questions or method from the proposal.  If there were no changes, simply state that.

C. Introduction
Present the context of the evaluation.  Describe the purpose of the evaluation and close with the specific evaluation questions.  Phrase the questions to include evaluation milestones.  (This section may be briefer than the corresponding section of the proposal.)

D. Method
Summarize the methodology used in the evaluation.  The primary focus of this section is on what data you collected, but it may also describe other aspects of method such as evaluation criteria or evaluation approach.

E. Results
Present the results for each evaluation question.  Consider tables and figures to present material more concisely.  (The APA Publication Manual offers helpful guidance regarding when to use text, tables or figures.)

F. Recommendations
State each recommendation in a sentence followed by text that elaborates and clarifies the recommendations and text that provides a rationale for the recommendation.  You may find it helpful to identify each of these two subtopics with a separate subheading beneath each recommendation.  Include at least four recommendations based on your findings.

G. Reference list
Do not let this take up too much space.  Focus on key references.

Evaluation criteria:
1. Overall clarity.
2. Overall completeness and informativeness.
3. Results are presented in a useful manner.
4. Recommendations are useful and well justified.


Grading:  Posters count 50% (25% each) and memos and travelogue count 50% (10% each).

Note on assignment due dates:  Students should make every effort to turn in assignments before the deadline.  On rare occasions, if a student needs more time, they should reach out to the instructor before the deadline, preferably well before, and provide a good reason.  Extensions are offered at the discretion of the professor.

 

Letter Grade
Percent Grade
A
92-100
A-
84-91
B+
76-83
B
68-75
B-
60-67
C+
52-59
C
44-51
C-
36-43
F
0-35


Academic Honesty:    
The Graduate Center of The City University of New York is committed to the highest standards of academic honesty. Acts of academic dishonesty include—but are not limited to—plagiarism, (in drafts, outlines, and examinations, as well as final papers), cheating, bribery, academic fraud, sabotage of research materials, the sale of academic papers, and the falsification of records. An individual who engages in these or related activities or who knowingly aids another who engages in them is acting in an academically dishonest manner and will be subject to disciplinary action in accordance with the bylaws and procedures of The Graduate Center and the Board of Trustees of The City University of New York.  

Each member of the academic community is expected to give full, fair, and formal credit to any and all sources that have contributed to the formulation of ideas, methods, interpretations, and findings. The absence of such formal credit is an affirmation representing that the work is fully the writer’s. The term “sources” includes, but is not limited to, published or unpublished materials, lectures and lecture notes, computer programs, mathematical and other symbolic formulations, course papers, examinations, theses, dissertations, and comments offered in class or informal discussions, and includes electronic media. The representation that such work of another person is the writer’s own is plagiarism.

Care must be taken to document the source of any ideas or arguments. If the actual words of a source are used, they must appear within quotation marks. In cases that are unclear, the writer must take due care to avoid plagiarism.

The source should be cited whenever:
(a) a text is quoted verbatim
(b) data gathered by another are presented in diagrams or tables
(c) the results of a study done by another are used
(d) the work or intellectual effort of another is paraphrased by the writer

    Because the intent to deceive is not a necessary element in plagiarism, careful note taking and record keeping are essential in order to avoid unintentional plagiarism.

    For additional information, please consult “Avoiding and Detecting Plagiarism,” available in the Office of the Vice President for Student Affairs, the Provost’s Office, or at http://www.gc.cuny.edu/CUNY_GC/media/CUNY-Graduate-Center/PDF/Policies/General/AvoidingPlagiarism.pdf.
(From The Graduate Center Student Handbook 05-06, pp. 36-37)

Note: Submitting work generated by a generative artificial intelligence program has been interpreted as both plagiarism and cheating under existing CUNY policy.  It constitutes submitting work from another source as your own work.  That is, you should not consider generative artificial intelligence to be a tool that you use to write, like a grammar checker or spell checker.  You should consider it a source like a book or article and apply the same rules for citation and quotation.

Top
Schedule
Date
Reading Assignments Due
Assignments Due
(due before start of class)
Week 1:
W 8/30
 Course Overview, What is program evaluation research?  Does it differ from behavioral science research?  If so, how?
  
Week 2:
W 9/6
 Program evaluation fundamentals and approaches.
Chen (C) Chapters 1-2.
  
Week 3:
W 9/13
 Evaluation as a transdiscipline
Scriven (S) Introduction: The Nature of Evaluation, "Logic of Evaluation".
Week 4:
W 9/20

 Program Theory and Program Scope.
C3-4, AEA Guiding Principles. 		Memo 1:
Stakeholders & Concerns
Week 5:
W 9/27
 Action Plans and Early Process Evaluation.
C5-6, S "Process Evaluation", "Evaluation Milestones" pdf handout
Week 6
W 10/4
 Implementation Fidelity and Program Monitoring (Go over evaluation proposal assignment)
C7-8 		Memo 2:
Evaluation milestones
Week 7:
W 10/11		 Outcome Evaluation and Dissemination.
C9, 15
Week 8:
W 10/18
 Holistic Effectuality Evaluation and Theory Driven Evaluation.
C11-12		 Evaluation Proposal
Week 9:
W 10/25
 Standards and Best Practices for Evaluation
Standards Introduction & Applying the Standards (YSHC). Morris, Chapter 1 (M1), S "Key Evaluation Checklist", "Ethics" & "Ethics in Evaluation."		 Thesaurus Travelogue
Week 10:
W 11/1
 Propriety Standards
YSHC:Propriety, M3, S "Conflict of Interest"
Week 11:
W 11/8
 Utility Standards
YSHC:Utility, M6-7 		Memo 3:
Theory of Change
Week 12:
W 11/15
 Feasibility Standards
YSHC: Feasibility & Evaluation Accountability, M2
Week 13:
W 11/29
(Classes do not meet 11/22)
 Accuracy Standards 
YSHC: Accuracy, M4-5, 8		 Memo 4: Standards
Week 14:
W 12/6
 Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007
Finals Week:
W 12/13
Come to class prepared to discuss your report.
 Evaluation Report




Source for Academic Calendar:  http://www.cuny.edu/academics/calendars.html

Top
Created 18 August 2008
Updated 20 August 2023