Keith Markus' Urban Sprawl


  EDPSYCH 73100 GC
  Program Evaluation Research
Course Information
Links
Syllabus
Blackboard Page 
Papers
American Evaluation Association
Schedule
Site Map

Syllabus
 
Fall 2013

Time:  Wednesday 6:30-8:30 PM
Room:  GSUC 5417

Office Hours:  Tuesdays and Thursdays 4:15-5:15pm

Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784
Office: Room 10.63.11, 524 W59 Street.  By appointment: GSUC Room 3204.02.
Address: Psychology Department, John Jay College of Criminal Justice, 524 W 59th Street, New York, NY 10019 USA

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured a bit since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provided practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course is equivalent to PSYCH U80103, Program Evaluation.

Text Books:
    Weiss, C H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

    Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Sage Publications. (Note, major online book sellers are currently offering the paperback edition at a substantial discount over the publisher's list price.)

    Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications.

    Morris, M. (2008). Evaluation ethics for best practice: Cases and commentaries. New York: Guilford.

Additional Reading:

   American Evaluation Association (2004). Guiding principles for evaluators. http://www.eval.org/Publications/GuidingPrinciples.asp.

   Fetterman, D. & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28, 179-198.

   Miller, R. L. & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319.


   Smith, N. L. (2007).  Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28, 169-178.

Books are available through the GC online bookshop: http://www.gc.cuny.edu/bookshop/. Ordering through the GC online bookshop helps to support the Mina Rees Library.

Course Flow:  Familiarize yourself with the reading material before the corresponding class.  Classes will summarize and clarify the reading.  In general, I would rather answer your questions than lecture. Come to class prepared for active discussion of the reading material.

Examinations:  There are no examinations in this course. A series of assignments (four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Evaluation Memos: Each memo should contain at least 500 words but not exceed 1000 words. Strive for clear and concise writing for an audience that may or may not have training in evaluation. Length is an evaluation criterion for the memos, so keep them within the prescribed range. Conciseness is an accomplishment that takes time and effort. Give yourself time to draft, review and revise each memo.

Memo 1: Stakeholders and their concerns.

Review the web page for the Mina Rees library (http://library.gc.cuny.edu/) and the 2005 CUNY LibQual survey report (http://libraries.cuny.edu/LibQualFinalReport.pdf), focusing on the material most directly related to the Mina Rees library. Consider how the library impacts (a) Graduate Center students, (b) Graduate Center faculty, (c) Library users from other CUNY campuses, and (d) Mina Rees librarians and other staff members. In the first section, briefly discuss each of these stakeholder groups and what you envision as their primary concerns with the activities of the library.

In a second section, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups not listed above, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final section, discuss the stake that the general public has in the activities of the Mina Rees library and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Memo 2: Evaluation Yardsticks.

Write a memo proposing three performance measures for each of two library outcomes.  Both outcomes should be impacts of library activities on library users.  The first outcome should involve students and the second should involve faculty.  For each of the two outcomes, (a) propose three outcome measures, (b) propose specific program yardstick for each measure, and (c) briefly describe the method of data collection required by each measure. (Note: You should have six separate outcome measures and provide three distinct pieces of information about each of these six outcomes.)

Evaluation Thesaurus Travelogue.

Pick an unassigned entry in the Evaluation Thesaurus. Follow up that entry with a related entry, and continue this process until you have read five entries. If you are not satisfied with the first five entries, continue your travels through the thesaurus until you strike upon a series of five entries about which you want to write. Write 500-1000 words based on your five-entry trek through the thesaurus. Do not summarize the entries any more than necessary to set up your observations or conclusions about them.  Instead, describe how the entries interrelate with one another and how the implications of the five entries for the practice of program evaluation are connected. You might also discuss how the material sheds light on things you have read in other courses, or your understanding of material from this course. Critical evaluation of the material offers another option. Whatever you include, aim to demonstrate that you have thought about the material.  Keep mere summary of the material to a minimum.

Memo 3: Program Model.
Write a memo outlining a proposed program model for a specific activity of the Mina Rees library. Describe inputs, program activities, outputs, and outcomes at a level most useful for the planning of an evaluation. Provide a diagram of your program model.  Clearly distinguish direct outputs of library activities from the outcomes that they facilitate.  Select three elements of your program model as most important for the evaluation, and propose evaluation questions that correspond to them.

Memo 4: Evaluation Standards.
Write a memo devoting one paragraph to each of the five main sets of program evaluation standards. Discuss the 2005 CUNY LibQual report cited above from the perspective of each of the five sets of standards.  Highlight strengths and weaknesses of the report from the perspective of each set of standards.

Mini-papers:  The course requires two mini-papers.  The first is an evaluation proposal and the second is an evaluation report.  The second paper builds on the first.

Evaluation Proposal: Respond to the following request for proposals.

Request For Proposals
Program Evaluation Research Text Book Evaluation
Fall 2013

I. Introduction
Four books are currently under consideration to replace the Weiss (1998) text.  The purpose of this call for proposals (CFP) is to solicit evaluations of these candidate replacement texts focusing on specific aspects of the text.  Proposals are due October 23, 2013.  Final evaluation reports are due December 18, 2013.  The four candidate replacement texts are as follows.

Bamberger, M., Rugh, J., & Mabry, L. (2012).  Real World Evaluation:  Working under budget, time, data, and political constraints (2nd ed.).  Thousand Oaks, CA: Sage Publications.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011).  Program evaluation: Alternative approaches and practical guidelines (4th ed.).  Upper Saddle River, NJ: Pearson.
Patton, M. Q. (2008).  Utilization-focused evaluation (4th ed).  Thousand Oaks, CA: Sage Publications.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004).  Evaluation: A systematic approach (7th ed.).  Thousand Oaks, CA: Sage Publications.

II. Eligibility
Applicants must be currently enrolled in EDPSYCH U73100.

III. Proposal Requirements
Proposals must contain each of the following elements and must not exceed indicated page lengths per section.  Proposals can be single spaced, and there is no need for page breaks between sections except after the title page.  All word counts refer to maximum word counts.

A. Title Page
List proposal title, author's name, and affiliation on the title page.  Begin page numbering at 1 on the title page.

B. Summary (200 words)
Summarize the proposal, including all elements from C to H below.  Convey the content of the proposal in condensed form.  Do not list topics covered without summarizing the content under the topics.  The summary should be a capsule statement of the proposal, not a description of it from an external perspective.

C. Evaluation Topic and Program Theory (500 words maximum)
Clearly state the aspect of the text books that you plan to evaluate.  Describe a program theory including inputs, aspects of the text, text book related activities, and outputs.  You may optionally include more distal outcomes.  Describe the various aspects of the program theory.  If your proposal is based on a particular educational theory or theory of learning, describe that here.  If your proposal draws upon existing literature, discuss that here.

D. Logic Model (1 page figure)
Include a logic model that illustrates your program theory.  You may refer to this in Part C.

E. Evaluation Questions (250 words)
State your evaluation questions.  Include at least five questions related to the topc described in Part C.  For each question, provide a brief rationale for why you chose this question and how the answer will contribute to text book selection.

F. Evaluation Methods (500 words)
For each evaluation question listed in Part E, describe the methodology that will be used to answer the question.  Provide as clear and precise as description as possible of the methods that will be used to evaluate the text books for each evaluation question, including any coding or rating methods used in the research.  Avoid overlap between your description of your methodology in this part and your description of measures in Part G.  The present CFP is primarily aimed at methods that involve drawing data directly from the books in question (content analysis, content coding, rating scales, objective performance markers).  However, other methodologies will be considered provided that they answer useful evaluation questions and are feasible within the time frame of the evaluation project.

G. Evaluation Measures and Performance Objectives (500 words)
Describe evaluation measures used in the evaluation.  Include verbatim rating scales or content coding instructions.  If necessary, summarize coding instructions here and include the full instructions as an appendix.  In necessary, long rating scales can also be included as an appendix.

For each evaluation measure, provide a performance objective (i.e., yardstick) that represents what you would consider minimally acceptable peformance for a text book on that dimension.

H. Guiding principles (500 words)
Relate your proposed evaluation to the Guiding Principles and explain how your evaluation responds to these principles.

I. Time budget (1 page table)
Your resources for this evaluation project are hours of work rather than dollars.  Provide a budget allocating work hours to various evaluation activities.  Include a total and any appropriate category subtotals.

J. Budget justification (250 words)
Briefly justify your buget allocation.

K. Time line (1 page table)
Provide a weekly time line for comletion of the project.

L. Reference list
Provide references for any literature cited in the proposal.  Use APA format.

M. Appendices
See Part G instructions for possible appendices.

IV. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.

[End of RFP]

Evaluation Project Report:
Complete the project described in your proposal, incorporating any feedback from the proposal.  Write up your report using the following format.

A. Title page.
List proposal title, author's name, and affiliation on the title page.  Begin page numbering at 1 on the title page.

B. Summary.

1. Summary of research report. (250 words)
Be sure to summarize all sections of the report including results and recommendations.  Like an abstract, the summary should provide a condensed statement of the content of the report.  It should not simply list topics covered in the report or describe the report.

2. Summary of changes.  (125 words)
Summarize any changes in the evaluation design, questions or method from the proposal.  If there were no changes, simply state that.

C. Introduction (500 words)

Present the context of the evaluation.  Describe the purpose of the evaluation and close with the specific evaluation questions.

D. Method (750 words)

Summarize the methodology used in the evaluation.  Coding instructions, coding sheets, rating scales, and other materials can be included as an appendix and do not count toward the 750 words.  The primary focus of this section is on what data you collected, but it may also describe other aspects of method such as evaluation criteria or evaluation standards.

E. Results (1500 words)

Present summary data comparing each of the four texts to your performance standards and to one another based on any evaluation measures used in your evaluation study.  Also present the results of data analyses for each evaluation question.  Tables and figures do not count as part of the word count, but should be included directly inside the text (not at the end of the document).

F. Recommendations (1000 words)
State each recommendation in a sentence followed by text that elaborates and clarifies the recommendations and text that provides a rationale for the recommendation.

G. Reference list

H. Appendices (optional)
See Part D above.

Evaluaiton criteria:
1. Overall clarity.
2. Overall completeness and informativeness.
3. Results are presented in a useful manner.
4. Recommendations are useful and well justified.


Grading:  Mini papers count 50% (25% each) and memos and travelogue count 50% (10% each).

 

Letter Grade
Percent Grade
A
92-100
A-
84-91
B+
76-83
B
68-75
B-
60-67
C+
52-59
C
44-51
C-
36-43
F
0-35


Academic Honesty:    
The Graduate Center of The City University of New York is committed to the highest standards of academic honesty. Acts of academic dishonesty include—but are not limited to—plagiarism, (in drafts, outlines, and examinations, as well as final papers), cheating, bribery, academic fraud, sabotage of research materials, the sale of academic papers, and the falsification of records. An individual who engages in these or related activities or who knowingly aids another who engages in them is acting in an academically dishonest manner and will be subject to disciplinary action in accordance with the bylaws and procedures of The Graduate Center and the Board of Trustees of The City University of New York.  

Each member of the academic community is expected to give full, fair, and formal credit to any and all sources that have contributed to the formulation of ideas, methods, interpretations, and findings. The absence of such formal credit is an affirmation representing that the work is fully the writer’s. The term “sources” includes, but is not limited to, published or unpublished materials, lectures and lecture notes, computer programs, mathematical and other symbolic formulations, course papers, examinations, theses, dissertations, and comments offered in class or informal discussions, and includes electronic media. The representation that such work of another person is the writer’s own is plagiarism.

Care must be taken to document the source of any ideas or arguments. If the actual words of a source are used, they must appear within quotation marks. In cases that are unclear, the writer must take due care to avoid plagiarism.

The source should be cited whenever:
(a) a text is quoted verbatim
(b) data gathered by another are presented in diagrams or tables
(c) the results of a study done by another are used
(d) the work or intellectual effort of another is paraphrased by the writer

    Because the intent to deceive is not a necessary element in plagiarism, careful note taking and record keeping are essential in order to avoid unintentional plagiarism.

    For additional information, please consult “Avoiding and Detecting Plagiarism,” available in the Office of the Vice President for Student Affairs, the Provost’s Office, or at http://web.gc.cuny.edu/provost/pdf/AvoidingPlagiarism.pdf.

(From The Graduate Center Student Handbook 05-06, pp. 36-37)


Top
Schedule
Date
Reading Assignments Due
Assignments Due
Week 1:
W 8/28
Course Overview, What is program evaluation research and how does it differ from behavioral science research?
 
W 9/4
Class does not meet.  No classes scheduled.

Week 2:
W 9/11
The purposes of program evaluation.
Weiss (W) Chapters 1-2.
 
Week 3:
W 9/18
Evaluation as a transdiscipline
Scriven (S) Introduction: The Nature of Evaluation, "Logic of Evaluation".

Week 4:
W 9/25
Understanding the program and planing the evaluation.
W3-4, S "Process Evaluation," AEA Guiding Principles.
Memo 1:
Stakeholders & Concerns
Week 5:
W 10/2
Roles of the evaluator, program measures.
W5-6

Week 6
W 10/9
Collecting data, Designing the evaluation. (Go over evaluation review assignment)
W7-8, Chapter Summaries for W9-10. (Note 1)
Memo 2:
Evaluation Milestones
Week 7:
W 10/16
Class does not meet due to American Evaluation Association conference.
Class will be online (asynchronous) using Blackboard.
Qualitative methods, interpreting data.
W11-12


Week 8:
W 10/23
Dissemination & Evaluation integrity.
W13-14, S "Ethics" & "Ethics in Evaluation."
Evaluation Proposal
(due at start of class)

Week 9:
W 10/30
Standards and Best Practices for Evaluation
Discuss Evaluation Review.
Standards Introduction & Applying the Standards (YSHC). Morris, Chapter 1 (M1)
, S "Key Evaluation Checklist"

Week 10:
W 11/6
Propriety Standards
YSHC:Propriety, M3,
S "Conflict of Interest"
Thesaurus Travelogue
Week 11:
W 11/13
Utility Standards
YSHC:Utility, M6-7

Week 12:
W 11/20
Feasibility Standards
YSHC: Feasibility & Evaluation Accountability, M2
Memo 3: Program Model
W 11/27
Class does not meet.  Classes follow a Friday schedule

Week 13:
W 12/4
Accuracy Standards 
YSHC: Accuracy, M4-5, 8
Memo 4: Standards
Week 14:
W 12/11
Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007

Finals Week:
W 12/18
Come to class prepared to discuss your report.

Evaluation Report

Note 1: The course is designed with the assumption that you have taken a basic research design course and are familiar with the designs discussed in these chapters. If not, you will want to read the full chapters. The last section of Chapter 10 may be useful in any event.


Top
Created 18 August 2008
Updated 31 July 2013