Keith Markus' Urban Sprawl

  PSYCH 80103 JJ
  Program Evaluation
Course Information
Links
Syllabus
Blackboard Page
(Links to CUNY Portal Log-in Page).
Papers
American Evaluation Association
Schedule
 Site Map

Syllabus
  Fall 2010

Time:  Wednesday 6:30-8:30 PM
Room:  2437N, John Jay College (445 W 59, second floor)
Office Hours:  Tuesday and Wednesday 4:00 PM to 5:00 PM.

Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com
212-237-8784
Room 2127N
Psychology Department, John Jay College of Criminal Justice, 445 W 59th Street, New York, NY 10019 USA

Course Description:  This course will examine approaches to evaluation and methods to evaluate the effectiveness of programs and projects providing educational services. Topics will include how to plan an evaluation, methods of collecting data, design and testing issues, data analysis, and the politics and use of evaluations. Techniques will be drawn from Anthropology, Economics, Psychology, Sociology and Statistics. (Note: Evaluation has matured a bit since this course description was written and the influences of the social and behavioral science disciplines listed are now primarily indirect, filtered through a substantial literature specific to evaluation. Evaluation as a transdiscipline will be considered. Both effectiveness and efficacy will be considered. Material likely to overlap introductory research methods courses will not be emphasized.)

Course Objectives:
1. Expose students to the basic theory and methods of program evaluation.
2. Expose students to professional standards for program evaluation.
3. Provided practice applying theory, methods, and standards to practical evaluation problems.
4. Provide practice with various forms of writing important to program evaluation.
5. Provide a strong foundation for further study of program evaluation either through additional course work or through self study.

This course is equivalent to EDPSYCH U731, Evaluation Research.

Text Books:
    Weiss, C H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

    Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Sage Publications. (Note, major online book sellers are currently offering the paperback edition at a substantial discount over the publisher's list price.)

    Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications.

    Morris, M. (2008). Evaluation ethics for best practice: Cases and commentaries. New York: Guilford.

Additional Reading:
   American Evaluation Association (2004). Guiding principles for evaluators. http://www.eval.org/Publications/GuidingPrinciples.asp.

   Fetterman, D. & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28, 179-198.

   Miller, R. L. & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319.

   Smith, N. L. (2007).  Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28, 169-178.

I have ordered the books through the John Jay College of Criminal Justice (CUNY) Bookstore, 841 10th Avenue, New York,  NY 10019-1199, Store telephone:  (212)265-3619.
Books are also available through the GC online bookshop: http://www.gc.cuny.edu/bookshop/. Ordering through the GC online bookshop helps to support the Mina Rees Library.

Course Flow:  Familiarize yourself with the reading material before the corresponding class.  Classes will summarize and clarify the reading.  In general, I would rather answer your questions than lecture. Come to class prepared for active discussion of the reading material.

Examinations:  There are no examinations in this course. A series of assignments (four memos and a short paper) take the place of take-home examinations. It is important that you keep up with the reading in order to make this examination-free approach work.

Evaluation Memos: Each memo should fill at least one page (single spaced) but not exceed two. Strive for clear and concise writing for an audience that may or may not have training in evaluation. Length is an evaluation criterion for the memos, so keep them within the prescribed range. Conciseness is an accomplishment that takes time and effort. Give yourself time to draft, review and revise each memo. You may find it helpful to review the RFP at the end of the syllabus before writing your memos.

Memo 1: Stakeholders and their concerns.

Review the web page for the Metropolitan Transportation Authority Office of the Inspector General (http://mtaig.state.ny.us/index.htm). Consider how the work of this office impacts (a) MTA management, (b) MTA employees, (c) contractors doing business with the MTA, and (d) MTA ridership. In the first section, briefly discuss each of these stakeholder groups and what you envision as their primary concerns with the work of the Inspector General.

In a second section, propose a cost-effective methodology ($500 or less) for (a) identifying any additional primary concerns among these stakeholder groups, (b) identifying additional stakeholder groups not listed above, and (c) identifying the primary concerns of any additional stakeholder groups.

In the third and final section, discuss the stake that the general public has in the work of the Inspector General and compare and contrast it with the specific concerns of the four stakeholder groups identified above.

Memo 2: Evaluation Milestones.

Write a memo proposing three performance measures for each of the following abstract outcomes: (1) Effectively identify issues that fall under the purview of the Office of the Inspector General and warrant investigation, (2) Effectively investigate known issues, leading to corrective action. For each of these two general outcomes, (a) propose three outcome measures, (b) propose specific program milestones for each measure, and (c) briefly describe the method of data collection required by each measure. (Note: You should have six separate outcome measures and provide three distinct pieces of information about each of these six outcomes.)

Evaluation Thesaurus Travelogue.
Pick an unassigned entry in the Evaluation Thesaurus. Follow up that entry with a related entry, and continue this process until you have read five entries. If you are not satisfied with the first five entries, continue your travels through the thesaurus until you strike upon a series of five entries about which you want to write. Write 400-500 words to summarize your five-entry trek through the thesaurus. Describe how the entries interrelate with one another and how the implications of the five entries for the practice of program evaluation. You might also discuss how the material sheds light on things you have read in other courses, or your understanding of material from this course. Critical evaluation of the material offers another option. Whatever you include, aim to demonstrate that you have thought about the material.

Memo 3: Program Model.
Write a memo outlining a proposed program model for a specific activity of the Office of the Inspector General. Describe inputs, outputs, and mediating mechanisms at a level most useful for the planning of an evaluation. Provide a diagram of your program model. Select three elements of your program model as most important for the evaluation, and propose evaluation questions that correspond to them.

Memo 4: Evaluation Standards.
Write a memo devoting one paragraph to each of the four main sets of program evaluation standards. In each paragraph, describes steps taken to conform to the relevant set of standards in conducting an evaluation of an activity of the Office of the Inspector General. Address the memo to Governor Patterson for whose office you can assume that you are to complete the evaluation. Hit the main points, and aim to assure the Governor that you will provide a good quality evaluation in accordance with each of the four basic types of standards.

Mini-papers:  The course requires two mini-papers.  The length of the two papers combined should not exceed what you would ordinarily write for a term paper (25-30 pages) The evaluation review will run about 10 pages, leaving the rest for the evaluation proposal.

Evaluation Review.  Download the 2009 full evaluation report entitled Evaluation Report: United Nations Trust Fund in Support of Actions to Eliminate Violence against Women from the UNIFEN evaluation unit web page. The main text of the report runs from page 45 to page 130, and there are pages more of supplementary materials. However, dealing with copious background material is something every program evaluator needs to learn to manage. Give yourself plenty of lead time to digest the evaluation report and think through your evaluation.

Part I: Header
List each of the following fields on a separate line: Your name, date, title of evaluation being reviewed.

Part II: Program Description (500 words maximum).
Evaluate the description of the program given in the evaluation report. Consider clarity, relevance of information provided, and the overall perspective of the description. Describe both strengths and weaknesses.

Part III: Evaluation questions (500 words maximum).
Evaluate the choice and framing of the evaluation questions including their relevance, importance, feasibility, and responsiveness to stakeholder concerns. Describe both strengths and weaknesses.

Part IV: Data collection and outcome measures (500 words maximum).
Evaluate the methods of data collection and the choice of outcome measures. Describe both strengths and weaknesses.

Part V: Data Interpretation and evaluation conclusions/recommendations (500 words maximum).
Evaluate the conclusions drawn and their support in the interpretation of the data collected. Consider both the justification, generality, and usefulness of the conclusions. Describe both strengths and weaknesses.

Part VI: Overall consistency with Guiding Principles for Evaluators (500 words maximum).
Briefly evaluate the reported evaluation (including the report itself) with respect to the five main elements (A through E) of the Guiding Principles. Assess this as best as is possible with the information provided. Rate the evaluation on each element (A through E) as either (1) Poor, (2) Adequate, (3) Good, or (4) Outstanding. Justify your rating in the narrative portion of Part VI.

Evaluation Proposal. 

Write a proposal responsive to the following request for proposals (RFP).  For the purposes of the course assignment, omit the supporting materials and turn in the paper by hand in class.

Request For Proposals
The State of New York
Office of Didactic Fictions and Pedagogical Exercises
Metropolitan Transportation Authority Office of the Inspector General Program Evaluations Initiative
Fall 2010

I. Introduction

New York State created the Metropolitan Transportation Authority (MTA) Office of Inspector General (OIG) to oversee operations of the MTA and its constituent entities. The activities of the OIG include solicitation of complaints, conduct of relevant investigations, provision of recommendations based on findings, reporting of findings, and reporting of evidence of illegalities to the relevant law enforcement agencies. The OIG is not part of the MTA transit police, but rather an independent office. The inspector general (IG) is appointed by the Governor of the State of New York. The OIG maintains a web page located at http://mtaig.state.ny.us/index.htm.

II. Eligibility

Only students registered in this section of Program Evaluation Research are eligible to apply for this grant. Supporting materials should demonstrate the capacity of the grantee to successfully complete the proposed activities.

III. Proposal Submission Process

Address proposals to Professor Keith A. Markus, Department of Psychology, John Jay College of Criminal Justice, CUNY, 445 W59 Street, New York, NY 10019 USA.  In order to receive consideration, all proposals must be received by Thursday, December 15, 2010. Electronic submissions will not be accepted.

IV. Available Funding

Five evaluations will be funded for an amount up to $100,000 over two years. The funding period begins August 1 2011 and ends August 1 2013.

V. Purpose

The purpose of this funding program is to generate evaluations of various OIG activities that explore their scope, effectiveness by a variety of criteria, and offer recommendations that can improve the effectiveness of the OIG in its general mission. Evaluation projects that put into place systems of assessment or evaluation that can be repeated in future years by the OIG for purposes of ongoing program monitoring will hold particular interest.

VI. Proposal Format

Double space your entire proposal exclusive of attachments.  Sections C through G of your proposal may not exceed 15 pages total. Use 1 inch margins on letter sized paper and Times New Roman size 12 font.

A. Title Page
Include the title of your proposal, the full names of the primary author(s), and "OIG Program Evaluation Initiative Fall 2010" on your title page.

B. Project Summary (200 words)
Summarize your proposed project here.  Be sure to cover each of sections C through G of the proposal.  Summarize the content of the proposal rather than describing what the proposal contains.  Be sure to convey a clear picture of the basic content of your proposal.

C. Focal OIG Activity

Identify an OIG activity to serve as the focus of the evaluation effort. Utilize information from the OIG web page or any other sources to provide a description of the activity, how it relates to the general OIG mission, the scope and methods of the activity, and specific examples drawn from OIG reports.

D. Evaluation questions.

Describe at least three evaluation questions that your evaluation research will address. Give the rationale for the selection of these questions. Moreover, if you intend to include a mechanism for considering additional evaluation questions as part of the evaluation process, describe that here.

E. Evaluation Design and Analysis

Describe your evaluation design including (1) research design, (2) data collection methods, (3) your intended analysis of the data collected, (4) the rationale for 1-3, and (5) an explicit statement of how 1-3 will collectively allow you to address the evaluation questions listed in part D. Number the subsections of section E accordingly.

F. Program Staff Involvement and Impact

Describe the involvement of program staff entailed by your evaluation plan. Describe approximate workload induced for program staff by the evaluation. Present a cost/benefit analysis to justify involvement of program staff in your evaluation. Where possible, consider specific benefits for individual participants or groups of participants as opposed to global benefits of the evaluation.

G. Dissemination and Utilization Plan

Describe a dissemination plan to communicate the results of your project. This should include an interim report due September 1, 2012 and a final report due September 1, 2013. However, it should also include additional efforts to disseminate the results  both within the OIG and beyond during the two-year funding period. Also discuss specific efforts to enhance the utilization of evaluation results.

H. Reference List

Use APA format for any references cited in the proposal.

I. Time Line

Detail a time line of activities during the grant period.

J. Budget

Provide a detailed budget listing proposed expenditures using the following outline.  Include 50% overhead on any expenditures that go toward personnel.  Include a budget total at the bottom and a subtotal at the end of each section outlined below.  (You may find it helpful to put this part of the proposal in a spread sheet rather than a word processor.)

1. Evaluation Costs
  a. Salary & Personnel
    i. Salaries & hours of grant personnel
    ii. 50% overhead
  b. Supplies and Materials
  c. Other costs (e.g., transportation, telephone, etc.)
2. Dissemination Costs
  a. Salary & Personnel
    i. Salaries and hours of grant personnel
    ii. 50% overhead
  b. Supplies and Materials
  c. Other costs (e.g., transportation, telephone, etc.)

K. Budget Justification

Provide a brief narrative justification for key budget items.

L. List of Supporting Materials*
1. Resume or Curriculum Vitae of key personnel.
2. IRB Permissions to be sought.
3. Letters of cooperation.
4. Any Additional materials.

VII. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of evaluation design.
3. Usefulness of potential evaluation results to decision making.
4. Justifications for design decisions and budget.
5. Overall clarity and precision of presentation.
(End of RFP)

*The actual materials are not required, only the list is required for the purposes of this assignment.

Grading:  Mini papers count 50% (25% each) and memos and travelogue count 50% (10% each).
 

Letter Grade
Percent Grade
A
92-100
A-
84-91
B+
76-83
B
68-75
B-
60-67
C+
52-59
C
44-51
C-
36-43
F
0-35


Academic Honesty:    
The Graduate Center of The City University of New York is committed to the highest standards of academic honesty. Acts of academic dishonesty include—but are not limited to—plagiarism, (in drafts, outlines, and examinations, as well as final papers), cheating, bribery, academic fraud, sabotage of research materials, the sale of academic papers, and the falsification of records. An individual who engages in these or related activities or who knowingly aids another who engages in them is acting in an academically dishonest manner and will be subject to disciplinary action in accordance with the bylaws and procedures of The Graduate Center and the Board of Trustees of The City University of New York.  

Each member of the academic community is expected to give full, fair, and formal credit to any and all sources that have contributed to the formulation of ideas, methods, interpretations, and findings. The absence of such formal credit is an affirmation representing that the work is fully the writer’s. The term “sources” includes, but is not limited to, published or unpublished materials, lectures and lecture notes, computer programs, mathematical and other symbolic formulations, course papers, examinations, theses, dissertations, and comments offered in class or informal discussions, and includes electronic media. The representation that such work of another person is the writer’s own is plagiarism.

Care must be taken to document the source of any ideas or arguments. If the actual words of a source are used, they must appear within quotation marks. In cases that are unclear, the writer must take due care to avoid plagiarism.

The source should be cited whenever:
(a) a text is quoted verbatim
(b) data gathered by another are presented in diagrams or tables
(c) the results of a study done by another are used
(d) the work or intellectual effort of another is paraphrased by the writer

    Because the intent to deceive is not a necessary element in plagiarism, careful note taking and record keeping are essential in order to avoid unintentional plagiarism.

    For additional information, please consult “Avoiding and Detecting Plagiarism,” available in the Office of the Vice President for Student Affairs, the Provost’s Office, or at http://web.gc.cuny.edu/provost/pdf/AvoidingPlagiarism.pdf.

(From The Graduate Center Student Handbook 05-06, pp. 36-37)

Top
Schedule
Date
Reading Assignments Due
Assignments Due
W 9/1
 Course Overview, What is program evaluation research and how does it differ from behavioral science research?
  
W 9/8
 No classes scheduled after 4 PM.
W 9/15
 The purposes of program evaluation.
Weiss (W) Chapters 1-2.
  
W 9/22
 Evaluation as a transdiscipline
Scriven (S) Introduction: The Nature of Evaluation, "Logic of Evaluation".
W 9/29
 Understanding the program and planing the evaluation.
W3-4, S "Process Evaluation," AEA Guiding Principles. 		Memo 1:
Stakeholders & Concerns
W 10/6
 Roles of the evaluator, program measures.
W5-6
W 10/13
 Collecting data, Designing the evaluation. (Go over evaluation review assignment)
W7-8, Chapter Summaries for W9-10. (Note 1)
 Memo 2:
Evaluation Milestones
W 10/20 Qualitative methods, interpreting data.
W11-12
W 10/27
 Dissemination & Evaluation integrity.
W13-14, S "Ethics" & "Ethics in Evaluation."		 Evaluation Review
(due at start of class)
W 11/3
 Standards and Best Practices for Evaluation
Discuss Evaluation Review.
Standards Introduction & Applying the Standards (YSHC). Morris, Chapter 1 (M1), S "Key Evaluation Checklist"
W 11/10
 Propriety Standards
YSHC:Propriety, M3, S "Conflict of Interest"		 Thesaurus Travelogue
W 11/17
 Utility Standards
YSHC:Utility, M6-7 		Memo 3: Program Model
W 11/24
 Feasibility Standards
YSHC: Feasibility & Evaluation Accountability, M2
W 12/1
 Accuracy Standards 
YSHC: Accuracy, M4-5, 8		 Memo 4: Standards
W 12/8
 Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007
W 12/15
Come to class prepared to discuss your proposal.

 Evaluation Proposal

Note 1: The course is designed with the assumption that you have taken a basic research design course and are familiar with the designs discussed in these chapters. If not, you will want to read the full chapters. The last section of Chapter 10 may be useful in any event.

Top
Created August 18, 2008
Updated August 13, 2010; October 28, 2010