Keith Markus' Urban Sprawl

  PSYC U77000 BAR
Training and Evaluation
Course Information
CUNY Blackboard
Note: Link may change due to Bb8 migration.
American Evaluation Association
Site Map

Fall 2008

Time:  Wednesday 4:30-6:30 PM
Room:  13-160 Baruch Vertical Campus
Office Hours:  Tuesday 4 PM to 5 PM.

Contact Information:
Dr. Keith A. Markus  (this is the best way to reach me)
Room 2127N
Psychology Department, John Jay College of Criminal Justice, 445 W 59th Street, New York, NY 10019 USA

Course Description:  This course provides a general introduction to both training in organizations and program evaluation.  The training material emphasizes the importance of planning and designing training as an organizational intervention based on needs analysis and assessed by a formal evaluation.  The program evaluation material surveys general principles of program evaluation applicable to other types of programs but uses evaluation of training programs as the illustrative case.  The course provides a basic foundation in the literature one each of these two topics.

Text Books:
Donaldson, S. I. (2007). Program Theory-Driven Evaluation Science: Strategies and Applications. Mahwah, NJ: Erlbaum. (Now from Psychology Press)

   Goldstein, I. L. & Ford, J. K. (2002).  Training in organizations (4th ed.).  Belmont, CA:  Wadsworth.

    The Joint Committee on Standards for Educational Evaluation (1994).  The program evaluation standards (2nd ed.).  Thousand Oaks, CA:  Sage Publications.

    Kraiger, K. (2002).  Creating implementing, and managing effective training and development.  San Francisco, CA:  Jossey-Bass. (Print on demand only.)
Additional Reading:
   American Evaluation Association (2004). Guiding principles for evaluators.

   Fetterman, D. & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28, 179-198.

   Miller, R. L. & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319.

   Reichardt, C. S. (2006). The principle of parallelism in the design of studies to estimate treatment effects.  Psychological Methods, 11, 1-18.

   Rogers, P. J. & Weiss, C. H. (2007). Theory-based evaluation: Reflections ten years on: Theory-based evaluation: Past, present, and future. New Directions in Program Evaluation, Summer, 63-81.

   Smith, N. L. (2007).  Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28, 169-178.

I have ordered the books through the Baruch College bookstore.

Course Flow:  Familiarize yourself with the reading material before the corresponding lecture.  Lectures will summarize and clarify the reading.  In general, I would rather answer your questions than lecture.

Examinations:  The course does not include any examinations.  Instead of examinations, your course grade will reflect 6 reaction papers and 2 mini-papers.

Reaction Papers:  Select any two readings due on or before the due date of the paper, but not on or before the previous paper if there was one.  Very briefly describe the perspectives of the two readings on a common topic.  (This can come directly from the reading or it can involve some interpolation or inference based on the reading.)  Focus the larger part of your paper on relating the two perspectives to one another.  This may involve contrasting the perspectives, drawing non-obvious connections between them, or some combination of both.  Feel free to draw on materials from outside the course to develop your ideas.  Finally, conclude the paper with your assessment of the common topic in relation to the two perspectives.  The grading continuum will run from papers that fulfill the requirement in a perfunctory but uninteresting way at the low end of passing grades (B-) to papers that pique my interest by leading me to think about something in a new way at the high end, with plenty of room in between the two extremes.  (Aim for 2-4 pages.  Don't go nuts.)

Mini-papers:  The course requires two mini-papers.  The length of the two papers combined should roughly correspond to what you would ordinarily write for a term paper (25-30 pages).

Mini-paper One:  Pick one of the case studies from the Donaldson (2007) text (Chapters 5-9, 11).  Once you chose, email me with your choice.  When you receive an email in return verifying the choice, you are ready to begin.  Each student must select a different case study and I will reserve them on a first-come first-served basis.  Once you have a case study, you have three tasks to fulfill in your first paper:  (1) convince me that you understand the material from the course related to the aspect of the case study that you choose to discuss, (2) convince me that you can integrate this material with relevant extraneous material of your choice, and (3) convince me that you can profitably apply both of these to the specifics of the case study in question.  (You do not need to discuss every aspect of the case study that you choose.)  You have a free hand in deciding how to convince me of these three things.  You can write a literature review, and essay, and analysis of the case study, a theoretical analysis of the methodological principles at play, a research proposal, an epic poem, a play, a graphic novel, or whatever you think best suits the task.  Whatever you write, however, keep your eye on the prize:  provide sufficient material to unambiguously convince me of the above three points.

Mini-paper Two:  Write a proposal responsive to the following request for proposals (RFP).  For the purposes of the course assignment, you may omit the supporting materials.  You may also turn in the paper by hand in class.

Request For Proposals
The City University of New York
Office of Institutional Development and Didactic Fictions
Distance Learning Training and Evaluation Initiative
Fall 2006

I. Introduction

The City University of New York comprises 21 separate units spread across five boroughs which combine to form the nations largest urban public university.  Although distance learning has historically served primarily rural populations in states with low population densities, it also plays an important role in urban settings where the advantages accrue more from flexibility with respect to time than from the ability to reach across vast distances.  Many CUNY students work full or part time while completing degree programs and stand to benefit from the flexibility afforded by asynchronous Web-based courses.  The University has invested in Blackboard software run centrally on servers located in the University Computing Center providing distance education infrastructure for use at all CUNY colleges.  Each campus has its own Blackboard coordinator who serves as the contact person for faculty at that institution within CUNY.  This university-wide initiative seeks to increase the use of Blackboard for Web-based courses within CUNY by training faculty in the use of Blackboard.

II. Eligibility

Only students in this course may apply for this initiative.

III. Proposal Submission Process

Address proposals to Professor Keith A. Markus, Department of Psychology, John Jay College of Criminal Justice, CUNY, 445 W59 Street, New York, NY 10019 USA.  In order to receive consideration, all proposals must be received by Wednesday, December 17, 2008.  Electronic submissions will not be accepted.

IV. Available Funding

One training program will be funded for an amount up to $500,000 over two years of which at least 20% must be dedicated to evaluation of the training program.  The funding period begins August 1 2009 and ends August 1 2011.

V. Purpose

CUNY does not deem it feasible to train all faculty in the use of Blackboard and many faculty already using Blackboard will not benefit from additional training.  The goal of this initiative is to target CUNY faculty who would benefit from training and who are also interested in and able to offer distance education courses.  The goal is to train a critical mass of these individuals in order to facilitate an increase in the number of Web-based sections of courses offered during regular semesters across the University.  A further task of this demonstration project is to provide evidence for the efficacy of the training and recommendations for continued training within CUNY.  It is not a requirement that proposals provide training to all 21 units of CUNY as part of this demonstration project.  However, they must provide a basis for sound decisions about continued training that apply to all CUNY institutions.

VI. Proposal Format

Double space your entire proposal exclusive of attachments.  Sections C through G of your proposal should not exceed 12 pages total.

A. Title Page
Include the title of your proposal, the full names of the primary author(s), and "Distance Learning Training and Evaluation Initiative 2006" on your title page.

B. Project Summary (200 words)
Summarize your proposed project here.  Be sure to cover each of sections C through G of the proposal.  Summarize the content of the proposal rather than describing what the proposal contains.  Be sure to convey a clear picture of the basic content of your proposal.

C. Target Population Analysis and Sampling/Recruitment Plan

Present a methodology for identifying individuals to receive training.  Clearly identify the number of proposed trainees and how you will select them.  Recognize that training is voluntary.  Include funding for training incentives in your proposal (e.g., budget $4000 per adjunct course for faculty reassigned time).

D. Training Needs Analysis

Present a plan for analyzing both the content needed to successfully use Blackboard technology and also an analysis of the learning needs of the target population identified in section C.  Describe how you will combine both sources of information to determine the specific training content.

E. Training methodology

Describe the intended training methodology at a general level that recognizes that you have not yet determined the precise content of the training.  Describe proposed instructional methods and explain how your choice of methods addresses the purposes of the initiative.  Describe the overall structure and logistics of the training.

F. Evaluation

Enumerate the research questions that the evaluation will address and justify your choices in terms of the purposes of the initiative.  Describe the research design and data analysis plan in sufficient detail to demonstrate that it will address the research questions.  Describe how different patterns of results will influence recommendations regarding future training.

G. Dissemination

Describe a dissemination plan to communicate the results of your project.  This should include an interim report due September 1, 2007 and a final report due September 1, 2008.  However, it should also include additional efforts to disseminate the results to individual CUNY institutions during the two-year funding period.

H. Reference List

Use APA format for any references cited in the proposal.

I. Time Line

Detail a time line of activities during the grant period.

J. Budget

Provide a detailed budget listing proposed expenditures using the following outline.  Include 50% overhead on any expenditures that go toward personnel.  Include a budget total at the bottom and a subtotal at the end of each section outlined below.  (You may find it helpful to put this part of the proposal in a spread sheet rather than a word processor.)

1. Needs Analysis and Training Costs (not to exceed 80% of total)
  a. Salary & Personnel
    i. Salaries of grant personnel
    ii. Any reassigned time for CUNY faculty ($4000/course)
    iii. 50% overhead
  b. Supplies and Materials
  c. Other costs (e.g., transportation, telephone, etc.)
2. Evaluation Costs
  a. Salary & Personnel
    i. Salaries of grant personnel
    ii. Any reassigned time for CUNY faculty ($4000/course)
    iii. 50% overhead
  b. Supplies and Materials
  c. Other costs (e.g., transportation, telephone, etc.)

K. Budget Justification

Provide a brief narrative justification for key budget items.

L. List of Supporting Materials (Actual materials not required)
1. Resume or Curriculum Vitae of key personnel
2. IRB permissions to be sought
3. Letters of cooperation
4. Any additional materials

VII. Evaluation Criteria

Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design:  A summary judgment of the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of training design.
3. Technical adequacy of evaluation design.
4. Usefulness of potential evaluation results to decision making.
5. Justifications for design decisions and budget.
6. Overall clarity and precision of presentation.

(End of RFP)

Grading:  Mini papers count 50% (25% each) and reaction papers count 50% (8.33% each).

Letter Grade
Percent Grade

Reading Assignments Due
Assignments Due
W 8/27
Course Overview, Training and Evaluation in Organizational Context
W 9/3
Beginning at the Beginning:  Training in Organizations
Goldstein & Ford Chapters 1&2 (GF1&2) and Kraiger Chapter 1 (K1)
W 9/10
Needs Assessment, Training and Diversity
GF3 and K2&5

W 9/17
Planning Training in Context
GF4 and K3&9
Reaction Paper 1
W 9/24
Introduction to Evaluation
G5, AEA Guiding Principles, and Standards Introduction & Applying the Standards (S)

W 10/1
No classes scheduled.

W 10/8 No classes scheduled.
W 10/15
Evaluation Design
G6, Reichardt 2006, K11
Reaction Paper 2
W 10/22
Utility & Feasibility Standards

W 10/29
Propriety & Accuracy Standards
Reaction Paper 3
W 11/5
Program Models and Theory-based Evaluation
Donaldson 1-3 (D1-3).
Mini-paper 1
W 11/12
Critical Reflections on Theory-based Evaluation
D 12,13,Rogers 2007, Weiss 2007
Reaction Paper 4
W 11/19
Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007

W 11/26 (really)
Training and Instruction
G7 and K7
Reaction Paper 5
W 12/3
Kinds of Organizational Training & Objectives
G8 and K6&8

W 12/10
Learning Organizations and Wider Contexts
G9 and K10
Reaction Paper 6
W 12/17
(Final Examination Period.)  Regular course meeting. Come prepared to talk about your proposal (Mini-paper 2).

Mini-paper 2

Created July 27, 2006
Updated August 25, 2008