Keith
Markus'
Urban
Sprawl
PSYC U77000 BAR
Training and Evaluation
Syllabus
Fall 2008
Time: Wednesday 4:30-6:30 PM
Room: 13-160 Baruch Vertical Campus
Office Hours: Tuesday 4 PM to 5 PM.
Contact Information:
Dr. Keith A. Markus
kmarkus@aol.com (this is the
best way to reach me)
212-237-8784
Room 2127N
Psychology Department, John Jay College of Criminal Justice, 445 W 59th
Street, New York, NY 10019 USA
Course Description: This course provides a general
introduction to both training in organizations and program
evaluation. The training material emphasizes the importance of
planning and designing training as an organizational intervention based
on needs analysis and assessed by a formal evaluation. The
program evaluation material surveys general principles of program
evaluation applicable to other types of programs but uses evaluation of
training programs as the illustrative case. The course provides a
basic foundation in the literature one each of these two topics.
Text Books:
Donaldson, S. I. (2007). Program Theory-Driven
Evaluation Science: Strategies and Applications. Mahwah, NJ:
Erlbaum. (Now from Psychology Press)
Goldstein, I. L. & Ford, J. K.
(2002). Training
in organizations (4th ed.). Belmont, CA: Wadsworth.
The Joint Committee on Standards for Educational
Evaluation (1994). The
program evaluation standards (2nd ed.). Thousand Oaks,
CA: Sage Publications.
Kraiger, K. (2002). Creating
implementing, and managing effective training and development.
San Francisco, CA: Jossey-Bass. (Print on demand only.)
Additional Reading:
American Evaluation Association (2004). Guiding principles
for evaluators. http://www.eval.org/Publications/GuidingPrinciples.asp
Fetterman, D. & Wandersman, A. (2007). Empowerment
evaluation: Yesterday, today, and tomorrow. American Journal of
Evaluation, 28, 179-198.
Miller, R. L. & Campbell, R. (2006). Taking stock of
empowerment evaluation: An empirical review. American Journal of
Evaluation, 27, 296-319.
Reichardt, C. S. (2006). The
principle of parallelism in the design of studies to estimate treatment
effects. Psychological Methods, 11, 1-18.
Rogers, P. J. & Weiss, C. H. (2007). Theory-based
evaluation: Reflections ten years on: Theory-based evaluation: Past,
present, and future. New Directions in Program Evaluation, Summer,
63-81.
Smith, N. L. (2007). Empowerment evaluation as
evaluation ideology. American Journal of Evaluation, 28, 169-178.
I have ordered the books through the Baruch College bookstore.
Course Flow: Familiarize yourself
with the reading material before the corresponding lecture.
Lectures
will summarize and clarify the reading. In general, I would
rather answer your questions than lecture.
Examinations: The course does
not
include any examinations. Instead of examinations,
your course grade will reflect 6
reaction papers and 2 mini-papers.
Reaction
Papers:
Select any two readings due on or before the due date of the paper, but
not on or before the previous paper if there was one. Very
briefly describe the perspectives of the two readings on a common
topic. (This can come directly from the reading or it can involve
some interpolation or inference based on the reading.) Focus the
larger part of your paper on relating the two perspectives to one
another. This may involve contrasting the perspectives, drawing
non-obvious connections between them, or some combination of
both. Feel free to draw on materials from outside the course to
develop your ideas. Finally, conclude the paper with your
assessment of the common topic in relation to the two
perspectives. The grading continuum will run from papers that
fulfill the requirement in a perfunctory but uninteresting way at the
low end of passing grades (B-) to papers that pique my interest by
leading me to think about something in a new way at the high end, with
plenty of room in between the two extremes. (Aim for 2-4
pages. Don't go nuts.)
Mini-papers: The course requires two
mini-papers.
The length of the two papers combined should roughly correspond to what
you would ordinarily write for a term paper (25-30 pages).
Mini-paper One: Pick one of the case studies from the
Donaldson (2007) text (Chapters 5-9, 11). Once you chose, email
me
with your choice. When you receive an email in return verifying
the choice, you are ready to begin. Each student must select a
different case study and I will reserve them on a first-come
first-served basis. Once you have a case study, you have three
tasks to fulfill in your first paper: (1) convince me that you
understand the material from the course related to the aspect of the
case study that you choose to discuss, (2) convince me that you can
integrate this material with relevant extraneous material of your
choice, and (3) convince me that you can profitably apply both of these
to the specifics of the case study in question. (You do not need
to discuss every aspect of the case study that you choose.) You
have a free
hand in deciding how to convince me of these three things. You
can write a literature review, and essay, and analysis of the case
study, a theoretical analysis of the methodological principles at play,
a research proposal, an epic poem, a play, a graphic novel, or whatever
you think best suits the task. Whatever you write, however, keep
your eye on the prize: provide sufficient material to
unambiguously convince me of the above three points.
Mini-paper Two: Write a proposal responsive to the
following
request for proposals (RFP). For the purposes of the course
assignment, you may omit the supporting materials. You may also
turn in the paper by hand in class.
Request For Proposals
The City University of New York
Office of Institutional Development and Didactic Fictions
Distance Learning Training and Evaluation Initiative
Fall 2006
I. Introduction
The City University of New York comprises 21 separate units spread
across five boroughs which combine to form the nations largest urban
public university. Although distance learning has historically
served primarily rural populations in states with low population
densities, it also plays an important role in urban settings where the
advantages accrue more from flexibility with respect to time than from
the ability to reach across vast distances. Many CUNY students
work full or part time while completing degree programs and stand to
benefit from the flexibility afforded by asynchronous Web-based
courses. The University has invested in Blackboard software run
centrally on servers located in the University Computing Center
providing distance education infrastructure for use at all CUNY
colleges. Each campus has its own Blackboard coordinator who
serves as the contact person for faculty at that institution within
CUNY. This university-wide initiative seeks to increase the use
of Blackboard for Web-based courses within CUNY by training faculty in
the use of Blackboard.
II. Eligibility
Only students in this course may apply for this initiative.
III. Proposal Submission Process
Address proposals to Professor Keith A. Markus, Department of
Psychology, John Jay College of Criminal Justice, CUNY, 445 W59 Street,
New York, NY 10019 USA. In order to receive consideration, all
proposals must be received by Wednesday, December 17, 2008.
Electronic submissions will not be accepted.
IV. Available Funding
One training program will be funded for an amount up to $500,000 over
two years of which at least 20% must be dedicated to evaluation of the
training program. The funding period begins August 1 2009 and
ends August 1 2011.
V. Purpose
CUNY does not deem it feasible to train all faculty in the use of
Blackboard and many faculty already using Blackboard will not benefit
from additional training. The goal of this initiative is to
target CUNY faculty who would benefit from training and who are also
interested in and able to offer distance education courses. The
goal is to train a critical mass of these individuals in order to
facilitate an increase in the number of Web-based sections of courses
offered during regular semesters across the University. A further
task of this demonstration project is to provide evidence for the
efficacy of the training and recommendations for continued training
within CUNY. It is not a requirement that proposals provide
training to all 21 units of CUNY as part of this demonstration
project. However, they must provide a basis for sound decisions
about continued training that apply to all CUNY institutions.
VI. Proposal Format
Double space your entire proposal exclusive of attachments.
Sections C through G of your proposal should not exceed 12 pages total.
A. Title Page
Include the title of your proposal, the full names of the primary
author(s), and "Distance Learning Training and Evaluation Initiative
2006" on your title page.
B. Project Summary (200 words)
Summarize your proposed project here. Be sure to cover each of
sections C through G of the proposal. Summarize the content of
the proposal rather than describing what the proposal contains.
Be sure to convey a clear picture of the basic content of your proposal.
C. Target Population Analysis and Sampling/Recruitment Plan
Present a methodology for identifying individuals to receive
training. Clearly identify the number of proposed trainees and
how you will select them. Recognize that training is
voluntary. Include funding for training incentives in your
proposal (e.g., budget $4000 per adjunct course for faculty reassigned
time).
D. Training Needs Analysis
Present a plan for analyzing both the content needed to successfully
use Blackboard technology and also an analysis of the learning needs of
the target population identified in section C. Describe how you
will combine both sources of information to determine the specific
training content.
E. Training methodology
Describe the intended training methodology at a general level that
recognizes that you have not yet determined the precise content of the
training. Describe proposed instructional methods and explain how
your choice of methods addresses the purposes of the initiative.
Describe the overall structure and logistics of the training.
F. Evaluation
Enumerate the research questions that the evaluation will address and
justify your choices in terms of the purposes of the initiative.
Describe the research design and data analysis plan in sufficient
detail to demonstrate that it will address the research
questions. Describe how different patterns of results will
influence recommendations regarding future training.
G. Dissemination
Describe a dissemination plan to communicate the results of your
project. This should include an interim report due September 1,
2007 and a final report due September 1, 2008. However, it should
also include additional efforts to disseminate the results to
individual CUNY institutions during the two-year funding period.
H. Reference List
Use APA format for any references cited in the proposal.
I. Time Line
Detail a time line of activities during the grant period.
J. Budget
Provide a detailed budget listing proposed expenditures using the
following outline. Include 50% overhead on any expenditures that
go toward personnel. Include a budget total at the bottom and a
subtotal at the end of each section outlined below. (You may find
it helpful to put this part of the proposal in a spread sheet rather
than a word processor.)
1. Needs Analysis and Training Costs (not to exceed 80% of total)
a. Salary & Personnel
i. Salaries of grant personnel
ii. Any reassigned time for CUNY faculty
($4000/course)
iii. 50% overhead
b. Supplies and Materials
c. Other costs (e.g., transportation, telephone, etc.)
2. Evaluation Costs
a. Salary & Personnel
i. Salaries of grant personnel
ii. Any reassigned time for CUNY faculty
($4000/course)
iii. 50% overhead
b. Supplies and Materials
c. Other costs (e.g., transportation, telephone, etc.)
K. Budget Justification
Provide a brief narrative justification for key budget items.
L. List of Supporting Materials (Actual materials not required)
1. Resume or Curriculum Vitae of key personnel
2. IRB permissions to be sought
3. Letters of cooperation
4. Any additional materials
VII. Evaluation Criteria
Proposals will be evaluated on the following criteria.
1. Overall conceptualization and design: A summary judgment of
the degree to which the proposal fulfills initiative objectives.
2. Technical adequacy of training design.
3. Technical adequacy of evaluation design.
4. Usefulness of potential evaluation results to decision making.
5. Justifications for design decisions and budget.
6. Overall clarity and precision of presentation.
(End of RFP)
Grading: Mini papers count 50% (25% each) and
reaction
papers count 50% (8.33% each).
Letter Grade
|
Percent Grade
|
A
|
92-100
|
A-
|
84-91
|
B+
|
76-83
|
B
|
68-75
|
B-
|
60-67
|
C+
|
52-59
|
C
|
44-51
|
C-
|
36-43
|
F
|
0-35
|
Schedule
Date
|
Reading Assignments Due
|
Assignments Due
|
W 8/27
|
Course Overview, Training and Evaluation in Organizational
Context
|
|
W 9/3
|
Beginning at the Beginning: Training in Organizations
Goldstein & Ford Chapters 1&2 (GF1&2) and Kraiger Chapter 1
(K1)
|
|
W 9/10
|
Needs Assessment, Training and Diversity
GF3 and K2&5
|
|
W 9/17
|
Planning Training in Context
GF4 and K3&9
|
Reaction Paper 1
|
W 9/24
|
Introduction to Evaluation
G5, AEA Guiding Principles, and Standards
Introduction & Applying the Standards (S)
|
|
W 10/1
|
No classes scheduled.
|
|
W
10/8 |
No
classes scheduled. |
|
W
10/15
|
Evaluation
Design
G6, Reichardt 2006, K11 |
Reaction Paper 2 |
W 10/22
|
Utility &
Feasibility Standards
Su&f |
|
W 10/29
|
Propriety & Accuracy Standards
Sp&a |
Reaction Paper 3
|
W 11/5
|
Program Models and Theory-based Evaluation
Donaldson 1-3 (D1-3).
|
Mini-paper 1
|
W 11/12
|
Critical Reflections on Theory-based Evaluation
D 12,13,Rogers 2007, Weiss 2007 |
Reaction Paper 4
|
W 11/19
|
Empowerment Evaluation
Fetterman and Wandersman, 2007; Miller and Campbell, 2006; Smith, 2007
|
|
W 11/26 (really)
|
Training and Instruction
G7 and K7
|
Reaction Paper 5
|
W 12/3
|
Kinds of Organizational Training & Objectives
G8 and K6&8
|
|
W
12/10
|
Learning
Organizations and Wider Contexts
G9 and K10
|
Reaction Paper 6 |
W 12/17
|
(Final Examination Period.)
Regular course meeting. Come prepared to talk about your proposal
(Mini-paper 2).
|
Mini-paper 2 |
Created July 27, 2006
Updated August 25, 2008