Appendix A: Report of the Subcommittee on the Evaluation of Teaching

Members: David Edwards, Susan Frost, Linda Gooding, David Kleinbaum (chair), Robert Pastor; Jodi Cressman (assistant).

In drafting this report, we have reviewed the literature on teaching evaluation and other materials on current related practices, criteria, and resources at Emory and at other universities. We also have considered comments and suggestions from faculty gathered during a 1995-1996 luncheon series organized by the provost, from the executive committee of the Commission on Teaching, and from subcommittee members. In making our recommendations, we recognize and affirm the diverse characteristics of the various schools at Emory. Consequently, these recommendations do not include strict criteria for all units to follow. Rather, we are setting forth global criteria and recommend that each school have some flexibility in designing its approach. For example, we recommend that each tenure and tenure-track faculty member have a teaching portfolio, and we list types of teaching activities and materials that may be included. We leave it to the individual school, department, or program, however, to specify how the portfolio is maintained, what information is appropriate, what form the information should take, and what schedule for review should be adopted.

I. DIMENSIONS OF TEACHING

Before making recommendations about teaching evaluation, it is important to determine, both generally and specifically, what characterizes teaching at Emory. Teaching at a university involves far more than lecturing in a classroom. The university represents a number of disciplines that require different kinds of learning in a variety of settings; therefore, different approaches to teaching are required. The following four dimensions seem to encompass teaching at Emory: (1) classroom teaching; (2) mentoring students and/or faculty, including thesis and dissertation supervision of graduate students; academic, career, and other advisement of students; helping other faculty improve teaching; (3) teaching-related contributions to one's field, including the development of textbooks and other teaching materials, and publications on teaching in one's field; and (4) other kinds of teaching at or outside Emory, including teaching in clinics, teaching in the field (e.g., to the general public), continuing education, and short courses.

We use the term dimensions to emphasize the variety of approaches that exist within each item listed. For example, teaching in the classroom can be carried out through lecturing, viewing and discussing audio-visual presentations, or using interactive approaches such as round-table seminars, supervised laboratory work, or hands-on computer tutorials. Mentoring can be carried out for either students or faculty and can involve thesis or dissertation supervision, academic or career counseling, or clinical consultation. Some schools or departments might consider textbooks, teaching materials, and publications on teaching as work done toward improvement of teaching. Even though they do not involve face-to-face contact with students, such scholarship or research may have an important impact on how teaching is carried out by professionals in one's field. Similarly, there are settings outside the traditional classroom where face-to-face teaching can have an important impact-- for example, in short courses at the national or international level.

II. Elements of "Good Teaching"

The form teaching takes can vary with the educational level of the student and the setting in which the teaching occurs. While some characteristics of "good teaching" apply to both secondary school and the university, one should not underestimate the difference in student capabilities and in the purpose of educational exercise. Similarly, the model of "good teacher" in an undergraduate liberal arts course at Emory College is quite different from one in the School of Medicine. Nonetheless, we suggest that general elements of "good teaching" exist in two categories: (1) the outcomes desired by the student, and (2) the outcomes expected of the teacher.

Outcomes desired by the student. The first task of a good teacher is to transmit information and knowledge to students in a manner that communicates the essential facts and the range of interpretation of those facts. Second, a good teacher ought to help students learn how to think and apply knowledge both to historical and contemporary problems, situations, or challenges. Third, the teacher ought to motivate students to learn more about the subject and-- what is even more significant-- should pique their curiosity and help them appreciate learning for its own sake. Fourth, students ought to be taught that a major part of education is learning to collaborate with other students and faculty. Fifth, students ought to learn how to be more responsible and to cooperate with the instructor and other students as part of the educational experience. And finally, a good teacher ought to instill in students a sense of the value of learning as important to their education and to their lives. Education should not end with the college or postgraduate degree; a student's curiosity should lead to further learning.

Outcomes expected of the teacher. A person is not born a good teacher, although there are people who bring to the profession certain qualities of inspiration, vigor, and enthusiasm that would make them good teachers. Whether the teacher is born with those qualities or tries to learn them, a good teacher needs to prepare thoroughly. We have identified the following eight indicators of good teaching. First, a teacher needs to be well organized; he or she needs to plan thoroughly for the course and choose the appropriate teaching methods. Second, the teacher should state clearly the goals of the course and should communicate the material to the students in a clear and effective way. Third, a good teacher should use the best teaching materials-- e.g., texts, case studies-- and should design assignments that evoke thoughtful analyses. Fourth, exams should be fair and appropriate; grading should be fair; and teachers ought to find ways to provide feedback to students regarding both correct and incorrect answers. Fifth, teachers ought to cultivate good rapport with students. Sixth, a good teacher will leave such a positive impression on students that they would recommend the course to other students. Seventh, students ought to master the material of the course. And finally, at the end of the course, students ought to have a more elevated view of the value of learning.

III. Goals of Teaching Evaluation and Focus of This Report

With the dimensions of teaching and the elements of good teaching in mind, the subcommittee has identified four distinct goals for the evaluation of teaching at Emory: (1) to improve the classroom teaching of individual faculty; (2) to assess teaching performance of individual faculty as regards appointments, promotion, tenure, and salary issues; (3) to rate the success of a teaching program in a department, school, or college; and (4) to provide a basis of information for rewarding outstanding teaching.

The first two goals are the foci of this report.1 As foundation for goals one and two, it is helpful to clarify two distinct purposes of evaluation. Traditional evaluation can be of two types: summative and formative. Summative evaluation is designed to provide a foundation for decisions about advancement (promotion, tenure, or salary enhancement) or, at times, about retention of a faculty member in association with a program or position. Formative evaluation is designed to bring about improvement in performance over time.2 It is important to note that the nature of goal one-- to improve the classroom teaching of individual faculty-- is formative. Goal two-- to assess the teaching performance of individual faculty with regard to appointments, promotion, tenure, and salary issues-- is by nature a gate-keeping activity that requires summative evaluation.

In the course of our work, the subcommittee has learned that, particularly when compared to evaluation of research performance, many Emory faculty view evaluation of teaching as a subjective process that has unclear or narrowly defined purposes or places too much value on one information source: student opinion. In this report, we espouse a more complete and objective assessment of teaching-- one that is collaboratively defined and purposeful. It may have both formative and summative aspects and-- in addition to student evaluations-- should include peer review, student exit interviews, and documentation of teaching activities in the four dimensions of teaching.

Peer review is a form of teaching evaluation that can provide valuable information to promote teaching improvement (a formative process). When carefully crafted and well understood by the evaluating unit and the teacher being evaluated, it also may be possible to use peer review to evaluate teaching performance for the purpose of tenure decisions, renewal of contract, and/or salary adjustments (a summative process). Although some respondents report success in using peer review to serve these dual purposes, others caution that peer review is weakened when the summative aspect is included. Those who use peer review seem to agree that its integrity depends in part upon preliminary discussion about the purpose or purposes of the review and the use or uses of the information gathered.3

IV. Recommendations Regarding Evaluation to Improve Classroom Teaching of Individual Faculty

The subcommittee has learned that support for quality teaching exists across the university in a number of forms and is designed to accomplish a number of purposes; however, there is no central repository of information about this support, no uniformly clear direction for making use of it, and in some cases no articulate purpose that the support is designed to accomplish. The subcommittee recommends that the university and the schools, through the following four measures, support evaluation of teaching for the purpose of improvement in the course of time.

First, the university should develop and maintain a listing of current Emory- and school-specific resources that support teaching and should give clear information about how individual faculty can access such support. The list should be updated at least yearly and made available to faculty. The subcommittee encourages the university, the schools, and other groups that provide tangible support for teaching to articulate the purposes that support is designed to accomplish. In some cases, the sponsoring group will need to define more clearly the nature of the support to assure that it will advance the intended purpose.

Second, the resources should be expanded to include a university teaching center . This center should be funded by the university and designed to serve its constituents. Faculty representing each school and certain interdisciplinary programs should be appointed to guide the center. The subcommittee urges the university to organize the center primarily around goals of nurturing and improving teaching over time rather than providing a basis for judging the outcomes of instruction. The center should provide instructional specialists or other educational consultants for those who seek their services.

Third, each school should develop a locally organized center or office for teaching support similar to the Emory College Center for Teaching and the Curriculum. These centers or offices should coordinate work with the university center and should enrich overall support for teaching by providing school- or discipline-specific expert help.

Fourth, all schools and departments should develop and implement peer review as a means to improve teaching performance. When used to improve teaching performance rather than punish poor teaching, peer review provides a way for faculty at all ranks to exchange ideas as well as offer feedback and suggestions in a helpful but nonjudgmental way.4 The exact structure of the review may vary widely. Some departments may find it most helpful to ask faculty to observe their peers in the classroom and offer feedback in conversation;5 others may ask faculty to provide a somewhat informal document that can be included in a teaching portfolio. Some universities with an informal peer-review system ask senior faculty to help junior faculty identify teaching weaknesses before they are reviewed for tenure; others have one or more faculty visit and critique every new course, regardless of the rank of its professor. In any case, the procedures of peer review should involve coordination among faculty being reviewed, faculty reviewers, and educational specialists or consultants who can provide guidelines for the review.

Although the subcommittee recommends that peer review take place across the university, it suggests that each school determine the appropriate structural design of the process and implementation of both kinds of review. It is important that each school, department, or program make clear the purpose of the review-- that is, whether it is for strictly developmental or for summative purposes.6

When peer review is used to inform decisions about tenure, renewal of contract, or salary adjustments, the process becomes summative and therefore a more formal structure is required to ensure fairness and reliability. Again, the diverse missions and goals of the schools and departments demand that the review be acknowledged as an activity that supports decision making and that its specific form be designed at the local level. When peer review is used to inform decision making, it is critical that (1) each school, department, or program include its faculty in the process of designing the peer-review system; (2) the system be tested before officially being adopted; (3) faculty are trained properly in the principles and practice of peer review; (4) peer reviewers work in teams of at least two colleagues and/or one colleague and one education specialist; (5) peer reviewers use a standard form to report in order to ensure maximum uniformity among observers; (6) procedures require preobservation and postobservation conferences between the visiting team and the teacher; (7) copies of peer-review reports are sent to the teacher and maintained in the teacher's dossier (for his/her reflection and response) and in the teacher's portfolio (for use in tenure, promotion, and salary reviews); and (8) procedures for departmental or school peer review are coordinated with university or school instructional-support resources when possible.

V. Recommendations Regarding Evaluation to Assess Teaching Performance of Individual Faculty

There is considerable variation across the university regarding the process and criteria used to evaluate teaching performance of individual faculty as well as the weight given to teaching for the purposes of determining salary and making decisions about promotion and tenure. We describe below the approaches that Emory's various schools take toward teaching evaluation. Then we address how the university can achieve some normalization of the evaluation process in order to allow for baseline quality control over all schools.

A. Current Practices of Teaching Evaluation at Emory

In Emory College, professors distribute the Emory College Course and Instructor Evaluation form to students in each of their classes. These forms are collected and scored by computer. Each department chair receives summaries for courses in his/her department. These summaries provide feedback to instructors about their teaching performance and are used to evaluate classroom teaching for purposes of decisions about salary and promotion/tenure. As a matter of policy,7 promotion and tenure are based upon either a combined strength in teaching and research requiring "excellent" achievement or promise in either teaching or research and at least "very good" achievement and promise in the other. With only a single departmental exception, faculty perceive teaching to carry substantially less weight than research in considerations about annual salary and promotion.8

The Graduate School of Arts and Sciences expects each department or program to have a formal procedure for evaluating teaching that is consistent with the principles set forth in the Report of the Committee on Evaluation of Graduate Teaching.9 The report recommended that all graduate courses and seminars be evaluated in a written form explicitly different from the form used to evaluate teaching at Emory College.

Teaching within the School of Medicine comprises a broad spectrum of activities, ranging from large, didactic lecture courses to clerkships in which students rotate through clinical disciplines in close association with residents and faculty members. The quality of individual teaching receives attention primarily within departments and varies accordingly. Considering the size of the school and the number of departments, it is not surprising that practice differs widely. For example, some departments do extensive student and peer faculty evaluations and give systematic feedback to faculty. Other departments present evaluation forms to professional school students as the sole source of information about teaching effectiveness and restrict peer evaluation to informal evaluation by course directors. Still other departments do nothing at all. The importance of teaching in salary and promotion/tenure decisions is determined by the departmental chairperson, thereby causing additional variation. For example, most faculty in School of Medicine basic science departments (rather than in clinical departments) do some or all of their teaching in the Graduate School Division of Biological and Biomedical Sciences (GDBBS). However, all review for promotion and tenure takes place in a faculty member's School of Medicine department, which may place less value on teaching in the GDBBS, as its students are not enrolled in the School of Medicine.

In the Law School, each student in every class is asked to complete a Student Evaluation of Teacher (SET) form, which includes several standard questions and invites general comments. These forms are available to review by the instructor, the dean and, where appropriate, the members of the promotion and tenure committee. SET forms are reviewed by the dean as part of the annual review of each faculty member and can be useful in identifying problems in classroom teaching. Members of the promotion and tenure committee regularly visit the classes of junior faculty; junior faculty are invited to visit classes of more senior professors; and tenured faculty periodically visit each other's classes. These visits are intended to lead to fruitful discussion about teaching methods. The candidate is given a candid summary of the evaluations both of students and faculty colleagues. Both forms and class visitations are important in the promotion process.

In the School of Nursing, student evaluations of course/teacher are obtained for each course on a standardized form. These forms are considered part of the annual merit review of each faculty member. The forms and criteria for promotion and tenure, as they pertain to teaching, are currently under revision.

Near the end of each semester, faculty in the Goizueta Business School distribute a standard evaluation form to students in their classes. Results are summarized and forwarded to the appropriate area coordinator and to the dean. Each faculty member writes a commentary on teaching as part of his/her annual report. Student ratings and faculty commentaries are used to evaluate teaching on an annual basis.

In the Rollins School of Public Health, new criteria for assessing teaching as part of the overall criteria for appointment, promotion, and tenure are currently under review by department chairs and center directors. Salary decisions are made by the department chair (with approval of the dean), and there are at present no formal guidelines for factoring teaching performance into this process.

In the School of Theology, junior faculty are required to have students complete teaching evaluation forms for each class taught. Summaries of these forms are submitted with other materials during tenure review. Senior faculty have one course per year evaluated, and summaries are used in promotion review. Peer reviews of academic performance are held every three years for non-tenure- and tenure-track faculty; peer reviews for tenured faculty are held every five years. Faculty are expected to report on their own teaching for purposes of this review. Each year the associate dean of academic affairs makes recommendations to the dean about annual salary adjustments for each faculty member. These recommendations are based in part on an evaluation of teaching performance.

At Oxford College, teaching ability is evaluated primarily on professional competence and on teaching effectiveness; however, also considered are other important activities, some of which are considered equivalent to teaching (for which released time is given). Evaluation of these activities may involve present and former students, other faculty members, the dean, and outside professional evaluators. In addition to the individual's self-evaluation, the college considers all teaching evaluations previously collected from students, former students, and peers; three letters of recommendation from former students; and two letters of recommendation from professional colleagues. (Letters of recommendation are submitted to the person being evaluated.) Other pertinent materials such as teaching aids (e.g., textbooks, lecture notes), student papers, other work supervised by the individual, the development of new courses, and attendance at training institutes also may be included.

B. Recommendations for the Evaluation Process

While it is essential to allow each school flexibility, we also recognize the need for the university to require that certain basic standards be met by each school or program. We therefore recommend the following: