Participants at HHMI's undergraduate science education conference discuss techniques for determining the impact of their efforts.
How do you know you are making a difference?
That question was at the heart of this year's Undergraduate Program Directors Meeting. The 140 participants from colleges, universities, foundations and other organizations focused on one of the hottest topics in education today: assessment. They explored the best ways to determine the impact of their efforts, and those of others, to improve undergraduate science education.
As Robert Lichter of the Camille and Henry Dreyfus Foundation noted, assessments have many different forms and purposes. They can probe for changes in students, in teachers, in programs or in higher education more generally. The tools used for assessment are similarly diverse. The challenge, according to Lichter, is to be clear from the outset "who wants to know what about whom, why, and for whose benefit."
The participants, who met at the Institute on September 30-October 2, had many lively discussions trying to answer these questions.
The Impact of Grants
For foundations and other funders, the purpose of assessments can be straightforward. "We assess to determine the value of our investments," said Martha Peck of the Burroughs Wellcome Fund. Funders also want to know where their investments will be most effective. If the goal is to produce superb scientists, for instance, should funders concentrate their support on students at the elementary, high school, college or graduate levels?
Colleges and universities also have investments to monitor. Yet not every campus has the same goal. Institutions serving large numbers of minority students may be most interested in retention and degree attainment, while those with a heavy emphasis on research may be more interested in how many students go on to graduate and professional schools. Students have different starting points, said Alan Gubanich of the University of Nevada, Reno, "and you can't assume that a single goal is the definition of success."
Too often, though, assessments measure the wrong things, or provide an incomplete picture. "Our language about assessment does not reflect the value of science graduates who do not go into science," said Aaron Ellison of Mount Holyoke College.
Examining these broader populations, however, is challenging--both statistically and practically. For instance, one of the most important things a faculty member might do for the future of science is to provide a research experience for a student who later becomes a legislator. Yet there is no easy way to quantify such an accomplishment.
Similarly, it can be difficult to evaluate science outreach programs for elementary school students or high school teachers. Participants at the meeting, however, did offer several ideas: Youngsters in elementary school programs might be asked some time later if they are interested in science, and their answers compared to a control group. K-12 teachers who receive training can be asked what materials and ideas they have actually incorporated into their classrooms.
Another important kind of assessment involves current and prospective faculty members who must be evaluated for appointment, promotion and rewards. Familiar measures such as grants, publications and scientific awards, which typically emphasize research accomplishments, are inadequate if one also wants to know how effective the person is with students.
Many measures exist to assess teaching, including student evaluations, peer review, class enrollments, teaching awards, exit polls and long-term tracking. But meeting participants said these and other tools must be refined before they can be adopted widely. "We have to find much better ways to assess teaching," said Susan Henry of Carnegie Mellon University, "because if we can't assess teaching we will never value it adequately." Participants also noted that teaching might be valued more if its products, such as innovative educational technology, were subject to the same process of outside peer review as scientific publications.
The problem of assessing successful teaching leads to the larger issue of how to identify "success" more generally.
Some measures of success, like number of majors or grade point averages, are fairly objective, said Mary Allen of Wellesley College. Others, like a person's attitudes or satisfaction with a career, may be more subjective. Participants discussed how to blend numerical and non-numerical measures, and also how to tap into promising assessment tools being pioneered at the K-12 level, such as portfolios of students's work.
Keynote speaker Daniel Koshland of the University of Cailfornia, Berkeley, the former editor of Science, said one assessment that needs a fresh look is the predictions of faculty members about their students. "Scientists who regularly have to choose students for admission to graduate school and grade them rarely spend time after the admissions meeting to evaulate their own selection process. Complete files should be evaluated to correlate individual success or failure with such items as grades in elementary school and high school, research experiences, letters of recommendation from research advisers and so forth," Koshland said. "We may find correlations that we didn't expect that would help us in giving advice to students."
Joseph Perpich, the Institute's vice president for grants and special programs, also emphasized the importance of tracking future careers of students as part of the assessment of science education programs. "Participants at this meeting stressed the need for sharing information on programs and assessment activities among both funders and the grantee institutions," he said. "Through grants program meetings, we have undertaken to do so. In addition, through the power of the Internet and the World Wide Web, we shall disseminate program information and assessment results to help inform future development of science education programs at all levels."
Measuring What's Measurable
According to Michael Gaines of the University of Miami, effective assessments require three things: focused objectives, methods used to achieve those objectives and preestablished outcomes that can be measured. "These may not be sufficient for success," said Gaines, "but they are necessary."
Gaines has used these criteria to assess a program he directs at the University of Miami that provides bridging experiences for potential transfer students from Miami-Dade Community College. Ongoing assessments have demonstrated lower attrition, higher student satisfaction and higher grade point averages among bridge students compared to a control group.
But the experience also demonstrates problems common to assessments, said Gaines, who spoke at the conference with one of his successful transfer students, Betty Blanco. First, he said, purely quantifiable outcome measures may not be able to capture the goals in which one is most interested. "You tend to measure things that are easy to measure, but those are not necessarily the most important things."
Also, because students themselves have different goals, their final outcomes reflect different rates of progress and therefore of success. Finally, asked Gaines, "How do you measure the unmeasurable?" Attributes like ambition and self-confidence can be the most difficult to quantify yet also may be the outcomes most desired.
A related question discussed extensively was whether scientists should attempt to perform assessments themselves or rely on outside expertise. "We're all amateurs at this," said Mary Ann Rankin of the University of Texas at Austin. "Each of us is reinventing the wheel."
Some meeting participants said they have worked with colleagues from their campus sociology or psychology departments to design and administer assessments. Others have relied on private firms or on higher education associations like Project Kaleidoscope. "There are only 24 hours in the day, so we need to rely on experts in assessment," said Donald Goldberg of Occidental College. "But it's our responsibility to work in partnership with them so they can apply their expertise to what we do."
The Limits of Assessment
Meeting participants also acknowledged that assessments cannot measure everything. Keith Amos, a student at Harvard Medical School and another of several students who spoke at the meeting, made that point with a story. As a student at a small junior high school in Louisiana, he wrote a letter to Xavier University of Louisiana asking about summer science programs. In reply, he got a personal letter from Deidre Labat, the head of the programs, telling him about them and asking him to apply. "I still have that letter today," he said, adding how difficult it is to quantify the value of that letter.
Assessments may be difficult and uncertain, attendees at the meeting concluded. Nevertheless, they can serve an important function quite apart from the results they produce. They can focus faculty members on the objectives of education and on the best ways of attaining those objectives. "It's important to build in assessment right from the beginning," said Rankin. "If I had thought about assessment first, I would have structured my program differently."