Indirect Assessments: How Useful Are They?

Student learning is typically measured using direct or indirect methods. Direct measures provide clear evidence of what students have and have not learned, evidence that assessment leader Linda Suskie says a skeptic would trust.   

In contrast, indirect evidence may reflect students’ perceptions of what they learned or what they probably learned. In the world of assessment, indirect evidence is considered less compelling and less reliable than direct evidence.

I do not dismiss indirect evidence as quickly as some of my assessment colleagues do. Grades are an example of indirect evidence that tell us what students probably learned (assuming the course had clear learning objectives). True, they often measure more than just what was learned in a course (e.g. participation, attendance), so there is a limit to how we might use them in our program assessments. But since so many crucial decisions are made based on grades—class rank, scholarships, financial aid, acceptance into graduate or professional school—we have to acknowledge that they are some measure of student knowledge and ability.

Similarly, students’ perceptions of their learning provide some insight into educational effectiveness. I’ve always been surprised when I’ve heard people dismiss findings as “student opinion.”  The opinions of our most important stakeholders should be respectfully considered.

Utica University’s Master of Business Administration Program (M.B.A.) is a case study for how indirect assessments may be used to identify areas for program improvement. For the past few assessment cycles, M.B.A. students have completed an exit survey at the close of their program. This  survey asks about the importance of specific knowledge, skills, and competence, each of which is a learning outcome of the program, and further asks them to indicate the extent to which the program helped them achieve these learning outcomes.

This outbound survey, an indirect assessment, measures students’ perceptions of educational gains in both the core curriculum and the areas of specialization. It allows for a systematic collection of assessment results using a sustainable process that yields actionable findings.

In the 2020-2021 assessment cycle, students reported lower ratings for the Accounting and Finance specialization than the desired target. This prompted the M.B.A. faculty to review the curriculum to ensure that accounting and finance concepts were being adequately reinforced in the appropriate classes, including those taught by adjuncts.

Another area the faculty identified as an opportunity for improvement was related to a diversity goal. The learning goal states the students will “Examine business solutions and problems from a global perspective and assess how cultural differences impact business."  21.8% rated this a “3” on a 5-point scale when asked how much their M.B.A. education helped them develop this knowledge. At present, cultural differences are discussed in the leadership class, and significant time is spent on the topic in a global consumer course, where the final project centers around various culture clusters of the world. However, based on this finding, the faculty is investigating ways they might weave cultural diversity more into the curriculum.

A direct measure of student learning in the M.B.A. Program is the Peregrine Assessment, a standardized test that measures graduates’ knowledge and provides benchmark data from peer institutions.

Similarly, biology is another case study in how direct and indirect measures might be combined to tell a meaningful narrative about student learning. Students graduating from this degree program take the Major Field Test in Biology, a standardized examination that measures a learning goal addressing key principles of biological fields. They also respond to a senior survey that asks them to rate how well they believe they achieved the program’s learning goals.

Direct assessments will probably always be considered more trustworthy than indirect ones. But indirect methods—survey findings, acceptances into graduate/professional schools, graduate employment—help us shape a more comprehensive narrative about student learning in our programs. Surveys have the added benefit of giving students an opportunity to share their feedback on a program or learning experience, thereby giving them agency in our assessment and planning processes.

 

Works Cited

Suskie, Linda. (2009). Assessing Student Learning: A Common Sense Guide. 2^nd ed. San

Francisco: Jossey-Bass.   

 

Underprepared or Underserved?

I have been hearing complaints for three generations about how poorly prepared college students are.   

It’s true that some students are not fully prepared for a post-secondary education. For a variety of complex reasons, some students may require extra services and support. They may even require additional coursework. What people fail to realize, however, is that the “underprepared” student is not a new phenomenon in American higher education.

In 1636, Harvard College opened in the American colonies to train clergy for the new commonwealth. Courses were taught in Latin; textbooks were written in Latin and Greek. Some of Harvard’s students benefitted from having been apprenticed to ministers prior to enrolling at the college. Through these apprenticeships, they learned Latin and Greek.   

Not everyone had this advantage, however, and many of those that didn’t were unschooled in Latin and Greek. In other words, they were underprepared for their course of study. The institution responded by providing tutorial services to assist those young men in learning the classical languages.

Fast forward 200-plus years, when Harvard faculty grumbled about how poorly their students wrote. To address the lack of students’ preparation in formal writing, Harvard faculty in 1874 introduced a freshman composition course, a staple in undergraduate education ever since.

The preponderance of preparatory programs in colleges and universities during the 19^th and 20^th centuries serves as evidence that a portion of American students entered higher education lacking the skills needed to compete. In her landmark text Improving Student Learning Skills, Martha Maxwell writes, “By 1915, three hundred fifty colleges in the United States reported to the U.S. commissioner of education that they had college preparatory departments” designed to help students develop the skills and competencies they would need to persist towards a degree.

Students less adequately prepared for post-secondary educations were enrolled in all types of institutions, from the public land-grant universities to private, highly selective ones. Maxwell states that in 1907, more than half the students matriculating at Harvard, Yale, Princeton, and Columbia did not meet the admissions requirements, and in the 1950s, experts reported that 2/3 of the students entering college lacked the reading and study skills necessary for success.

In the mid-1960s, larger numbers of traditional-aged college learners sought admission to post-secondary institutions than in previous years, and colleges and universities were opening their doors to a more diverse group of learners. These changes, too, resulted in the need for support services for those students who might have been less prepared than some of their peers for academic success.   

We know the pandemic has had a significant impact on student performance. A New York Times report (https://www.nytimes.com/2022/09/01/us/national-test-scores-math-reading-pandemic.html) cites findings from the National Assessment of Educational Progress testing that showed a drop in math and reading scores of the 9-year olds who completed the assessment. Assessment reports from academic departments for the 2019-2020 and 2020-2021 assessment cycles provide evidence of declining student performance, motivation, and satisfaction.

That said, students referred to as “underprepared” have had a place in colleges and universities from the very start, because, as Maxwell writes, “American higher education has historically had an egalitarian thrust.” An equity-minded approach recognizes that underprepared doesn’t mean unqualified or incapable. An equity-minded approach recognizes that being underprepared is often a consequence of being underserved or, like those Harvard students in the mid-17^th century, not having all the advantages enjoyed by other students. Institutions committed to the principles of democracy, diversity, equity, and inclusion are committed to serving these students without judgment.  

 

Works Cited

Maxwell, Martha. Improving Student Learning Skills. Clearwater, H & H Publishing Company, Inc., 1997.

Mervosh, Sarah. “The Pandemic Erased Two Decades of Progress in Math and Reading.” New York Times, 1 September 2022, https://www.nytimes.com/2022/09/01/us/national-test-scores-math-reading-pandemic.html. Accessed 27 September 2022.