Indirect Assessments: How Useful Are They?

Student learning is typically measured using direct or indirect methods. Direct measures provide clear evidence of what students have and have not learned, evidence that assessment leader Linda Suskie says a skeptic would trust.   

In contrast, indirect evidence may reflect students’ perceptions of what they learned or what they probably learned. In the world of assessment, indirect evidence is considered less compelling and less reliable than direct evidence.

I do not dismiss indirect evidence as quickly as some of my assessment colleagues do. Grades are an example of indirect evidence that tell us what students probably learned (assuming the course had clear learning objectives). True, they often measure more than just what was learned in a course (e.g. participation, attendance), so there is a limit to how we might use them in our program assessments. But since so many crucial decisions are made based on grades—class rank, scholarships, financial aid, acceptance into graduate or professional school—we have to acknowledge that they are some measure of student knowledge and ability.

Similarly, students’ perceptions of their learning provide some insight into educational effectiveness. I’ve always been surprised when I’ve heard people dismiss findings as “student opinion.”  The opinions of our most important stakeholders should be respectfully considered.

Utica University’s Master of Business Administration Program (M.B.A.) is a case study for how indirect assessments may be used to identify areas for program improvement. For the past few assessment cycles, M.B.A. students have completed an exit survey at the close of their program. This  survey asks about the importance of specific knowledge, skills, and competence, each of which is a learning outcome of the program, and further asks them to indicate the extent to which the program helped them achieve these learning outcomes.

This outbound survey, an indirect assessment, measures students’ perceptions of educational gains in both the core curriculum and the areas of specialization. It allows for a systematic collection of assessment results using a sustainable process that yields actionable findings.

In the 2020-2021 assessment cycle, students reported lower ratings for the Accounting and Finance specialization than the desired target. This prompted the M.B.A. faculty to review the curriculum to ensure that accounting and finance concepts were being adequately reinforced in the appropriate classes, including those taught by adjuncts.

Another area the faculty identified as an opportunity for improvement was related to a diversity goal. The learning goal states the students will “Examine business solutions and problems from a global perspective and assess how cultural differences impact business."  21.8% rated this a “3” on a 5-point scale when asked how much their M.B.A. education helped them develop this knowledge. At present, cultural differences are discussed in the leadership class, and significant time is spent on the topic in a global consumer course, where the final project centers around various culture clusters of the world. However, based on this finding, the faculty is investigating ways they might weave cultural diversity more into the curriculum.

A direct measure of student learning in the M.B.A. Program is the Peregrine Assessment, a standardized test that measures graduates’ knowledge and provides benchmark data from peer institutions.

Similarly, biology is another case study in how direct and indirect measures might be combined to tell a meaningful narrative about student learning. Students graduating from this degree program take the Major Field Test in Biology, a standardized examination that measures a learning goal addressing key principles of biological fields. They also respond to a senior survey that asks them to rate how well they believe they achieved the program’s learning goals.

Direct assessments will probably always be considered more trustworthy than indirect ones. But indirect methods—survey findings, acceptances into graduate/professional schools, graduate employment—help us shape a more comprehensive narrative about student learning in our programs. Surveys have the added benefit of giving students an opportunity to share their feedback on a program or learning experience, thereby giving them agency in our assessment and planning processes.

 

Works Cited

Suskie, Linda. (2009). Assessing Student Learning: A Common Sense Guide. 2^nd ed. San

Francisco: Jossey-Bass.   

 

Underprepared or Underserved?

I have been hearing complaints for three generations about how poorly prepared college students are.   

It’s true that some students are not fully prepared for a post-secondary education. For a variety of complex reasons, some students may require extra services and support. They may even require additional coursework. What people fail to realize, however, is that the “underprepared” student is not a new phenomenon in American higher education.

In 1636, Harvard College opened in the American colonies to train clergy for the new commonwealth. Courses were taught in Latin; textbooks were written in Latin and Greek. Some of Harvard’s students benefitted from having been apprenticed to ministers prior to enrolling at the college. Through these apprenticeships, they learned Latin and Greek.   

Not everyone had this advantage, however, and many of those that didn’t were unschooled in Latin and Greek. In other words, they were underprepared for their course of study. The institution responded by providing tutorial services to assist those young men in learning the classical languages.

Fast forward 200-plus years, when Harvard faculty grumbled about how poorly their students wrote. To address the lack of students’ preparation in formal writing, Harvard faculty in 1874 introduced a freshman composition course, a staple in undergraduate education ever since.

The preponderance of preparatory programs in colleges and universities during the 19^th and 20^th centuries serves as evidence that a portion of American students entered higher education lacking the skills needed to compete. In her landmark text Improving Student Learning Skills, Martha Maxwell writes, “By 1915, three hundred fifty colleges in the United States reported to the U.S. commissioner of education that they had college preparatory departments” designed to help students develop the skills and competencies they would need to persist towards a degree.

Students less adequately prepared for post-secondary educations were enrolled in all types of institutions, from the public land-grant universities to private, highly selective ones. Maxwell states that in 1907, more than half the students matriculating at Harvard, Yale, Princeton, and Columbia did not meet the admissions requirements, and in the 1950s, experts reported that 2/3 of the students entering college lacked the reading and study skills necessary for success.

In the mid-1960s, larger numbers of traditional-aged college learners sought admission to post-secondary institutions than in previous years, and colleges and universities were opening their doors to a more diverse group of learners. These changes, too, resulted in the need for support services for those students who might have been less prepared than some of their peers for academic success.   

We know the pandemic has had a significant impact on student performance. A New York Times report (https://www.nytimes.com/2022/09/01/us/national-test-scores-math-reading-pandemic.html) cites findings from the National Assessment of Educational Progress testing that showed a drop in math and reading scores of the 9-year olds who completed the assessment. Assessment reports from academic departments for the 2019-2020 and 2020-2021 assessment cycles provide evidence of declining student performance, motivation, and satisfaction.

That said, students referred to as “underprepared” have had a place in colleges and universities from the very start, because, as Maxwell writes, “American higher education has historically had an egalitarian thrust.” An equity-minded approach recognizes that underprepared doesn’t mean unqualified or incapable. An equity-minded approach recognizes that being underprepared is often a consequence of being underserved or, like those Harvard students in the mid-17^th century, not having all the advantages enjoyed by other students. Institutions committed to the principles of democracy, diversity, equity, and inclusion are committed to serving these students without judgment.  

 

Works Cited

Maxwell, Martha. Improving Student Learning Skills. Clearwater, H & H Publishing Company, Inc., 1997.

Mervosh, Sarah. “The Pandemic Erased Two Decades of Progress in Math and Reading.” New York Times, 1 September 2022, https://www.nytimes.com/2022/09/01/us/national-test-scores-math-reading-pandemic.html. Accessed 27 September 2022.

 

 

 

Using Multiple Assessment Methods to Measure Student Performance: An Equitable Practice

 When reviewing annual assessment reports from academic departments, members of the Academic Assessment Committee look to see if a department uses multiple methods to assess its learning goals. To do so constitutes an exemplary practice. The reason for this is simple: every assessment method has its limitations, and it can be misleading to make judgments or develop plans based on data gathered from one instrument. If we want reliable results that can be used to shape a compelling narrative and assist our planning efforts, we need more than one approach to tell us about students’ successes and failures.  

This holds true for course-level assessment as well. Linda Suskie asserts, “The greater the variety of evidence, the more confidently you can infer that students [in your classes] have indeed learned what you want them to.”

It’s also a matter of equity. Different learners demonstrate their learning in different ways. To voice a commitment to the principles of diversity, equity, and inclusion but measure student learning using a single instrument that favors some learners over others is a contradictory practice. 

Using multiple assessment methods in the classroom aligns with the guidelines for Universal Design for Learning, which consider the diverse needs and abilities of all students enrolled in a classroom. These guidelines emphasize multiple means of having students engage with learning, recognize the what of learning, and demonstrate the how of learning.

Some educators advance the idea of giving students choice when it comes to how their learning will be measured. Examples include having students complete fill-in-the-blank type questions or completing multiple choice questions, writing a long essay or composing three short answers.

I have no experience with this particular approach, so I cannot testify to its merits (and quite frankly, I see it as potentially problematic). A more plausible strategy would simply be to use a variety of assignments—papers, presentations, objective tests, surveys—to assess performance.  That is, do not rely solely on one method, such as the mid-term and final exam. 

Understandably, some programs need to prepare students to be successful on a single summative assessment: a certification or licensing examination, a standardized admissions test to graduate or professional school. It is unlikely we will see professional boards or state agencies change how they measure knowledge and skills. Developing students’ ability to be successful on a single test makes sense.

But more often than not, we have the opportunity to use multiple assessment methods, both summative and formative. Doing so is not only a better way to measure actual learning, it is also a more equitable practice, one that recognizes diversity as the norm in our classes.

 

 

Works Cited

CAST (2018). Universal Design for Learning Guidelines version 2.2. Retrieved from

  http://udlguidelines.cast.org

Suskie, Linda. (2009). Assessing Student Learning: A Common Sense Guide. 2^nd ed. San

Francisco: Jossey-Bass.   

 

When the Results Are Good, Showcase Them!

Good assessment helps us determine where improvements are needed. We know this. It’s basic to good teaching, and it happens all the time in courses. It might mean rephrasing or eliminating exam questions, redesigning a rubric for greater clarity, or implementing active learning strategies to help students learn difficult material.

Continuous improvement is not the sole purpose of assessment, however. Assessment, by definition, is a way to identify weaknesses and strengths. It should, and often does, yield results that might be celebrated and used to tell others about our programs. This aspect of assessment is often overlooked, though. In part, that is because it has been drummed into our heads that assessment is a way to ferret out where our efforts are unsuccessful. Another reason is that humans are hardwired for negative bias. We look at survey results, for example, and tend to see only where students are dissatisfied or unhappy.

Assessments at Utica University, particularly those done by academic departments, have produced celebration-worthy findings that that underscore the value of our programs, and we need to share these results with prospective students, their parents, potential employers, advisory boards, and community partners. They are what distinguishes us.  

In biology, for example, the results earned on the Major Field Tests in molecular biology and genetics, cell biology, organismal biology, and population biology, evolution, and ecology are benchmarked with national institutional means. Utica University’s mean scores were higher than the national mean in each subject area.

Likewise, physics students continue to surpass the national average on a survey tool measuring conceptual knowledge of mechanics.

English majors demonstrate marked improvements in their writing abilities at the end of or close to the end of their academic tenure at the University, evidence of the value-added of this program.

On a standardized pre/post assessment, students in undergraduate business programs and the MBA program demonstrated significant growth in their knowledge and skills, and the mean scores earned on the post assessment were higher than those earned by an Accreditation Council for Business Schools and Programs peer group.   

A student satisfaction survey administered in November 2021 showed high levels of satisfaction with support offerings, namely library resources and services, career services, tutoring, academic advising, and the availability of counseling.

What’s most encouraging about all these findings is that they came from a year when pandemic-weary students showed signs of disengagement and poor motivation.

Assessment leader Linda Suskie writes, “The higher education community has a long-standing culture of keeping its light under the proverbial bushel basket and not sharing the story of its successes with its public audiences.”

It’s time we changed that. It’s time we move the narrative past clichés and use assessment results to showcase the strengths of our programs and our students, as well as to improve our educational effectiveness.  

                                                                        Work Cited

Linda Suskie.  Assessing Student Learning: A Common Sense Guide. 2^nd ed. 

                San Francisco: Jossey- Bass, 2009.   

Standardized Testing: A One-Size-Fits-All Assessment

 By Michelle Boucher

When we first started with formal department-wide assessment in our department, we identified the  skills we want our students to develop as chemists. These are the very skills that the American Chemical Society (ACS) expect students to demonstrate in a certified program. It made sense for us to assess these skills using ACS standardized exams as final examinations in those classes for which ACS offered a standardized exam.

This assessment strategy wasn’t perfect, but the exams offered the potential for a beautiful tool for assessment. They are created by a committee of faculty from diverse institutions, they are checked by the Examinations Institute Board of Trustees, and data on student performance is collected and collated by the ACS and distributed to every school administering the specific exams. Specific means and medians for each question and the exam overall are available from a reasonably large pool of students who took the exam nationwide in a given year. The exams are refreshed every 5-10 years through committee, and fresh student outcome data is collected.

While this all sounds fantastic in theory, the reality of the exams proved less utopian. The committee of faculty from “diverse institutions” who write the exam do not typically include equal representation from schools that serve first generation students or schools that are historically black schools or from community colleges. The faculty represented are typically a mix of faculty from Ph.D. granting institutions, highly selective small liberal arts colleges, and perhaps only 1-2 faculty (out of 17-20) from all the “other types of schools” (community colleges, small comprehensive colleges like ours, etc.). All the exams are political in one form or another. The ACS is clearly dictating, through this exam, what it feels to be of importance in the specific course. For example, one iteration of the organic exam had two questions (out of 70) on green chemistry, when there was a faculty representative on the committee who was a green chemist. The most recent organic exam has 4 carbonyl reactions that are specific subfields and also “named reactions” (after the chemist(s) who discovered or publicized the use of the reactions) in part as a direct answer to a recent push in chemical education to minimize the use of “named reactions” in the interests of promoting a more inclusive classroom experience.

Additionally, there are issues concerning equity and standardized exams. It has been shown, time and again in sociological and pedagogical literature, that there are inherent equity issues around standardized exams. There continues to be discussion and research nationwide around what root causes exist that lead to underserved students (students of color, students from lower economic brackets, first-generation students) and students who identify as women to perform at lower levels on standardized exams. Regardless of the reason, the faculty in our department believe disparities exist. We see it in our incoming students, who benefit greatly from the holistic application review that Utica University offers and are often high achieving students with poor standardized test scores, and we see it on our final standardized exams, where students who have performed exceptionally well all semester choke on a standardized exam.

This past year, we experienced another issue with these standardized exams:  we could not vouch for their reliability. The standardized exam results, in fact all of the assessment data, showed little to no impact on our student education and the 2020-2021 COVID-19 experience. While that makes us go “Yay!”, we know that our students right now have fewer skills than they would typically have at this point in their education. We know that our students are faring better than some cohorts at other schools; we all talk with multiple people at other institutions and are active in ChemEd circles nationally, and we can see where our students place. We know our students have a much smaller “COVID-lag” than cohorts at other institutions. We are proud of that. But we know there is a knowledge and experience gap, and our assessment methods do not show that.

We are making the move away from the ACS standardized exams, or we plan to use them the way we want to use them and bend them to our own wills. There is absolutely no good reason for our program to be dictated to, our learning goals determined and defined, by a committee of homogenous professors protecting a status quo that we have dedicated our professional lives to overthrowing.  In our department, the age of one-size-fits-all assessment and the lies it propagated is over.

Undergraduate Intern Assesses How Well Students Learning During the Pandemic

On a Student Voice survey administered to college students in May 2021, 52% of respondents said they learned less during the 2020 – 2021 academic year than they had in pre-COVID years, and close to a quarter of first-year students reported feeling very underprepared for college.

Students might have perceived that they learned less in the first year of the pandemic, but did they?

Senior psychology major, Jacqueline Lewis, posed this question in her internship experience at James Madison University (JMU) in summer 2021. Jacqueline was one of three undergraduate interns selected to work on an independent project supporting the mission of JMU’s Center for Assessment and Research Studies.

Her initial training included becoming familiar with assessment and learning how to use statistical software. Working with a mentor, a professor affiliated with the Center for Assessment and Research Studies, Jacqueline then designed a project to measure how well first-year students developed information literacy skills, one of JMU’s general education competencies. Her research also considered the role motivation plays in learning.

At James Madison University, an institution serving more than 19,000 undergraduates, students participate in two assessment days: one during their first-year orientation and the second after earning 45 – 70 credits hours. This allows the university to implement a pre-and-post test design that examines total score growth, objective level growth, and item growth over time.

Jacqueline collected the pre-test data and the post-test results, the latter of which were generated after students completed an online curriculum in information literacy. She also gathered data on a Student Opinion Scale, an instrument that measures two aspects of motivation: effort and importance.

An analysis of both sets of results showed an increase in the mean total score from the pre-test to the post-test, and a statistically significant mean difference in effort and importance, leading her to conclude that yes, despite the adverse impact of the pandemic on the student experience, learning was happening!

This internship experience exposed Jacqueline to an area of study she was not familiar with: assessment and the scholarship of teaching and learning. In addition, it expanded her opportunities for graduate study and gave her access to a professional network.

In November 2021, Jacqueline Lewis presented her work at the Virginia Assessment Group Conference, thus contributing to the body of scholarship in the field of assessment. Her work also added to the growing narrative about the pandemic’s impact on college students’ learning, a topic that will attract researchers for at least the next five years. Jacqueline herself is continuing her research in this area by collaborating with her Utica College advisor and mentor, Dr. Kaylee Seddio, on measuring ADHD and anxiety in college students during the pandemic. 

"I am extremely grateful for my experience at JMU and for those who helped me get there,” Lewis said. “I value the opportunity and all that I have learned, but more so the feeling that I helped make an important contribution to understanding learning during this unprecedented time."

 

 

AAC's Response to the North Pole's Gift-Giving Process

The Academic Assessment Committee recently reviewed the gift-giving process practiced at the North Pole. What follows is the committee’s feedback and suggestions regarding these practices.

Santa has articulated an outcome ("Children should be nice"), but the wording is too ambiguous, and, therefore, hard to measure. Further, he has not specified an appropriate target. Should 100% of the children be "nice" 100% of the time? 90%? 75%? Without a clearly defined target, Santa risks being arbitrary and inconsistent in his assessment of children's behaviors. 

 The methods Santa uses to assess children's behavior and distinguish between "naughty" and "nice" are not apparent. How often is he able to observe each child first-hand?   Are there others who are involved in this assessment? Santa's elves or Mrs. Claus, for example? Given how high stakes this assessment is, there should be multiple individuals engaged in observing and evaluating children's behavior, and there should be 90% inter-rater reliability. Further, each individual child's behavior should be observed on numerous occasions throughout the assessment cycle. This may not be a sustainable plan, however, particularly given the recent cuts made to the workshop staff as a result of the pandemic and the shortages caused by the supply chain problem.  

 It is also not clear what instrument is used to document children’s' behaviors. Does Santa use a rubric that articulates clear criteria regarding the kinds of behaviors he regards as "nice" versus "naughty?" Do these criteria take into account cultural differences?  In other words, are Santa's assessment practices equitable?  

 The results of Santa's assessments have never been published or analyzed. What percent of the world's population under the age of 10 gets what they asked for? What percent receives coal in their stockings? Are there specific trends that Santa has observed over a period of time--say the last 100 years?  Has the percentage of naughty children increased? Decreased?

Since the purpose of assessment is to inform improvement, how are Santa's findings shared with and used by parents? Or do his "naughty" and "nice" lists remain in a drawer in his office at the North Pole and referenced only during the Christmas season as a prop?  How might parents (and perhaps even teachers) use them to help develop children’s character?

The committee recommends that Santa reflect more deeply on his assessment processes and consult with other characters such as the Easter Bunny and the Tooth Fairy to get ideas for how he might design an assessment plan that is fair, useful, and sustainable.