How to use aggregated group score reports to advance educational effectiveness, increase retention, improve student success, and prepare for accreditation self-studies
This “message to colleagues,” although composed in the first person, is a distillation of multiple conversations with deans and academic vice-presidents. The specific critical thinking skills referenced are actual metrics reported by the California Critical Thinking Skills Test (CCTST), and by our professional school related Health Sciences Reasoning Test (HSRT), Business Critical Thinking Test, (BCTST), and the Test of Everyday Reasoning (TER). The activities and plans described are real examples of the kinds of action steps many of our clients take in response to the data reported about their students.
Peter A Facione, Ph.D.
Dear Colleagues,
I’m an academic vice president at an institution with the typical mix of different academic departments and professional programs. Student retention is important for us – as I know it must be for you. And, with accreditation around the corner, we wanted to take an objective look at educational effectiveness.
So, we contacted Insight Assessment about using one of their higher education critical thinking skills tests. We did a pilot project with entering freshman. 166 of them completed the assessment online. With their mix of majors and undecideds, and other factors, we regarded that sample as reliably representative. When we downloaded the aggregated results, we quickly saw the learning gaps that needed to be addressed to improve our student retention. We also look forward to using this data in our upcoming accreditation self-study.
Overall, our students looked about as we expected. Most students scored in the “moderate” range, and there was a nice spread of scores across all five qualitative levels. We knew that the assessment was very well suited to our student population. The average of our students’ “Overall Score” put us at the 48th percentile of the comparison benchmark we elected to use. We wish that our percentile had been higher, but at the undergraduate level we cannot afford to be as selective as we would like to be. Some students showed strong and superior critical thinking skills. These students individually were near the top of the percentile benchmark. Other students displayed weak skills. For privacy reasons, in our pilot project we tested anonymously. Yet, looking at the group aggregated scores, we knew right away that we would have to make some instructional changes to help our students achieve greater academic success through stronger critical thinking.
We knew that critical thinking is reflective judgment that relies on a core set of interacting skills. So, we looked at the data on specific skill metrics to learn where we should concentrate our efforts.
For example, our students were stronger on analysis, but weaker on interpretation. That meant that our students could identify the parts of a problem, but they did not know how those parts fit together into something meaningful. To address this, we encouraged faculty to show students not just how to identify, but also how to correctly interpret key data in their respective disciplines. Since analysis of the problem and interpretation of findings go together, we urged faculty across the disciplines to fold analytical and interpretive questions into class discussions, course assignments, and exams.
The data on evaluation showed unexpected weakness. Evaluative skills are used to assess the credibility of the claims people make, and to assess the quality of the reasoning people display when they make arguments or give explanations. We suspect that our student’s evaluation skills may have been suppressed by the long pandemic isolation and an increase in the spread of misinformation. We have decided it would be valuable for our students for faculty to use class time, class assignments, and exams to teach our students how to make informed and reasonable evaluations. We needed to model for our students how thoughtful people query the evidence and consider the reasons being offered for one point of view or another in a fair-minded and objective way.
It became obvious to us that sensible and informed evaluation was one critical thinking skill, along with problem analysis and the interpretation, that we could address in all the subjects that we taught and in our academic community as well.
The largest potential for improving student learning was in numeracy. Vital for many professional fields and disciplines including business, health sciences, and STEM programs, numeracy is applying critical thinking to problems and decisions involving quantitative information. Numeracy uses all the critical thinking skills: analysis to identify key quantitative relationships, interpretation to figure out what they mean, inference to discover their implications, evaluation to check the quality of the earlier analyses, interpretations, and inferences, and then explanation to share the basis for whatever conclusion or decision is reached. Doing all this in contexts involving precise quantitative considerations was a major challenge for our students.
Looking at our undergraduate students’ critical thinking skills test scores, we know where we to focus our educational effectiveness energies. And we know that by tweaking the kinds of questions we use in classroom discussions and on assignments and exams, we can demonstrate, teach, and focus our grading more on each of the specific critical thinking skills. With access to the free instructional resources Insight Assessment offers for faculty, and with some optional summer workshops through our Teaching and Learning Center, we plan to test our entering students each year to discover how best to support their learning. And we will post-test graduating students to measure the impact of our efforts. That plan is going into our accreditation self-study too. Our aim here is to provide every student with the tools, like critical thinking, that will optimize their academic success.
For our accreditors and for our alumni, faculty, and benefactors we will continue to celebrate not only the individual success stories that make us proud, but also the growth of all our students on the one learning outcome that seemed to be unanimously embraced – critical thinking: learning how to think well when making decisions about what to believe and what to do, in whatever context life presents.