Defining Questions, Measures and Predictions
In working towards an institution’s goals, it is good practice to lay out specific questions that the improvement team wants to address through data collection and analysis. Identifying improvement questions ahead of time will also help keep colleges on track and minimize the burden of data collection in terms of time and resources. These questions should be focused on gathering the critical information that must be known to determine whether the change being evaluated is leading to improvement.
Improvement questions can focus on many different aspects of the change that is being tested, including how specific practices or processes are being rolled out, the experiences and perspectives of students and college staff involved in the rollout of the change to corequisite remediation and outcomes for students or staff.
How to
- Develop concrete questions that will be addressed through the evaluation.
- Identify data sources and measures.
- Make a prediction based on how the phase is supposed to go.
Example
Question
How many hours will instructor pairs be subject to the new alignment strategy meet in order to align syllabi?
Data Source
Weekly instructor survey
Measure
Hours of time collaborating by phone, email
Prediction
Instructors will meet for at least four hours.
Question
Will students be more likely to perceive that the developmental education part of the corequisite model is valuable?
Data Source
End-of-course satisfaction survey
Measure
Likert question: “The developmental education support was important to supporting my success in the college course.”
Prediction
Students in courses with the instructor pairs testing the alignment strategy will be at least 10 percentage points more likely to agree or strongly agree compared with other students.
Question
Will student success rates in the college course increase?
Data Source
Administrative data
Measure
Percentage of students passing
Prediction
Course passing rates will increase by five percentage points relative to prior semesters.