Defining Questions, Measures and Predictions

In working towards an institution’s goals, it is good practice to lay out specific questions that the improvement team wants to address through data collection and analysis. Identifying improvement questions ahead of time will also help keep colleges on track and minimize the burden of data collection in terms of time and resources. These questions should be focused on gathering the critical information that must be known to determine whether the change being evaluated is leading to improvement.

Improvement questions can focus on many different aspects of the change that is being tested, including how specific practices or processes are being rolled out, the experiences and perspectives of students and college staff involved in the rollout of the change to corequisite remediation and outcomes for students or staff.

Download the Document

How to

  1. Develop concrete questions that will be addressed through the evaluation.
  2. Identify data sources and measures.
  3. Make a prediction based on how the phase is supposed to go.
See the following chart for an example.

Example

Question

How many hours will instructor pairs be subject to the new alignment strategy meet in order to align syllabi?

Data Source

Weekly instructor survey

Measure

Hours of time collaborating by phone, email

Prediction

Instructors will meet for at least four hours.

Question

Will students be more likely to perceive that the developmental education part of the corequisite model is valuable?

Data Source

End-of-course satisfaction survey

Measure

Likert question: “The developmental education support was important to supporting my success in the college course.”

Prediction

Students in courses with the instructor pairs testing the alignment strategy will be at least 10 percentage points more likely to agree or strongly agree compared with other students.

Question

Will student success rates in the college course increase?

Data Source

Administrative data

Measure

Percentage of students passing

Prediction

Course passing rates will increase by five percentage points relative to prior semesters.