Affordable Course Materials Initiative (ACMI)

Impact on students can be assessed through a variety of measures. Before beginning the assessment, you should consider what factors you wish to study and what data collection method will be needed to assess those factors.

Common measures:

• How has the OER/new materials impacted student learning outcomes?
• How has OER/new materials usage impacted student success?
• What do students think of the OER/new materials ? How does their perception compare to commercial textbooks?

Sample Assessment Questions for Students

1. Do you feel that the open textbook used in this course supported your learning? Why or why not?
2. How well did the open textbook align with what was taught in the course?
3. How did the open textbook compare to more traditional textbooks that you have used in other courses?
4. How easy or difficult was it to access and navigate the open textbook?
5. What changes need to be made to make this OER mores successful going forward?
6. Any additional comments?

(courtesy VCU OER Guide & Getting Started with Open Educational Resources)

Things to consider: 

1. When comparing grades between courses with new content/OER and traditional courses there are many factors to take into account. (See page 6 of the Guide for Research on OER adoption.) Generally, demonstrating that students achieved as well as students using the traditional content on measures such as a common final, rather than overall course grades is the best approach.
2. Asking students' general perceptions of the new content/OER can be problematic in that typically students are not asked how they feel about the textbook in courses using commercial textbooks. Look at sample instruments (from resources below) and consider what you are trying to learn. 

The COUP Framework (Cost, Outcome, Usage, and Perception) is the recommended set of measures to use for evaluating the impact and effectiveness of adopting OER in your course. This framework helps to break down the different types of data, methods, and analyses you may want to collect and conduct for your project. The Open Education Group provides more detailed information on the COUP Framework if you are interested in learning more about it.

Sample assessment tools from previous ACMI stipend awardees:

• Rebecca Malatests - PSY 1000 - assessment tool 
• Kate Rougeau  EmilyVanWasshenova - HS 2000 - assessment tool

The OER Research Guidebook - includes sample questions and measures for evaluating:

• Final Grade
• Course Throughput Rates
• Enrollment Intensity And Persistence
• Comparing Use Of Traditional Materials And OER
• Measuring Student And Teacher /Perceptions Of OER
   

The OER Research Toolkit - includes sample surveys from previous studies and data collection methods. 

• A sample student survey based on Bliss, T., Robinson, T. J., Hilton, J., & Wiley, D. (2013). An OER COUP: College teacher and student perceptions of Open Educational Resources. Journal of Interactive Media in Education, 1–25.