2023/24 — San José State University

— Preparation for Quality Matters Certification

Proposal Summary: At San José State University, we proposed to provide training along with a faculty-mentor support structure for the faculty participants in the program. We were able to complete the goals of this project through a combination of Quality Matters Workshops, Canvas-based resources and faculty/staff facilitators. The campus continues to expand our support with guidelines that faculty can use as they develop their online courses. The quality assurance program provides extensive training, faculty expertise, and guidance on the redesign of a course that takes into account best practices as identified in the Quality Matters (QM) rubric. This program focused on revising courses in preparation for Quality Matters Certification.

Campus Goal for Quality Assurance

The EOQA goals are in line with the CSU’s Graduation Initiative 2025 specifically addressing the measure of ensuring effective use of technology is part of every CSU student’s learning environment. Additionally, our approach follows guidelines proposed in SJSU’s Four Pillars of Student Success that promote the development of:

• richer and more readily accessible on-line supplemental study materials;
• more elaborate and interactive homework and self-check instructional materials;
• and more engaging in-class teaching strategies.

This proposal focused on developing a standard with which online and hybrid courses can use as a way to reflect upon their current course design and make the necessary revisions to reflect best practices. There are multiple needs of the campus to support Quality Assurance Efforts.

• 1: Develop materials and resources that faculty can access and use to guide course design.
• 2: Provide professional development opportunities to increase faculty awareness regarding quality assurance.
• 3: Build a group of faculty that can become experts in quality assurance and provide mentoring for new faculty.

Quality Assurance Lead(s)

• Jennifer Redd, Program Cooridinator
• Yingjie Liu, Project Facilitator
• Megan Chang, Faculty Reviewer
• Sulekha Anand, Student Quality Assurance Impact Researcher

Campus Commitment Toward Sustainability of QA Efforts

• Developed multiple Canvas course templates based upon Quality Matters Principles
• Encourage quality assurance principles in instructional design consultations

Summary of Previous QA  Accomplishments

This quality assurance program was the tenth iteration on campus. The previous cohort participated during the 2022-23 academic year. A variety of quality assurance efforts continue to expand on campus.

• Effort 1: A faculty cohort completed one or two Quality Matters trainings: Applying the Quality Matters Rubric and Improving Your Online Course. A previous cohort completed the Peer Reviewer Course. Additional information about last year's effort can be found in the Quality Assurance ePortfolio.
• Effort 2: Increase awareness through outreach activities. This includes posting resources on the eCampus website and through participation in informational webinars. It also includes promoting workshops and encouraging attendance through flyers and presentations at campus events.
• Effort 3: The rubric is provided as a resource for faculty in a password-protected Canvas course. Also, it serves as a guide when instructional designers consult with faculty members on course design.
• Effort 4: Encourage faculty and staff that have completed Peer Reviewer Training to become a Quality Matters Peer Reviewer.
• Effort 5: A Canvas course template that adheres to Quality Matters Standards is available to all faculty.

Course Review and Certifications

The program focused on providing training and support for faculty with online or hybrid courses and preparing those courses for Quality Matters Certification. The program itself was called Getting Ready for QM Certification (GRQC). The program was asynchronous and was broken down into six modules in a Canvas Course. For example, Start Here, QM Workshop information, All about Learning Objectives about Alignment, All about Activities and Assessments, All about Course Technology, Learner Support, and Accessibility and Usability, and QM Review. The faculty participants met with either the lead instructional designer and/or a faculty member for additional guidance to support the best practices learned in the Quality Matters workshop and within the Canvas course modules. They also received detailed feedback through an information course review to help assist them with submitting their courses for a formal review. The next stage involved submitting the course for a formal QM review and then making course revisions per the feedback received. 

Faculty Participants

Development of Campus QA Resources

Canvas Course Templates are available to SJSU faculty

Next Steps for QA Efforts 

• Expand the number of courses that are Quality Matters certified
• Provide guidance to department chairs and faculty interested in having their courses certified
• Encourage department chair and faculty participation in Quality Matters Trainings

Training Completions

The following table summarizes all of the SJSU faculty and staff Quality Assurance training completions that occurred during the 2023/24 academic year.

The CSU QA Student Online Course Survey was distributed via Qualtrics to the classes taught by the AY 2023-2024 EoQA grant program participants.  The survey was completed by 23 students in 3 upper division courses.  For 74% of respondents, the course was required rather than elective.   Thirty-nine percent of respondents were Hispanic or Latino, 30% were Asian, 17% were Caucasian, and 13% were Two or More Races.  The group of respondents was primarily female (87%), and most respondents were juniors or seniors.  Seventy-four percent of respondents had previously taken at least four online courses.  

In addition to the questions pertaining to course details and student demographics, there were 25 questions asking students to rate their agreement with a statement on a six-point scale from Strongly Disagree (1) to Strongly Agree (6).  There were 4 questions pertaining to Course Overview and Introduction, 5 questions pertaining to Assessment and Evaluation of Student Learning, 4 questions addressing Instructional Materials and Resources Utilized, 3 questions addressing Student Interaction and Community, 2 questions pertaining to Facilitation and Instruction, 2 questions pertaining to Technology for Teaching and Learning, 2 questions addressing Learner Support and Resources, and 3 questions addressing Inclusivity and Accessibility.  Descriptive statistics are presented in the Table.    

All averages were close to the Strongly Agree rating and individual responses were uniformly high (see minimum values in the Table).  The average responses for the Course Introduction and Overview questions were in the Agree to Strongly Agree range (Figure 1).   The average responses for the Assessment and Evaluation of Student Learning questions were also in the Agree to Strongly Agree range (Figure 2).  

Figure 1.  Mean responses to questions about course overview and introduction.  In all graphs, error bars depict the standard deviation. 

 

Figure 2.  Mean responses to questions about assessment and evaluation of student learning.

For the questions on Instructional Materials and Resources Utilized, the average responses were in the Agree to Strongly Agree range (Figure 3).  For the questions addressing student opinions on Student Interaction and Community, the average responses were also in the Agree to Strongly Agree range (Figure 4).

Figure 3.  Mean responses to questions about instructional materials and resources utilized.

 

Figure 4.  Mean responses to questions about student interaction and community.

 

There were two questions on Facilitation and Instruction (Figure 5).  The average responses were 5.7 for clarity on how long it would take to receive feedback on assignments, and 5.86 for sending reminders of due dates.   The average ratings for the two questions pertaining to Technology for Teaching and Learning were 5.55 for the use of a variety of technology tools to engage the class and encourage them to interact, and 5.65 for providing clear information on how to access/acquire the required technologies (Figure 6).  

Figure 5.  Mean responses to questions about facilitation and instruction.

 

 

Figure 6.  Mean responses to questions about technology for teaching and learning.

 

The average responses for the questions on Learner Support and Resources were in the Agree to Strongly Agree range (Figure 7), as were the average ratings for the Inclusivity and Accessibility items (Figure 8).    

 

Figure 7.  Mean responses to questions about learner support and resources.

 

Figure 8.  Mean responses to questions about inclusivity and accessibility.

 

Table.  Descriptive statistics for the 25 items in the Student Impact Survey completed in AY 2023-2024, n = 23

Question	Minimum	Maximum	Mean	Std Deviation
Course Overview and Introduction
The instructor provided clear and detailed instructions for how to begin accessing all course components, such as syllabus, course calendar, and assignments.	5	6	5.83	0.388
Detailed information about the instructor was available and included multiple ways to contact him/her, times s/she was available, a brief biography, and a picture or welcome video.	5	6	5.82	0.395
The course description included the purpose and format (e.g. fully online, blended; schedule/calendar specifies dates/times) of the course, as well as any applicable prerequisite knowledge (e.g., prerequisite course).	4	6	5.83	0.491
The instructor clearly defined academic integrity and/or provided a “code of ethics” and provided institutional policies and/or links to those policies (e.g, academic dishonesty, cheating, and plagiarism).	5	6	5.87	0.344
Assessment and Evaluation of Student Learning
The instructor provided specific, well-defined, and measurable learning objectives. I understood what I was supposed to accomplish both weekly and by the end of the course. For example, each week there were specific learning goals and I knew exactly what I was supposed to learn/accomplish (e.g., there were bulleted list of activities to complete each week).	5	6	5.91	0.288
I understood how the learning activities (including the assignments and ungraded activities) helped me achieve the learning objectives each week. For example, I understood how a discussion forum could help me prepare to develop a “reaction paper” on a topic.	3	6	5.74	0.689
The instructor made it clear how individual papers, exams, projects, and/or group contributions would be evaluated. For example, I was given grading sheets or detailed descriptions of how points were distributed for major assignments.	5	6	5.83	0.388
The instructor provided a course grading policy that clearly defined how much each assignment or category of assignments contributed to my overall course grade.	3	6	5.83	0.650
I was given opportunities to receive feedback from my instructor and to self-check my progress in the course. For example, my instructor posted grades regularly, provided comments on my work, had us self-grade assignments, allowed us to submit drafts of projects for comments, and offered discussion forums for feedback and practice tests.	3	6	5.70	0.703
Instructional Materials and Resources Utilized
The instructor gave me adequate notice and time to acquire course materials. For example, I received information on how to obtain the course textbook/materials prior to the start of the course via email, or the instructions for how to acquire the materials were in the syllabus or elsewhere in the course.	5	6	5.78	0.422
The instructor offered a variety of course material types (such as audio, video, and readings) and perspectives. S/he did not over-rely on a single way to deliver content such as via text or from a single source/textbook or author.	5	6	5.87	0.344
The materials supported the content of what I was learning in the course. For example, the textbook, articles, audio recordings, and videos were all tied to the course topics and objectives.	5	6	5.91	0.288
The instructor provided a good explanation to show how the instructional materials (e.g., textbook, videos organized by topics) support the course objectives or competencies.	5	6	5.83	0.388
Student Interaction and Community
The instructor provided an opportunity at the beginning of the course for students to introduce themselves. This created a sense of community among course participants.	4	6	5.83	0.491
The learning activities (e.g., discussions and activities) encouraged me to log on and interact with my fellow classmates often.	4	6	5.61	0.722
The course learning activities helped me understand fundamental concepts and build skills that will be useful in the real world. For example, the activities made connections with real-world problem solving, and involved real-world scenarios.	4	6	5.78	0.518
Facilitation and Instruction
The instructor was clear on how long it would take to receive feedback on assignments.  I received feedback about my coursework and progress in a timely fashion.	3	6	5.70	0.703
The instructor sent reminders of due dates (email, weekly announcements) and other information and instructions to help keep me on task.	5	6	5.86	0.351
Technology for Teaching and Learning
The instructor used a variety of online technology tools to engage me and encourage me to interact with others in the course and I felt the tools used supported the course objectives.  Examples include, but are not limited to, web meetings, online discussions (e.g., VoiceThread), online collaboration tools (e.g., Google Docs), social media tools (e.g., Twitter).	4	6	5.55	0.739
The instructor provided clear information about how to access or acquire the technologies required to successfully complete the course. Examples include, but are not limited to, web authoring software (web pages, blogs, wikis), proctoring software, printers, scanners, browser plug-ins or media players.	4	6	5.65	0.573
Learner Support and Resource
The course syllabus listed and/or the course website linked to a clear explanation of the TECHNICAL support provided by my campus and provided information about when and how I can access it. For example, the syllabus had links to the technical support website, Help Desk contacts, and online tutorials.	3	6	5.73	0.703
The course syllabus listed and/or the course website linked to ACADEMIC support services and resources, such as Supplemental Instruction, Writing Center, Math Center, Tutoring Center, testing services, and library resources.	4	6	5.74	0.541
Inclusivity and Accessibility
The course syllabus or course website provided or linked to the campus policy regarding accommodating students with disabilities.	4	6	5.84	0.512
The course materials (whether created by the instructor or from external sources) were in accessible formats (e.g., videos were captioned and/or had text transcripts).	4	6	5.78	0.518
It was easy to navigate the online components of the course. For example, the module or weekly organization was easy to follow and course headings and links were clear and easy to understand.  It was easy for me to locate respective course resources/components.	4	6	5.78	0.518

 

 

 

Three of the four faculty participants in the 2023-2024 EoQA program were interviewed upon completion of the program.  They were asked about the changes they had made to their courses or planned to make based on their EoQA training and specific modules in the EoQA program, namely the QM IYOC workshop, Learning Objectives module, Alignment module, and Course Technology/Learner Support/Accessibility & Usability module.  Participants unanimously reported that the changes they made to their courses and Canvas shells as a result of the program improved instruction.   

The reported changes include: 

• Course map or updated course map
• Evaluation of rationale for learning activities and assignments
• Mapping of assignments and assessments to learning outcomes or learning objectives 
• Updated module LOs 
• Refinement of CLOs and mapping them to module LOs
• Use of Bloom’s taxonomy for LOs and alignment
• Connecting LOs with learning activities and PLOs
• Text headers within modules to better organize content
• New assessments to improve learning
• Video for course project with details about the assignment
• Improved accessibility of images
• Planning to make module introduction videos

All participants had positive comments about their experience in the EoQA program and would recommend it to colleagues.  It motivated participants to bring professionalism and consistency to their classes, and enables standardization of online courses.  Deans and the Provost should recommend that faculty complete the program.  Because the program is time-consuming, it may not be feasible for a faculty member with a full load.  

Suggestions for improvement include a better timeline of due dates and estimated time to complete assignments (this suggestion applies to the QM workshop as well and was suggested in last year’s EoQA report as well); starting the program before the semester begins or a week after the start of the semester instead of when instruction starts; a template for Canvas that all faculty can use and insert their own assignments and videos; encouraging all faculty who teach online to complete the program; and a follow-up meeting with a summary, wrap-up, discussion on what worked, a list-serve, and regular meetings (quarterly, bi-annual, or annual).  It was also reported that the QM standards and workshop are very rigid.