Hear what students are saying...
Standard
Two: Assessment System
and Unit Evaluation
Assessment System and Unit Evaluation
The teacher preparation unit at Lewis-Clark State College has a thorough assessment and evaluation system designed to collect and analyze information on applicant qualifications, candidate and intern performance, alumni follow-up, and unit operations to improve the unit and its programs. In addition to the data collected through our formal assessment system, we rely heavily on data collected informally. Our size is an advantage in that it allows us multiple opportunities to interact with our candidates, our campus colleagues and our field-based partners. Through these informal interactions we gain insights that are equally important in guiding our decisions as we seek to improve the quality of our programs.
In 1998 we received a three-year grant from the J.A. and Kathryn Albertson Foundation to recreate our teacher preparation programs. Throughout the three years of the grant, we met weekly to determine changes needed in programs and assessment systems. In addition, we met periodically with members of the Humanities, Social Sciences, and Natural Science/Mathematics divisions as well as with teachers and administrators from K-12 schools in the region to solicit suggestions and feedback. External reviewers from the J.A. and Kathryn Albertson Foundation also consulted with us as we developed a process for analyzing and evaluating our programs. As a result of this initiative, we established an assessment model for each program, the Candidate Assessment Map: Program for Elementary School Teacher Candidates [PDF] and the Candidate Assessment Map: Program for Secondary Education Candidates [PDF].
A list of individuals who actively participated in the revision of the assessment system during the redesign initiative can be found at Assessment System Revision Participants. All current faculty members are regularly involved in ongoing revisions as deemed appropriate and necessary.
We continue to solicit suggestions for improvements to our assessment system through our Elementary and Secondary Advisory Board meetings (Elementary Meeting agendas, Secondary Meeting agendas), our end-of-program surveys by interns, and our informal interactions with colleagues in the field. Both elementary and secondary teams hold regular team meetings (Elementary Team minutes, Secondary Team minutes); one frequent agenda item is program review and revision. We regularly consider feedback data from these various sources when making program revisions. For example, electronic portfolios, a significant part of the assessment plan, are now required in the secondary program as a result of recommendations from both candidates and faculty members. The decision that elementary candidates may submit either electronic or traditional portfolios was based on feedback from elementary candidates. Requiring an entrance interview for admittance into the programs is another example of our response to external suggestions for improvement. The suggestion came from an evaluator hired by the J.A. & Kathryn Albertson Foundation. The elementary and secondary faculty members discussed the idea, reached consensus and the entrance interview has become a valuable part of the entrance procedure for all candidates.
Our assessment system reflects our conceptual framework which is anchored in professional and state standards. During our redesign initiative, we thoroughly discussed and clarified the INTASC (Interstate New Teachers Assessment and Support Consortium) standards and indicators, the ISTE (International Society for Technology in Education) standards, and the content-specific standards of professional organizations. We then crafted our conceptual framework, a mission statement supported by seven standards, which reflected our beliefs about teaching and learning and would guide us in preparing teachers to work effectively in K-12 schools. Finally, our teacher education faculty and our professional community reviewed and approved our conceptual framework. When the Idaho State Board of Education adopted the Professional Standards for the Initial Certification of School Personnel in 2000, we carefully aligned these new state standards with our existing program standards. The Alignment of Professional Standards for Teaching [PDF] grid outlines the alignments currently in place. Most recently, from Fall 2004 to Spring 2005, we met monthly to review key elements of our conceptual framework, to consider adjustments and revisions to the standards, and to update our research-based resources. As a result, we areconsidering the addition of two new standards: one related to communication and one related to cultural responsiveness.
Our program level assessment forms reflect the principles outlined in our conceptual framework. Candidates must demonstrate competency in seven areas before graduating and being recommended for certification. Consequently, these standards frame our expectations for candidate performance and guide our discussions concerning each candidate’s growth and development as an educator. Our success in helping our candidates develop these competencies is a measure of the effectiveness of our programs.
Our programs are divided into three phases. Phase I consists of pre-professional coursework, Phase II focuses on professional studies and Phase III is the professional internship. The Program Assessment Maps (Elementary [PDF] and Secondary [PDF]) indicate the sequence of program assessments required at each transition point or phase. Multiple and varied assessments are used by elementary and secondary programs to monitor and evaluate candidate/intern performances and to determine whether the candidate/intern has developed the knowledge, skills and dispositions required by the standards of the profession, state, and LCSC teacher education programs for initial certification. Sources of evidence at the program level include standardized tests, GPA requirements, entrance interviews, portfolios, teaching performances, candidate work samples, and exit interviews with evaluation forms or scoring rubrics linked to standards.
Course syllabi provide clarification and links to how standards are met in required teacher education coursework. Examples of sources of evidence at the course level are papers, tests, quizzes, lesson plans, and projects. The alignment of courses with standards is outlined on the charts: Professional Standards by Course (Elementary) [PDF] and Professional Standards by Course (Secondary) [PDF]. The Instructional Events website shows specifications of a variety of instructional events, supporting resources, assessment criteria and evaluation procedures, samples of candidate performance, and summaries of candidate performances for courses required in the programs.
The program maps (Elementary [PDF] and Secondary [PDF]) are designed to guide decision-making at the following transition points: 1) program admission, 2) entrance into Internship II, and 3) program completion.
The first transition point is Admission. Candidates submit their Application to the LCSC Teacher Education Program the semester prior to entering the program. Admission requirements include:
- Having a cumulative 2.75 GPA and for secondary candidates, a 3.0 in their major field.
- Achieving qualifying scores on the COMPASS, a test of basic skills in math, reading and writing: 66 on the pre-algebra exam, 75 on the writing exam, and 80 on the reading exam. The COMPASS is waived for candidates who have Bachelor degrees from accredited institutions and meet all other admission requirements.
- Submitting a lucid writing sample dealing with insights they have gained from working with children or adolescents.
- Receiving positive recommendations from their practicum experiences in ED 214 - Principles of Education.
- Having two recommendations from faculty members who share their perspectives regarding the candidate’s knowledge, skills and dispositions for teaching.
- Obtaining a passing score of 13 or above on the Entrance Interview conducted by a team of teachers and administrators from the professional community. Candidates scoring below 13 are required to meet with the Director of Field Experience to discuss the problems identified and consider possible solutions. Candidates may be counseled out of the program, or another Entrance Interview may be scheduled. Questions asked during the Entrance Interview are designed to screen for dispositions related to teaching, culturally responsive demeanor, and respect for children. The Entrance Interview is scored using the Entrance Interview Evaluation [PDF] form. In addition to the oral interview, an extemporaneous writing sample is collected and assessed.
The assessment results are entered into the unit’s database by the Program Managers; the forms are then placed in candidates’ files. When the application process is complete, candidates’ scores and files are forwarded to the appropriate program coordinator. Each team reviews the data to determine which candidates qualify for full admission, which qualify for provisional admission, and which should be denied admission into their respective programs. To be fully admitted to the program, candidates must have successfully completed all requirements. Provisional admittance is the most common level of entry into the program. Candidates are provisionally admitted awaiting the posting of grades for the semester in which they apply. Another reason a candidate may be admitted provisionally is due to needing one or two required courses from Phase I of the program. There is room in the program plans for candidates to complete one or two courses from Phase I during Phase II. This allows flexibility in our program that we find necessary due to being a small institution where not all courses are offered at a variety of times each semester. When candidates have passed all required classes they are granted full admittance. A candidate may be provisionally admitted if s/he does not have the required GPA but has demonstrated potential as a teacher candidate. When candidates have fulfilled all the requirements they are notified that they have been fully admitted into the program. Candidates must be fully admitted prior to beginning the Internship II experience.
Applicants may be denied admittance if they have not met one of the requirements or if they failed to achieve a cumulative score of fifty points through the application process. Candidates may be denied based solely on the Entrance Interview if the interview team deems the responses to questions inappropriate for teaching. In most cases candidates who are denied admittance receive a letter advising them to schedule a meeting with their advisor to develop a plan for meeting the requirements. If denial is based on the Entrance Interview, a meeting is scheduled with the Director of Field Experience.
The second transition point is Entrance into Internship II. Candidates must:
- Earn a B- or better in each Education course required in Phase II of the program.
- Successfully complete all requirements as outlined on the Intern I Progress Evaluation Conference form (Elementary form [PDF], Secondary form) to proceed into Internship II.
- Meet the required qualifying scores on the PRAXIS exams required for their program prior to beginning their Internship II experience.
Candidates who do not meet these requirements must delay Internship II until all requirements are met.
The third transition point is Program Completion. Candidates must earn:
- Scores of 3- or better (on a 4 point scale) on all elements of the final Evaluation of Teaching [PDF] form from the on-site teacher educators.
- Scores of 3- or better on all elements of the final Evaluation of Teaching [PDF] form from the faculty mentor.
- Scores of 3- or better on all elements of the Evaluation of Professional Portfolio [PDF] form.
- Scores of 3- or better on all elements of the Elementary Program Exit Interview [PDF] form or a “yes” on all elements of the Secondary Program Exit Interview [PDF] form.
- Passing scores on the six tasks of the Idaho Technology Performance Assessment.
- Passing scores on all three standards of the Idaho Comprehensive Literacy Assessment (a requirement for all elementary candidates and for secondary candidates who will teach reading).
Candidates who are not successful in meeting these requirements 1) must retake courses until the required grade is earned, 2) may be required to extend their time in their internship setting, 3) must retake state exams until established cut scores have been achieved, or 4) may be counseled out of the program.
The unit takes effective steps to eliminate sources of bias in performance assessments and works to establish the fairness, accuracy, and consistency of its assessment procedures. The reliability and validity of assessments is a priority for improving the LCSC Teacher Education-Program Evaluation Model. Both Elementary and Secondary Education Coordinating Teams frequently discuss the sources of bias, fairness, accuracy and consistency of their assessment procedures. Inter-rater reliability is examined during scoring of the entrance interviews, portfolios/e-folios, and exit interviews. Faculty members and OSTEs complete lesson evaluations and compare their assessment scores when discussing an intern’s progress. A system of triangulation is in place. When interns experience difficulties, if assessments of a faculty member and an OSTE are not consistent, or if assessments of a portfolio/e-folio differ by more than one point, a third rater is called upon for assessment purposes. Due to our size, we have been able to deal with issues of bias, fairness, accuracy and consistency in assessment using faculty discussion and consensus. Team minutes provide evidence of when individual candidates’ situations are discussed to determine fair and appropriate assessment decisions.
During the internship experience assessments directly related to the functions of teaching are evaluated by both faculty members and OSTEs. Since the number of candidates and interns in cohorts and cadres is reasonably small, the Elementary and Secondary Education Teams are able to consider individual needs and accommodations required for fairness. When an individual struggles with a requirement, faculty members consider the reliability and validity of each assessment causing the individual difficulty. Accommodations may be made if deemed appropriate. A comparison of mean scores between the final faculty supervisor assessments and final OSTE assessments during the internship experience for the previous two years provides evidence of consistency in evaluation procedures.
Table 22
Final Internship
Assessment Comparison:
Dedicated Professional,
Content Specialist,
Educational Designer,
Educational Facilitator,
Educational Evaluator
Reflective Professional
4=Exemplary, 3=Proficient, 2=Needs Improvement, 1=Inadequate
Source: Candidate Files in Field Experience Office, prepared by gt
* Based on 97% of candidate files ** Based on 93% of candidate files
Each program is presently looking at admission criteria, as well as gates in the programs, to determine their ability to predict candidate success. For example, both the elementary and secondary programs have examined the patterns of specific candidates or interns who have failed to complete the program to determine whether predictors such as Entrance Interviews, course grades/incompletes, or para-competencies could have predicted difficulties earlier.
In addition to monitoring the knowledge, skills and dispositions of our candidates as they progress through our programs, we monitor the effectiveness of our programs in developing the competencies required for success. We use data from the following assessments to manage and improve the quality of our programs:
- Teacher Education Graduate Questionnaire
- Teacher Education Administrator/Employer Questionnaire
- Student Reactions to Instruction (Fall 2005 – Student Course Evaluations)
- Mid Program Evaluations of Candidates
- Exit Interviews
- End of Program Candidate Evaluations
- On-Site Teacher Educator Evaluations of Interns and Cadre Leaders
- Intern Evaluations of OSTE and Cadre Leaders
- PRAXIS II results for candidates
Candidates are asked to complete Student Reaction to Instruction (SRIs) questionnaires at the conclusion of each semester. These data are reported to instructors after the semester is over. This process provides instructors and the division chair with information about student perceptions and ways in which courses could be improved. Beginning Fall 2005 a revised form named Student Course Evaluations, SCE, was implemented.
During the mid-program conference near the conclusion of the Internship I experience, candidates are apprised of how they are meeting program expectations. During this conference candidates are asked to share how their experience has been, and how their faculty mentor might be able to enhance their learning experience during Internship II.
The exit interview is the final program requirement. After the candidate has addressed the questions posed by the faculty members and OSTEs, s/he is asked to share insights regarding changes to the program. These comments are later discussed at team meetings.
Follow-up telephone surveys are conducted with graduates who have one year’s experience, as well as with their administrator/supervisor. Information gleaned from this process is shared with the division chair and coordinators. If concerns surface during this process, they are shared with the appropriate team to determine if program changes are in order.
At the completion of each semester a questionnaire is sent to each OSTE asking for a summative assessment of her/his intern as well as the faculty supervisor. These data are available in a binder in the Field Experience Office. The division chair and Director of Field Experience review these data, and if any problems are identified, the division chair discusses the situation with the faculty member involved.
During the final week of the final semester interns are asked to complete questionnaires assessing their OSTE, faculty supervisor, and internship experience. The forms used to gather these data are: Intern Evaluation of OSTE, Intern Evaluation of LCSC Mentor, and Longitudinal Evaluation of the Teacher Education Program. This information is shared with the division chair and the Director of Field Experience. If any problems are identified, the division chair discusses the situation with the faculty member involved.
Qualifying scores for the PRAXIS II were established as a state certification requirement for the 2004-2005 academic year. Candidates’ scores have been analyzed to determine where program changes might improve candidate knowledge for passing these exams.
Data Collection, Analysis, and Evaluation
The Division of Education maintains an assessment system that provides regular and comprehensive information on applicant qualifications, candidate proficiencies, competence of graduates, unit operations, and program quality. Due to the size of our programs much data collection, analysis and evaluation is done in an informal manner, but we believe it to be thorough and to meet our needs. The schedule for collecting data is outlined in the table below.
Table 23
Unit Assessment
System: Transition
Point Assessments
Source: Field Experience
Office, prepared
by tm
The college maintains a system for tracking students’ progress and faculty can access information about their advisees using the campus database. The division maintains a written file on each candidate as well as an electronic database of student information. Elementary and secondary teams use the written files and reports from the division files to make decisions about the candidates/interns during transition points and during discussions about program improvements. The coordinator for each program assumes the responsibility for presenting the data at team meetings. These reports also provide useful information for discussions of potential changes in Teacher Education program structure and program components. The Field Experience Office maintains an additional filing system, where information pertaining to each candidate’s practicum and internship experience is housed.
In addition, WarriorWeb provides faculty advising tools such as transcripts, schedules, program evaluations, test summaries, and student profiles as well as lists of advisees, grades and class rosters. Students may access their grades, grade point average, transcript, or schedule through WarriorWeb. All official correspondence occurs through LC Warrior Mail.
As data are collected throughout the program and results compiled, teams are responsible for discussing and making decisions regarding student performances and program effectiveness. Decisions are recorded in Elementary and Secondary Coordinating Team minutes located on EduShared. (Elementary Team minutes; Secondary Team minutes)
Use of Data for Program Improvement
The minutes of coordinating team meetings provide evidence that data are referenced regularly when making decisions about candidates and their progress, as well as when recommending program changes.
Examples of data-driven program improvements include the following:
- When Idaho initiated the Idaho Comprehensive Literacy Assessment requirement for certification, our candidates’ pass rate after the first two administrations of Standard I, Language Structures, was 39% (N of 51). In response, we added a 1-credit course, RE 401 - Phonemic Awareness/Phonics Seminar, to the elementary program. Our candidates’ pass rate for Standard I has improved in subsequent administrations.
- When Idaho initiated the Idaho Technology Performance Assessment requirement, our candidates were having difficulty passing all six tasks. Based on this data, we implemented help sessions to assist our candidates in preparing for this series of exams.
- The Elementary Advisory Board has voiced concern regarding some candidates’ ability to teach math well. Discussion of this issue led the team to request that a one-credit math practicum be developed and added as a program requirement. The possibility of adding a math practicum is currently being researched.
- Informal conversations with on-site teacher educators and graduates have led to changes in textbooks for classes.
- We are presently examining PRAXIS II results to determine if we need to make changes in program requirements.
- Data gathered from Student Reaction to Instruction forms for RE 401 Phonemic Awareness/Phonics prompted discussion between the division chair and the professor regarding a change from a one-credit course to a two-credit course.
- Reviewing institutional enrollment data which showed declining enrollment in the secondary education program prompted the design, development and implementation of the Secondary PACE program. Its success prompted the design, development and implementation of the Elementary PACE program. PACE programs provide alternative scheduling and instructional delivery for students who cannot regularly attend scheduled on-campus classes.
- Peer review of the RE 422 Content Area Reading (K-12) syllabus prompted changes in the course including a strong emphasis on teaching literacy from a diversity perspective.
Candidates also use data to improve their performance. For example, interns improve their teaching performance after receiving feedback following observations by faculty supervisors or OSTEs. Candidates who score poorly on the PRAXIS II meet with their advisors to discuss how their content or pedagogical knowledge might be improved prior to internship. Tutoring sessions are available to candidates who are not successful on their first attempt taking the Idaho Comprehensive Literacy Assessment.
Institutional assessment reports for each program are submitted annually to the Office of Planning, Research & Assessment. Decisions regarding infrastructure support are based on the data provided.
