Leveled Literacy Intervention Program

I have decided to design an evaluation for a literacy program that has been in existence for the last 6 years at the elementary school where I teach. The program is called Leveled Literacy Intervention Program (LLI). I have seen colleagues use the program but being a junior teacher, I have not had the need to use this program in my class as it is designed for the primary grades.  I am interested, however, in the effectiveness of the program as I anticipate using it in the future when I have the opportunity to work in the primary grades.

Step 1:  Select and describe program context

Focus: Leveled Intervention Program (LLI) is a supplementary intervention program designed for teachers to use with small groups of students that are struggling with reading and writing. Fountas and Pinnell created this program with the belief that powerful early intervention, in terms of a child’s literacy skills, can change a child’s path in life. It is designed to prevent literacy difficulties in students rather than dealing with reading and writing deficits in junior and intermediate grades.

Goals: The goal of this program is to bring struggling readers up to grade level. The program is designed to improve students’ reading ability in terms of accuracy, fluency and comprehension as well as improving their writing skills.

Resources: The program provides the teacher (resource teacher/support teacher) with a kit that comes in a series of boxes. Included in the kit is the following:

• a program guide
• a lesson guide which includes 120 thirty minute lessons that focus on reading texts, writing and phonics
• small reading books, 4 copies of each
• writing books
• take home books, which are black and white versions of the reading books
• student folders and take home bags
• a prompting guide as a quick reference for language the teacher can use
• a data management CD, which allows for tracking and printing reports

Selection of Participants for the Program: The LLI program is designed to support the lowest achieving students in the class, including those with English as a second language and those who have been identified as having special needs. The ideal target level for the intervention program is for Grade two students who are struggling with reading and are one or more grade levels behind in literacy skills. Student selection for the program happens with the observations of the classroom teacher and the results from a reading assessment tool, such as PM Benchmarks, where the student’s instructional and independent reading levels are determined. To form an LLI group, three students with similar reading levels are chosen and meet with the LLI teacher daily for intensive instruction.

Program Delivery: Once an LLI group has been formed, as close to the beginning of the year as possible, the LLI teacher delivers a thirty minute lesson daily, for a length of eighteen weeks with the possibility of extending up to twenty-four weeks if necessary. The length of program is dependent upon the progress of the participants. It is possible to exit students from the program as early as after eight weeks of instruction, however on average, students who are reading up to one year or slightly more below grade level, require eighteen weeks of instruction. The LLI program is best suited to be administered by a teacher other than the classroom teacher, although possible, such as a support or resource teacher. The program is intended to be a supplement to the classroom reading instruction and if administered by a support/resource teacher, it is possible to facilitate multiple LLI groups in one day, allowing for a larger number of students to receive support.

Community demographics/students accessing the program: The school community is very diverse with high needs, including a high ESL population, low income and subsidized housing and a family shelter. There is a large number of newcomers from Syria in the community and many of these students have had limited education.

Step 2:  Purpose of evaluation and specific evaluation questions

The purpose of evaluation of the LLI program is determine its effectiveness in improving students’ literacy skills. I would like to examine and assess the outcomes of the program to determine if the program was influential in bringing students’ literacy skills up to grade level.

The following key evaluation questions are paired with enabling questions for further elaboration.

1. To what extent has the LLI program improved literacy skills of the participants?
   1. What is difference in participant’s reading levels from the entry point to exit point of the program?
   2. What are the trends in progress rates amongst participants?
   3. What percentage of participants are at grade level upon completion of the program?
2. In what ways has the program impacted the literacy skills of the participants?
   1. Have the participants’ reading levels improved?
   2. Have the participants’ writing skills improved?
   3. Have the participants’ systematic phonics skills improved?
3. To what extent do the activities and resources impact the program and the desired outcome?
   1. Do the provided program resources support the desired outcome of the program?
   2. Are the program’s activities engaging and beneficial for the participants?
   3. Are the program’s resources and activities feasible for the teachers/those who administer the program?
4. In what ways do the conditions of the program impact its success?
   1. Where is the ideal location for the program to take place?
   2. What are the optimal conditions (room size, table shape, lighting, noise level) of the location?
   3. What is the ideal number of students in a group?
   4. Who is the ideal person to administer the program?
   5. How does the nature of the particular participants and the make up of the group of participants impact the success of the desired outcome of the program?

Step 3:  Constructing a program theory

Program Theory for LLI Program:

If students who are achieving lower than grade level in literacy, engage in daily, intensive, small group instruction in reading, writing and word work for a period of 24 weeks, then they will improve their literacy skills, achieving grade level in terms of reading and writing.

Theory of Change: Logic Model for LLI Program

Variables that contribute to impact and outcomes:

1. English Language Learners-in the context where this program is being used, there is a large number of English Language Learners (ELLs) where following instructions and accessing the reading texts may be challenging. It is important to assess students’ reading levels and abilities prior to the start of the program to form small groups of students with similar reading abilities.
2. Scheduling/timing: daily LLI lessons are important for the impact of the program. This is a challenge in any classroom as there are many programs and activities going on regularly. Committing to a 30-minute lesson can be challenging as there can be many surprises and obstacles that occur when working with children.
3. Different rates of progress: naturally, students will accelerate at different rates. There is an option to move students out of one group once they have progressed past the group’s reading level and into a higher group. As well, students who do not make significant progress in the 24 weeks, can continue in the program.
4. Attendance: if students are absent, they will miss the text that was introduced that day and will be working on the following day. Students can catch up at home by reading the texts they have missed. Multiple absents can impact the progress of a student in the program.
5. Home connection: it is impossible to monitor and ensure that additional reading, writing and word work is being done at home. Without the support from home, the rate of acceleration of progress can be impacted.
6. Students: each student is unique and comes to the program with their own attitudes, feelings and behaviour styles. These aspects of a student can impact the outcome and success of the program for the individual student as well as the others in the group.
7. Teacher or support staff: the facilitator of the LLI program is an integral part of the success and impact of the program. Their attitudes, approaches and preparedness for the program can greatly impact the outcome and success of the program.

Step 4:  Evaluation approach and rationale

The evaluation approach I will use for the LLI program is the Contribution Analysis approach. This approach is designed to help leadership or administrators determine the contribution a program is making towards a desired outcome. This approach aids in understanding of role the program plays in achieving the documented outcome. This approach is helpful in situations where the program being evaluated is not an experimental program but rather one that has been implemented for its clear and well-developed theory of change.

I feel the Contribution Analysis approach is a good fit for designing an evaluation for the LLI program for a number of reasons. First, the LLI program has been in existence at my school for several years and comes with a clear and reputable theory of change. Second, I am not interested to see whether or not the program achieves the goal of increased literacy skills but rather what role the program plays in the desired outcome. Finally, this approach is designed to confirm and revise a theory of change, not pick it apart and find fault in it. The Contribution Analysis approach will provide me with evidence and reasoning as to the contribution the LLI program plays in improving literacy skills in the lowest achieving students.

Step 5: Data collection methods and analysis strategies

Step 6: Promotion of evaluation use

The goal of the LLI program is to provide intensive support for the lowest achieving students to improve their literacy skills to be at grade level. The program evaluation design is intended to determine the contribution the LLI program makes in improving literacy skills. Upon completion of the data collection and analysis portion of the evaluation, the promotion of evaluation use will determine how the findings will impact the program, the stakeholders and intended users, their knowledge and understanding of the program as well as the value of the program itself.

Given the elementary school context of the LLI program, the presentation of evaluation findings would be fit for a staff meeting at an elementary school, a grade level meeting or an ISSP(In school support program) meeting. Given the stakeholders and intended users being teachers, support staff and school administrators, a presentation including statistics, visual representation of data and an entry/exit data comparison would be well received and easily understood. In terms of evaluation use and impact, evaluation communications and reports should be designed to grab and hold readers’ attention (Torres, 2005). A simple, one page, snapshot overview of the evaluation findings, using elements of design to engage the audience and enhance readability could be presented to stakeholders, intended users and school administrators. To share findings of the key evaluation questions concerning the quantitative data (i.e. improvement in reading levels and writing levels), a broken-line graph would communicate the level of improvement over time amongst the program’s participants. A creative way to illustrate findings to the key evaluation questions concerning qualitative data (i.e. conditions of the program) would be to share photographs of conditions that positively impacted the program. Photographs provide an opportunity for stakeholders to construct their own meaning of the findings. The photography can be a catalyst for dialogue that stimulates audiences’ participation in interpreting important events and experiences, enabling the findings to be used for developing action plans (Torres, 2005).

The challenge in promoting evaluation use in the context of an elementary school is not convincing stakeholders and intended users of the program’s worth, as the data analysis (graphs and charts showing progress over time) and the program’s reputation will speak for itself but instead, getting intended users to see its value as worth taking the time to integrate another program into an already brimming school day. The solution to this challenge would be to put the onus on the support or resource teachers to deliver the program to eligible students, working with the support of classroom teachers but taking on the lead role in the program. The challenge with relying on support or resource teachers to administer and deliver the program is the lack of staff members in the support or resource role, which in turn, spreads their availability too thin to deliver the program in its optimal manner. As a collaborative staff, support or resource teachers, classroom teachers and administrative staff need to be creative problem solvers, thinking outside of the box, to find ways in which the program can be delivered.

Powerful early intervention can change the path of a child’s journey to literacy. A growing body of research shows that reading difficulties are preventable with effective intervention programs (Fountas and Pinnell, 2009).

Step 7: Commitment to Standards of Practice

This program evaluation design adheres to the Program of Evaluation Standards as follows:

Utility: The evaluation is designed with key evaluation questions and processes that are meaningful and valuable to the stakeholders and intended users, intending for them to rediscover, reinterpret, or revise their knowledge, understanding and attitudes about the program.

Feasibility: The evaluation is designed to fit within the parameters of the program, collecting data for key evaluation questions that is responsive to the way the program is intended to operate, thus increasing the effectiveness and efficiency of the evaluation.

Propriety: The promotion of evaluation use provides visual representation and clear readability in findings, limitations, and conclusions to all stakeholders and intended users, offering support that is fair and just.

Accuracy: The presentation of evaluation findings are derived from interpretations, analysis, conclusions and judgements that are clearly and completely documented in ongoing data collection as to increase the dependability and truthfulness of the evaluation.

Evaluation Accountability: The evaluation design includes developed key evaluation questions, a theory of change, a logic model, data collection method and tracking and analysis strategies, demonstrating a fully documented evaluation.

References

Canadian Evaluation Society. (2014). Program Evaluation Standards. Retrieved from https://evaluationcanada.ca/program-evaluation-standards

Fountas, I. and Pinnell, G. (2009) Leveled Literacy Intervention: Program Guide. Heinemann, Portsmouth, NH.

Mayne, J. (2008). Contribution Analysis: An approach to exploring cause and effect. Retrieved from http://www.betterevaluation.org/plan/approach/contribution_analysis

Saunders, M. (2012). The use and usability of evaluation outputs: A social practice approach.

UNICEF Innocenti. (2014). Data Collection & Analysis. Retrieved from   https://www.youtube.com/watch?time_continue=34&v=HFGVJJMDo4I

Wilder Research. (2009). Program theory and logic models. Evaluation resources from Wilder Research. Wilder Foundation.