The controversy over Superintendent Mark Johnson’s unilateral adoption of the Istation K-3 reading diagnostic tool against the wishes of a team of experts continues to swirl as the public waits to see if the superintendent will release related records as required by law.

One reason the backlash has been so intense has to do with Istation’s limitations as a screener for dyslexia and other specific learning disabilities.  

Identifying reading difficulties as early as possible is crucial in allowing schools to provide the targeted interventions necessary for student success.  Effective screening for dyslexia requires having a reader produce sounds and read words to determine whether phonological processing problems are present.  

Because Istation is an online tool, its phonological awareness measure is limited to having students listen to a sound and match it with an answer choice instead of actually segmenting individual sounds out from a spoken word by producing it on their own. This approach is inconsistent with the International Dyslexia Association’s recommendations on dyslexia assessment and appears to fall short of meeting the North Carolina General Assembly’s mandate that students “receive the necessary and appropriate screenings” as well.  The mClass tool which has been in statewide use since 2013 aligns much more closely with those requirements.

For its part, Istation continues to claim its tool is “capable of identifying and supporting students with learning disabilities” including dyslexia.  

Now more detailed claims by both Istation and Amplify–parent company of mClass–have been provided by the North Carolina Department of Information Technology.  Actually, detailed is probably not the right word to use to describe Istation’s claims.

In the original RFP, both companies were asked by the Department of Public Instruction to provide a description of how measures “adequately and accurately identify indicators of risk for dyslexia in grades K-3.” 

Here’s the description Amplify provided in its Response to RFP:

Amplify and the co-authors of DIBELS Next support the use of a universal screener (the DIBELS Next measures assess decoding, accuracy, fluency, and comprehension) for identifying students with reading difficulties, including students at risk of dyslexia. Amplify has developed additional screening measures of Vocabulary, Spelling and Rapid Automatized Naming (RAN), that may also be administered to obtain additional information on risk for dyslexia and possible impact of difficulty on related skills.

The Vocabulary(VOC) measure is an indicator of a student’s general vocabulary knowledge. It assesses a student’s depth of knowledge of a set of grade level high frequency and high utility words that are used across domains (Tier 2 words; Beck, McKeown, & Kucan, 2002) and content specific words and whether a student has strategies for making meaning of words encountered in text. It incorporates a variety of tasks that vary across grade levels and allow for assessing multiple dimensions of vocabulary knowledge across multiple contexts. The tasks included in each grade level are those that produced the most reliable results for that grade. The number of items increases in each grade level as students’ ability to sustain attention increases with age.

The first task administered to all students is the two-question vocabulary task adapted from the work of Kearns & Biemiller (2010). In this task, students are asked two yes or no questions about a target word. This requires deeper knowledge of a word than many traditional tasks as the same word is included in two contexts or questions. Evidence of the reliability, validity and sensitivity to differences in students vocabulary skills has been documented (see Kearns & Biemiller, 2010). In grades 1 through 3, students also complete traditional fill in the blank questions. They read a sentence with a missing word and select the word that best completes the sentence from a set of four words. Distractor or incorrect responses for each item include words that look or sound similar to the target word, words that mean almost the same thing as the target word but are not correct in the context of the sentence, or are related to the target word or sentence context but are not correct in the context of the full sentence.

Finally, grades 2 and 3, students complete items that require matching a word to its basic definition. The words included in this question type are words that are included in earlier portions of the assessment. The purpose is to see if a student has basic knowledge of the definitions of these same words. 

The Spelling measure is an indicator of a student’s level of general spelling skills. It is designed based on the principles of General Outcome Measurement and Curriculum-Based Measurement (CBM; Deno, 1992). Assessments from this approach are designed to efficiently screen for students who are at-risk for difficulty – they are brief assessments of critical skills that are sensitive to student learning and growth overall. CBM measures do not assess all skills within a domain but provide a snapshot of a student’s skills in a given area using tasks that are instructionally useful and can be reliably administered (Deno, 2003).

The Amplify spelling measure incorporates the key features of CBM Spelling measure design and administration. It is administered on a computer or tablet so typical procedures for administering the measure were modified to fit the software environment. The target word is spoken (by the computer) and the student uses letter tiles to spell the word. Both correct letter sequences (CLS) and words spelled correctly (WSC) are calculated.

The Rapid Automatized Naming (RAN) measure indicates how quickly students can name aloud numeric symbols. While there is not strong agreement in the field of exactly which cognitive processes RAN is measuring, a large body of evidence has documented RAN as one of the best predictors of overall reading skill, including word reading, text reading, reading fluency, and comprehension and an area difficulty for students with reading disabilities (Araujo, Reis, Petersson, & Faisca, 2015). Deficits in rapid automatized naming have also been shown to be a robust indicator of risk for dyslexia in children (Gaab, 2017). 

Because of the strong predictive relationship that RAN displays with tasks that measure various reading skills, researchers hypothesize that completion of RAN tasks requires the coordination of multiple processes. “The seemingly simple task of naming a series of familiar items as quickly as possible appears to invoke a microcosm of the later developing, more elaborated reading circuit” (Norton & Wolf, 2012, p. 427). The full circuit requires coordination of attention, working memory, visual processing, phonological processing, etc., individual processes also required for reading. 

The addition of these measures is in line with the definition of dyslexia included in NC HB 149. These measures provide additional information about a student’s processing, spelling, and decoding abilities. In addition, the Vocabulary screener and the existing mCLASS:Reading 3D measures allow teachers to continue to understand the overall reading skills of students and potential “secondary consequences” or problems in reading comprehension or vocabulary as described by North Carolina’s definition of dyslexia. The addition of these measures in the screening process aids in the development and implementation of targeted interventions with ongoing progress monitoring through a multi-tiered system of supports. Progress monitoring data should then be used to determine whether additional assessment and evaluation are needed for the student.

The Vocabulary measure is administered on the computer. Each item and all answer options are spoken (by the computer) to the students who then select their answer choice. The Rapid Automatized Naming and Spelling measures are administered on the online student testing platform. To administer this assessment, an educator enables the measures from Online Assessment Management portal then logs into the student account to launch the RAN and Spelling assessment with the individual student. Both the teacher and student can view the screen and listen to the audio prompts that guide the student through a model, practice, and each assessment item. The teacher controls the input device and selects student responses.

For students with adequate self-regulation and computer skills, teacher assistance may not be needed for Spelling measure and students can interact directly to enter their response.

A student may be flagged as demonstrating additional risk for reading difficulty including dyslexia when the DIBELS Next composite result is in the Well Below Benchmark range and results from either the Spelling and RAN measures are in the Well Below Benchmark range. Educators will have the option to include this information in the Home Connect Letter to share the student’s performance on these additional measures with parents. Please see Appendix G (pgs. G-1 – G-23) for more information on the Vocabulary, Spelling, and RAN measures.  

Now take a look at the “meh, we got that covered” response from Istation:

That’s literally all they had to say about it.  And somehow, it was enough for Mark Johnson.

Newly released documents obtained through a public records request with the NC Department of Information Technology reveal that the Department of Public Instruction was fully aware of Istation’s dyslexia screening shortcomings–and chose to purchase the tool anyway.

When the North Carolina Department of Public Instruction originally issued its Request for Proposal (RFP) for a K-3 literacy assessment tool, one requirement was that vendors explain how their product would identify students at risk for dyslexia.  

The image you see below is from the RFP that DPI released in the fall of 2018.  Business Specification #8 requires vendors to provide a description of how measures “adequately and accurately identify indicators of risk for dyslexia in grades K-3.” 

The RFP includes a link to legislation passed by the General Assembly in 2017 which mandates that students with specific learning disabilities such as dyslexia “receive the necessary and appropriate screenings” and tasking local boards of education with reviewing “diagnostic tools and screening instruments used for dyslexia…to ensure that they are age-appropriate and effective.”   

Clearly it was a priority in the fall of 2018 that the Department of Public Instruction procure a reading assessment tool that can be used to identify students who are at risk for dyslexia.  That was the understanding of the two robust evaluation committees that thoroughly reviewed the available vendors before recommending to Superintendent Mark Johnson in December of 2018 that mClass was the best choice for North Carolina’s children. It’s also an approach which is consistent with this Dyslexia Topic Brief produced by DPI’s Exceptional Children division in 2015, which points out that “Assessments that serve as screening tools can provide early warning indicators of students who are at risk of reading failure.”

After the RFP evaluation committees both recommended mClass, the RFP process was cancelled.  DPI has yet to explain what led to the cancellation, although last month a spokeswoman for the department referred in a cryptic statement to “actions that jeopardized the legality of the procurement.”  

Whatever the true cause for the RFP cancellation, the procurement process was restarted in the spring with a new evaluation team, and Requests to Negotiate were sent to two vendors:  Amplify (the company which produces mClass) and Istation. DPI’s Contract Award Recommendation document lays out the following timeline for what occurred:

According to DPI’s letter, the Evaluation Committee that chose to award the contract to Istation consisted of the following individuals:

Note that it’s a much narrower team than the roughly 20-25 knowledgeable statewide education leaders–including specialists in general education, special education, and English language learner services, school psychologists, representatives of Institutions for Higher Education, and dyslexia experts–who made up the two committees that originally evaluated the available screeners before recommending mClass to Mark Johnson.  You know, right before the RFP was cancelled.

The team that chose Istation specifically referred to the program’s lack of a separate dyslexia component as a weakness of the program (while also holding up Amplify’s dyslexia component as a strength of mClass):

In explaining the final choice, Istation’s deficiencies with regard to screening for dyslexia were noted by the Evaluation Committee again–and then dismissed as being outside the scope of the procurement.  

The law referenced here by DPI (GS 115C-83.1) is the primary goal of the Read to Achieve program:

The goal of the State is to ensure that every student read at or above grade level by the end of third grade and continue to progress in reading proficiency so that he or she can read, comprehend, integrate, and apply complex texts needed for secondary education and career success.

I’d argue that effective dyslexia screening is part of ensuring “that every student read at or above grade level by the end of third grade,” and that it should have been considered a “primary obligation of this procurement.”

For a young child, failing to have a learning disability detected early on can completely alter his or her life trajectory.  

Our students deserve better than this.

The “e-learning” company Istation has been engaged in a public relations offensive ever since NC Superintendent Mark Johnson’s early June announcement that he had awarded a three year, multimillion dollar contract to the company for use of its K-3 literacy assessment tool led to massive pushback from North Carolina’s public school parents, teachers, and superintendents. 

It isn’t going very well.

Johnson’s unilateral decision disregarded input of two teams of professional educators who overwhelmingly recommended students continue using the mClass tool which has been in North Carolina schools since 2013.  The aftermath has seen both public outcry and official protest by Amplify, the company that produces mClass.

One key difference between the two products is that mClass requires one-on-one interaction between student and teacher, while Istation has young children sit and work alone on a computer.

When he announced the news about Istation, Johnson referred to the company’s “proven results of helping students grow.”

Durham school psychologist Dr. Chelsea Bartel, whose research focused on identifying and implementing effective interventions to improve skill deficits, reached out to Istation to find out more about those proven results.

Istation obliged by providing links to a handful of studies, three of which were written by a Dallas professor who it turns out actually works for Istation:

But it was an independent study by Tarleton University professor and education researcher Rebecca Putman that caught Dr. Bartel’s eye.

As provided by Istation, Putman’s study “Technology versus teachers in the early literacy classroom: an investigation of the effectiveness of the Istation integrated learning system” is behind a paywall.  You’d have to pay $39.95 to actually read anything beyond an abstract that mentions Istation’s “statistically significant effect” on kindergarten literacy skills.

If you don’t have that much cash to spare, you could just read the Istation.com summary of the article entitled “Does Istation’s Technology Improve Learning? Research Says Yes!” by Istation’s digital marketing manager Rachel Vitemb.  Not surprisingly, the Istation summary of Dr. Putman’s research focuses exclusively on positives, crediting the software with “improved students’ letter-sound knowledge as well as their ability to hear and record sounds and write vocabulary.”

But if you only read the Cliff’s Notes version provided by Istation’s marketing team, you’d be missing out on some of the study’s most important conclusions.  Fortunately, the study’s author graciously provided Dr. Bartel with the full piece free of charge.

Dr. Putnam’s article explains that she gave kindergarten students the ISIP-ER assessment at the beginning of the study, then had them spend 135 minutes per week using computer-based Istation reading interventions.  The study lasted 24 weeks. At the end of the study (so, after approximately 54 hours of kindergarten children working alone, wearing headphones and clicking boxes on a computer screen), the students took the ISIP assessment again.  Comparing the results, Dr. Putnam found the program to be effective in teaching students early literacy skills such as letter sound recognition “that require drill and repeated practice”.

I’d bet $39.95 you could get even better results if you spent those 54 hours working on sound recognition with a human teacher.

Here’s what Istation left out about Putnam’s research:

In the full study, Dr. Putnam explains that, in addition to assessing Istation for its effectiveness in supporting early literacy achievement, her goal was to determine whether Istation served as what she refers to as a “more knowledgeable other” when compared with a classroom teacher: 

 Another purpose of this study was to investigate whether Istation is an adequate substitute for the more knowledgeable other (MKO) in the classroom. In other words, did this particular application of technology scaffold students’ learning as effectively as a classroom teacher and serve as a MKO? Generally, a MKO refers to a person who has a higher level of understanding and knowledge about a particular topic or concept (Vygotsky 1978).

Her conclusion?  It isn’t.

Istation does not appear to be an adequate substitute for the MKO when it comes to creating meaning and applying early literacy skills to more complex literacy tasks. Based on the data from this study, early literacy skills that require the integration of a variety of literacy skills and strategies, such as reading and comprehending a book, understanding concepts about print, and reading words, seem to require the instruction and feedback of a human, one who is able to interact, provide multidimensional feedback and allow for the student to take on a more active role in the social interaction.

At the end of a study in a section titled “Implications for use of Istation in early literacy education,” Dr. Putnam acknowledges that technology is often seen as a quick fix for literacy problems and calls for more independent research so that it can be incorporated into early childhood classrooms in a way that is healthy for students:

There is increasing pressure on school districts to find quick and efficient solutions to perceived problems in reading achievement, and often, the focus is on improving early reading skills (Paterson et al. 2003). A popular solution to these problems is educational technology. As the use of technology becomes more prevalent in elementary schools, and particularly in early childhood classrooms, there is an increased need for independent research on the relationship between technology and literacy in order to justify (or discourage) districts’ large expenditures and inform their decisions about how to integrate technology into the instructional curriculum (Tracey and Young 2007).

Istation is the wrong choice for North Carolina’s children, and the research that the company itself is sharing just confirms it.

Note:  Chelsea Bartel has conducted an in-depth review of the available research on Istation.  You can read her summaries of the studies and takeaways in her piece “Try Again, Istation.”