NN4DA Authorizer Evaluations: Connections and Resources
The NN4DA hosted a discussion of authorizer evaluations. A video recording of the forum is available. We benefited from outside guests: David Hartman, from Venn Education Group; and William Haft, Tandem Education Partners, as well as the NN4DA’s own Lauren Iannuccilli and Michelle Doane.
This forum was designed to inform several efforts underway in NN4DA states to develop authorizer evaluations. To support that effort, the NN4DA has a shared folder where materials related to authorizer evaluations are posted. View the NN4DA Google folder for authorizer evaluation materials here, and feel free to upload more materials you would like to share.
Included in that folder is a draft document that captures questions that may be useful in a generic evaluation that states could draw from. Feel free to comment or edit the document. If you upload something to the folder, please shoot me a note so that I can keep track of it and help other people find specific materials that may be helpful.
We are also hosting a monthly meeting to discuss authorizer evaluations. We will meet on the first Thursday of the month, at 12:00 noon MST. We will use the same zoom link that we use for the monthly ISP call (which is at 11:00 am MST on the first Thursday of each month). Logistics for the first call are:
Date/Time: Thursday, Dec. 5. 12:00 noon to 1:00 pm MST
Login: Zoom Link
Meeting ID: 891 4883 3773
Passcode: 970912
One tap mobile: +17193594580,,89148833773#
_________________________________________________________________________________________________
A Few Reflections on Our Discussion
I personally learned a lot from the discussion and was impressed by the wealth of experience we have among our group and our friends.
What is Common and What is Different:
Context and Purpose
This is clearly another issue where state context is super important, and any evaluation’s details depends on the specific purpose. At the same time, the practices and tools we are trying to evaluate have much in common.
There is a growing convergence on what is considered “best practice”, and any evaluation can learn from others. This includes the sense of importance of broad principles, as well as many technical tools in the suite of materials that produce strong performance management and rigorous review of charter applications and appropriate review of a comprehensive body of evidence during the charter renewal process. Shared materials from other states can be helpful in thinking through this shared cannon of modern authorizing expertise.
While leveraging what is common across all states, any evaluation should be built around the local context and purpose.
That is the kind of project where I believe a network, that shares ideas and brings people together but is ultimately focused on supporting each state’s independent local work, is a helpful structure.
Georgia’s Bottom-Up Process
I was struck by the innovative approach Kelly is using in GA. While my understanding of it is probably superficial, as I understand it, they have a looming state evaluation that is based on the state’s authorizing principles. They are building a pre-test that helps districts get into a position where the state evaluation is less threatening. But despite being on the books for several years, the state doesn’t have experience with these third-party evaluations.
The districts have the opportunity to create their own pre-test that gets at what they think is the way to meet the state’s, rather streamlined, authorizer principles. Their development is informed by the state’s materials that are already developed, but there is still a lot of room for interpreting what those materials mean.
Kelly is leading an iterative process, where the Georgia authorizers are meeting in a workshop to focus a discussion on one of the principles during each meeting.
They are collectively defining what it would look like to meet that principle. This will provide a better sense of what quality authorizing really means in Georgia, as well as engagement and buy-in among the districts involved about the importance and relevance of that expectation.
This could be an approach other states would find useful. I look forward to seeing what they come up with and how the authorizers in Georgia feel about the process and its result.
Weakness and Attraction of Self-Assessments
Other states could use an evaluation or a self-evaluation to engage a large number of small or unengaged districts (e.g., CA, WI). When the NN4DA partner faces scores or hundreds of authorizers, third-party evaluations will never touch most of them. Self-evaluation may be the only realistic way to touch many of the districts.
While our experts were highly skeptical of the validity of self-assessments, state partners were interested in how to use evaluation to engage large numbers of districts, including those that may suffer from regular turn-over of charter liaisons or suffer from severely under-resourced offices.
It is interesting to see what can be done with a new self-assessment tool and how to address the issues of validity. Experienced evaluators, working on multiple states’ summative evaluations, found the authorizers almost always had an inflated sense of their practices. They found appraisals from self-assessments to be wildly unrealistic.
It is interesting to explore whether our projects, coming from a bottom-up approach, might yield more candid self-appraisals, or if our self-assessments are doomed to the same delusions of quality and “happy talk”.
That optimism/delusion may not be the best lever to engage a new district in an NN4DA partnership. But if self-evaluation helped build a sense of what is possible, or expected, maybe it could lead to engagement and improvement. We’ll see.
Evaluation as Entry-Point for District Technical Assistance:
and Fee-for-Service Revenue
Meanwhile, NN4DA initiatives could conduct third-party evaluations themselves, and use it to dive deeper into a district’s work (CO).
While leading CACSA, I used funding from a local foundation to conduct an evaluation of Denver Public Schools (DPS). They found the evaluation helpful. They used its recommendations to justify some changes they were already hoping to make and pointed at the evaluation to persuade district’s leadership about the desirability of changes, at a time when the politics of authorizing were “problematic”.
Ultimately, conducting an evaluation may lead to opportunities for the NN4DA partner groups to provide direct technical assistance, both to conduct the evaluation and then to help a district address any deficiencies that the evaluation identifies.
This combination of evaluations and linked technical assistance, could advance state efforts to provide fee-for-service assistance to districts. Evaluations could be a starting element of support that eventually creates a new funding stream for state partners.
Next Steps
Clearly, there are a lot of land mines on the road to authorizer evaluations, but also some great ways to engage an influence districts. Please join us in this ongoing discussion, and let’s share what we come up with across the network!
Alex