#FailForward: The Pilot

An entire chapter of the book Go Blended by Liz Arney is dedicated to running successful pilots of blended learning initiatives. For those looking for an in-depth explanation of a successful pilot, order her book (seriously, it’s fantastic). This post will outline a few of the lessons I learned the hard way from our most recent pilot.

At ReNEW, we run a year-long schedule in an attempt to prevent the ‘summer slump’ for our students. When October rolls around, we take a two-week long ‘break’ called Intersession.  This 10-day period is an opportunity for a small group of our students to receive remediation – to fill a few gaps and help catch them up before the next quarter starts. Due to the personalized nature of intersession, blended learning software is used to meet students on their level. We decided to pilot an adaptive math program to compare its effectiveness with a program currently in use on many of our campuses. Specifically, we wanted to see how a truly ‘adaptive’ program would affect student achievement, and to determine if the program would be worth piloting on a larger scale later on in the year.

In attempting the pilot, I learned a number of lessons the hard way; many of them are quite obvious, and all of them are covered in depth in Liz’s book.

1. Plan Ahead
I have all sorts of excuses for this one, but ultimately it became clear that I needed to find a way to set-up everything as far in advance as possible. We didn’t know who would be on the rosters until just days before Intersession started, which put us in a holding pattern. In retrospect, I could have pushed for a full SIS integration weeks in advance, and followed-up with the unique intersession classes once we had rosters in hand. Instead, when it came time to load the students, the SIS integration took a shockingly long time – leaving teachers and students hanging without access to the program they had been promised for the first three days of Intersession.

2. Vendors Make Mistakes
Despite my egregious frustrations from our set-up this fall – I had no idea vendors could be so difficult – I trusted our new friends when they told me we could be up and running in the short timeframe we needed. I even trusted the automated email on Sunday night alerting me to a successful student data upload, only to find out it unexpectedly that rosters were not working Monday morning. This is a consistent problem with edtech companies. So what seems to be happening? For one, the sales and support departments never seem to be on the same page. As our data guys continue to remind me, I need to start talking more directly with the folks that make the software work – not the ones who are selling me on it. We could have alleviated a number of problems along the way with more streamlined and accurate communication from the vendor.

3. Communication Matters
To build on that second point, accurate, timely communication is critical to the success of a pilot (and most of what we do!). School leaders need to communicate their intentions – curriculum needs, student rosters, etc. – ahead of time. Vendors need to communicate the actual over the theoretical. Most importantly, I have come to learn how important it is for me to communicate more often and with greater clarity with everybody: leaders need deadlines and updates; teachers need to know what to expect when; vendors need to know our needs.

4. Clarify Expectations
Liz does a wonderful job of explaining the how and why of this. How we communicate the purpose, desired outcomes, and strategy behind a pilot is as important as the pilot itself (sort of). Mis-communicating can cause a good pilot to go bad – quickly. Controlling the expectations of all involved will allow your pilot to give you the information you need to make decisions moving forward. I’ve learned this lesson best from the vendors themselves, actually – nearly every single one has promised something – a timeframe, SIS integration capabilities, program features, etc. – that was unrealistic and ultimately fell short; instead of planning appropriately and communicating early with teachers, we get frustrated as our expectations aren’t met. This disregard for the experience of the teacher forces me to rethink the usefulness of the product. From this experience, I am learning to push vendors to provide more accuracy and clarity, in hopes that I can do so with our leaders and teachers.

If I take one thing away from my experience with vendors this year, it’s to expect delays and false promises. Every yes comes with an asterisk – “yes, we can integrate with Clever” could have any number of caveats, from “if you pay for it,” to “if you create extremely specific structures that mimic our system (and don’t work with yours),” to “if you wait 6 months!”

Our pilot will conclude in the coming days. Hopefully, we’ll learn about the impact of the product itself on student learning – and not more about how not to pilot.

Fail forward!