How Do You Know It’s “The One”?
I have a post-it on the front of my Moleskine notebook that accompanies me everywhere: “What is the ONE metric that matters?” I keep this visual to remind myself to constantly keep high-level programmatic goals top-of-mind as I navigate our decision-making as a start-up non-profit. But how do you know what metric matters most in a field that has, to date, been using inadequate proxies for measuring financial literacy? In our field, most of us measure impact by looking at student-reported data from surveys centered around knowledge gain and attitude change. Can you define a mortgage? Are you more likely to set a budget because of this program? How much money will you earn if you put $500 into a savings account with .05% interest? (By the way, some measure impact even more imprecisely by simply reporting the number of youth they have reached.) But none of these accurately tell you whether your student will actually be a more financially responsible adult.
Part of this is because it’s very difficult to identify a financially literate adult. It’s far easier to point out financial illiteracy than it is the inverse: we know when we see someone spending beyond their means or paying off the minimum on their credit card for years or splurging on a want despite more pressing immediate needs. But identifying a financially literate adult is far more difficult because it’s often what they’re NOT doing — how do you measure NOT spending? In short, you can’t know whether someone is making good financial decisions based on a bank account statement. It may be safe to say that most financially stable adults save regularly, spend less than they make, have positive net worth, and pay off their credit card bills on time — but for many of the students we serve — students living below the poverty line — these metrics may not be possible, despite their best efforts. The other part is that no one knows what kinds of interventions will put teens on the path to good financial practices in the future. Is it opening a savings account for a child? Is it pushing kids to go to college? Is it teaching kids how to budget? These are moving targets that we think about — and extensively — on a daily basis. I have had more conversations than I can count exploring the intricacies of this thinking — and that’s just trying to figure out what the end goal should look like. The waters get muddier when you think about the early indicators that help us know our kids are on track, and, beyond that, the one metric amongst many that really matters in making that determination.
We’re right in the muck right now, talking through and around a refined impact measurement framework that encompasses a lot of these questions and concerns, but we had a phenomenal conversation with one of our board members, Shayne Evans, Director of both the University of Chicago Charter School and Urban Education Institute, that proved helpful to me and that I in turn wish to share out. He showed us the framework he uses at his charter high school. The end goal for him is college graduation for all of his students. He then backwards designs from that goal by identifying various metrics that help him determine whether his kids are on track while in high school. He clusters these metrics into high-level categories — High School Success, Student Progress, School Values — and then provides primary and secondary measures for each that he can then report on regularly. Despite the fact that he tracks a number of different metrics, he advised us to track many, but commit to one — and then really build it out and reflect growth. You can always move on to another metric once you’ve significantly moved the needle, but focusing on, say, 15 different metrics results in confusion and imprecision. You can also begin by using imperfect measures — proxies, almost — and continue to iterate until you have the right measure and the right data coming in. But the end goal is to communicate continuous growth on one important metric and then be able to explain both the importance of that metric and the reasons for its growth in a convincing way.
Returning to my initial question: how do you know your metric is “the one”? I think part of this is thinking carefully through desired end outcomes. What do we want our kids to report back to us regarding their financial situations when they are adults? And then how do we backwards track those goals to what we can realistically accomplish within the classroom? If we want all of our kids to grow up and budget, what do we need kids doing when they are 17 years old and have no regular income? What’s realistic? I think we’re getting close to identifying some leading metrics, but part of it, as one of our other board members said, will be “a big leap of faith.”