Video highlights

Transcript

Daniel Emmerson 00:01
Welcome to Foundational Impact, a podcast series that focuses on education and artificial intelligence from a nonprofit perspective. My name is Daniel Emmerson and I'm the Executive Director of Good Future Foundation, a nonprofit whose mission is to equip educators to confidently prepare all students, regardless of their background, to benefit from and succeed in an AI infused world.
Daniel Emmerson 00:28
Hello everybody, and welcome to another episode of Foundational Impact. I'm here today with Elizabeth, who's joining us from the Gates Foundation. Elizabeth, it is wonderful to have you with us today. How are you doing?
Elizabeth Moore 00:41
I'm doing great. It's wonderful to be here.
Daniel Emmerson 00:43
Really, really appreciate being able to talk to you about your work and the incredible initiatives that you're responsible for. I think first of all, it'd be wonderful to hear a little bit from you about what Gates foundation is.
Elizabeth Moore 00:55
Yeah, absolutely. So. So I work in the US on the education team. The team is called K-12 and we are focused on improving the rigor and the quality of the instructional materials that are used in the classrooms, researching to ensure that things do have an evidence base and there's efficacy of those same products. Also on, I would say demand side, but also in signaling awareness of what you should be looking for related to the evidence base, the quality and the rigor. So this looks like a whole lot of different things. You know, I have a portfolio. We make investments, we keep an eye on the market, keep an eye on new products, keep an eye on what the evidence is saying and look for the ways to accelerate, incentivise influence where we can use some leverage either to exert some market pressure, to improve qualities, to scale qualities, scale product quality, end products. So it can look a whole lot of different ways. But US context, K-12 market, and I'm both doing things on supply side and demand side.
Daniel Emmerson 02:16
And how much of that relates directly to the way schools are addressing AI?
Elizabeth Moore 02:24
Well, every day it relates more and more, which I think is what everyone would say just the pace is, you know, we spent years working on implementing common core standards and getting instructional for us, you know, updated standards or curriculum I think would be more the way it would translate for the UK. Every couple of years there's a new edition of a textbook and it's updated and it might have a new sort of pedagogical tilt to it given whatever either the states are looking for or has been promoted. And now we're in this world where every week the new models are released and they're better at this and they're not so good at this and there's know sort of single point solutions using AI popping up all over the place. So it's like a bit of the wild west and it's moving at a pace that we're challenged to keep up with in like a thoughtful way. Right. So what is, where are the, where are the ways that we can continue to do the work we were doing related to just instructional materials.
Daniel Emmerson 03:39
Right.
Elizabeth Moore 03:39
And improving learning. Right. That, you know, goes with that, like high quality, rigorous instructional materials. What does that look like in AI? So we're figuring it out too.
Daniel Emmerson 03:48
And how much of that figuring out is connected with what's happening at Microsoft? Is it specifically related to that or are you looking at a broader strategy?
Elizabeth Moore 03:58
We're working with all the frontier labs, right. OpenAI and Google and Anthropic. We have relationships with all of them. Our specific lane is sort of how can we with those relationships influence the, have some pressure to improve the educational use cases of the models, which again can look very different depending on how we're working. So that's the piece we are leaning in on. And also, you know, with philanthropy, which, you know, there's some things the government is going to take care of, there's some things that the market pressures are going to naturally sort of incentivise and solve for. And then there's this gap of, of things that are a great place for philanthropy to lean in. But this is a new world too for philanthropy. You know, we can. And it's moving fast. I mean, I don't know if you're experiencing this too. Just my, like being up to date on the market in the field and the research and the evidence is requiring different muscles I think than it used to for people to sort of have a point of view on what was research based. 
Daniel Emmerson 05:18
That's for sure. And we certainly get asked quite a lot about, you know, how do we address this or what's the best or the, the best tool that we might use for this situation. How do we navigate this new innovation? And you can't always be on top of everything that's out there. I don't think that's possible these days. There's a new model dropping every day or two. But is that the sort of thing that schools are coming to Gates foundation with as well, or is it rather that you're going to them? I'm just trying to understand a little bit more.
Elizabeth Moore 05:51
Yeah, no, it's, it's, it's not transparent. We. So where we might have tried and lean in there would be not in a school, by school or district, by district or even there is some advocacy and policy work happening that I'm not, I'm not leading on that work. So the ways that we would lean in are public goods that, you know, guidelines and standards. Either that Gates has invested in another credible, someone with a lot more expertise than just Gates Foundation to author, we would help sort of boost that through different channels, teacher networks, superintendent networks, statewide networks where everyone's asking these same questions. So we would sort of try and operate by providing really evidence based guidelines and standards that states and districts and schools can pick up and use. That's one lever. The other is on the supply side, like sort of working with certain developers to improve their products and test their products so that there's a little bit more evidence in the market of what's having an impact on learning and then, and making that evidence generalisable because you know, that's sort of like, okay, well everyone now should know that. Here's some, you know, here's what we've learned from this study around best practices for making the tutors more interactive. The other place we're leaning in is on that evaluation of the models. Sets and benchmarks that again, ideally would both exert some pressure on the developers to like pay attention when the models aren't really great at something related to education and support like raising awareness on the school leaders, on the people making policy, on the people using the products that, you know, you should be demanding this, you should be looking for this.
Daniel Emmerson 08:00
So when it comes to sort of responsible deployment or responsible adoption of this technology, where are the main bottlenecks here? Is that with the models themselves or is it with the product or is it with the school's ability to be able to utilise them effectively?
Elizabeth Moore 08:19
That's a really great question. I wouldn't say that the models are a bottleneck. Keeping up with the model. There might be a bottleneck in keeping up with evaluating the models, which is a different kind of bottleneck. They're getting better all the time. Sometimes they sort of all get better in some like sort of burst. And then, you know, sometimes they actually get better at some things but drop in quality of other things. Which is why I think benchmarks, validated benchmarks from third parties are just like so essential in education. I don't know that they're getting better at specific education use cases at the same pace. They're getting better at other things. Right. So not so much a bottleneck, but as like I was saying earlier, like a place for philanthropy, I think, lean in. Right. Like how do we, how do we keep elevating the importance of these models which are going to underpin a lot of educational products getting better. On the supply side, I would say it's less of a bottleneck and more like the wild west, right. I mean, we're seeing single point sleep solutions using AI in a way that, like when ed techs, you know, like there was sort of, you know, there was that upsurge in digital products, right. And now it sort of seems, you know, people are vibe coding things left and right here. Teachers are doing it and like making their own lesson generator. So you've got like one end of people just embracing AI to do the thing they've always wanted to do in their classroom. Quizzes, item generators, lesson generators. You have these single point solutions that are using AI, like multimodal solutions to give feedback on something that's visual or verbal, but very specific. Just providing feedback on the thing you wrote on a pad.
Daniel Emmerson 10:13
Right.
Elizabeth Moore 10:14
And then you have the huge publishers with platforms that have an embedded AI tutor now. So it's,
Elizabeth Moore 10:23
you know, it's not a bottleneck. It's like too much and it's all fragmented and it's not coherent. And the quality and the rigor is across, you know, across the entire spectrum. So it's exciting in one way, but also, you know, it's, it's the risk of, you know, we've spent a long time in the United States sort of raising the floor of what quality cricket particular materials, instructional materials would be. Right. And that's been pretty successful. And now I almost feel like we're going backwards in some ways because the content that can be pulled is all the content ever related to math, for example. So the quality of the content that's getting served up is now back to this place where how is it being curated? How is it being, how is the quality of that content being judged? So it's, it's a weird time in the market. It's.
Daniel Emmerson 11:22
That's for sure.
Elizabeth Moore 11:24
Yeah. I was thinking about this the other day. You know, it's early on the supply side and on the demand side and sometimes you sort of get one or the other. You know, people are asking for something that's, people are like, oh, we got a solution for that and they develop it across sectors. Right. Or people have a solution and they're sort of trying to raise awareness of how this solution could solve your problem. And I feel like we're sort of like doing this on both sides of that now really fast, really quickly.
Daniel Emmerson 11:56
So that's for sure. I think that, yeah, certainly when thinking about bottlenecks to responsible deployment and responsible adoption, that looks very different to the floodgate almost when it comes to access to different options. Right. As you said, there are so many different options available to schools and school leaders who are making decisions about the technology that they deploy. So I'm wondering from your perspective, what does good leadership look like in that space where everything else seems so fragmented and perhaps disjointed?
Elizabeth Moore 12:32
Well, I'll start with what I don't like seeing, which I think is harmful, which is sort of a blanket no AI use policy.
Daniel Emmerson 12:40
Right.
Elizabeth Moore 12:40
And we're. I don't know how much of that you're seeing in the UK, in England, but it's a little, you know, it is. And of course it's tied up in safety, you know, data safety, safety of students interacting with an AI agent, cheating, screen time, cell phone bans, which is, you know, continue to be a thing here in the States. So it's. Those are all legitimate concerns, but it's sort of wrapped up into all of those concerns. And my worry is when there is this blanket, you know, as a school or as a school leader or as a district we're not using or as a state, we're not going to have AI, it's increasing this digital divide. Right. Like so. So those are kids in schools that are not going to have the same advantages of being prepared to discern what do you use AI for? How do you figure out if it's helping you or not? They're not getting the opportunity to think about and determine for themselves how to use it. So that's the number . It's not that. It's not the positive answer to your question. It's like, don't. Having those blanket no AI use policies I don't think is a helpful approach from leadership. The places I've seen work, and this is both with parent and student engagement and sort of setting vision is like this informed embracing of it, but with, you know, curiosity and skepticism and staying on top of the research and like the evidence. So the school leaders I've seen that both have a vision and can explain to parents and to teachers, here's how I want to see us use AI in the school. Here's how I don't think we should until we have more evidence. Here's how I want to protect our students. Here's how I want to measure if it's having a positive impact on learning. I've seen that sort of be the most positive way to approach it, not being afraid of it, not canceling it. And then I've also seen a lot of really inspiring sort of school grassroots like Thursday evening in the library. Come learn about AI. Because there are a lot of parents out there that have never used ChatGPT.
Daniel Emmerson 15:12
Right?
Elizabeth Moore 15:13
No, there's. Their kids know more than they do. So then it. We're back in this cycle of like, well, you know, it's sort of, you know, social media. A while ago, you know, I remember like I'm not. My kids use Snapchat. And I was like, I'm never going to figure out Snapchat. Like it's beyond me. I don't get it. But I wasn't able to have an opinion about it because I didn't actually understand what they were doing in Snapchat. I don't know where it went. Yep. So there's. I think a responsible school leader's approach has to involve parents and students and teachers and there's like a degree of transparency but also having an informed point of view. And that's true in all school leadership. Right. Like you.
Daniel Emmerson 15:57
And starting with purpose as well. Right.
Elizabeth Moore 15:59
Why do we want to use it? What do we think we're going to. How are we going to monitor that? The thing we thought it was going to do is actually. Actually doing that for us. Right.
Daniel Emmerson 16:10
And is that you seeing that happen in some of the schools that you're speaking with? Is that the typical approach or the majority, do you think, like the shiny new technology? Let's onboard this.
Elizabeth Moore 16:21
I. It's. It's better than that, but it's not quite the ideal state yet. It's somewhere in between. It's sort of what I'm seeing now. And we saw this with ed tech too. Right. Abundance of data. Right. AI helps give you individual student learning progress, research, class level. You can get it all much quickly. You don't have to put it in an Excel spreadsheet and then crosswalk it to the scope and sequence. And then all the things that data driven instruction sort of inspire people to do back when it was more old school. It does it all quickly and it can do it well. What I'm seeing mostly is like a use of the data, but not so much a mindset of is this the data we really need to make decisions. So. And we see this all the time. I'm sure you've seen this too. Like the pretty shiny thing comes with data that like you're like, this is great. But is the data correct? Is it telling? Does it match what the teachers judgment about the students are and teachers can be biased too. So like the, the. I guess that's what I meant with skepticism but I don't mean it in a. Don't believe it won't work. But like how are you, how are you going to be confident that it is working? And I don't see a lot of that right now. Doesn't mean it's not happening. It just means that's not. When I go visit schools, people are showing me the product that we've invested in and the school that we've invested in to help pilot the product. And so I try and visit schools not with my Gates hat, you know, but even from what I read and what I am aware of, people are using the products. But whether they have a plan for testing, are the products doing the thing that they hope they would do. I would love to see more of that.
Daniel Emmerson 18:20
So that stakeholder management piece as well that you mentioned with, with getting parents on board about the decisions that are being made maybe at a procurement level but also at a pedagogical outcomes level.
Elizabeth Moore 18:34
Yeah.
Daniel Emmerson 18:34
Are you seeing some like good use cases of that, that stakeholder engagement happening? Is that something that schools are investing in?
Elizabeth Moore 18:43
Yes, and some of this is coming from the developers and some is coming from the schools. The information that parents can get about their students learning, which they can also then interrogate has a lot of the AI use has enabled parents to see a lot more of what their students work is and sort of insights about their students work. I've seen more parent engagement tools that are really promising which I think is an under. What was the word? It's not getting the same kind of attention as some of the, like an AI tutor or a teacher tool. I've seen some great products where they, because they can do it with it, you know, sort of generate like one question to ask at night, you know, and they send it on WhatsApp or they send it out through text and it's about math and it's like, you know, hold up your phone in the grocery store and we'll give you a question to ask about, you know, adding or you know, play and that's, you know, that's a resource intensive thing that used to be like a really heavy lift either for a developer or for a parent to do on their own. So I think there's a lot, I think part of the engagement of the parents is actually being impressed that there is a benefit to it. And so if they know that their kid used it to write an essay, but the kid didn't go back and like, you know, see if those were the changes they really agreed with in their essay, they're going to have a negative impression and they're going to say you're using it for cheating. If it helps them have a conversation about counting with their second grade, you know, the kid's not on a screen, the kid's not doing anything, but it's sort of supporting the parent. I think there's real potential there as it gets better too, as the model gets better.
Daniel Emmerson 20:36
So thinking about math in particular, and I know this is an area that you're doing some, some work in.
Elizabeth Moore 20:42
Yes, I missed that in my intro. Take everything I said and then layer math on top of it.
Daniel Emmerson 20:49
Well, let's, let's address that now. It would be great to hear more about it. So, I mean, historically, if we look back historically, it feels like a long way has come since, you know, mainstream models became mainstream. But, where are you exactly? I mean, during that time, you know, models have got a lot better at what they can do on the math side of things. What can you tell us about the greatest promise or potential for maths when it comes to AI use?
Elizabeth Moore 21:21
I believe that the greatest potential is accurately diagnosing where there are learning gaps and then sort of effectively and meaningfully in a way that sticks, you know, helping address the learning gaps. And that's the part that is the most exciting to me. I was in the classroom for years. You know, if you, if you had a sixth grade student who had not mastered some of the skills they needed that they should have learned in fourth grade, and for whatever reason, right. They, they don't have those skills. You, you would have to go first, you'd have to figure out like, what skills they didn't have because it can't be all of fourth grade math. Right. But some of the approaches were like, well, just teach you at the fourth grade math in fourth grade level now. Right. You had to go get a fourth grade textbook from another classroom to get the exercises or the workbook or the less, you know, the little, you know, the worksheets that were the right level. And you sort of had to go find the content that would have been the fourth grade content that can all take place within a single, you know, second now.
Daniel Emmerson 22:30
Right, right.
Elizabeth Moore 22:31
The prerequisites for that being effective are the accurate diagnosing. And I think, you know, the assessments are all over the, you know, there's not, there's the same, there's the same need to continually sort of evaluate. Are the assessment items that have been generated by AI actually measuring the thing that they're intended to measure. So that was like a whole separate new problem and we've always had that problem, but now they can make a lot of items. So like what's, what's the rubric behind them that's evaluating? Like does this actually, you know, everyone will say it evaluates misconceptions in math. Right? Misconceptions in math is now like the buzzword. I'm like, okay, well sometimes they're procedural errors and sometimes they're an actual conceptual misunderstanding. Like the items should be able distinguishing between. To distinguish. Did you make an actual procedural mistake and not carry something or do you actually not understand place value? Because there's an actual underlying misunderstanding of the concept. So assuming that you've got quality assessments that are accurately diagnosing and then you have quality appropriate content that's helping either support the teacher and like here's, you know, let's go back to the fourth grade math. You know, here's, you know, let's go back to how this works or a video and then the teacher or you know, whatever and then assessing like did they actually with the re. The instruction is that gap, that skill gap now filled. I think that's the potential. But like with all technology there's some people doing it really, really well in a very rigorous way. And there's also a lot of claims that they're, you know, people are doing, you know, a product does that but
Daniel Emmerson 24:28
you're seeing an increasing capability as far as the models are concerned in being able to differentiate in that way.
Elizabeth Moore 24:33
Right. There's, there's a product here that the called Learning Commons, which is fund partly with Chan Zuckerberg Institute (CZI). That's a knowledge graph that's based on. There's a. Do you have an open education resource here called Illustrative Math and it's a public good basically. So it's you know, high. A highly research, highly research based, proven knowledge and graph of how math is learned, how concepts are developed down to the sub, sub level. So what we're starting to see and it was meant to help improve so developers don't go out and make their own assessments by you know, standards are always really broad and people start to sort of assume, you know, like say like oh, if you get this right, you understand this entire standard. So this is back to the argument of where philanthropy providing this open source knowledge graph that's really high quality, that Any developer can then put underneath an assessment item or content generation or a recommendation. It seems like you don't know this. We should go back to this prior skill. I'm seeing really exciting stuff in that regard.
Daniel Emmerson 25:58
So let's maybe hone in on while we're wrapping up. Just from the perspective of a head of department ahead of a math department in a school that's really interested in this and they're thinking about, okay, what are the best options here? What are the kinds of things that you would suggest they look for in an AI solution that's going to help them with their math provision?
Elizabeth Moore 26:24
I think the safety standards that I've seen from the UK give a really nice roadmap of thoughtful, just not just data safety, but student engagement pedagogy. Like things that are, you know, like a first. Like, what would the word, you know, first tier of things you should be looking for. 
Daniel Emmerson 26:42
The product safety standards. Yeah.
Elizabeth Moore 26:44
Yes. Yes. Sorry, I was like, is there another word that goes in there? The product safety standards. I like to see schools give it to teachers and ask them to break it.
Daniel Emmerson 26:59
Okay.
Elizabeth Moore 27:00
You know, and sort of, you know, but not, not in a, you know, like, let's find out what's wrong with this kind of way, but sort of. And I think involving teachers in those decisions is really important because then you're, you know, it's one level of resistance that you've sort of already, you know, accounted for. Right. Like have them give your, you know, everyone takes the same lesson that they're teaching normally, uses the product and comes back with some kind of a, you know, very lightweight. Right. A very lightweight thing. Not a curriculum review. 
Daniel Emmerson 27:35
But it helps you find edge cases as well, I guess if you're doing that.
Elizabeth Moore 27:37
It does, it does. And your trust, and it helps demonstrate that you trust their content expertise as educators, which I think we are guilty of not doing frequently in this in the States, it's sort of a decision is made and then the teachers, you know, have to adopt a new something and they also have really great insights. Right. And I'm not talking like user research, like, oh, it'd be great if we had a button that allowed me to export, you know, whatever. I'm talking like, is it instructionally sound to your point? You know, if I, I already know a kid who is struggling with this, with this product, also identify that this kid is struggling with that. So something that is correlating with their own judgment also builds their confidence that the product can be trusted to some degree along with the human judgment.
Daniel Emmerson 28:33
Awesome.
Elizabeth Moore 28:34
Does that answer your question?
Daniel Emmerson 28:35
It does, I think.
Elizabeth Moore 28:36
You know, I forgot your question.
Daniel Emmerson 28:37
Well, we're looking at the head of department math decision.
Elizabeth Moore 28:41
Right.
Daniel Emmerson 28:41
When thinking about, particularly relating back to what we were saying at the beginning about the number of options that are available to school, it's unlikely that it's going to be solely a leadership decision around what tools, what solutions they're looking to explore. But I think at a department level, you've engaged with the tools that address the issues you're facing in the classroom. And so with math.
Elizabeth Moore 29:08
Oh, I think you said something earlier that I would add. I would just wrap up with this knowing what problem they're trying to solve for and what the purpose of it. The teachers and the department head should all be aligned on that. Right, right. And I think there's often a disconnect of what one person is solving for versus what teachers believe needs to be solved for. So getting alignment on that and then bringing in all the stakeholders to judge, is this solved, solving that problem, That's a great approach that we do not, you know, I don't know how often it's approached that way.
Daniel Emmerson 29:41
A very, very good place for schools to start, I think, along with, of course, with the Gen AI product safety standards that came out earlier this year. Some really, really great advice as well for teachers and leaders. Elizabeth, thank you so, so very much for being with us on Foundational Impact today. It's a joy speaking with you as always.
Elizabeth Moore 30:02
Same. Take care and have a great day. 
Voice Over
That's it for this episode. Don't forget, the next episode is coming out soon, so make sure you click that option to follow or subscribe. It just means you won't miss it. But in the meantime, thank you for being here and we'll see you next time.

‍