This week we ask how we can teach future policymakers to use evidence well.
Good policy depends on evidence, but the statistical methods behind the best research are complex and few policy-makers can master them in depth. So how do we equip people to engage critically with research without being trained statisticians?
A new module on UCL's Masters programmes tackles exactly this, teaching students to think rigorously about what conclusions can and cannot be drawn from research - from measurement and causal inference to the gap between credibility and real-world meaningfulness.
Host Alan Renwick is joined by the module's creator, Dr Julia de Romémont, Lecturer in Quantitative Research Methods and Political Science at the UCL Department of Political Science.
Mentioned in this episode:
[00:00:05] Alan Renwick: Hello, this is UCL Uncovering Politics and this week we are looking at how we teach future policymakers about using evidence. What core features of good evidence analysis do they need to know and how do we get them across?
Hello, my name is Alan Renwick and welcome to UCL Uncovering Politics, the podcast of the School of Public Policy and Department of Political Science at University College London.
Policymakers hopefully base their decisions on careful assessment of the available options, and at least part of that assessment is likely, in most cases, to involve systematic analysis of relevant evidence.
But such analysis at its best often uses complex statistical methods and few policymakers, whether they're politicians or officials, or indeed campaigners or commentators, have the capacity to learn those methods in depth. So how can we ensure that policymaking can be well grounded?
Well, a module recently introduced on our master's courses here in the UCL Department of Political Science and School of Public Policy aims to square the circle. It teaches our students, many of whom will have great careers as policymakers in the future, how to be excellent consumers of policy-relevant evidence.
Even if they're not ready to apply methods of statistical analysis themselves, it helps students think through what conclusions they might be able to draw from any given piece of research. So in a slightly different episode from the usual podcast, we're going to explore this module, what's it for? How does it work? And also how much of the content can you, dear listener and I lap up over the next half hour or so?
To explore all of these things we're joined by the module's divisor and convenor and my wonderful colleague, Dr. Julia de Romémont, who's a lecturer teaching in Quantitative Research Methods and Political Science here in our department. Welcome Julia to UCL Uncovering Politics. It's fantastic to have you on your first appearance on the show.
And let's go straight into the heart of the matter. What's this module for? What was the need that you perceived for which you saw this module as the solution?
[00:02:19] Julia de Romémont: So this module was in response to the fact that at this point we teach our master students an intro to quantitative methods where we teach them both a little bit the methods, but a lot of the coding as well.
And what we've realised over the time is that only a minority of those students actually take to it and end up doing more advanced methods and actually applying it. And while obviously it makes sense to understand how the methods work to be able to then, you know, read evidence, often it doesn't cover enough ground for them to be actually able to do that effectively once they have a job where they're presented with a lot of evidence and also a lot of the time is taken over by the coding and things that also some students feel very stressed about and therefore like, don't engage as much as we would want to.
And the discussions that me and colleagues had about this is just like, how can we cover more ground and really focus on the intuitions rather than the mechanics of how these methods work? Because ultimately you don't need to understand the mechanics in order to assess whether a given piece of evidence is making a credible leap of faith or not, because as much as some of these methods are extremely advanced, they still are trying to make a leap of faith in terms of what they're claiming, right? They're all gonna be wrong in some way at describing the very complex social reality, they have to decomplexify, sorry, that's a very weird word that I chose to say in a podcast, decomplexify reality. And so I think it's like really focusing on those intuitions that we wanted to build the module around, or that I wanted to build the module around. And, yeah, so that's what I did.
[00:03:55] Alan Renwick: That's fantastic. And, um, go a little bit further into how this might be useful for people. So what are you imagining are the kinds of situations in which the material that people are covering in this module might turn out to be helpful?
[00:04:07] Julia de Romémont: Yes. So, especially in the UK Government, there's been a big push for doing evidence-based policymaking. I think the UK has been generally quite good at that. So that means that there's a lot of impact evaluations that are produced by different parts of ministries and so on. And some of these people are themselves data scientists and know how to do the analysis, but ultimately they then bring that evidence forward to other parts of the government who themselves, like other people in the civil service or higher ups or politicians themselves, who then need to make decisions based on those things.
And a lot of our students who do a master in Public Policy or a master in International Public Policy, they might end up in these kinds of positions where they're presented with these reports and have to make decisions based on them. And the danger often, especially when we talk about quantitative evidence, which this module focuses on slightly more or quite a bit more because that's more my personal expertise than qualitative methods, is that there's a danger that like, oh, there's numbers, it's maths, it's objective, and I can just believe what comes out of it. But there's actually a lot of things that we need to question around that. And it's basically giving students a confidence in the tools to know where to look without having to look at the mathematical formula in those.
[00:05:19] Alan Renwick: Mm-hmm.
[00:05:19] Julia de Romémont: And so the idea that this would then empower them to be able to make decisions even though they feel that they haven't understood the depth of the methods.
[00:05:28] Alan Renwick: And you were suggesting there that this may be useful for students if they go onto careers in the civil service.
[00:05:33] Julia de Romémont: Yes.
[00:05:34] Alan Renwick: But presumably also it can be useful if they themselves become politicians or journalists, commentators. And, you know, a lot of maybe people like us.
[00:05:42] Julia de Romémont: Yeah.
[00:05:42] Alan Renwick: Are often quite kind of disparaging about politicians and the degree to which politicians actually understand the evidence that is before them. But I guess part of the point of this module is that, yes, those politicians, it would be great if they did have the confidence to engage with this sort of evidence, even if they don't have the full grasp of the methods.
[00:06:02] Julia de Romémont: Exactly. And ultimately the idea is that this would lead to better policymaking in the future, that people are able to better assess also the state of the evidence, because it's one thing to only understand one study very well, but it's another thing being presented with a whole, like loads of different studies and try to make informed decisions on that basis.
And I think the idea of the module is to give people, at least the starting tools on like how to develop those skills beyond just the analysis and try to, um, yeah, feel more confident. Because also a big problem with these methods and the pressure on the stats and everything, like people feel, oh, that's just not my thing and therefore I'm not gonna engage with it and I'm gonna trust other people to do it. But ultimately we should all be thinking about what we can learn from that.
And yes, policymakers, civil servants, but like politicians themselves, also think tanks, people who work in like regulation as well. So public service adjacent positions as well. I think that there's a broad, like a very broad population of people who this might be useful for.
[00:07:05] Alan Renwick: And do you think there's acknowledgement of these ideas among people in those positions? And do you get a sense that people in the civil service or in public service more broadly, that they see that there is a deficit and that something like this could be helpful in filling that?
[00:07:20] Julia de Romémont: Yeah, I think so. I mean, I don't have direct personal experience from that, but one of our colleagues, Dr. Jack Blumenau, who's like, has been doing a lot of work with the Cabinet Office, he's told me that even people in there are conscious that they're lacking. They have people who are data scientists who can do the evidence based, like the impact evaluations and so on, but they really lack the capacity to actually understand what can be done with that, with those methods.
So while there's been a huge development of doing impact evaluations all around, we are missing the link between those impact evaluations being made and then people actually being able to assess the, like we're missing, I mean, that link needs to be strengthened.
[00:08:01] Alan Renwick: Mm-hmm.
[00:08:01] Julia de Romémont: So that we then go to like actual decisions.
[00:08:04] Alan Renwick: Mm-hmm.
[00:08:04] Julia de Romémont: Yeah.
[00:08:04] Alan Renwick: Yeah. So you've said that you're seeking to develop various.
[00:08:07] Julia de Romémont: Yeah.
[00:08:07] Alan Renwick: Skills and intuitions on the module, do you want to go a bit further into the content? What are these skills and intuitions that are at the core of the module?
[00:08:14] Julia de Romémont: Yes. So I structured the content around three different blocks. So the three different types of inferences that we generally do with evidence. So one would be measurement inference, it's trying to characterise what the things we observe might mean about the things we observe. So, the example that are often used is in the context of survey methods, we think about, we ask people a series of questions about what their attitudes to certain things in society are. And then based on their patterns of responses, we infer that they are more or less right wing. So that would be trying to under like, characterise what these people are.
And we do the same thing with, you know, democracies. We observe democracies or like we observe countries doing a certain type of things, elections, that they're free and fair, that there's a regular change of government and we infer that they are democracies. And that measurement inference is really at the core of quantifying a lot of the things that we then use in subsequent analyses. So that's the first type of inference.
Then there is the causal inference. So once we have measured certain things, it's like assessing the causal relationships between those two things. And essentially it's all about the age old adage at this point: correlation is not causation. And it's trying to give people an intuition: when can we credibly claim that two things that go together, go together because one causes the other. And it's really structured around the idea that we need to think in terms of alternative universes to help us identify whether we are actually comparing things that are nearly the same except for the one thing that's different, whose effect we want to assess.
And then the third block, is about population inference. So most often we only have access to data from a slice of reality. We only have access to a sample of respondents. We only have access to a sample of participating local authorities in some kind of programme and so on. And so we try to generalise from what we observe on those smaller populations, so the samples, onto a broader population. So what would happen if we apply this policy that we applied in this subset of local authorities to all of them and so on. Or what can we conclude from a sample of 2000 respondents saying that they would vote for a particular party in one election to the next one?
So those are the three inferences that I focus on in how I structure the course. And the really important point that I try to make from the beginning on and throughout is that the fact that you're doing inferences means that you are trying to make claims about something that we never directly observe. And regardless of how complicated and how complex and correct in, quotation marks, our analysis seems, is that we are always still making a leap of faith.
We are always still trying to just say something about something we can never directly observe. And understanding that any analysis makes a little bit of that leap of faith, I think also helps. I think that that's what students really took away in some ways that they felt like, okay, so just because those people are like better able to do these statistical analyses doesn't necessarily mean that they're more correct. There still is uncertainty for them as well.
[00:11:19] Alan Renwick: Mm-hmm.
[00:11:19] Julia de Romémont: And basically structuring the course not on getting things correct, but getting people to a point where you say, like, my leap of faith is as credible as it gets. I try to do my best to make an educated guess about what actually something that I might not see.
[00:11:34] Alan Renwick: Hmm. That's great. Thank you. Really good overview. Let's go into the first of those areas of inference. So the measurement
[00:11:39] Julia de Romémont: Yes.
[00:11:39] Alan Renwick: Inference about what is the thing. So, as you were describing that there, I guess I was particularly thinking about survey research
[00:11:47] Julia de Romémont: mm-hmm.
[00:11:47] Alan Renwick: Where we have questions that we put to people and certainly something that I'm always thinking about whenever I look at a survey, I always want to see exactly what was the question, what was the question wording? And sometimes you're not given that and you're just kind of told kind of vaguely what was the question about. But actually the question wording can make such a big difference. So is that the kind of thing that you're talking about when you're exploring that aspect of the overall module?
[00:12:09] Julia de Romémont: Yes, absolutely. So I, because I use surveys quite a lot in my own research, I actually often use survey questions as examples of those. And I mean, I completely agree. I also have some research ongoing with one of our colleagues, Tom O'Grady, where we actually assess the effect of question wording.
Maybe it might be useful to, um, say how we structure the seminars in this course. So like me with the two PGTAs who joined me on this module, they helped me a lot. So Joelle Tasker and Marco Cappelluti. One of them was a PhD student at Essex and Marco, a PhD student in our department. They were really helpful at helping me design the seminar. So basically our idea was that in the seminars, the students would take a piece of evidence, it generally wouldn't be a pure research paper, but actually something produced by a think tank or a research institute that was trying to produce research or a report whose audience is generally like the wider public.
And the students in the module, in the seminars first, like a group of students were asked to present it and explain what the measurement exactly was. And then the students would critically engage with all of the aspects of that evidence report. And in the measurement weeks we had a lot of survey based written research and it's exactly that where the students at the beginning, you'd often write in the report, it's like people's attitudes to democracy is this, or people's attitudes is that so immediately going, rather than saying our measure of this thing, they say like this thing, they are not explicit about the fact that they have only a measure of the thing that they want to measure.
[00:13:39] Alan Renwick: Mm.
[00:13:39] Julia de Romémont: And students really started looking into the, how the questions were phrased and started thinking about like, oh, but does that really mean what the people who are reporting on it actually, um, say it means and
[00:13:51] Alan Renwick: And actually this democracy, I mean you mentioned democracy as an example there. This always seems to me to be one of the clearest examples where this is problematic, that people are often making claims about what people think about democracy.
[00:14:03] Julia de Romémont: Yeah.
[00:14:03] Alan Renwick: On the basis of a question that is worded something like, you know, do you agree that we need a strong leader who is able to get things done?
[00:14:11] Julia de Romémont: Yeah.
[00:14:11] Alan Renwick: And you know, we in our world think of strong leader. Aha. That means something alternative to democracy.
[00:14:16] Julia de Romémont: Yeah.
[00:14:16] Alan Renwick: But there's absolutely no reason to think that someone who's never come across that question who's just got a
[00:14:20] Julia de Romémont: Absolutely
[00:14:21] Alan Renwick: Survey question in front of them is thinking of it in those terms.
[00:14:23] Julia de Romémont: Yeah, exactly. And I think I often try to tell the students, especially when we're talking about surveys, which I often find really interesting, it's like, how easy do you find it to agree with that statement? Because a lot of these statements are in this direction also like trying to measure populism for instance, like questions such as like the will of the people is important, which we associate in Political Science with adhering to populist ideology but it's extremely easy for anyone to agree with that statement.
[00:14:50] Alan Renwick: Yeah.
[00:14:50] Julia de Romémont: So it's trying to think in those terms. But what I also try to make the point is like there is no perfect measurement. So I think students then start engaging critically with those things. And of course we need to really think about like, was the wording of a question or any kind of measurement tool we devise too leading and therefore not useful. But ultimately, none of these questions will be able to capture the complex ways in which people think about democracy or relate to democracy on a day-to-day basis which might not be really that present in their minds. And when we use survey questions, we try to get a survey question in order to distinguish between people.
And I try to focus on the fact that ultimately those survey questions are not to get at the right answer, because we don't know exactly what the right way of measuring people's attitudes to democracy is. What we want to do is devise measurements that are useful for purposes. And the purpose often in survey research is try to find out where do people disagree? Where's the variation in how people respond to those things? And yes, we might say like, the problem with a question, so like, do you think a country needs a strong leader that you might not get enough variation to be useful because it's extremely easy to agree with, but it might help you to identify those who are very much in support of this kind of thing, or it might help you identify those who are very much against that.
And then ideally you would ask also other types of questions that cover different aspects of these ideas. And none of them by themselves actually gets close to what we want to measure, but the combination of them might be useful enough for us to identify patterns and try to get a sense of like, where is there differences in the population and how they feel about it.
And then obviously we then go on and think, does that also correlate with how they vote or how they would react in the face of certain events and so on. So I try to get them at the same time to critically engage with those things, but also say like, at one point all of these will be wrong. We need to think about like, where do we stop to try to make them useful enough?
[00:16:53] Alan Renwick: Yeah.
[00:16:53] Julia de Romémont: For the purposes.
[00:16:54] Alan Renwick: Yeah, we could keep on talking about measurement inference for the whole of our half hour and you take three weeks just focusing on that in the module, but I'm also really keen that we talk about causal inference.
So you said earlier that there's a kind of fundamental intuition to get there around correlation not being causation. Where further can we go in thinking about causal inference without going into the depth of the statistical analysis?
[00:17:19] Julia de Romémont: So. Yes. I think it's clarifying for a lot of people, like for a lot of the students, like obviously we have methods in which we like develop methods such as where we use regression to assess those correlations.
And then we think about like, what are the things we could control for to do it? Or do we design experiments in a particular way and so on. And I think what I try to really focus the intuition on is that regardless of how you do it, ultimately to any causal statements, regardless of how complex the method is, the underlying idea is really are you comparing, like, because obviously we cannot observe two different states of the world for any given person unless you know you are an author or a movie maker and so on, and then you, the world is your oyster.
And the idea is then to say that, okay, everyone, though implicitly, makes those kinds of statements. So when people said like during COVID for instance, like, oh, if our country only implemented these policies that this other country did, they're implicitly saying like, those countries, those two countries are the same and the only difference is the policy that they're implementing. But that obviously doesn't seem very intuitive because those countries are not the same. There are many other things that are not the same that might have consequences on how effective those policies will actually be.
So I really try to focus on this like what is called the potential outcomes framework, but without all of the maths and everything, just really thinking like, okay, ultimately for a causal statement, to be truly credible, you need to say the thing that I'm comparing, like the control group thinking in terms of like, one group gets treated and one is like something doesn't happen, need to be as comparable to each other as possible except for the difference in the status of being treated or getting some benefit or getting some policy on not, and so on.
[00:19:08] Alan Renwick: Mm-hmm.
[00:19:08] Julia de Romémont: And so the idea then would start with randomised control trials, because that always clarifies the intuition quite well, is that if we randomly assign a treatment, obviously the people who were assigned to treatment cannot choose whether they're treated or not. And therefore, in general, a randomised experiment will generally ensure that the treatment and control group or like different levels of treatment are comparable to each other so are essentially similar in every other respect other than the treatment.
The problem is, though, obviously we cannot randomise everything and it would be unethical. Sometimes it's impractical and well, the really important bit really is the ethics of it.
[00:19:47] Alan Renwick: Mm-hmm.
[00:19:48] Julia de Romémont: And obviously as much as researchers and scientists would want that, I don't think it would be great for the world if we randomise everything.
And so then we go on like, what can we do when we don't have a rand? If we don't have access to randomisation as researchers, how can we take the data we observe in the world and try to find out where can we make good comparisons basically? And that generally involves trying to, rather than doing complicated statistical models, is actually trying to think about ways in which, by chance, certain things are more comparable than something else.
So we go through various different designs that are often used in impact evaluation. There's what's called difference in differences, where the idea is that we compare two groups over time, that yes, they are different to begin with, but because we just observed the trajectories, we can argue that like the trajectories would look the same had they not been treated. And obviously then someone who makes claims based on that will have to convince the reader that that's the case.
[00:20:46] Alan Renwick: Do you want to give an example of that just to help us get our heads around how that difference in differences design works?
[00:20:53] Julia de Romémont: Yes, so, I'll take the kind of classic example that Jack Blumenau and I also use in a module that's dedicated only to that. So, there's a paper from Ladin Lens from 2009, I think, and they looked at the effect of the Sun endorsing Tony Blair in 1997.
And the Sun quite famously claimed that they had won it for Tony Blair because they had endorsed him, whereas they hadn't endorsed a Labour candidate in the previous election. And, basically what those people in that paper did is that they used survey data from the British Election Study, which has a panel of people, so basically the same people get interviewed at regular intervals and their vote choice gets observed at these different elections. And they compared those respondents who are Sun voters who at one point they were asked what kind of newspapers do you read?
And a set of people said, I read the Sun regularly. And then they compared the trajectory of vote choice for Labour among those people who said they were reading the Sun and those who didn't. And the idea is that they can compare what's the baseline difference already before the Sun changed their endorsements.
So when actually they weren't pro-Labour or like didn't say they were endorsing the Labour candidate in 1992. And then looked at how the rate of the Labour vote increased for the Sun voters relative to those who are non Sun voters. And what they argue is that, obviously both of these vote shares increased because 1997 was a big landslide election for Labour, but it increased more for those Sun voters.
And then what the authors try to explain, like try to argue is that like nothing else except the Sun endorsement changed for that group of people who read the Sun.
[00:22:36] Alan Renwick: Mm-hmm.
[00:22:36] Julia de Romémont: So that's an example of a diff-in-diff.
A more policy related application is generally observing, for instance, unemployment outcomes in different local authorities. And then some kind of policy gets put in place, but only in some local authorities. And what we compare then the trajectories of unemployment, for instance, for those where there wasn't a policy put in place and for those where it did, and then just whether the rate of change is different helps us get a sense of what the policy's effect might have been.
But again, while this is like technically quite clever, a big part of those papers or like those research reports, like what they need to really argue, and it's not that much a stats problem than an actual common sense problem, is try to convince that really nothing else has changed other than the treatment for those local authorities where the policy was put in place. So for instance, if any other policy was put in place is those same local authorities, you cannot really argue that it's due to that policy happening.
[00:23:34] Alan Renwick: Shall we say a very quick word as well about, um, population inference? So that's the third kind of leg of the module.
[00:23:41] Julia de Romémont: Mm-hmm.
[00:23:41] Alan Renwick: And we don't have terribly much time on this, but you want to just very quickly give us a sense of what are the key things that students learn?
[00:23:47] Julia de Romémont: Yes. So, I try to get them to send really this idea that often, especially in survey data or something, we only make claims based on a small subset of observations. And people say like, well, we find this in this, you know, in this sample of respondents. And that means that, so in the sample of respondents the vote share, like in the next general election, like what they would vote in general election for Labour is so and so much percent. We then infer that this would be the same in the population because the sample itself and they argue then is representative of the population.
So the module, the population inference leg is really around the idea of representativeness because basically we take samples because we cannot observe everything. So we only observe a sample and we say that sample is representative, so therefore can stand in for the whole of the population.
And what we focus on in here is, first of all, thinking about like, are the people or are the units that are used in the sample representative of the broader population that the producers of that research say it is? Right? So, the students are encouraged to think about the differences between what the population might be and those people who end up in a sample, but we also think in terms of representativeness of, for instance, of how we measure those things on them, right?
Like, for instance, we can say like a survey question about democracy is not really representative also of like all of the ideas that someone might have in their head about democracy. So I try to talk about representativeness not just in the pure statistical terms of like random sampling, and therefore we have a sample that's representative, but also in a slightly more qualitative way as well.
So we also focus on how focus groups can be extremely representative of a wider range of views, whereas a sample of 2000 respondents with a survey question that are like more contrived and therefore, like they're gonna be less representative in a slightly different way. So I also try to make the case of also mixing the methods to get a different, like looking at the problem from different way because we only observe one slice of reality. We need to think about how can we like pierce through that, like veil in different ways and try to get a fuller picture of what might be actually happening in the public's mind.
[00:25:57] Alan Renwick: This is sounding terrific. I'm feeling very envious of all the students who get to take this module. How does it actually go in practice? How do you find students respond to it? Do they find things particularly challenging or eye-opening or empowering?
[00:26:11] Julia de Romémont: Yes. So I did, as we mostly do in all of our modules the continuous module dialogue, where throughout the term, once at specific moments, we asked them, well, I do a survey, uh, where they can put in like what they feel like.
And I asked them what they feel like they've learned the most at that point in the module. And one thing, that I always find, that I found great this year is like after a couple of weeks when I first talked about regression and tried to give them a sense of what really is regression, literally it's just drawing a line through a cloud of dots. I mean, that's what it is. And then someone wrote, when I asked like, what's the most interesting thing you've learned and someone responded, regression and how it's actually not that scary.
[00:26:50] Alan Renwick: Mm-hmm.
[00:26:50] Julia de Romémont: And I think that that was generally a theme where they felt like it felt great to be able to critique all those pieces of evidence that they read.
And I observed also the teaching of my PGTAs. And in that I saw how the students said like, oh, here are the methods. I don't understand all of it, but here's what I take from it. And they still felt that they could critique and critique what those people had done because ultimately I asked them to focus, what they should focus on is like, what is the data that goes into those really complex calculations because that's where all of these inferences really hinge on. And I think it's that empowering moment where first they read that and go like, well, these are methods I don't understand. Let me just like put that paper away and forget about it.
And it's the moment where they say like, no, actually I have things to say about this and I think at one point in the module they got all really into the critiquing part. And me and the TAs were like, okay, now we need to go from the critiquing part to like the useful enough kind of parts of our philosophy of teaching there where it's really like, okay, all of these things are wrong, we still need to see what do we learn from that?
And I think, yeah, and I think students at the end felt like you, they wrote, um, they wrote, their assessment was choosing a piece of evidence, like anything of their choice could be a report from, from a think tank or it could be report from a government and so on and then they critique that evidence. And honestly, it was great to see a large majority of the students really engaged really well with it and a lot of them came to talk in office hours with either one of us at one point about their ideas. And they had wonderful ideas and had really interesting things to say.
And I would say like, it's their first step on actually like, okay, I'm giving this evidence and writing down like what they think they can learn from that or what they think should be done better and so on. And I feel like a lot of them felt really scared, like that's why they chose to take this module and not the introduction to quantitative methods where I have to do all the coding was like, I'm not gonna like any of this.
And then in the end they had like, covered so much ground. And yeah, overall student, at least those who talked to me and answered the surveys, were generally like the survey were generally very positive about that feeling.
[00:28:58] Alan Renwick: Fantastic. Of course, one might think there's a population inference there from the people who choose to answer the survey.
[00:29:03] Julia de Romémont: That's always the problem.
[00:29:05] Alan Renwick: Yes. I guess we should also think a little bit about the limitations of this approach.
[00:29:08] Julia de Romémont: Mm-hmm.
[00:29:09] Alan Renwick: And I suppose, I am maybe someone who is a bit like the students at the end of this module.
[00:29:15] Julia de Romémont: Mm-hmm.
[00:29:15] Alan Renwick: In that my stats training was many decades ago. And so it's all very rusty. And frankly, quite a lot of the stats methods that are used now hadn't even been invented when I was, uh, being taught stats in the 1990s. And so, you know, I think feel that I can read lots of stats heavy papers and get a sense of what I should be getting out of them. But I think also, you know, if there are specific problems in how they are applying the stats in those papers, then I'm not gonna notice that frankly.
[00:29:46] Julia de Romémont: Mm-hmm.
[00:29:46] Alan Renwick: Someone else has to notice that for me. Is that a limitation that we have to be aware of and that students have to be aware of? Are there other aspects of limitations that you want the students to be aware of at the end of the module as well?
[00:29:57] Julia de Romémont: Yes, of course, the students obviously will, or if they're ever becoming, come into these positions where they'll read loads of evidence. They also have to trust that the people who are doing the analyses are competent in doing it well.
So that also means that like the other type of quantitative methods training that we do, where we do more like actual coding, and I teach like a causal inference methods module that's more advanced. We have other colleagues teaching quantitative text analysis. I teach an undergraduate module only about measurement inference, where like those people who do more data science, like they need to also be trained very well, but also trained with this kind of conceptual thinking as well, and not just the methods.
I think another limitation of this is, again, something that I've talked a lot with my colleague Jack Blumenau. So I wanna give a shout out to him, about the fact that there is a danger, especially with evidence-based policymaking, that we will focus on things that we can measure and evaluate. And in some ways, policymaking requires creative thinking and making bold policies and changing people's lives will mean that you will again, make a leap of faith in terms of like, we might not have evidence about whether this is gonna work or not, but we want to take that step.
And when you think about like the big advances of the last 20th century when it comes to like the NHS or I mean the, the New Deal in the US and so on, like all of these like bold social policy who really transformed people's lives for the better. Those were things where there was no evidence to know whether it was gonna work or not.
And there's a little bit the danger that focusing too much on these, like small, like not small, they're big in when in the accumulation, but like focusing on small steps and small policies and testing all of that. There are good things to it, but it also is the danger that therefore, so policymakers become themselves a little bit too cautious in what they try, and I tried to talk about that in the last part of the module to say there's limitations to it.
First of all, obviously politicians or policymakers might not listen to that because they have their own ideological agenda, but that's not necessarily always a bad thing. And I think I'm trying to make them aware that understanding better evidence still means that there's a leap of faith. And that doesn't mean that we shouldn't try to make the leap of faith in those circumstances where we can't close the gap that well between what we think it might do and like the evidence that we have and the thing we might want to do, if that makes sense.
So I think that that really is a limitation I try to bring home, and, and I guess that that would be dedicated a whole other module or whole, if I had more weeks, I would talk about like doing decisions under uncertainty and also how that relates to people's risk preferences and values and norms and so on but unfortunately we only have 10 weeks.
[00:32:36] Alan Renwick: So always the constraint. Final question: have you learned anything yourself from teaching the module?
[00:32:43] Julia de Romémont: Yes. What have I learned? First, I mean, one thing that I really enjoyed is the thinking about also the seminars. Like a lot of the modules I teach involve coding, and so the seminars are kind of standard in, not standard. I mean, I find it exciting, but it's, they get a, a data set, they get some questions, they have to code it, it's in front of the computer. And this one was really one where there was much more discussion.
And sure, I've taught like a substantive module on welfare states before where there is discussion, but not on methods as much. And I really liked also involving the PGTAs who like came both from different, they have their own different experience and they were the ones, like we together found the materials to talk about in the seminars.
And what I realised is how great it was that it's not only me choosing those seminar tasks to discuss because it meant that they had their own ideas and they found like very creative things. And the students, one thing they really praised is like the materials that we had chosen for the seminars because they covered a lot of ground. And that is something that me on my own, like would not have been able to give that set of diverse ideas. So like the collective effort of us three really helped there.
Then, I think I learned, I mean obviously this was the first time of like really writing lectures from scratch for that, is like learning how to do the pacing. And also I was always quite ambitious in all of the material that I could cover that I wanted to cover. And so I had to sometimes moderate a little bit, but I realised that if you really focus on those intuitions, like students get more excited about it than if you go into the details of the maths, which I personally obviously find exciting, but I understand that others do not.
Yeah, and I feel like obviously it's very empowering for myself also, like having done really a complete module from scratch and feeling that you have yourself empowered people, which I do feel with the other modules, but like the students were more vocal about how they said that.
I mean, one student in one of those CMD things said like, it's a, it's crazy.
[00:34:37] Alan Renwick: That's the continuous module process
[00:34:38] Julia de Romémont: Yeah. Sorry,
[00:34:38] Alan Renwick: Dialogues.
[00:34:38] Julia de Romémont: Yeah.
[00:34:39] Alan Renwick: The process by which we assess how the module's going. Yeah.
[00:34:41] Julia de Romémont: Yes. So in that, where I asked the students how things were going. One comment was that I can't believe that this module hasn't existed before, which obviously is very flattering, but also makes me very proud that like, it feels like it was actually filling a need that was there. And I had been thinking about this for a while, but it took me a while to go like, okay, I'm gonna do it. It's like trusting that gut a little bit that I have. I think that, that, that is something I learned as well.
[00:35:05] Alan Renwick: Well, it sounds like you're doing really fantastic work on this module. As I said, I'm feeling very envious of all the students taking it. You've given us a wonderful little taster over the last few minutes, but it would be, uh, great to hear more. So thank you so much, Julia.
Normally at this point I give you the details of the publication that we've been discussing. Clearly, I can't do that this time because we've been discussing a module instead. But I can say that if you enrol on one of our master's courses here in the UCL Department of Political Science and School of Public Policy, then you too can opt to take Julia's module in full and then you will thereby gain from all of her wonderful insights and energy. Well worth doing.
We will be back next week when we'll be looking at the tradition of political parades in Northern Ireland. To make sure you don't miss out on that or other future episodes of UCL Uncovering Politics, all you need to do is subscribe. You can do so on Apple, Google Podcasts or whatever podcast provider you use. And while you're there, we'd love it if you could take a moment of time to rate or review us too.
I'm Alan Renwick. This episode was produced by Matthieu Dinh. Our theme music is written and performed by John Mann.
This has been UCL Uncovering Politics. Thank you for listening.