81. Brooke Macnamara: Growth mindset, deliberate practice, and the benefits of diverse experiences

Brooke Macnamara is an associate professor at Case Western Reserve University. In this conversation, we talk about her research on growth mindset and deliberate practice, whether deliberate practice is falsifiable, the benefits of diverse experiences, and much more.

BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith.

Support the show: https://geni.us/bjks-patreon

Timestamps
0:00:00: How Brooke started working on mindset and deliberate practice
0:02:10: (Growth) mindset: does it matter?
0:21:10: Mindset interventions
0:36:48: Deliberate practice
0:47:06: Benefits of diverse experiences
0:56:20: Is the theory of deliberate practice unfalsifiable?
0:59:36: What can we take practically from the growth mindset and deliberate pratice research?
1:01:06: A book or paper more people should read
1:02:10: Something Brooke wishes she'd learnt sooner
1:04:32: Advice for PhD students and postdocs

Podcast links

• Website: https://geni.us/bjks-pod
• Twitter: https://geni.us/bjks-pod-twt

Brooke's links

• Website: https://geni.us/macnamara-web
• Google Scholar: https://geni.us/macnamara-scholar
• Twitter: https://geni.us/macnamara-twt

Ben's links

• Website: https://geni.us/bjks-web
• Google Scholar: https://geni.us/bjks-scholar
• Twitter: https://geni.us/bjks-twt

References/links

 Brainology mindset intervention: https://www.mindsetworks.com/programs/brainology-for-schools

Trello: https://trello.com

Burgoyne, Hambrick, & Macnamara (2020). How firm are the foundations of mind-set theory? The claims appear stronger than the evidence. Psychol Science.
Dweck (2006). Mindset-Changing the way you think to fulfil your potential.
Epstein (2021). Range: Why generalists triumph in a specialized world.
Ericsson & Harwell (2019). Deliberate practice and proposed limits on the effects of practice on the acquisition of expert performance. Frontiers in Psychol.
Ericsson, Krampe & Tesch-Römer (1993). The role of deliberate practice in the acquisition of expert performance. Psychol Rev.
Gladwell (2008). Outliers: The story of success.
Macnamara & Burgoyne (2023). Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. Psychol Bull.
Macnamara, Hambrick & Oswald (2014). Deliberate practice and performance in music, games, sports, education, and professions: A meta-analysis. Psychol Science.
Macnamara & Maitra (2019). The role of deliberate practice in expert performance: Revisiting Ericsson, Krampe & Tesch-Römer (1993). Royal Society Open Science.
Macnamara, Moreau & Hambrick (2016). The relationship between deliberate practice and performance in sports: A meta-analysis. Perspec Psychol Science.
Macnamara, Prather & Burgoyne (2023).  Beliefs about success are prone to cognitive fallacies. Nat Rev Psychol.
Sisk, Burgoyne, Sun, Butler & Macnamara (2018). To what extent and under which circumstances are growth mind-sets important to academic achievement? Two meta-analyses. Psychol Science.

Transcript

[This is an automated transcript with many errors]

Benjamin James Kuper-Smith: [00:00:00] It's funny, this is in a way, like, I guess, quite a typical episode for me, but also kind of quite different, um, because, you know, I'm talking about science to a scientist, that's, that's pretty normal, I'm, I'm used to that, but in terms of topics, I'm talking, talking about, we'll be talking about two topics that I encountered, I know them basically from Pop Sci, or kind of from general popular culture, that kind of thing, which is quite unusual. 
 
 

Um, and these are basically two factors that I thought were very important for success in various domains. And I guess your research kind of questions that to some extent. I'm very curious, uh, were you also, how were you exposed to these topics? Was it also kind of because they're fairly, they're kind of everywhere in, in every culture or was it a more specific route to the topic? 
 
 

Brooke Macnamara: Yeah, uh, let me think. So I, I've always studied performance variance and expertise and been interested in individual differences and predictors of achievement. So deliberate practice, I believe I came across [00:01:00] fairly early in getting interested in the topic. So pre, uh, it being so. Popularized say by Malcolm Gladwell's outliers. 
 
 

This is where the 10, 000 hour rule quote unquote, um, really kind of gained a lot of popularity mindset. I believe I. Did first come across through popular media and then also read Dweck's book that she put out and then started reading Academic articles from there. So one was academic and then popular and then the other one was popular and then academic. 
 
 

Benjamin James Kuper-Smith: Okay, okay. Um, yeah, I thought we could start with, with a growth mindset. Yeah. So maybe, maybe I make sure for, for both topics, um, I thought I should maybe, um, for, for growth mindset and deliberate practice, I guess I'll just say we, in some sense, you'll be contradicting some of the, or at least some of your research contradicts some of the kind of popular claims that are out there. 
 
 

Um, and maybe because this may be slightly controversial, I should maybe say that I'm just not from the field, so I'm not [00:02:00] going to be able to assess all the claims and all that kind of stuff. I'll just try my best to kind of be, uh, to ask some decent questions. Yeah, maybe I thought we could start with what is growth mindset, uh, somewhere you wrote, uh, that I think Dweck said it's a success to weight loss and peace in the Middle East. 
 
 

Um, not quite that strongly, uh, but yeah, kind of what is it, um, and how, maybe how is it supposed to work or kind of what effects does it have? 
 
 

Brooke Macnamara: Yeah, so it started out only referring to human attributes, so intelligence, personality, talent, and meaning in that case that people either believe Uh, and it, the, the vocabulary has changed. So it used to be called implicit theories and also sometimes lay theories and also sometimes self theories. Um, and then mindset is what kind of hit the popular press. Um, and so that is more often used [00:03:00] even now in the academic literature. So I'll just say growth mindset and fixed mindset, uh, even if I'm referring to older literature that use different terms. Um, so originally it. Um, it was saying, if you have a growth mindset, then you believe this attribute, whatever the attribute in question was, uh, can develop and grow with effort. 
 
 

Uh, and if it's fixed, then you believe that it's relatively stable. So, mindset now has become so popular that it's attributed to all sorts of things now. So, after attributes, which are still very common and, and Mindset of intelligence is still almost certainly the most studied and talked about, uh, type of mindset, but then there was mindset of willpower, especially when Roy Baumeister's work was popular, mindset of stress, the [00:04:00] mindset of belonging, so mindset of culture. 
 
 

So there's now kind of all these other phrases. that are used that don't necessarily mean the same thing. Um, so most of my work is focused on the attribute side because there's fewer studies in each of these other areas and it's unclear. Well, it doesn't seem to be exactly the same thing, even though it's the same words used. 
 
 

So I haven't studied those other areas. I've, I've pretty much just focused on attributes. 
 
 

Benjamin James Kuper-Smith: And what does This mindset, like what does a mindset do to these attributes, let's say intelligence or educational achievement or whatever. 
 
 

Brooke Macnamara: Sure. Well, according to Dweck and Yeager and, uh, and a lot of proponents of, of the view, Uh, your mindset is the core of your meaning system that depending on your mindset, it creates an entirely different psychological world, whether it's growth mindset or fixed mindset, and that it's the [00:05:00] precursor to how you make meaning out of everything, other beliefs, what you think about effort, your learning goals. So whether you want to learn versus whether you just want to look good, if you embrace challenges or whether you avoid them and want to give up, if you. face a challenge and then therefore Academic achievement is the biggest area. So it's also been as you as you mentioned implicated in weight loss peace in the Middle East racism all sorts of different areas, but academic achievement is the Area with the most work behind it. Um, oh, and also corporations. So for example, Dweck used to have on her website that, uh, a fixed mindset caused the Enron scandal. And so if you remember that from, from way back in the day, and, and essentially that a growth mindset is more or less the, the secret to success, the way to unlock your potential recently. One of the commentaries on, uh, on the [00:06:00] meta analysis sort of pointed out that the, that there's not a clear process model. So how exactly a growth mindset gets you to academic achievement and kind of every paper has a different downstream effect and what's direct and what's indirect. So I can't speak exactly to what they say it is, but roughly the idea is that mindset leads to the types of goals that you have and how you think about effort and challenge and resilience. And all of those factors lead to academic achievement differences. 
 
 

Benjamin James Kuper-Smith: Yeah, I mean, so the way I understood it, maybe you can kind of comment on what's, it sounds, it sounds kind of like the definition you're offering is a bit broader than what kind of, maybe I initially Thought it was. So I thought it was basically just the growth or fixed mindset kind of dichotomy. I mean, or continuum. 
 
 

The rhetoric dichotomy here was that kind of if you have a growth mindset you believe you can grow in something and then you Do, or you're more likely to, and if you have a fixed mindset, you think, for example, intelligence is completely fixed, [00:07:00] I'm stupid, therefore I'm not going to achieve anything, or, you know, there's a, maybe at least an inherent limit to what I can achieve and that kind of stuff, but it sounds a little bit as if what you said is a little bit broader than just the growth fixed mindset, or is, 
 
 

Brooke Macnamara: Um, well, I think you're right in terms of, and, and even whether it's a dichotomy or continuum is, is debated.  
 
 

Um,  
 
 

Benjamin James Kuper-Smith: I mean, I always assumed that the dichotomy was just for rhetoric purposes, that it was a kind of like, this is very easy to understand, but obviously it's a continuum, but, so that's not even, 
 
 

Brooke Macnamara: no, people don't agree on that. Um, so, so Dweck refers to it as a, as a continuum. Other people have said that these are independent constructs. Um, so yeah, that's, that's also unclear. 
 
 

Benjamin James Kuper-Smith: Yeah, I wanted to say, I mean, you already said that the mechanism is kind of unclear because that's kind of what I was curious about in the original story, because it was never entirely clear to me kind of how this was supposed to work. I mean, to some extent, I mean, some of your, the results from your meta analysis that we'll get to in a second, I guess, so surprising [00:08:00] because it seemed to me kind of almost self evident that this was. 
 
 

That a growth mindset is more beneficial than fixed mindset because in the kind of model that I implicitly assume people had is like, well, if you don't believe that you can improve, you're just going to put in a lot less effort and you're, you know, maybe, as you said, the goals you have are going to be lower because, you know, I'm, well, at least on the low end, maybe for people who think they're smart, it's very different, but on the low end, maybe let's say, let's take that as an example. 
 
 

If people think. fixed mindset and they think of themselves as stupid because they can't do something or couldn't do something in the past, then I'm not going to try because I'm stupid. I'm not going to do it anyway. Um, there's no point. Uh, I'm, what is down to a life of, uh, whatever negative attribute people might want to go to. 
 
 

It's funny. Uh, as I just said that I realized, I guess on the flip side for people who think of themselves as very intelligent, it might be very beneficial. 
 
 

Brooke Macnamara: So there's been very little work [00:09:00] looking at, say, benefits of fixed mindset, but there is a study that specifically looked at exactly that. So if people had high confidence in their abilities, um, then a, a growth mindset intervention actually made them worse at the task. Um, and a fixed mindset made them better because they say, yeah, that's right. 
 
 

I am good at this. I'm naturally good at it. I will succeed. So yeah, there might be differences in terms of, uh, where you are in your confidence and your ability of the, or the attribute that's, that's in question. 
 
 

Benjamin James Kuper-Smith: Yeah. Although it still seems to me that people who think they're smart. Yeah. I actually would still kind of probably assume that. If you tell them what you can get even smarter, you know 
 
 

Brooke Macnamara: Well, so recently that, so that's also an interesting question recently, uh, some researchers, LeMary et al kind of asked this question. So they pointed out that the measure used for mindset. First of all, it was designed for children and then wasn't really [00:10:00] adapted or it was adapted, but it wasn't validated and in older samples, but it's pretty much used for for all developmental stages and in the main measure, that's most often used. 
 
 

It's asking about intelligence. Um, so these researchers asked. People, well, how are you interpreting the word intelligence? And it turns out that there's a response process validity problem. So some people are interpreting intelligence as ability. Some people are interpreting intelligence as knowledge, in which case it's logically reasonable that you can increase your knowledge. 
 
 

And so everybody who is interpreting it that way is very, very high on growth mindset. And so it's almost like you're giving different measures to different people. So even, and I, I come from a lab where, uh, we look, we looked at individual differences in cognitive abilities. And so even saying, okay, well, and even from my view of saying, okay, well, are you talking [00:11:00] about fluid intelligence or crystallized intelligence and those? the evidence suggests that one does develop and the other one is more stable, but of course you don't expect the average person completing this form to know those differences. So it turns out that the measure itself is, uh, is problematic. 
 
 

Benjamin James Kuper-Smith: Yeah, I guess it's a classic psychology problem, especially when assessing people who Have certain ideas about a certain, you know, when you're using words that people think they already know or they have some sort of understanding about The measurements get very tricky. I guess we've already been alluding to the meta analysis a little bit maybe we should just kind of say a kind of a brief summary of kind of what you found to not leave people hanging there too much. 
 
 

Yeah, maybe the I don't know whether we should go through them one by one or do you kind of maybe can you summarize them all together kind of? I don't know what's easiest for you 
 
 

Brooke Macnamara: Sure. Um, so in 2018 colleagues and I conducted two meta analyses, one, we looked at the [00:12:00] association between a person's mindset and their academic achievement. And then the other, we looked at the. mindset interventions. And then more recently we did a much more in depth, um, series of meta analyses on growth mindset interventions. So in the one where we just looked at the association between mindset and academic achievement, we did find a positive effect where mindset accounted for about 1 percent of the variance in academic achievement. So not huge. Uh, you would think that Um, maybe based on that number, of course, it was the very first meta analysis on the topic. 
 
 

So maybe people didn't realize, but if you're dealing with 1 percent variance, implementing interventions to try to nudge that, uh, you know, might not have amazing returns. Um, but you know, who knows? So, um, so we did the second meta analysis looking at growth mindset interventions. [00:13:00] And we looked at potential moderators such as the age of the students, socioeconomic status, level of challenge, the factors that growth mindset proponents have suggested are important. And I believe the only significant moderator we found in that meta analysis was developmental stage of the student. So children, adolescents, or young adults. What we also found, so then we also looked at methodological moderators and a concerning result is we looked to see how many of those growth mindset interventions had actually tested. whether the intervention changed students mindsets. So the idea behind a growth mindset intervention is that you go in, you teach them, or in some way, you push people towards more of a growth mindset. Um, and then because of that, those in the treatment condition versus the control, who presumably have not changed their mindset, will get better grades, or score higher on standardized tests, or whatever the [00:14:00] academic achievement. outcome measure is. So we found that many studies had not done this kind of simple manipulation check to see if the treatment had any kind of effect at all. And that's where we started seeing some differences. So when there was no measure of it, those studies on average found a significant effect. But when they did test for it, there wasn't a significant effect. And then when you look at just the studies that showed that they changed participants mindsets. So you had some studies that didn't even test, and then of those that tested, how many worked? Um, and it was fairly small, so I'd have to look at the numbers, but I want to say about half didn't test, and then of those that did test, about half worked. Um, and so if you look at those subsets of studies, the ones where they changed, effectively changed students mindsets, There was no significant outcome on academic achievement, but the ones [00:15:00] where they failed to change students mindsets, then they found an effect on academic achievement, which suggests something else is happening, that there's some other factor that is not growth mindset change responsible. So, based on those results, we had a lot more questions about... The studies. Um, and so we decided to conduct a much more in depth meta analysis. So for a couple of reasons, primarily that reason, uh, trying to really see what was going on and if it's about methodological issues in the literature. Right. So is it that there's bias or bad practices that are maybe causing spurious, spurious results? 
 
 

Or plus the other reason was that. Growth mindset interventions have exploded in popularity. So even from when we completed that [00:16:00] meta analysis, um, and it was published in 2018, so many more studies had come out. So it was just a completely changed field in terms of the literature that was out there. So the, the more recent meta analysis was very in depth looking at. best practices criteria that we laid out. So we had best practices and intervention design. So this was if they tested whether the treatment group's mindset changed. Uh, if there was an active control group comparison, uh, if students were randomly assigned a condition at the individual level, whether there was blinding and, and how far this was. 
 
 

So, in these cases, when you go into a classroom, it's not even just double blind that you have to think about it's triple blind. So is the student blind to condition? Is the administrator? Uh, of the experiment, blind to condition, [00:17:00] and is the teacher who's doing the grading blind as well? So, we also looked at that, and essentially if there were no confounds, so if the only difference between the treatment and control groups was being taught a growth mindset, uh, and then finally if an a priori power analysis was conducted. Then for analysis and reporting, we looked at whether the results were provided for the whole sample or all subsamples or whether they just reported for some of the subsamples or removed participants, whether they reported the effect on those who were treated as opposed to, this would, we'll probably have to get into this in more detail if you're interested in it, but there were some odd cases where, where studies had, so for example, there's this one huge study where. A large number of schools never got the intervention [00:18:00] materials, but they were included in the analysis, and when you do that, suddenly the effect is much larger if you include the students who never got the intervention than if you only include the students who got the intervention, so just some odd things going on with that, so, um, you know, it, it, there are reasons to include everybody, but I think it's important to, when you're unsure about how effective the intervention is, it's even when it's administered correctly, that you need to also report, uh, if it's effect on. students who were treated. And then if the study was pre registered. And then our third category was avoiding conflicts of interest. So if, whether any authors on the paper had financial incentives to find a positive effect. So those were our 10 best practice criteria that, that we laid out and examined. So we did a systematic review, just evaluating those [00:19:00] features, and then we conducted three main meta analyses. So the first one was all studies, regardless of adherence to any of those best practices, um, plus theoretical and methodological moderators. The second one was only looking at studies. That provided evidence that the intervention changed students mindsets as intended in the treatment group. And then the third one was evaluating the best available evidence. So all the students. There was evidence that students in the treatment group had a change in mindset from the treatment and they adhered to what we wanted to do was all best practices criteria. There were no studies that, that met that. 
 
 

So we had actually preregistered how we were going to lower that threshold if that indeed was the case. Um, so we ended up including studies that met at least 60%. best practices because that's the best that we could do. And then we did [00:20:00] actually 200 other models just looking at every number and combination of best practices criteria with and without, showing that they change students mindsets, etc. So in that first meta analysis with everything, we found a small effect. Um, we ran several publication bias analyses, uh, which all suggested publication bias. Um, and when you correct for that, there was no significant effect. None of the theoretical moderators were significant. In the second meta analysis, looking at sort of, this is essentially kind of, minimal evidence for the mechanism, at least, that, that the, the reason for any change observed is due to the growth mindset. We found no significant effect. And again, no theoretical moderators were significant. And then in the third sort of best available evidence, there was no significant effect and there weren't enough studies to look at moderators. 
 
 

Benjamin James Kuper-Smith: Sounds pretty damning. [00:21:00] Um, I wanted to ask, maybe as a, as a kind of first question about interventions, about mindset. I mean, I don't know how widely these interventions vary, but could you describe maybe some examples of what they might do with students, or what students might do, or, um, kind of how, how are they trying to change growth mindset? 
 
 

I mean, you're surveying lots of studies, but maybe there's some sort of commonalities, or kind of typical examples. 
 
 

Brooke Macnamara: Sure. Um, so one thing I'll say is that, uh, the vast majority, I believe it was 97%. Nope. Sorry. 94 percent had confounds. So there was more than just the difference of, uh, the critical ingredient of growth mindset between the two groups. Um, 
 
 

Benjamin James Kuper-Smith: So what kind of confounds are we talking about here? 
 
 

Brooke Macnamara: Yeah, so it could be anything from students in the control, uh, sorry, in the treatment group were also [00:22:00] encouraged to work harder, which there's evidence that that has an effect in and of itself. They were given extra, uh, time, tutoring time with the teacher. They were, which obviously also would presumably have an effect. 
 
 

They were given specific study strategies, um, you know, just other things that, that might have an effect. 
 
 

Benjamin James Kuper-Smith: Yeah, I was curious how kind of, that sounds like a fairly major confound to have. Yeah, it didn't, it doesn't, that doesn't sound too minor yet.  
 
 

Brooke Macnamara: It does. It does. Um, so I would say a typical intervention. And again, I don't know if there's a typical, but, um, usually between one and say three sessions would be typical. There's a lot of single session interventions. So they go in and it's 45 minutes or an hour and they teach students about a growth mindset. 
 
 

They say. Yeah, they have them read or depending on the age level, uh, give a presentation about [00:23:00] evidence of how essentially talking about neuroplasticity and saying that it's true that you can, your brain can grow smarter, uh, if you. work hard. Um, and then that's going to have these effects. It's fairly typical for them to do a, what they call saying is believing exercise where they, um, then write a letter to a future student explaining what a growth mindset is and why they should have one. Um, the commercial products for growth mindset interventions that are sold to schools. So brain ology is, is a common one. So students kind of do this gamified. Where they follow students talking about a growth mindset. Um, but of course also has study strategies and, and, and other things in it. So, um, so that's a little bit different than the one-off, um, 
 
 

Benjamin James Kuper-Smith: Yeah. 
 
 

So what kind of timeframe are we talking about? So you have students and I'm assuming they just take what their grades from last year, or I don't know, they do some tests at the beginning. Uh, then they do the [00:24:00] intervention one to three times, whatever. And then like, well, what timeframe is this that they then make that kind of final measurement? 
 
 

Brooke Macnamara: Yeah, good question. We also looked at that as one of our, um, methodological moderators. So not all studies even looked at baseline grades. Um, sometimes it was just post intervention grades. Um. 
 
 

Benjamin James Kuper-Smith: and they're just comparing two, the two groups or, okay,  
 
 

Brooke Macnamara: yeah. 
 
 

Um, so when we could, we took, uh, into account baseline differences because of course you can have it where it just happened to be that the treatment group had higher grades or vice versa. 
 
 

Right. The, so often it was. Mid semester and we're looking at grades at the end of the semester or standardized test at the end of the semester Occasionally, it would be say the following semester. There were not too many long term if I recall I believe nothing went out further than a year post intervention, so there's not really Any way to know if there's [00:25:00] any sort of long term effect, but we didn't find any I don't believe we found any, uh, significant effect for how long between, so the interval between the end of the intervention and the outcome measure, but I can double check on that 
 
 

because there were, we had a lot of measures. 
 
 

I don't 
 
 

Benjamin James Kuper-Smith: Yeah, yeah. 
 
 

Brooke Macnamara: that that was wrong. Um, yes, there was no, so I'm looking at the, the. First meta analysis that had all of the studies and there was no difference between Short intervals that were within a semester time frame and long intervals that were longer than that So the the moderator was not significant. 
 
 

Benjamin James Kuper-Smith: Okay. I mean, so the reason I asked this is that in a way it seems that to me, to me, it seems that of course this is not going to work. I mean, like independent of whether a growth mindset makes a difference on anything or not. I just don't see how this, how this could possibly [00:26:00] work. I mean, this. I mean, I had this really when, uh, what did I write down? 
 
 

I think I wrote down whilst I was reading an article that this whole idea is just seriously ridiculous. Because, I mean, I don't, this isn't even, again, this is not even a criticism of growth mindset. This is just, like, the, the problems of doing this kind of research. It's more about that. Um, because, maybe you can correct me on this, but kind of what I'm thinking is that, okay, basically they're giving people a long TED talk. 
 
 

And some variant thereof. And then that's supposed to make a difference. In their grades half year later, I mean, there are millions of people who've watched millions of TED talks, including the one on growth mindset. Knowledge is often not necessarily a problem. I mean, uh, you know, I've read lots of books about weight loss and all sorts of things. 
 
 

The problem is not necessarily just having the information, being given the information once. And so, to me it just seems that it's really problematic if the intervention is... [00:27:00] Not only so short by itself, you have such a short time frame that's supposed to have an effect over the course of months when so much else is happening in people's lives. 
 
 

Uh, I mean, I'd be curious how many people could even remember vaguely what, what they were taught in the beginning. So basically, in a sense, I'm almost saying like, just because this, these meta analyses or individual studies don't find that effect. I mean, to me, that almost doesn't really say that much necessarily about whether interventions work or not. 
 
 

Um. Does that make sense? Or, 
 
 

Brooke Macnamara: Yeah, and so proponents have kind of dialed back their claims in recent years so it used to be about claims of how this was a huge influencer, you know, has profound effects, largely influences academic achievement. And then now it's a lot more about saying, well, this, this, you know, D equals 0. 06 is actually a huge effect. 
 
 

Um, [00:28:00] um, and, you know, so more recently, for example, um, one proponent was saying, Well, you're only going to find effects when it's the right environment, and the teacher also has a growth mindset, and then another proponent said, you're only going to find these effects if it's the right environment, and the teacher starts out with a fixed mindset, um, so kind of saying opposite things, uh, but sort of trying to find these other factors that are, that are necessary to have it work, so it went from Um, this needs to be made a national funding priority and policies need to change and it has huge effects to, well, let's try to find where we can find an effect. 
 
 

Benjamin James Kuper-Smith: yeah, because I mean, like, in a way, what I'm almost saying, what I'm suggesting is that like, if you actually have a 1 percent improvement over the course of three months from people watching a talk, I think that is a pretty big effect. I mean, what else can you do within 45 minutes to change people's [00:29:00] performance by that much? 
 
 

Um, 
 
 

Brooke Macnamara: True, except when we 
 
 

Benjamin James Kuper-Smith: assuming it's robust. Yeah. 
 
 

Brooke Macnamara: right, right. So, so we did end up concluding that, um, that it appears that the effects, uh, might be due to. researcher bias, flawed reporting, and problematic study design. Um, so for example, when one or more authors of the paper had a financial incentive to find positive effects, they were two and a half times more likely to find significant effects in their papers. Of course, when we look at studies that demonstrated that they changed students mindsets, we don't find a significant effect. Um, so there's concern, and of course what I told you the, the 96 percent having confounds, right? Like, so it's, it's... 
 
 

Benjamin James Kuper-Smith: yeah,  
 
 

Brooke Macnamara: There's not strong evidence. There's not compelling evidence that, uh, the growth [00:30:00] mindset interventions are effective. 
 
 

Benjamin James Kuper-Smith: Yeah, yeah, definitely. I mean, I completely agree that the, I mean, basically, yeah, I kind of, it seems quite clear to me that the interventions themselves don't have a huge likelihood of working. Let's put it that way, maybe. I mean, after a bit, I still don't know what to make of growth mindset, in a sense. 
 
 

Still don't really know what, uh, again, what this means for the actual concept. Um, but I think, yeah, these short term interventions, yeah, it's a question what that exactly is supposed to do. 
 
 

Brooke Macnamara: Yeah. So, I mean, and in some ways, in some ways this is all one issue because part of the theory is saying that mindsets are malleable and therefore it should have an effect if we change somebody's mindsets on, on these theoretically important. Constructs, you know, and outcomes on the other hand, you know, it's been argued that these are, these are separate and that growth mindset theory might have, um, explanatory power that, but [00:31:00] an intervention doesn't, for example, or interventions might sometimes have an effect that has nothing to do with that, the mechanism that we think it does. 
 
 

Right? So there might just not be internal validity, but there could still be an effect. Um, yeah. So I have with a colleague looked at some of the underlying premises of mindset theory. Um, cause it's, it's surprising. There's a lot of foundational claims that, uh, in many cases were never tested. Um, or the evidence was largely anecdotal. Um, and so we set out to test some of these. Concepts, some of these foundational premises, such as people with growth mindsets, hold learning goals, people with fixed mindsets, hold performance goals, that people with fixed mindsets believe that talent without effort creates success. People with growth mindsets [00:32:00] persist to overcome challenges and that people with growth mindsets are more resilient following failure. So we did a fairly large, um, just study where we tested all of these premises. And we found generally really sort of weak associations between mindset and these theorized, um, Rather sort of proximal outcomes. And so the largest effect that we found was for people with growth mindsets, being more resilient following failure. 
 
 

So what we did there is we essentially manipulated failure. So we, we tested, um. We tested, of course, their, their mindset, and then we gave them these really challenging Raven's problems, and then we gave them their actual feedback, which was all pretty bad, because we just started right at the top, so they got zero, 25 percent,  
 
 

um, [00:33:00] correct. 
 
 

And then, then we tested afterwards sort of how well they were able to recover from that and, and do other problem solving. Um, so that was the strongest effect that we found. It was a 0. 12. But it was in the opposite direction than hypothesized by, by mindset. So people with a growth mindset actually were worse following failure feedback, but a weak effect overall. 
 
 

Benjamin James Kuper-Smith: Yeah. What do you kind of make of that when you have a something that's been touted to be such a huge influence and then you just keep getting at going at it with all these analyses and collecting all the data from different studies and then you just always come up with, you know, you end up empty handed. 
 
 

I don't know. Does that actually, maybe a slightly different question. Kind of what's next in this research area, because in a way you've done like the big meta analyses of the, of all the [00:34:00] available evidence. Are you just moving on to a different topic, or? 
 
 

Brooke Macnamara: Yeah, so what I'm interested in now, so we just had a paper come out, uh, a couple of days ago. So this was a comment in, um, in Nature Reviews Psychology about the... The cognitive fallacies that we think underlie some of these theories and, and the beliefs and, and that those cognitive fallacies make the theories really popular despite weak evidence. And that, that's also the reason that weak evidence kind of keeps going, uh, essentially by the wayside because it reduces scrutiny because they're so appealing and people want to believe them. There's these. feel good ideas. And so people don't look too closely most of the time. So, uh, somebody named, uh, Oysterman who wrote one of the commentaries talks about culturally fluent ideas versus culturally disfluent [00:35:00] ideas. 
 
 

And, um, and people just sort of accept very readily ideas that are intuitively appealing. Um, and then if you go against the grain, then. people, uh, get upset. Um, so I'm, I'm interested in sort of why these are so appealing and, and what makes them appealing. And, and this is a, this is something in psychology very broadly, especially anything related to success. 
 
 

We have lots of theories that people get so excited for, and there's this feedback loop. With the self help industry, so anything I, so I believe now people generally believe in sort of a self determination with, with success, you know, there's this pull yourself up by the bootstraps with hard work. 
 
 

Anybody can do anything. Um, we really like those stories. in many cultures. Those, [00:36:00] those are appealing. And the self help industry, of course, markets this, and they market these theories, which also gets into the issue of financial incentives. Um, but it's a way to kind of keep those popular. Um, so there's, there's market demand for these ideas. And so research that supports those ideas, gets perpetuated in the market. Research that doesn't support it, just doesn't show up there. And like you said. Um, earlier that you heard of these concepts through the popular media and that's where most people hear about them. And they're of course framed in the most ideal terms that are readily accepted and embraced without looking at the evidence behind them. 
 
 

So that's where a lot of people are. 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah, so, uh, as I said in, in the beginning, kind of speaking of things I found out through the media. Uh, deliberate practice, the other topic that you've, uh, done some work on. Maybe [00:37:00] again, what is deliberate practice? Again, where does it come from? 
 
 

Brooke Macnamara: Um, so deliberate practice was a theory proposed by Erickson, Krumpe, and Tesh Romer. The seminal paper came out in 1993. So in that paper, they. We looked at violin students at the Music Academy of West Berlin, um, and they were divided into the best violinists, there were ten of them according to teachers, uh, ten good violinists, and then ten violinists in a different department, uh, that had weaker criteria to, to get in. And they did some interesting statistics. Um, so what they did is they first compared, um, well, first of all, so they defined, theoretically, they defined deliberate practice as practice alone. That was designed by a teacher. Operationally, they defined deliberate practice as [00:38:00] practice alone with the violin. So regardless of whether the teacher designed the practice or the performer themselves designed the practice. And then they compared the best and good violinists and left out the least accomplished violinist, but they used the full sample degrees of freedom, which is going to erroneously lower the F critical value in order to find a statistically significant effect. Um, and then they conducted a separate analysis. combining the best and good as one group and comparing them to the least accomplished and found a significant effect there. And so concluded, hence there's complete correspondence between amount of deliberate practice and the groups. Interestingly, years later, in kind of a defense of not including any kind of, uh, variance. statistics in the paper, no standard deviation, no 95 percent confidence intervals, et cetera. Erickson gave the 95 percent confidence interval for the best group, which [00:39:00] was huge. I think it was, um, just under 3, 000 hours to, just under 12, 000 hours. This is up till age 18. So the, the 10, 000 hour was based on when they were 20, but that's not what the analyses were done in the paper, which encapsulated the means of the other two groups. 
 
 

So it's kind of surprising that they found anything significant at all. Um, anyway, so that was the seminal paper, uh, or that was the seminal study. Then there was a second study where they looked at pianists. Um, that was supposed to be essentially a replication of that study. And then, so there were big claims in this as well. 
 
 

So they said that deliberate practice largely explains variance in performance, even among elite performers. That they rejected any role of innate ability. They applied this to all domains and not just music, even though that's all that they had studied. They gave the exception of, uh, height [00:40:00] and body size for some sports, but other than that, Erickson later said that every healthy child's DNA contains, uh, what's necessary to become an expert, that it's just going to be essentially unlocked by. 
 
 

So in 2014, we published, uh, the first meta analysis on deliberate practice. So we looked across all domains where it had been studied. This was. Games, primarily chess, but a couple on Scrabble, sports, music, academic, so academic achievement and, um, other professions. So things like being a salesperson or a fighter pilot or a referee. 
 
 

And so it, the overall amount of variance explained by deliberate practice varied considerably by domain. So it went from about a quarter of the variance. to not a significant amount of the variance. So that is deliberate practice broadly. [00:41:00] So Erickson then, um, had a reply that didn't get published, but he rejected all but one of the studies that we included, saying that they weren't deliberate practice, including his own studies, where he had argued that they were deliberate practice. Such as his seminal study introducing deliberate practice. Um, and the reasons were primarily that this wasn't deliberate practice. It did not meet the definition of deliberate practice. So we ended up going through and just using a lot of quotes of his and his definition of deliberate practice would change depending on the paper, depending on what the results said. 
 
 

So for example, sometimes it was. It's the theoretical definition that a teacher had to be involved. Sometimes it was closer to his operational, uh, definition where, uh, a teacher didn't need to be involved. And sometimes he explicitly said a teacher needed to be involved. Sometimes he explicitly said that a teacher [00:42:00] didn't need to be involved. Sometimes he said that a teacher is usually involved. But which definition he used just changed. each time he was either writing a new paper or responding to others. So he could always point to a definition and say, see, this is what I've said. Um, but they were at complete odds with each other. 
 
 

Benjamin James Kuper-Smith: Yeah, I mean, on the definition part, I mean, yeah, I found that a bit weird. I read parts of his 2019 review in Frontiers, and what I found slightly weird is that to me, it seems like also there, he was very specific that it had to be not teacher driven, let's say. But what I found really weird, I mean, so I've never been a musician at the time. 
 
 

Yeah. To the, to the point where I wanted to be a professional classical, or where I was studying to be a professional classical musician. But I did in my teenage years consider that as a career option. And to me, what was really weird is that I don't know what kind of deliberate practice you have outside the [00:43:00] context of what you discussed with your teacher. 
 
 

Because, I mean, sure, you can have other things on the side, but basically... I mean, that's, you have a teacher, you practice a thing, you discuss things you do, and then the week later you come back, so I don't even know what's supposed to fall into that. At least I always discussed across various instruments with my teachers the things that we were practicing, whether it was alone or in an orchestra or whatever. 
 
 

So I don't even know. It was really weird reading the way he wrote about it, and it really made me question whether he ever played an instrument, because it just, I don't know, it just sounded... Yeah, it was a bit weird. But anyway, um, we don't have to focus too much on the, on, on his particular, uh, definitions, maybe in this case. 
 
 

Um, maybe as a, as a kind of brief, almost aside, but as a commentary, you tried to replicate his original study. Um, kind of what, what, what did you find there? 
 
 

Brooke Macnamara: Yeah. So, so one of the things, um, Um, is [00:44:00] we looked at whether it mattered how you defined deliberate practice, if it was practice alone, how it was operationally defined originally, um, or if it was practice designed by a teacher that he theoretically defined it as, um, and there was no difference in those two. 
 
 

So along the lines of, of what you were saying, um, Yeah, we didn't replicate it. Um, we found that actually even the good violinists had on average accumulated, um, numerically more. I believe it was not significantly more than the best violinists, but the majority of the best violinists had, had accumulated less practice than the good violinists, which also counters one of his claims that it's impossible for somebody. with less practice to catch up to the best individuals. Um, and so, 
 
 

Benjamin James Kuper-Smith: Although, can I, uh, brief comment on that? Just, [00:45:00] uh, I, I found that part really interesting. I mean, I didn't, uh, I just looked at the, at the figures of kind of how the, you have this, like, Uh, lifetime estimated trajectory, I believe, of, um, how much these three different categories practice. What I took from that is that the good people actually over practice. 
 
 

That was my take for that. Because if you look at it, they had, if you add up, This is from memory now, but if you added up the, I think you had practice alone and practice with a teacher, and if you added up to that, that up, it seemed to me that around the age of like, when was it, 15, 16 or something, there was, they started to diverge the good and the best, where the good just practiced way more, and they were practicing like 42 hours a week or something like that, on average, I think they claimed, if you add everything up, which is seven hours a day, which, uh. 
 
 

I don't know if you've, I certainly have never been able to practice four hours a day intense, well, more than four hours a day intensity, so to me it actually seemed, I mean, this is maybe where my personal kind of intuitions [00:46:00] come in more than any, anything I've read, but it seems to me that like practicing much more than four hours is actually detrimental to, 
 
 

Brooke Macnamara: clarify some of those measures. So those are not independent. So practice alone, um, could have been teacher designed or not. And then teacher designed practice could have been alone or not alone. 
 
 

Benjamin James Kuper-Smith: I see, okay, so, 
 
 

okay, I just added, okay, that was my misunderstanding then. Um, yeah. Okay, but what I still find interesting is that it seems to me that the best violinists stuck to roughly what I would assume the best people would stick to, which is just like slightly under four hours a day on average if you, you know, occasionally maybe have a day where you do less and that kind of thing. 
 
 

Yeah, I don't know whether Ericsson actually says that there's, I mean, I'm assuming he also thinks that there isn't a linear increase in deliberate practice, uh, you know, if you practice 20 hours a day deliberately, obviously it's not going to be twice as good than 10 hours a day, uh, et cetera. 
 
 

Brooke Macnamara: sure. Yeah, he did talk about that there [00:47:00] needed to be rest and he would talk about a monotonic increase in accumulated practice But it's it's interesting what you bring up So I've done some more work looking in sports sports is a really nice test bed because a it's very popular around the world You have objective measures of performance Anyway, and what we find there is that a lot of What we think about deliberate practice and well, let me, let me rephrase this. 
 
 

It depends a lot on the developmental stage of the athlete. So if we're looking at junior athletes. Everything seems to align with the deliberate practice view. So the, 
 
 

Benjamin James Kuper-Smith: So junior means,  
 
 

Brooke Macnamara: uh, 
 
 

usually it depends on the sport, but usually up to age 18, up to age 
 
 

Benjamin James Kuper-Smith: I see. Okay. 
 
 

Brooke Macnamara: Yeah. Um, where they're playing, you know, 12 and under 14 and under, uh, whatever [00:48:00] their sport is. Um, so in those cases, the best junior. Players have practiced more than less accomplished junior players in the same sport as you would expect and a lot of the research We have on deliberate practice and sports is on junior performers because they're a lot easier to Get a hold of, um, than your professional Olympians. However, when you start looking at professional Olympians, world class athletes, national class athletes, at the adult, you know, true world class level, you find the opposite pattern. So world class athletes... have accumulated less practice in their main sport compared to national class athletes in the same sport. 
 
 

So, very good, but not world class athletes. So, world class athletes, less accumulated practice. This does not say that they're not practicing a lot. They've done lots of practice, but they started later. They accumulated less practice in their main sport. They accumulated more practice in [00:49:00] other sports. compared with their national class counterparts. Um, and then the next question people say is, oh, okay, well, they were just the better athletes. They were more talented. Originally, they didn't have to practice as much, but they also reached milestones later. So it didn't seem that they were just good. So compared with who would later become national class athletes, those national class athletes were outperforming them. To a point at the junior level and then things change. So we can't extrapolate from junior athletes and probably this is the case in, in many other domains that we seem to have thought that we could. Which really sort of throws a wrench in the oversimplistic way that we like to think about success and expertise in general. 
 
 

Benjamin James Kuper-Smith: Yeah, that's actually a really interesting point. I hadn't thought about in relation to this is that. Uh, from the sports that I follow, um, it seemed it's very difficult. [00:50:00] I mean, sure there are the cases where people know like this kid is 15 and he's going to be world class and they are, you know, that obviously happens, but the correlation between like youth performance and adult performance is. 
 
 

It's so low, I mean it figs out just the silliest example, if you look at tennis at the junior Grand Slam winners, most of those people you have never heard of. And these are the, you know, there are the occasional people who then go on to win actual Grand Slams. But, um, 
 
 

Brooke Macnamara: It's very rare. Like, we could probably name them on one hand. And even at those cases where they say they're the number one juniors and they become number one, uh, at the professional level. So the correlation is a little bit better at the oldest junior age category. There's a little bit more. Anything younger than that... There's, there's no association. In fact, if you go very young where a lot of places are starting to do quote unquote, talent identification, there's, if anything, a negative correlation between [00:51:00] how they're doing and the success rate that they have as adults. Yeah. So absolutely that it's, it's very rare to have, uh, right. 
 
 

A junior Grand Slam champion who also becomes a professional Grand Slam champion. 
 
 

Benjamin James Kuper-Smith: yeah, I mean, it does happen and there are complicating factors like the best juniors often stop playing juniors very early because they start on the professional tour. So there are lots of complicating factors, but, um, it's. It's always a bit surprising that it, that it isn't more obvious early on who, who becomes great. 
 
 

Um, I just briefly, I guess we don't have much time, but, um, you mentioned some of the things that really reminded me of, what's his name, David Epstein? His book Range. I'm 
 
 

assuming you're familiar with that. Uh, his general argument being that, you know, we, we highlight the, uh, Tiger Woods kind of figure who specializes at the age of... 
 
 

one or whatever, uh, but actually most successful athletes develop much later. Um, can you comment a little more on that related to kind of what you discussed earlier? 
 
 

Brooke Macnamara: Yeah, absolutely. Um, so [00:52:00] what's fascinating is these paragons of the deliberate practice view. So Tiger Woods, uh, the Williams sisters, uh, their myths. So Tiger Woods played multiple sports. The Williams sisters played multiple sports. And even when people discuss this in interviews. When other people write their biographies, that's left out. 
 
 

They talk about the single minded determination because that's the culturally valued way that we think about success, and so that's what makes it into the books. It's pretty wild how biased... Uh, this gets that we so want to believe that it's just about working hard and that's how they made it. And so therefore anybody who works hard can make it, but that's not the case. 
 
 

So they did have this diversity of experiences that we found in all of our data sets that seems to predict better performance at the adult level. We also found this outside of sports. So we did a comparison of Nobel. Laureates in the sciences and [00:53:00] national, uh, awardees in the sciences and found the same pattern. 
 
 

So the Nobel laureates who we'll call sort of the world class. Scientists were more likely to have studied multiple fields or and or worked in other fields compared to the national class awardees. So there absolutely does seem to be something about this diversity of experiences that either changes how well you are able to learn, allows you to problem solve in certain ways. 
 
 

In the case of sports, it could eliminate burnout and injury. Um, so there definitely does seem to be some, some benefits of diversity of experiences as opposed to this domain specific, single minded experience factor, 
 
 

Benjamin James Kuper-Smith: Yeah, I always find it interesting because it goes so against the narrative. I mean, is it just because early on you do see these differences? Is it because, you know, if you have someone who, who really focuses on one sport, yeah, they're going to be better at that sport by the age of [00:54:00] eight than the other eight year olds. 
 
 

But then after that, you know, it just tops off at some point and then we kind of. I don't know, lose track of those people. I don't know. Yeah. Yeah. Why is it so counterintuitive again? 
 
 

Brooke Macnamara: right? Yeah, it's counterintuitive. Um, again, because I think we have an intuition to want it to be a certain way. Um, and so that, therefore that's what we push, right? So the, the kids who work the hardest are going to be the ones that are invited into. talent identification programs where they will increase their amount of practice, um, which is, seems to not actually be doing them any favors. 
 
 

And think about all the resources that are in these types of programs. But yes, I think, right, if you're, so one thing to clarify just in general about deliberate practice is that there is a difference between intra individual change and inter individual differences. So if you practice, You will get better. 
 
 

Almost certainly, right? If you don't practice, you are unlikely to get better, [00:55:00] but we all have different starting points and Learning rates and apogees. So an hour of practice for one person is not the same as an hour of practice for somebody else So it's it's Not to say that, um, because I've had kind of angry emails going, How dare you say practice doesn't matter? 
 
 

When I practiced the piano, I got better. Absolutely. You will get better. That does not mean if you practice more than somebody else, you will be better than somebody else. So... It's very different if we're talking about within an individual as opposed to individual differences. 
 
 

Benjamin James Kuper-Smith: Yeah, that's interesting. I mean, because I, uh, one thing I want to say at the beginning is that unlike growth mindset with deliberate practice, I still feel like there's a very clear, I mean, maybe just, I think maybe in music, I think it was also the clearest example from the ones you studied from correctly. 
 
 

Is that true? That it had deliberate practice had the biggest effect in music. 
 
 

Brooke Macnamara: Um, games, music, and sports were the the biggest [00:56:00] ones. Um, I think music was not the biggest, but they were all in the, you know, 18 to 25 ish percent of the variance explained. And that's overall, so that's not broken down like we did later with sports, separating out juniors and professional athletes. 
 
 

Benjamin James Kuper-Smith: Yeah. I guess maybe because I come from music, there's this kind of, you know, you're just not going to make it without lots of. Practicing deliberately. I mean, it's just a necessary condition. But 
 
 

one thing that I wonder about, one thing I found kind of tricky is that, for example, I've noticed from having different piano teachers that some piano teachers teach you stuff that allows you, allows your deliberate practice to be so much more effective. 
 
 

So like, you know, if I had stayed with my first piano teacher, I just wouldn't be as good as if I'd switched just because a deliberate practice wouldn't become as effective. And I guess that's kind of probably why the variance of the deliberate practice, or the variance explained is so low, I'd assume, because you have to be taught how to practice [00:57:00] and what to pay attention to and all these kind of things. 
 
 

Brooke Macnamara: Yeah, and that's a tricky point of the theory that ended up making it essentially unfalsifiable. So whenever there would be evidence saying, Hey, look, these people practice less, but had better performance, uh, you know, or this person, I guess stated the other way, it makes it a bit easier. This practice person practiced so much, but did not make it as high as other people who practice less. Erickson would respond, well, then it wasn't actually deliberate practice, you know, then it wasn't high quality practice. There wasn't a, you know, well then how do you define quality practice? Well, that was never defined. So you could always say. This is deliberate practice and even, and he did this frequently, then say the same thing was not deliberate practice. And so if you don't have a definition of your key construct, then it's unfalsifiable. Uh, you can't test the theory 
 
 

Benjamin James Kuper-Smith: Yeah, yeah, I mean, I guess it's [00:58:00] just, um... I mean, it seems to me that that's just an inherent problem with the field, because even if you have a teacher, they might be the wrong teacher for a certain person. But I guess it's just, um, if I had a very kind of dictatorial, you do what I say kind of teacher, I would just stop playing the instrument. 
 
 

Like, yeah, but I guess those are, I mean, are those just like inherent problems? Um, I mean, it's not even that it's necessarily, I mean, yeah, to some extent it's unfalsifiable. But this also seems to me, it's just impossible to measure in any systematic way. I mean, way. Yeah. Is that just a limit of this kind of science almost or? 
 
 

Brooke Macnamara: mean, you can think of it as a, as a limitation, which how the research had been done certainly was. It's also a good reminder that It's nuanced and complicated, and that we shouldn't fall prey to the oversimplification fallacy where we tend to want to explain complex phenomena with a simpler explanation than is warranted, right? 
 
 

To just say, go out and practice more, [00:59:00] the end, is not the, 
 
 

it's not the whole story. It doesn't take into account societal factors, in terms of resources available, it doesn't take into account... Social structures in terms of support. It doesn't take into account the relationship with the teacher and what's best for that person based on, say, their personality or motivation or talent level at that time, how that changes over time. 
 
 

It's, it's complicated. Um, and I think one of the biggest issues with this area of research is that we just want to simplify it too much. 
 
 

Benjamin James Kuper-Smith: Yeah. So kind of from deliberate practice and the growth mindset, um, from your work, kind of what do you take from this practically? If, if you do take something on practically, I don't know whether you have children or whatever you teach your PhD students or whatever, uh, does any of this, yeah, kind of what, what do you, what do you make of all of this? 
 
 

How does this affect the way you try and, uh, get good at stuff or? 
 
 

Brooke Macnamara: Yeah, [01:00:00] um, well, I guess it doesn't given that there's not strong evidence for it. It doesn't really. Um, and I think that's going to lead into application of, uh, of by professional life, uh, for example. And so I think, you know, there are people and times where it is going to be best to give them the quote unquote process praise. 
 
 

And there are going to be people and times and contexts where it's going to be better to say, you know what, you are great at this, keep going. Um, and I also think that they're not mutually exclusive, right? So if you. Say, believe that you are smart or have a high ability, then it stands to reason that you also believe that you can learn and possibly learn even, learn very well in this particular domain, for example. 
 
 

So, um, so I don't think they're mutually exclusive as they've, as they've kind of been defined. That's funny. 
 
 

Benjamin James Kuper-Smith: So it's the classic of [01:01:00] just pay attention to the specifics. Okay, um, I have a few recurring questions. First one is, uh, a book or paper that you think more people should read. Uh, this could be, yeah, could be widely read, could be not widely read, uh, could be old or new. Um, yeah, 
 
 

Brooke Macnamara: you already brought it up. I was going to, uh, suggest David Epstein's range. 
 
 

Benjamin James Kuper-Smith: yeah, it's a great book. I really enjoy that too. 
 
 

Brooke Macnamara: Mm hmm. 
 
 

Benjamin James Kuper-Smith: Is that also because it kind of confirms some other things that we want to believe in the same way, or is it just, is it actually very, very well? Um, I can't actually remember too well. I read it a few years ago. Is it just very well, um, researched and fit the data or yeah. 
 
 

What kind of makes it good? 
 
 

Brooke Macnamara: Yeah, I thought it was, is well done, and if nothing else, it just gives you a counterfactual to, to what you want to believe and to say, oh, wait, hey, let me check this out. [01:02:00] Evidence actually suggests something completely different, and so I think it can be fairly eye opening to read it. 
 
 

Benjamin James Kuper-Smith: Yeah, definitely. Um, okay. Then, uh, something you wish you'd learned sooner, uh, or, uh, yeah. And maybe how you, how you dealt with it. 
 
 

Brooke Macnamara: Um, yeah, I sort of thought about this one when you... said when you sent that question, I thought, well, gosh, everything that. I didn't know. I sort of wish that I had learned earlier. Um, but the, the maybe not super profound answer I'm going to give is Trello. So using Trello to organize projects and, and everything. 
 
 

So, um, I use it more than just the sort of simple Kanban of to do doing done. Um, but I have boards for projects to organize ideas. I have it for papers. I have it for to do lists in different areas of my job. I have ways that they all come together. Um, and that made a big difference in terms of controlling or feeling [01:03:00] under control with, uh, lots of different projects. 
 
 

Benjamin James Kuper-Smith: Hmm. Interesting. Uh, just a quick, uh, kind of follow up there. I feel like when I try and be really organized on the internet, like use, not the internet, but like using an app or using a website or whatever, I feel like I do it and then I just forget that it exists because it's somehow not like in the physical world. 
 
 

Uh, I guess you haven't found that, or did you find, did, did you do something to kind of make Yeah. Make it so that you use it more or that it works better for you? 
 
 

Brooke Macnamara: Yeah, I do have to have a system where it's my system for everything, so I couldn't live without it. Um, sort of an idea. I do have one other way when I'm planning out my, what I'm doing that day, I'm looking at my boards and then I plan that on, on something else. So each time I'm looking at my day, well, I primarily plan out my week. Um, and so I, I look at the boards for that and then each day I [01:04:00] kind of look the night before and sort of see if I need to do any updating. Um, so I'm, I'm looking at at least my kind of overview board every day and just have the, the sort of system to, to put that in place. So I see what you mean. I've done that as well with, with other apps. 
 
 

Um, and I think you really just have to. make it part of your routine or part of a system, uh, to make it work. And if it's, if you're not doing it, well, then it's not working for 
 
 

Benjamin James Kuper-Smith: Yeah, yeah. Okay. Uh, final question, uh, any advice for people kind of in their late PhD, early postdocs? Yeah, any advice there? 
 
 

Brooke Macnamara: Yeah. So I never did a postdoc. I kind of missed out on that, uh, experience, but I had a postdoc. Um, uh, so what I've heard from people who liked their postdoc is just how great it was just to be able to focus on research. So I think a big thing is enjoy it. Um, and kind of appreciate that time. I would say [01:05:00] that kind of general advice for for near graduating or or postdoc is, um, you know, start thinking about the identity you want to have for a career. 
 
 

Um, so thinking about the job talk that you will do or if academia is not what you want to do and because there's That's more and more common, um, and I think it's more and more acceptable. So sometimes faculty don't, you know, want somebody going into a, an all tech career, but um, there's more and more resources out there now. 
 
 

So just thinking through, planning out your job talk, planning out your research statement. This is assuming you're going academic earlier than you think and kind of help shape a, okay, this is who I am as a researcher. This is who I want to be as a researcher. I think sometimes people wait until they're applying and then are having to figure it out kind of under more pressure.[01:06:00]  
 
 

Benjamin James Kuper-Smith: Okay, great. Well then, thank you very much. 
 
 

Brooke Macnamara: You're welcome.