BJKS Podcast

108. Robert Wilson: 10 simple rules for computational modelling, phishing, and reproducibility

Share episode

Robert (Bob) Wilson is an Associate Professor of Psychology at Georgia Tech. We talk about his tutorial paper (w/ Anne Collins) on computational modelling, and some of his recent work on detecting phishing.

BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith.

Support the show: https://geni.us/bjks-patreon

Timestamps
0:00:00: Bob's strange path through computational cognitive neuroscience
0:07:37: Phishing: a computational model with real-life applications
0:25:46: Start discussing Bob's paper 10 simple rules for computational modeling of behavioral data
0:32:15: Rule 0: Why even do computational modelling?
0:46:24: Rules 1 & 2: Design a good experiment & Design a good model
1:02:51: Rule 3: Simulate!
1:05:48: Rules 4 & 5: Parameter estimation and recovery
1:18:28: Rule 6: Model recovery
1:25:55: Rules 7 & 8: Collect data and validate the model
1:33:15: Rule 9: Latent variable analysis
1:36:24: Rule 10: Report your results
1:37:46: Computational modelling and the open science movement
1:40:17: A book or paper more people should read
1:43:35: Something Bob wishes he'd learnt sooner
1:47:18: Advice for PhD students/postdocs

Podcast links


Robert's links


Ben's links


References

Episodes w/ Paul Smaldino:
https://geni.us/bjks-smaldino
https://geni.us/bjks-smaldino_2

Bechara, Damasio, Damasio, & Anderson (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition.
Feng, Wang, Zarnescu & Wilson (2021). The dynamics of explore–exploit decisions reveal a signal-to-noise mechanism for random exploration. Scientific Reports.
Grilli, ... & Wilson (2021). Is this phishing? Older age is associated with greater difficulty discriminating between safe and malicious emails. The Journals of Gerontology: Series B.
Hakim, Ebner, ... & Wilson (2021). The Phishing Email Suspicion Test (PEST) a lab-based task for evaluating the cognitive mechanisms of phishing detection. Behavior research methods.
Harootonian, Ekstrom & Wilson (2022). Combination and competition between path integration and landmark navigation in the estimation of heading direction. PLoS Computational Biology.
Hopfield (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS.
MacKay (2003). Information theory, inference and learning algorithms.
Miller, Eugene & Pribram (1960). Plans and the Structure of Behaviour.
Sweis, Abram, Schmidt, Seeland, MacDonald III, Thomas, & Redish (2018). Sensitivity to “sunk costs” in mice, rats, and humans. Science.
Walasek & Stewart (2021). You cannot accurately estimate an individual’s loss aversion using an accept–reject task. Decision.
Wilson & Collins (2019). Ten simple rules for the computational modeling of behavioral data. Elife.

[This is an automated transcript that contains many errors]

Benjamin James Kuper-Smith: [00:00:00] By the way, before we, I mean, so yeah, well, you know, I wanted to talk as I mentioned before we started recording mainly about the the defining paper of your career, the computational modeling paper. But before that, I just had a couple of kind of slightly random questions just about stuff I saw on your Google Scholar and in your website.

So I was just curious whether, because I guess RC Wilson isn't the least common abbreviation. So that's on your Google Scholar. There's one book from 1958.

Robert Wilson: Oh, really? That wasn't me.

Benjamin James Kuper-Smith: Yeah. So I was about to say either you're in phenomenal shape for your age, or it was someone else

Robert Wilson: Yeah. There, there are a lot of Robert Wilson's. There are two Nobel prize winners called Robert Wilson. Um, So far. Exactly. Exactly.

Benjamin James Kuper-Smith: So, you know, that was, that's one thing in 58 and then it actually started you know, a couple of years before your. You got your PhD. But there was there was a couple of papers or proceedings or something like that on gate recognition, which might've been you, but there was also some [00:01:00] on sedimentary

Robert Wilson: Sedimentary rocks. Yep.

Benjamin James Kuper-Smith: Was that also you?

Robert Wilson: That was me. That

Benjamin James Kuper-Smith: That was also you. Okay. Okay. So then my question is number one, what's cool about gate recognition and sedimentary rocks? And number two, how do you go from there to your next thing? I think your next publication was on Hopfield

Robert Wilson: Parallel Hopfield networks. Yeah, I've

Benjamin James Kuper-Smith: Become more relevant

Robert Wilson: Yeah, maybe that'll boost my citations on that paper.

Benjamin James Kuper-Smith: But yeah, I was just curious, you know, I know you as someone who does mainly for reinforcement learning stuff that kind of area, let's say roughly. But how do we, how do we go from, from all of that, that we just mentioned to reinforcement learning,

Robert Wilson: Yeah, so so I have this sort of strange path Through computational neuroscience or I don't even know what you call the field computational cognitive neuroscience, right? I guess that's the name of the field that we're sort of settling on in that I started out as a chemistry undergraduate.

And so then as part of that you know, to my eternal shame, I did a an internship [00:02:00] in an oil company, Schlumberger in Connecticut. And, uh. they've since moved up to up to Boston, but, um, that's that internship I was looking at rocks and doing NMR of rocks. So they are very interested in essentially using, you know, MRI technology.

Or, you know, magnetic resonance technology to probe the structure of rocks down an oil well to sort of figure out how much oil is in there, how easy is it to get that oil out of there. You know, if it weren't like destroying the planet, it's it's really, you know, fascinating science.

And so my project summer project there was probing diffusion in, in rocks in very low field. So it was permanent magnet MRI. So not you know, the big MRI machines that you see, it was like this desktop thing. That's, you know, about a meter across. And we probed diffusion in different rocks and in a thing of [00:03:00] glass beads as well.

And then, I can't even remember what we were doing the two dimensional inverse Laplace transform, it was all, all about that. To try and find the structure of these rocks. And in particular the role of the role of, oh God, what was it? It was like Magnetic impurities like changing the changing how you interpret these things So that was you know, that was very early on when I was still in the chemistry world and

Benjamin James Kuper-Smith: by the way, you know, what's kind of a, I mean, it's maybe not that surprising, but so I've always found chemistry to be the most boring topic in the world. And somehow I think you're like the. Fourth guest or something. I know now about who, who's mentioned their chemistry undergraduate somehow. I mean, you know, it's by now becoming a pretty reoccurring thing.

People are still. Well, doing chemistry and then not sticking with it, which, you know, I support, but um,

Robert Wilson: that's

Benjamin James Kuper-Smith: I've, I've, I'm surprised that it happens so frequently, but again, it's [00:04:00] not that surprising

Robert Wilson: right? Yeah

Benjamin James Kuper-Smith: Interdisciplinary neurosciences and

Robert Wilson: Yeah, I think that's it. And especially back then, sort of 20 odd years ago, training for this stuff was just not established, you know, even really at the graduate level. I mean, there was Gatsby at UCL, right? 

Benjamin James Kuper-Smith: Again, come into lots of fame this week.

Robert Wilson: yeah, exactly. Exactly. You know, um, but there wasn't much. There wasn't, you know, you couldn't do a computational neuroscience PhD.

You know, really, anyway, Gatsby was probably the closest you could get. And and certainly at the undergraduate level, There's really nothing, you know, And so that's why you have all these folks that came in from physics or chemistry or math you know all biology right the more traditional bio route or psychology route, but You know a lot of the quantitative stuff just wasn't taught on that side So, you know a lot of the modelers come from this this sort of physics background that's changing now, you know that education undergraduate education is changing.

We're hopefully at Georgia Tech getting a major on computation and [00:05:00] cognition which is going to be, you know, entirely this stuff, right? You know, computational neuroscience, mathematical psychology training people to do you know, to do the job that that I do, that my generation, we kind of just had to figure it out as we went along.

What were the useful things to know? So, yeah, I mean, I guess that's where that weird paper comes from. The gate stuff is actually, it was after I made the transition. So I was already doing a PhD in bioengineering at that point at Penn. This guy, Lee Finkel, who sadly has passed away since He was making biophysical neural network models and models of single cells.

And trying to, you know, trying to understand all sorts of things, but the project that we were working on as part of that was biological gait recognition, right? So this is kind of this you can show You know, you show the classic example of this is you show the points from like a motion capture device.

So you put points on the on the [00:06:00] joints. And you know, you look at those when they're not moving and it just looks like a random collection of dots. But then the minute they start moving, you instantly recognize that as as a person walking. You can even, you know, do things like say the age, gender.

You know all sorts of things there's all you know, and they even recognize individuals you know if you know them from how they move 

Benjamin James Kuper-Smith: isn't that also how they, I don't know whether they still do it, but at least how they used to do a lot of the animation. For films

Robert Wilson: Yes, right with like motion

Benjamin James Kuper-Smith: to do it. And then they use that

Robert Wilson: Exactly. Yeah, exactly and I remember one thing that we did as part of the phd was to create a motion capture data set where I remember this is when I used to run a lot and i'd been out for a really long run in the morning And then I came into the lab and they put me on a treadmill in a, one of these funny motion capture suits and made me do more running and more walking for hours.

I remember being completely exhausted at the end of it. So that, that was a fun project, kind of the way in to that stuff. So I was [00:07:00] somewhat involved in some of that. Not wasn't. The main project for the PhD and then I sort of transitioned over to these more neural network models and Bayesian models.

You know, still pure model and then from there transitioned into the, to the actual, you know, sort of, you know, Psychology, neuroscience side, where I was doing experiments on people.

Benjamin James Kuper-Smith: Yeah. So. Once you had your excursion at the gate and rocks it was basically. I mean, from then, I think it was fairly

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: to

Robert Wilson: Yeah,

Benjamin James Kuper-Smith: what I might expect.

Robert Wilson: At that point. Yeah, it's computational neuroscience and then really cognitive neuroscience since about 2009.

Benjamin James Kuper-Smith: Yeah. And then the last kind of thing that I had there, which is interesting. And this, I know you do because it's also a new website is the phishing thing. So how does that fit into it and where does that come from?

Robert Wilson: That's a good question. I mean, I guess, I guess the,

Benjamin James Kuper-Smith: So by

Robert Wilson: yeah. So, yeah. So yes, by fishing is with pH. Yes. It's not a, no, I'm not an angler. No, this is studying email phishing, right? You know, [00:08:00] where you get a, an email from the Nigerian prince or, you know, whoever it is, right? And you click on the link.

And then you're in trouble. So that I guess comes from I mean what it came out of was one of these. Um, events held by the mcknight brain research foundation, which I was part of when I was at the university of arizona and that mcknight is in a bunch of different areas, but the brain research foundation is a set of four universities university of arizona Florida, Miami, and um, uh, Birmingham, Alabama.

And they organized this event to get a bunch of researchers together talking. And you know, that basically connected me with This researcher, Natalie Ebner, who's at University of Florida studying fishing. And she just had this beautiful task where she actually fished people for real.

So they, you know, people sign up for an experiment under some cover story. And the real experiment is I send you a bunch of phishing emails and I Check whether you click on the link or [00:09:00] not. And I just thought that was such a beautiful measure and so, you know Obvious question from there was can we do that in the lab, right?

And when we do that in the lab, does it relate to the behavior in the real world which You know in some ways it's kind of a holy grail for psychology, right? You would like to be able to measure something in the lab and have it tell you something about you know Is someone going to get fished in the real world?

So that was a really Interesting hook and then You know, I guess more generally, my research is,

Benjamin James Kuper-Smith: metaphor.

Robert Wilson: yes exactly. You know, my research is all about like computational modeling of behavior and this is a really, this is a behavior, right? And you can build a computational model of it, especially now that you have large language models, you can actually handle verbal report data.

And you can ask things, you know, the first thing we did, With just the in lab behavior was asked, do we see order effects in that? So, you know, there's classic order effects in all sorts of [00:10:00] psychology tasks where the last trial influences this trial. And a classic effect on these kinds of judgment tasks.

Is you see a sort of repetition of the last response and a negative effect of the last stimulus and you get that if you're essentially comparing the last stimulus to the new stimulus and you're using your last response and anchor and. You know, we, we see exactly those order effects in that task.

That was kind of cool to see from that perspective. But yeah now we're sort of throwing LLMs at this. You can start to ask, can I predict behavior? You know, can I predict who's going to fall for a particular email? Is this person going to fall for that email? So that's really interesting.

And then you can also ask things. By doing sort of dimensionality reduction, you know, kind of like factor analysis, basically you can ask what's the dimensionality of fishing susceptibility, right? Is it a high dimensional you know, individual differences determined by a high dimensional vector, right?

Or is it relatively low [00:11:00] dimensional? They're just people who fall for it and people who don't, right? And you know, In our unpublished stuff so far we're seeing it's about five dimensions for the behavior on this task. Right. And that's one issue is we only have you know, about an hour's worth of data from one person, which is about 200 emails.

Which is, you know, limited in its scope to a certain extent, but it's just an interesting behavior and something where maybe, you know, I felt like I could add something both in terms of the experiment, but also in terms of the analysis.

Benjamin James Kuper-Smith: Okay. Yeah. So it was a bit of a random origin in that sense that it wasn't, you know, something you thought made complete sense from what you did before, but it was kind of this um, what's the word like serendipitous meeting

Robert Wilson: I think that's right. Yeah, it was sort of serendipitous. I think it was also, but also. You know, I think once, once you have this skill set to, to make computational models of behavior you can [00:12:00] apply it to a lot of different situations. You know, it's not just the situations in the 10 simple rules paper, right?

Like binary choice and reinforcement learning. You could build a computational model of navigation, right? We've done that recently head turning, right? You can build a a computational model of perceptual decision making. You can build now computational model of. how people process verbal data, right?

Like that becomes you know, it's the same set of skills. So I think. It fits in that respect, and it also fits in this sense of what are the real world applications, right, of what you're doing, what's the real world behavior. And this is a case where we could actually answer that question.

We, you know, Natalie just published this paper where you combine both tasks. And there is a correlation between the in lab task and the real world task at the individual difference level, but it's like point two. You know, it's it's a big effect by psychology standards, but but a relatively small effect which is interesting, right, in its own right.

I mean, that, you know, that, there's a case where [00:13:00] you know, obviously when you're getting phished in real life, it's a very different situation to sitting in the lab but we tried to make the lab task as, similar in, in, you know, similar as we could, right? If they're reading emails, they're deciding whether they're phishing or not you know, they're doing all of these things that, that in principle should tap into a lot of the same cognitive processes and let yet the best you can do is, you know, a correlation of point two between their in lab behavior and their real world behavior.

So that's a really interesting finding in its own right. Right. That goes beyond You know, sort of speaks to this general issue, right? You know, I think a bunch of people have said this. Russ Baldrack is one of them, right? That these in lab tasks that we have maybe don't capture the same, you know, for whatever reason don't capture the cognitive processes that we that we thought, or at least don't capture individual differences in the way that we thought they should.

Yeah,

Benjamin James Kuper-Smith: yeah, I mean, that's, I mean, [00:14:00] when we say that's a huge problem, it's to me, that's one of the, it's almost like one of the main pillar problems of experimental psychology, right? Like, how do you, I mean, for me, like for the stuff I do, which is often, I mean, for the, for now it's fairly simple, like gamble decision making kind of

Robert Wilson: yeah, 

Benjamin James Kuper-Smith: I mean, I think it might be a similar thing. It's just usually people make one of those decisions and then they wait three hours and then they make another one, then they wait two days and then, you know, rather than 200 in a row. And my behavior, you know, I took part in these as a participant first back in my master's in one of these studies.

And I mean, it's interesting. So actually, I think this was one of Rob Rutledge's study when he was still at UCL and he later on, then I was, I think they had a at UCI, like a PhD program. And I wrote him an email about you know, interest in PhD kind of things. And we talked a little bit about this.

I was kind of then randomly interested in the thing that I think I took part in. And so I, I basically said I didn't like this setup of doing 200 decisions in a row [00:15:00] because then that takes chance out of it. And so I, before I knew about the whole economics, that just like stubbornly calculated the expected value of the options.

And I actually asked the experimenters whether they could just calculate that for me, because that's all I'm doing. And now this becomes a test of like my ability to do calculations in my head. And apparently, yes, I had to finish it to get paid. And no, that is not normal behavior because apparently no one does it.

That's what then Rob said later on. Most people don't do that.

Robert Wilson: right. But that's so interesting, right? You know, and if you not told him that you are a subject that looks like expected value decision maker, you know, if he did model selection on that behavior, you would hopefully find that you're a pure expected buy decision maker, right? and what.

Benjamin James Kuper-Smith: which I was in that task.

Robert Wilson: Yeah, which you were. Right.

Benjamin James Kuper-Smith: But not in real life.

Robert Wilson: right, exactly. What does that mean? Yeah, for your real life behavior, right? And what has been missed? [00:16:00] Yeah. And so, I mean, so much about a task is, you know, is, you know, you're not doing the behavior that you're doing in the real world. You're doing a psychology experiment in a lab, right?

And there's expectations on what you think is expected of you, you know, and there's all these sort of strange, you Processes that come in, you know, and I think people have tried to study them, but they're it's very Yeah, makes interpreting these things hard.

Benjamin James Kuper-Smith: I mean, I think actually the best example of this is actually not joking. So I, when I draw, so I'm now at the University of Zurich and the part of the economics department, even though we're doing neuroeconomics. But I'm not joking day one. I received an email from the head of department

Robert Wilson: Oh, wow

Benjamin James Kuper-Smith: asking for a meeting and I thought, huh, that's interesting.

That was quick.

Robert Wilson: Right,

Benjamin James Kuper-Smith: And then I told the person sitting next to me, the office, I was like, is that normal? Which is like, Oh no, there's like standard like [00:17:00] spam emails that we get pretending to be the head of department.

Robert Wilson: right. Yes.

Benjamin James Kuper-Smith: again, if I was an experiment, Well, I was supposed to detect phishing.

I would behave very differently in that moment.

I was thinking I just arrived at the department.

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: I mean, it's surprising that they would, that he would mail me,

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: but it wasn't completely out of the, you know, I just had a completely different mindset and it wasn't completely unreasonable that he would mail me. You know, I don't know what the, maybe it's just very personable.

Robert Wilson: No, that's right. I got semi fished by an email like that from my department head and they've done a great job. They've gotten her picture off the internet and they, you know, sort of spoofed her email address. And the, it was like a one word email. It was like available question mark and or available to talk or something.

And the thing is she sends emails like that, you know, and I was like, I'll be right down. So I went to her office and apparently I was one of many that had been there that day.

Benjamin James Kuper-Smith: a congregation in front of our

Robert Wilson: Yeah, exactly. You know, and clearly that, you know, they're sophisticated. Right. And there's an interesting question of [00:18:00] you go down the rabbit hole with them.

Right. You know, you may be suspicious at first, but not suspicious enough to not respond. Right. And then, you know, you know, once you started engaging, you end up going, you know, getting locked into a response and you can't do the reversal. Right. And I think that is something that was not captured in our task of just, can you detect these things or not?

Benjamin James Kuper-Smith: To me, what's also surprising is just that from what I understand, and I don't know whether it's just like common myth or something, but from what I understand with most phishing emails, you try and filter out the skeptical people by having like obviously fake emails kind of thing. So like only the.

To not be mean, but like only the idiots would kind of click through

Robert Wilson: Oh,

Benjamin James Kuper-Smith: these things. I think they're real, right? Because you have like spelling errors. Every when I, I've heard that's intentional just so the people who might be skeptical, don't pick up on 

Robert Wilson: Don't waste your time.

Benjamin James Kuper-Smith: out the people.

Yeah, exactly. So when I actually get as a phishing email, that's like perfect. It just, you know, I'm approaching it from a completely different angle.

Robert Wilson: Right. [00:19:00] Right. Yeah. That's interesting. I mean, and obviously a big thing that's changed phishing emails is chat GPT, right? Like now you can, you know, there's no excuse for them except if they're trying to weed out, you know, the sort of smart people, right. Is there's no excuse for them to have spelling mistakes anymore, right.

Or bad English, right. They can just put it into into chat GPT and get a perfectly worded email. So that's completely, you know, completely, you know, and that was a lot of, you know, what people are picking up on is spelling mistakes is bad grammar is weird request, right? Those are the things that people pick up on as being you know, as being indicative of phishing.

And a lot of those things they can get rid of now. Yeah, it is interesting. And it's amazing How many smart people and how many people in general just fall for these things? You know, there was the community group in tucson where we were like the neighborhood association. They lost five thousand dollars.

You know by Someone just being you know pretending to be the president [00:20:00] and emailing the treasurer saying I need you to send this check to these people and You know that you know, it's a voluntary organization, right? People are busy and you know this is a sort of standard interaction for that.

It's not a huge amount of money and there you go 5, 000 is gone, right? Like it's you know, and it happens a lot and there's a lot of shame associated with it as well and that makes it worse, right? People are ashamed They've lost, you know thousands of dollars and then they don't tell people and then they can get sucked in.

I mean, it's, you know, there's a huge human cost to it. Not just money, but but mental health issues health issues, you know, and there are a lot of people who are getting sucked into these things and, you know, and obviously, you know, so, so Natalie is an aging researcher. And aging, Alzheimer's disease, cognitive impairment, you know, it really is, you know, I mean, these folks are despicable, right?

Like they're cheating people out of their money, very vulnerable people, usually the people who have fallen for it are vulnerable people. And yeah, [00:21:00] so it's, you know, so that, that aspect of it as well, just, you know, in terms of like why phishing is, you know, You know that it is Interesting to think about something that might have a practical application, right?

So the hope was and maybe there's still a hope but the hope is you could develop some kind of test that you could give to someone that's you know as part of a standard cognitive battery that's like You know, it might be time for you to You know, not have such easy access to your bank account, right? That you can just ship off money to random people on the internet, or there needs to be a second, you know, a second check, or they need, you know, something like that.

So, so I do think that it's kind of exciting to have a, you know, something that has such a clear practical component to it.

Benjamin James Kuper-Smith: Yeah, I guess, I guess one of the common should put it I guess, at least as at the PhD level, probably also, but beyond that, there's the sense of Occasionally you get, it's what am I doing here?

Robert Wilson: Right.

Benjamin James Kuper-Smith: I'm creating this like silly task and you know, and then you go [00:22:00] maybe I should do something applied.

That's the solution to my existential angst, you know? So I guess you've leaned into that.

Robert Wilson: I leaned into that, but

Benjamin James Kuper-Smith: No, but that's, it's

Robert Wilson: yeah, it's nice. It's nice. But also, I think there are, I think the applications are important, but also, I think the applications can drive interesting questions as well. You know, you start to ask questions that maybe you wouldn't normally ask. I mean, you know, certainly dealing with verbal data, you know, you asked me five years ago or 10 years ago, if I'd be I'm, You know, trying to model verbal data, I'd be like, no way in a million years.

And now, partly because of this phishing stuff, partly because of the AI, you know, it's, you ask interesting questions. And I do think that's, there's an importance to that. I think there's a huge importance to the basic science as well. And I think really that, you know, The exciting part is where you know, take the applied stuff as inspiration.

But I do think there is something to be said for thinking about what it, you know, what the [00:23:00]actual application of this stuff is. And also real world problems. I do think real world cognition, and there are a lot of folks, you know, on this now, right? People are trying to measure all sorts of things.

You know, cognitive processes in the real world. I do think that's a huge, you know, huge area and hugely important. And I think inspirational as well to the lab, you know, I think you know, a couple of years ago there was this work on foraging. Not a couple of years ago, but you know, this it's ongoing, but folks like Ben Hayden Michael Platt, Dave Reddish, right.

Started doing work on foraging. And You know, inspired by real world foraging, right, of animals, and that's a very sort of, relevant behavior and realistic behavior and also, from a modeling perspective, super cool, right, you know, the whole marginal value theorem type stuff of patch foraging.

So this is beautiful research and but what they did was they used it to inspire lab tasks that the animals in the lab would learn a lot faster. I mean, sort of [00:24:00] anecdotally, I think they learn these tests a lot faster than they do these sort of arbitrary tasks that we force, force animals to do.

And then you end up with some of these beautiful things like that restaurant row task from Dave Reddish's lab is just absolutely beautiful. And you know, it's got a theoretical component to it as well, but also just this really you know, ecologically valid behavior. All right. So I do think there is, you know, that would be an example of this, where you have this, you know, You know, realistic behavior.

I mean, maybe it's not really applied research, right? But it's like this realistic behavior driving development of in lab tasks that are maybe getting it, you know, just, you know, getting it in a, getting it at a different kind of behavior that you can get a computational handle on. So I thought that was, yeah, I do think that's kind of interesting.

Benjamin James Kuper-Smith: Yeah, I guess it's, I guess there's two sides to it, right? In some sense, I feel like going the complete abstract way and just doing stuff that are increasingly separated from reality, like from a [00:25:00] physical reality can actually be interesting because then you end up in places you never would. But I think, yeah, often it's good to be anchored to, you know, actual behavior.

And so you don't go, you know, isn't it like the case with like lots of these like logical, You know, problems for humans that are often only not solved correctly when you give them in weird examples. But if you give them in different examples, most people solve them correctly.

Robert Wilson: Right.

Benjamin James Kuper-Smith: You know, so it's like sometimes losing connection to what people actually do usually can just lead you completely astray.

Robert Wilson: right. That's right. But there, you know, equally there is you're right. I mean, there is a place for this sort of abstract basic thinking and computational thinking. I mean, you know, I made my career around computational models and abstract computational models of these things.

So, you know, there's definitely a place for that as well.

Benjamin James Kuper-Smith: Okay. So, uh, Let's, let's start talking about well, the paper about competition modeling, I guess we've already been talking about some parts of modeling for the last couple of minutes. Yeah, I mean, as I said, before we started recording, I feel like [00:26:00] your 10 steps 10 simple steps paper in eLife is a paper that most people I know I've read.

And I mean, it was, I mean, I'm starting to learn computational modeling. It's also one of the first things I read Just to get a kind of nice overview of kind of roughly how the whole thing works and yeah, maybe as a as an introduction why write the paper? I mean, there's, I mean, there's so many textbooks about this. There's lots of stuff about this already out there. And I feel like occasionally I've seen, so I've interviewed, had some other interviews where people did these kind of tutorial esque papers, and they have said that they've faced quite some they've had some difficulty getting it published because people, you know, would say well, do we really need this?

We already know how this works. And then they will often, you know, like yours very well appreciated papers. So I'm actually just curious, like why write this paper? And did you face anything like that or was it 

Robert Wilson: yeah, so i'll yeah that so the publication barriers are real and definitely want to thank elife for for [00:27:00] publishing that yeah, we I won't say which journals but we went to two journals one of which gave a full peer review That was positive And still rejected it was a desk reject after peer review at one journal, which was bizarre but Yeah, we put it online as a pre print.

We already knew it was going to be really popular just from the, you know, number of downloads. You know, it was great that eLife, you know, sort of open minded enough in their in their publication model that they were willing to take it. I mean, I guess why write a paper like this? You know, there's sort of two reasons.

One more practical was I'm often working with students who are new to computational modeling, and I just wanted to be able to give them something that's okay, read this so that I don't have to answer, you know, the first 90 percent of questions. So that, that was one component but the other has a sort of older sort of motivation and it's as I came up in science, in the.

2010s really, you know, sort of in my postdoc, it was sort of like the [00:28:00] heyday, the early 2010s and, you know, sort of late 2000s was the heyday of this kind of stuff or when this stuff was really exciting and new and you could get these high impact papers you know, publishing these model based analyses and, you know, there are all sorts of claims that you can do things with the model that you can't do with just simple measures.

And. I just remember being kind of, when I got to doing it myself, being kind of skeptical of the claims that are being made and how rigorous the process is. So you, you fit these models. And I just remember, you, you know, you, it almost feels like magic when you do it, right? You write down a likelihood function, and then you throw it into some minimizer or whatever, right?

And you get some parameter values, and then you can simulate the behavior which people at that time didn't do so much. They just, you know, throw it into the, you know, just report the parameter values, right? And which model was the best fitting. And it does feel very black boxy in that respect.

I mean, obviously it's not a black box, right? But it [00:29:00] feels very black boxy. And I just remember thinking when I was doing it is this real? You know, and so the first thing you do then is start simulating it. And you say, well, if I know the ground truth. Do I get the ground truth back?

And a lot of times you don't. You know, if your experiment isn't good enough, or if you have bugs in your code, right, that's another reason. But certainly if you have an experiment that, that's not got enough trials or not got the right kinds of trials. You can't get your parameters back or you can't tell the difference between two models and You know, so I guess I guess it kind of came out of a frustration there and there's one There's actually there's one lab meeting.

I remember at Princeton where We had this big old debate because there was you know, people were fitting different models to a task it was sort of like a grid world task and you know, when you're moving around in, in, in a grid space, you know, like a really, you know, terrible video game, right?

And, you know, they fit all these different models and the model [00:30:00] that won was like a pure choice kernel model, which says what you do is you just repeat what you did last time. And, you know, I just remember thinking, well this, okay, maybe it won on like BIC or whatever you know, whatever sort of model comparison measure you used.

But this model can't actually do the task. This model just repeat, you know, if you were to simulate this model, it produces nonsensical behavior, because it just repeats what it did on the last trial, the last movement. Um, and we, you know, I do remember it was a big debate, right.

In the lab meeting as to, well, maybe we should. Do this model because if it's best, but I'm like, clearly not can't do the task, right? You know, just a simple simulation would rule it out. Um, that's when I realized like this really like needs needs to be, you know, someone needs to sort of.

Say something about this kind of thing and that, that was sort of the deeper drive for this was like trying to sort of boost the quality. And then, [00:31:00] you know, the 10 simple rules format, you know, originally it was just going to be a short comment. There are these 10 simple rules papers in um, PLOS.

Yeah, exactly. And, you know, I was thinking of it being that kind of format. A relatively short paper, and then it just kind of took on a life of its own. And you know, it sort of sat on the back burner for a little bit, and then I went to visit Berkeley and chatted with Anne, and we were like let's just finish this, finish writing this paper.

And yeah, it just became this big thing. And you know, it's since, yeah, I think, you know, this will be the thing that people remember me for since this paper.

Benjamin James Kuper-Smith: It's funny.

There's Modeling things where a model wins, but doesn't actually do the task. This is like a less sophisticated level than what you were describing, but I recently just fiddled around a little bit with evolutionary simulations and trying to you know, get something simple going there and see how it works.

And I ended up with something quite interesting that kind of supported my hypotheses. And then at the end, I realized like, Oh no, wait, I just have a mold that doesn't do anything. And because I don't have it [00:32:00] in build cost for each decision, the ones that wins just don't do anything. And yeah, it wins the competition in this sense, but only because the competition isn't realistic in that sense.

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: But yeah, the yeah, I guess we'll be. Kind of going over most of the steps or some of them, we'll see how far we get. I thought maybe to start off with well, I mean, this is also what I use. I say, if it was my great idea, it's the beginning of your paper, which is kind of like a step zero, which is a little bit of like why.

Do this kind of competition modeling. What do we gain from it? I mean, I've had some episodes about this with Paul Smodino in particular about the use of formal models, but yeah, just in the context of this you know, I mean, I come from a psychology background, so my thinking is very much in, You have some sort of fake hypothesis and then you create categories and compare them to each other and then something significant or not.

And, you know, that's it. Why do I need this whole modeling thing? It sounds very complicated. What are the main advantage of doing competition modeling and what does it give us that [00:33:00] we can't do without it?

Robert Wilson: yeah, I mean, I think the big thing with anything computational mathematical is it forces you to be very explicit about what you mean when you're describing a cognitive process and Yes. If you want that simulation to run on a computer you can't fudge anything, right? You have to specify your parameters.

You have to specify what you think is going to happen under these conditions. And that's in and of itself without, you know, any of the other benefits just makes you be very explicit about what, you know, forces you to think at a much deeper level about your theory. You can't hand wave anything away.

So that, that's really cool. I think it's also important from a from a communication perspective as well. It's unambiguous, right? You've written down your model, you've written down some equations you've, made it clear how that model is going to apply in different situations. And if somebody else wants to go [00:34:00] and run that model hopefully you've shared the code, so they can.

But if you've shared the equations, you know, it should be the exact same model. that someone can implement years in the future. You know, and this is fun to do, you know, you know, you, I mean, it's the Nobel prize you know, in the last couple of weeks. Right. And if you go back and look at Hopfield's paper, You can look at those equations and you can simulate that model.

I did it actually in my class the other day, you know, the day after they announced it. We just dropped everything. We're like, let's talk about hot field networks. And you know, it, it behaves exactly as he said it does. And you can simulate that. And that's very explicit about what you mean.

It's very explicit about what, in his case, what a memory is, right? And what it means to recall a memory. You know, in that model. That, that's there's a power in that allows people to build on what you've done, right? And I think that is a big part of the reason for modeling.

I think there are other things, you know, sort of these downstream things like parameter estimation, right? [00:35:00] Picking apart cognitive processes. That's a lot, you know, sort of the nitty gritty of this paper, right? You know, and characterizing individual differences. I think all of these things are really really good.

Yeah and really useful, but I think the bigger picture is yeah, once you've written something down and a bunch of equations, it's unambiguous as to what you're doing.

Benjamin James Kuper-Smith: Yeah, especially when I talk about the parameter recovery for me is an interesting one that I think it's yeah, we'll talk about that later, but I think that's a really interesting one that just gives you a precision about your experiment that you would otherwise not get. Well, Would be difficult to get.

Just briefly before we, you know, get into the details of modeling, I was just curious kind of when, a sense, should one start thinking about this? I mean, is this the kind of thing one should start doing right from the beginning of, I mean, of the, of an experiment, not like in your career or whatever, I mean, like in a project or does it require a kind of prior. and precision in knowing what you're actually [00:36:00]doing before you can actually nail it down, you know, to first get like a vague sense of the shapes, shape of what you're doing and then nail it down or would you actually start with the model from the beginning.

Robert Wilson: That's a big question. And I think a lot depends on your experience level. I think for me and someone who's, you know, done a PhD in this, right. Or, you know, has got an experience in modeling. I think you sort of always thinking about it from that perspective and how can I cast this, you know, what do I mean with this, psychological question and how could I make a test, how can I make a test to get it?

And then how would I model that? And then. The model is a part of that and oftentimes, especially when you're coming at things from a computational perspective the question is driven by the computation, right? So, you know, you want to study the explore exploit dilemma and you know, from theoretical work that you can do that in at least two ways.

You can seek information explicitly, or you could just randomize your choice. And that, that's something that you don't even I think you don't even have those concepts if you [00:37:00] are not thinking about it in a modeling way. Um, and then from there the experiment flows, I think a more traditional approach, especially for folks that are less experienced.

But also if you're new to an area, I think. It's much more, yeah, you start somewhere, you make some kind of experiment. You know, you want to make, if you take this approach, you want to make sure that you can analyze your experiment in a traditional way, and it's going to tell you something. Because oftentimes when you just throw a model at it, it's not going to tell you much more.

You know, because you didn't design the task in a way that allows you to get at the parameters that you want, right? So, I mean, a classic example of this would be the Iowa gambling task, right? You know, you want to study decision making, and so you come up with this Iowa gambling task and it's a very engaging task.

People will play it and they're,

Benjamin James Kuper-Smith: So just very briefly, what is it

Robert Wilson: oh, sorry, the Iowa

Benjamin James Kuper-Smith: Just like a, you know, 30 second overview and not

Robert Wilson: so you choose between four card decks and each card [00:38:00] deck pays out probabilistic rewards. There's the four card decks are set up such that the first two, A and B, are on average bad, but they pay out higher rewards. in the short term.

And then C and D are an average good, but they tend to pay out smaller rewards, right? So that, but their punishments are less. So, you know, that, that's a task to study decision making. It's hugely important in the, in, in the field, in the neuropsychology of decision making, you know, it detected deficits in patients where, like OFC lesion patients, where you couldn't otherwise see Decision maker couldn't see cognitive deficits, so it's a hugely important task.

But if you try to model that task, it's just a mess, you know, and there have been heroic efforts to model it. But it's really limited in what you can say because of the design of the task. You know, if you look at the. Distribution of rewards, the gains are either plus 50 or plus 100. The losses can be anywhere down to [00:39:00] minus a thousand.

So very unequal distribution of gains and losses. You get a gain on every trial, but losses are probabilistic. There's an exploration exploitation component as a risk taking component. There's all these other, you know, there's all sorts of weird things going on in that task, but the reason it was successful is.

There is a very simple way you can analyze it, which is just how often do you choose the good decks as a function of time that allows you to detect deficits in certain populations that makes sense, right? And so I think that's a really great example of a task where, you know and I feel, I mean, it sort of predates the, you know, sort of computational modeling of behavior, at least the sort of modern model fitting of behavior.

 But it's sort of like you, you develop a task that, that's a great assay of decision making or gets at some aspects of decision making. Then you think about it and try to model it and realize actually it's terrible for modeling decision making. You can't get parameters very well. You have to make all sorts of assumptions to do it.

And that's not to say there [00:40:00] haven't been incredible efforts to try and do that. But you're ultimately limited by the task. And so if you want to. Go to the next step you start making different kinds of tasks like a task just for explore exploit or a task just for risk Or a task just for ambiguity or a task to measure gains and losses, right?

You know in a way that you can actually back out a loss aversion parameter and it's actually when you do parameter recovery, you can get it reliably out of there. So I think that is Yeah, that trajectory of that field sort of maps the trajectory of how most people end up doing their work Sort of approaching cog, you know, computational modeling is you have some task that's based on good psychological intuition for what might be wrong or what might, how people might be made, you know, that should give you interesting results in the simple statistics and then you apply a model to it and you realize, Oh, if I really wanted to apply this model, I should have designed a better experiment.

And then you go back and design the better experiment. And that may be, you know, that could be two or three [00:41:00] papers by the time you're down there. That's a PhD, right? But I think that is the wise approach to doing it. And I think especially that second stage, it's important to be kind of realistic in what you can expect the model to do.

And this was a lesson, a hard lesson for me. And I think for everyone in the field, you know, there's sort of this realization that if you can't see something in the raw data. You usually can't see it. The model is not going to let you see it either. The model can get you more precision, um, but you can't, I remember in my PhD, we did this thing with, you know, very briefly had this voltage sensitive dye data set that people were sharing with us.

 We were applying all these crazy image processing tools to try and extract Interesting information out of it. And the only thing we could get was on one trial where you could see very clearly a propagating wave in the raw data. Could we get anything interesting, any of our [00:42:00] sophisticated tools to actually pick anything up on the other trials?

where you couldn't see anything with your eye. It was basically noise. You couldn't see anything with a model either. And I think that, that is a real lesson. I mean, there are sometimes where you can get things out that you can't see in the raw data. I'm of a firm opinion that the window between things that you can see with simple analyses and things that you can see with a model is not as big as as People would like to believe.

Benjamin James Kuper-Smith: so I mean, if I understand it correctly, then the, I mean, I guess for one, you need the simple analysis also to kind of check that, you know, some basic sanity checks of what you're doing actually makes sense. And that the, you know, the model roughly agrees with that. But it seems to me then that from, if I understand it correctly, that kind of the model is always something.

It's always a consideration on top of an existing experiment. Something that can, you know, you have something and it's not like the model completely replaces it. The model analysis completely replaces everything else, but it's something that adds something that [00:43:00] the original couldn't

Robert Wilson: exactly. Exactly. So yeah, exactly. When you're fitting a model to an experiment, it's getting more information out or more precise information. Then you could say with just a more qualitative analysis of the data you know, sort of, , means and standard deviations.

I think where the model really comes into its own though, is when you take that next step and you design the experiment with the model in mind. Because then you, you're asking a question that you may not have even conceived before. You know, an example of that we did some stuff on exploration, exploitation.

And, you know, being random in your exploration, that's the sort of standard Epsilon greedy, you know, softmax type exploration that people talked about in the RL world, in the, you know, computational side of that world. But there's an interesting question. Is that, actually random?

Is that choice behavior, you know, behavioral variability that you see you know, is it actually random? And we have a [00:44:00] task that seems to suggest that under conditions where people should explore, they become more random. But then you can think about this mechanistically in terms of a model and in terms of a drift diffusion model, right?

Where randomness in a choice in the drift diffusion model can actually be modulated in two ways. It could be modulated by you just adding more noise to the whole process, right? Or changing the signal to noise ratio, or it could, you could be more random in your choice in your button presses by reducing the threshold in this model, right?

So you're, you know, The drift diffusion model, you accumulate noisy evidence to a bound. If that bound is really low then you're more likely to make a wrong decision because you went to the wrong bound. That's two different mechanisms by which you could get random exploration. Right. And that's, you know, you wouldn't even without thinking about the DDM first, you wouldn't even have posed that question.

And then you can ask in the experiment that you design, you can ask, well, what does that mean for the experiment? [00:45:00] Well, those two things mean something quite different. If you're reducing your signal to noise ratio it means you should, you're, you, in conditions where you're more random, you should be slower.

If you reduce your threshold in conditions where you're more random, you should be faster, right? So you don't even need the model fit at that point to test that hypothesis. It's just, do they speed up or slow down in, in situations where they're more random? And in fact they they slowed down, so it's a signal to noise story.

The model, then you add the model and say, Okay, let's estimate the parameters. And C, does the threshold change as the signal to noise ratio change? And it's mostly the signal to noise ratio. There's a little bit of threshold. And then you can go on further and you can simulate and say, well, how much of the behavioral noise variability change, the choice variability change, is driven by the threshold and how much is driven by the signal to noise ratio?

And you can see it's about 80 percent is on the signal to noise ratio. So, so that was a case where, you know, You wouldn't have even thought to do that analysis, [00:46:00] or at least to understand what that now, that reaction time analysis is telling you without putting it in the context of a drift diffusion model.

So that I think is a case where, you know and sort of illustrates the power of these models is it, and thinking about these models is it's, it drives you to experiments and analyses and simple analyses that, that you wouldn't have without having that context of the model.

Benjamin James Kuper-Smith: I guess we've already, you know, hit on a couple of points that we'll probably refer to later, but to kind of maybe go a little bit through the steps. I mean, so we've talked in, I guess, in a lot about the importance of experiment, that kind of thing. But you know, again, before we start recording, that was the point you wanted to hit on a bit more, because I guess most people seemed I mean, it's a paper modeling.

So like, why do I have to care about the experiment? I wanted to see about the modeling stuff. But Yeah, we already mentioned some parts of it, but maybe to be a bit more explicit in this context yeah. Why is step one and two still design experiment and build models? Why is that especially the first design experiment wise?

[00:47:00] Why is that still the most important step?

Robert Wilson: I really think it is the most important because you know so much especially my early career as a modeler You someone comes to you with an experiment saying can you model this and it can even be yourself? I have this experiment. I want to model it and That's where you really run into the limits of what you can say about these things.

You know, you very quickly realize that the experiment can't tell you much about this model, you know, or this model applied to this experiment doesn't really tell you much. And so I think that really is such a key component, if you're going to be building computational models of behavior, is that you build them on an experiment that can actually tell you something and this is, you know, True in any experiment, right? If you don't, you know, measure the thing that you're trying to measure or, you know, you're not, you know, Designing experiment that can answer the question you're trying to ask, then you're never going to get anywhere.

And, but I guess people get these [00:48:00] blinkers on when you have a model, they think it's like this magical thing that you can apply and understanding comes out the other end. And it doesn't, right. It is garbage in garbage out. If you put in experimental data from an experiment that can't tell you anything, then no model is going to save that, right.

Right? I mean, models can do cool things and maybe even an experiment where, you know, seemingly nothing's going on. Maybe there is interesting things you could model. Maybe it's order effects you could model or whatever, right? You know, maybe not the actual question you wanted to answer. But I do think experimental design is just, it's just so, so critical a component of it.

You know, I think it's not what people read this stuff about. This paper for, you know, and certainly when you first encounter this paper, I think folks are reading it to learn how to do reinforcement learning models, right? And I think that's fine and good and a good use of the paper. But I think the next step, if you're serious about a modeling, You know career and having that as part of [00:49:00] your research is to then say okay I've modeled this one experiment that wasn't really optimized for the model now let's actually think about what the correct experiment should be and think more carefully So maybe it's 0.

1, but maybe the way people actually encounter is it's 0. 11, right when they come back to do it the second time right when you actually have a model in mind Now you can ask the question that, that actually tests key, you know, the key assumptions in the model.

Benjamin James Kuper-Smith: Yeah, I mean, how do you and of course, as you mentioned, this is an iterative process. It's not like you go through step one to 10, then you're finished. This is in your paper, you have a couple of loops of where you might go back. And of course, this is also just a schematic in reality is probably a little bit messier than

Robert Wilson: Yep.

Benjamin James Kuper-Smith: but um, maybe., It seems to me like it's a little bit of a chicken and egg situation where how do I know what a good experiment is without having tried whether the model works on it, in a sense. I mean, like, [00:50:00] how do I know what's a good experiment for my model?

Robert Wilson: Yes. Yeah, that's a great question. And it's a hard question, right? It's an art as much as it is a science, right? And You know, I think the practical approach to it is try to build a model of the experiment that you have and you'll start to see what the issues are with it, right?

So you start to build something, say, of the Iowa gambling task, for example, and you see that, oh, hang on, there are outliers In the loss domain, but there aren't outliers in the gain domain. And maybe I want to understand like, how do people process outliers in the gain domain as well. and maybe when I'm simulating this, I, Oh, I have to understand the explore exploit trade offs.

As well, because there's an exploration component, especially early in this task. Okay. Well, I can model that in different ways. But maybe I want to focus just on the explore exploit component, or maybe I don't, maybe I care about ambiguity or I care about risk or I, you know, so I think it really is.

 It's as different as a [00:51:00] scientific question is different, right? What do you want to say from your experiment? But I do think, trying it is a key component to it. And then trying to boil it down to its essence. I think that's a key component, especially if you're doing lab tasks.

I think really trying to boil things down to their essence is important. You know, so so in exploration exploitation, you know, a big thing we did was just try to find the simplest possible way you could get an explore exploit dilemma, and the simplest possible way you could change the relative value of exploration.

And our, you know, what we sort of settled on was this horizon manipulation. If you have a long time horizon, it makes sense to explore early on. If you have a short time horizon, it doesn't make sense to explore. And so then the whole task flows from that, right? called the horizon test. Another example in this is what we've been doing on head direction, path integration in, in, in head direction system is how do, how does your path integration system, you know, how, you know, [00:52:00] navigated in the dark, how does that combine with visual input?

To, you know, determine your location, or in this case, your heading. And so we had a very simple VR task where people turn in the dark and we give them a flash of visual information. And now we can ask, and we can, because it's in VR, you can offset the visual information. relative to where they actually are.

And now you've got a prediction error, right? And it's a predict, you know, visual prediction error and that, you know, it almost maps perfectly onto a kind of Raskola Wagner type model. In angle space now, right? We're interested in how do those things get combined and is it in a Bayesian way?

Is it in a, you know, a Bayesian way, but sampling as opposed to averaging, you know, and those are questions that you can start to address when you Have an experiment that's like this. It's very simple. So there isn't really a There isn't really a sort of easy recipe for it.

It really is, think [00:53:00] deeply about the task that you have, think deeply about the models of the task that you have, but also think deeply about the question that you actually want to answer. And is there an alignment between the task that you have, the model that you have, and the question that you want to answer?

And if not, right, something's got to change there. It could be the question. Maybe you're interested in a different question, right? Maybe you're interested in one group versus another, right. And in general decision making not in, you know, in which case the Iowa gambling task is great.

You know, if you just care, does this group have a decision making deficit relative to this other group? That's fine, right? But if you want something more fine grained, you're going to need to have a task where you can ask those fine going questions. So yeah, it is unfortunately a Circular but as you do it, right and it's certainly if you're doing this as a phd student as a postdoc, right?

You have a few years you go through You know one task and you try it you go through another task and you try it and the second time around you already I've learned so much from the first time that you start to get an intuition. [00:54:00] It really, you do start to get an intuition for what's a good experiment and what kinds of things can I easily model versus what kinds of things am I not going to be so, you know, not going to be so easy to

Benjamin James Kuper-Smith: know, I mean, I have to admit, it's a little bit disappointing that it's not easy, given that your paper's called 10 Simple

Robert Wilson: I know

Benjamin James Kuper-Smith: you know,

Robert Wilson: that's yes. Maybe that's the hardest rule of all. Yes. I mean, I think it's, it's important to keep it. It's a simple rule to keep in mind. Um, but yes, it's not a simple rule to just, to execute right? 

Benjamin James Kuper-Smith: Yeah, exactly. Yeah. Everyone can be easy to state, but hard to follow.

Robert Wilson: yeah it's you know, write a beautiful symphony, you know, or a beautiful piece of music.

It's easy to have that as a goal and you would know it when you see it as well or hear it, right? But to actually create it is hard, right?

Benjamin James Kuper-Smith: Yeah. What's the , so this is kind of the thing that uh, I think is important if I understood you correctly, and that I also kind of would like to be important because it's the kind of thing I [00:55:00] also like doing which is kind of, what's the role of like intuition and introspection about the task?

Because I'm assuming now this is, I mean, this is now for I guess maybe also for animal research, but you know, this question has come mainly from human research, where you actually do the task yourself and think about like, how would I do this task, right? Or like, how might people do this task and all that kind of stuff.

I'm just curious, how does that play into the whole thing?

Robert Wilson: I think that is a big role to sort of imagine yeah, to do the task yourself and think about it. I mean, in some ways you know, a source of a model is your own introspection of how might this task be done? And obviously introspection is biased and incomplete, right? And you don't have access to all the computations that are going on.

But certainly doing the task, getting a feel for the task, Okay. Is an important thing and that can, I mean, that can help with, you know, even just surface level features like if you have lots of delays in a task, it can get really frustrating and slow and disengaging, you know, just doing a [00:56:00] task, having a task that's engaging is pretty important.

I think pretty important actually, right? You know, I mean, a lot of these tasks when you have to have lots of trials, just by definition, they become unengaging but at least trying to make it, you know, as, you know, minimize the boredom level is an important part. So I think there is a role for doing the task yourself, for thinking about it but also being aware that you are in a very weird position as an experimenter doing this, right?

You're, you know, obviously a highly educated person for a start, right? So you may think, you know, and if you're a modeler, probably fairly at ease with mathematics, right? Certainly more than the general population. So you do have to be aware that people will solve your task in completely different ways than you could imagine.

And I do think that is a, that is something to be aware of, right? That, that you live in a bubble and you know, the way you might think about a [00:57:00] task It's just completely different to how 90 percent of the population thinks about a task.

Benjamin James Kuper-Smith: Yeah that's one of my problems with, I mean, I have to do it, but with learning probability theory in a bit more detail it's, Is that people doing a task are not going to know probability theory. So by default, if you learn it, you're going to think about the task differently than pretty much anyone you want to study.

Robert Wilson: Yes.

Benjamin James Kuper-Smith: And this is, it feels to me like it's almost like any study that deals with uncertainty should have one person who has no clue, probability theory, collaborate with someone who actually knows how to model it. Because yeah, I mean, most people. You know, don't think about all the intricacies and the ways of how you might even consider probability, you know, it's just ah, sometimes stuff happens

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: it's like, it's

Robert Wilson: Yeah. Yeah. I mean, people, yeah, people, yeah, it is weird, right? People are really bad at probability, especially when it's explicitly described, but then people can be almost Bayesian if it's an experience, you know, something [00:58:00] they're experienced with, right? Their intuitive reasoning.

can be really good, but if you put numbers on it yeah, people are notoriously bad with probabilistic reasoning. You know, it's all these, you know, Monty Hall problem, right? Things like that, right? That you know, I mean, that's what a case where people even with statistical training get confused, right?

So, um, that's probably the best example I think I've heard of, A case where as a researcher, you are just unqualified as a, you know, sort of general member of the public to, to understand the decision process. I mean, it's even bigger for animals, right? If, you know, you know, animals obviously experience the world in a completely different way to humans, right?

And experience the lab in a completely different way to humans. I've certainly heard anecdotes of animal researchers trying to think You know, think like a rat, right? What is the rat actually doing? Yeah, so, so

Benjamin James Kuper-Smith: but I think that's the nice, I mean, I think the, I mean, you're obviously right, but I think one thing that's kind of. nice about animal research is that it's a different species. So it's more [00:59:00] salient that they have a different way of thinking about it. Whereas with humans, I think, especially when people think that there are normative ways about thinking about these things, you get frustrated with why they aren't doing it that way.

Even though the participants are always correct, basically.

Robert Wilson: right, right.

Benjamin James Kuper-Smith: I mean, it's

Robert Wilson: Yeah. It's yeah. It's really hard to imagine that people think differently about it. Right. I mean, you see this. In election politics, right? How can someone vote for this person or that person, right? Like, But even in, in the dress, right? You know, the, that famous optical illusion, right?

Is it blue or black or white and gold? And you just can't, you know, in your bones, you feel like, how can anyone see it any other way? Yeah, and that is a bias that you can take into a behavioral experiment very much. So when you're designing it

Benjamin James Kuper-Smith: Yeah. So, I mean, yeah, one thing I think is super important, which I guess comes out of this, then also is pilot your data a lot and ask participants what they were doing.

Robert Wilson: Yes. Yes. I mean verbal report So so we've got a whole line of research now on explicit verbal [01:00:00] report thinking aloud And and then after the fact reporting what people have been doing And in fact, we're trying to model that right with all these LLMs. But that is a, I think it's just so, so huge, you know, you know, I remember, you know, one of the things that drove us to do this was again, one of these Explorer exploit tasks where we've found this really surprising result in older adults where there's a population of about 20 percent that will never choose an uncertain option, even if it is Clearly the better option, right?

You've seen one reward from it and it was 50 and you've seen three rewards from another one that were, you know, mean 30. They never choose that 50 option. And, What's interesting, you know, there was this anecdotal report, you know, the grad student was sort of sitting in there with the subject doing the task and she was like, Oh, I want to choose this one, but I'm just not sure, you know, and she went back to the other one.

[01:01:00] And and, you know, at that point, we were like, Oh, we've got to do think aloud in this experiment. And we, you know, we have a data set now not looked at the data yet, but I would love to know. What do these folks who, 20 percent of the time, never, you know, 20 percent of the people that never choose the uncertain option what do they say, do they say things that are qualitatively different than the other folks, because, you know, for, and again, it's my own bias.

I look at that decision. I'm like, why on earth would you do that? Not choose that one. It doesn't make any sense to me. But clearly they're doing it and they're doing it, you know, 160 160 times, but 80 times in the in the experiment. Which, you know, clearly there's a very strong drive not to choose that option.

Benjamin James Kuper-Smith: Yeah, I mean, there's lots of stuff we could uh, you were speaking, I had lots of other examples of silly stuff participants told you told me about just them doing the tasks intentionally the way you didn't want them to do it, even though you told them to do it a different way, there's all sorts of other issues, but Yeah, still [01:02:00] going is trying to stick a little bit to the actual modeling and not just idiosyncrasies of humans.

I mean, I guess they have to be taken into account, but you can't model everything. Um, like that one guy who found Jesus um, and cooperated on every single trial.

Robert Wilson: Oh, wow.

Benjamin James Kuper-Smith: And God. Really badly exploited by one person.

Robert Wilson: Oh, no.

Benjamin James Kuper-Smith: to see in the data, just completely rinsed him.

Robert Wilson: Oh, no.

Benjamin James Kuper-Smith: you almost have to have respect for just so consistently taking advantage of someone being nice.

Anyway that's a different story. Um, I guess we've kind of gone a little bit over steps one and two, roughly. I mean, this is all I mean, this is also, you know, representative in some sense of, as we said, the actual. Modeling process from what I understand, you know, if these aren't clearly separated steps maybe three simulate model and experiment is, I mean, it seems fairly straightforward, I guess.

Robert Wilson: Yeah. Yeah. Yeah. I think again, you [01:03:00] know, all of this paper is sort of like trying to raise the standard, right? And if you have a model you can simulate that model, right? And I think that, you know, is something that, you know, I'll admit you know, Most people don't do this before they've run the experiment itself or at least a pilot version of the experiment including myself you know, I'm not a perfect follower of my ten rules, but I You know,

Benjamin James Kuper-Smith: commandments,

Robert Wilson: right it is exactly exactly but it

Benjamin James Kuper-Smith: are a sinner

Robert Wilson: is exactly exactly this but this is you know It is something you can do you can simulate the experiment And get a sense like can you?

Answer the question even in theory, right? And what kinds of effect sizes are you going to get oftentimes? You can get an intuition for this or you have an intuition for this. The Models we have in the paper sort of win stay lose shift and riscola wagner and choice kernel you kind of get a sense for how those are going to behave right, you know, win stay lose shift After I [01:04:00] lose, I should be switching, and after I win, I should be staying, right?

And you know, choice kernel, you know, I should be repeating my previous action. So, sometimes this step gets replaced with intuition, right? I know how this is going to look. You know, certainly that was the case in this in this head turning procedure, you know, that we did this navigation task was, I was like, yeah, I kind of know what this is going to look like.

I know if we plot on the X axis, the offset and the Y axis, the error, I know there's going to be you know, if there's an averaging, there's going to be some sort of linear relationship between those two things, at least in the region that they're they're doing that. So. So, but, you know, if you don't have that intuition about the task and about the model, it can be really useful and it gives you a sense, right?

Now, there's a lot of uncertainty there because you don't necessarily know what the parameters are, right? But it does give you a sense of the kinds of behaviors that you can get out of a model. And especially if you have a [01:05:00] weird model that can be useful to, to test.

Benjamin James Kuper-Smith: and I guess sometimes you can also have, you know, slightly counter intuitive predictions from a model. I mean, I, so I mean, for me, the interesting thing is that basically my PhD, I did very much a like psychology approach, let's say, and now I'm moving over to more of a modeling approach even though, you know, they're not mutually exclusive and there was a little bit of simulation in my PhD, but what was kind of interesting is that.

With a little bit of simulation, I did do once or twice. It's oh, I didn't think of that, but yeah, it makes sense. Once you, once you, Once you show me the, oh yeah, that is what would happen. I just didn't think about it that way.

Robert Wilson: right, right. Yeah. And that, that's the beauty of modeling, right? Is everything is explicit and once you've made it all explicit, the model will behave as the model behaves. And yeah. You can see that in a simulation. Mm-Hmm.

Benjamin James Kuper-Smith: So to, to go to, to the steps that I'm currently. In some sense, a little bit stuck in parameter recovery. Um, Which I I mean, I think one, one reason I'm I'm quite stuck in it. I mean, it's [01:06:00] also because I did like grant applications. So that's the main reason, actually, I've just been doing other stuff, but to me, this is like one of the, this is one of those steps that I hadn't really thought about before that I think is super powerful.

Um, So, uh, well, what is parameter recovery? Maybe what are parameters? I guess if people are very new to this, maybe we should also define that. But,

Robert Wilson: Yeah. Yeah. So parameters are the numbers that you put into your model that. You know, determine individual differences in behavior. So, you know, learning rate roughly says how quickly do you learn from experience, right? A softmax parameter tells you how random you are in your choices.

And often what we want to do when we fit a model is extract those parameters, right? And you'd like to say this person, you know, has a fast learning rate and this person has a slow learning rate. And maybe this group has a fast learning rate and this group has a slow learning rate, right? And maybe this

Benjamin James Kuper-Smith: a clinical thing or something like

Robert Wilson: yeah.

These [01:07:00] folks with depression have a slow learning rate or a high softmax parameter or whatever it is, right? And maybe that sheds light on the, mechanisms of the disease, right? That would be the sort of, you know, perfect story from a kind of computational psychiatry perspective. 

Benjamin James Kuper-Smith: I mean, in a way, isn't the computational psychiatry thing, just you take the standard approach of letting them do a task and then you have the mean differences of behavior. And now we say, well, we have mean differences and parameters.

Robert Wilson: right. I mean, that's definitely one way of doing it, right? But I guess the idea, right, and the promise is, You know, it's like Iowa gambling task again. Sorry to keep coming back to this one task, right? You can see there are all sorts of groups that behave differently in that task. But there are all sorts of ways that you can get that deficit.

And the model tries to figure out, well, is it a learning rate? Is it an exploration parameter? Is it a risk aversion? Is it loss aversion? That, that is, you know, if it is to the extent to which these cognitive models, reflect cognitive [01:08:00] processes in the brain, right? The, to the extent to which you can separate those things.

I mean, that, that is potentially adding value, right? You know, if it's a loss aversion thing, that's a different thing potentially than if it's a reinforcement learning thing. Sorry, a learning rate thing. So you would like to be able to do that with your model. You'd like to be able to say, you know, This is, this person has this learning rate parameter, or this softmax parameter, or this bias.

But the question is, can you actually do that, right? Because the way you're getting parameters is you take a model, And you fit that model to behavior, you basically tune the parameters until you find the PR value of the parameters that best fit the behavior of the subject in some sense. Right. Well, does that work in the best case scenario where you have simulated data, where you know the parameters 'cause if it doesn't work in that situation, you can forget it in any other situation.

Right. And so that, that was a. You know, and I wasn't the first to do this by any means [01:09:00] but. It was something that was pretty neglected, I think, in the field. You know, before this paper. Not that people didn't do it. There are definitely examples of people doing it. But I don't think it was routinely done.

And what you find when you do it, I mean, this is where you, Very quickly run up against the limits of your experiment is sometimes you can't tell a parameter you, you know, you put in a learning rate of, you know, 0. 5 and you get out a learning rate of, you know, 0. 9, right? And, or 0. 1, you know, and it's completely random as to, or, you know, it's not random, but it appears random, the learning rate that you get out.

There seems to be no relationship between the input learning rate and the output learning rate. If you don't have that, or if you do have that, where there's no correlation between those two things, you know, you can't say anything about parameters at that point. And then a parameter recovery analysis is trying to do that more Systematically, you simulate [01:10:00] a range of different parameters and you ask, can I recover?

You know, do I get them back across this range? And You know, you could even do things like try to quantify, like, how much information do I have, right? The correlation is a simple way to do that. But it allows you to say how, you know, how, what's, it puts an error bar, in some sense, on the parameter that you get out, in the best case scenario, right?

And obviously, there's a proviso to all of this, which is, in real life, they're almost certainly not using your model. Right. But at least in the best case scenario where they are using your model, the parameters, you know, are meaningful that you get out.

Benjamin James Kuper-Smith: Yeah, I mean, so this, the parametric everything for me is, Again, as I said, like that's it's a really interesting and cool thing because in a way to me, it presents a how should I put it? I mean, it's like a minimal quality control of your experiment in a way, right, where you can say, okay, [01:11:00] we have idealized situations.

I'm simulating. I mean, the thing that I'm doing, we have three, four parameters that, that we're trying to estimate. We, so we say, okay, we let these different parameters just vary. We, you know, we're basically pretending we have a bunch of random people. with all sorts of different behavior types doing this. Can we get it out of this or not? And yeah it's, it's not that trivial.

Robert Wilson: No, No, that, that's the thing that's surprising. When you first do it, you're like, oh, I actually can't get this parameter out

Benjamin James Kuper-Smith: Yeah. And what's for example, I mean, for me, so I do stuff on losses and gains and risky decision making and there isn't a, there's so what's interesting to me is that I mean, I still have to do the systematic, like reading all the different approaches to doing it, but there's a bunch of different ways that people actually loss aversion and that kind of stuff. I've not seen the word parameter recovery in the couple of papers that I've scanned so far. Which doesn't, you know, doesn't necessarily mean they're bad at estimating the parameters. It's just, it seems like they didn't test it. And there is [01:12:00] this one paper by Valasek and Stewart that is, I mean, it's a relatively small paper, I guess, in decision, but it's a paper that just says that's basically a parameter recovery exercise and says, Hey, you know, this except for like tests that people use. It's not that great.

Robert Wilson: Yep.

Benjamin James Kuper-Smith: Doesn't have a great recovery. There's a lot of noise around. And I mean, depending on what you do, that might not even be a problem. I think depending on what your task is, maybe Bapro isn't the, you know, maybe you have a design where that's not that much of a problem, but for many designs, that is going to be

Robert Wilson: Yeah. Yeah. I mean, it's great that the papers can get published doing this, right? Because it's. It's a big deal, right? If you can't get a parameter out and you want to say I mean it does depend on the Question that you're asking if you question somehow sidesteps the parameter and parameters not important for the question although You know, how often that happens.

I'm not sure. 

Benjamin James Kuper-Smith: Mean, the only thing that I could really imagine where it might not matter, or like something that comes to mind is where maybe you just for whatever, I mean, it's, if it's a fairly obvious [01:13:00] question, we expect fairly large differences. So like maybe a parameter coverage isn't great. There's a lot of imprecision around it, but you're pretty sure it's going to be, you know, the difference is going to be large enough that it's not.

But then again, that sounds also like you already know the answer in advance.

Robert Wilson: big enough. Yeah. Oh, you only care about differences that are big enough. Yeah, I mean, yeah, so it's great that they're doing that. And it is just amazing. And how many different tasks and experiments that the parameters don't. Don't recover, you know, yeah, and certainly a lot of the, you know, a lot of the prospect theory parameters It's hard to get those parameters, you know, the curvature hard to design tasks where you can estimate the curvature

Benjamin James Kuper-Smith: And you get papers with very different,

Robert Wilson: and you

Benjamin James Kuper-Smith: Which could also be for good reason, right? 

Robert Wilson: Oh, yeah. Oh, yeah.

Benjamin James Kuper-Smith: thing and different populations and whatever, but.

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: you would imagine that to be slightly less of a spread,

Robert Wilson: Yeah, no it's, I mean, that, that gets, you know, and that gets at a question that, you know, is the model, you know, is there something [01:14:00] fundamentally missing from the model? I always remembered John White's PhD thesis at Princeton. He did temporal discounting experiments and all he did was like scale the amounts and scale the times by factors of 10 or a hundred or a thousand.

And you know, if you have a fixed discount rate, right, a fixed discount That's you know, you your behavior should drastically change between those scalings, right? If I just scale everything up all the delays by a factor of ten You know, I ought to scale my behavior in an appropriate way And he found that they largely don't, and, you know, they're sort of, yeah, so it was an interesting question as to, is discount factor like now a function of, you know, magnitude of delay and everything else, right?

So, so it was a very you know, or is it just the way we measure discount factor? And he was doing it with these where you put a bunch of choices in a row, and then you sort of pick the difference point out. 

Benjamin James Kuper-Smith: Right?

Robert Wilson: So, but it, [01:15:00] you know, that was a case where, oh, hang on, you know. If you fit the discount factor in these different conditions, you get very different answers depending on the overall scale of the delays and the rewards, right?

Which, if it's a single parameter, is not the case. Which suggests something's missing from the model, right? You know, or, I mean, his story, I think Partly, and I don't want to, you know, put words in his mouth, but partly was, it's just demand characteristics of the task that people expect to respond in these tests somewhere in the middle.

And you know, that, that implies a certain kind of normalization. Or gives you something that looks like normalization when it's just demand characteristics of the task. And But yeah, parameter recovery can be, you know, and that I think was in a task where you do have good parameter recovery but it was, you know, something fundamental about the parameters that weren't that weren't lining up.

And that's a case of you know, a good experiment, right? If there's a single discount parameter, it ought to apply across all different scales. Right. And if it doesn't, then [01:16:00] you have to change your theory in some way.

Benjamin James Kuper-Smith: Yeah, maybe actually, I think I maybe also found like a better motivation for why parameter recovery is such a cool thing, because I guess, I mean, you can just say a few huge amounts of time and complicated problems that come from having a bad experiment. I mean, that's kind of the obvious consequence of this, but I mean, if I You know how many people I know who've and I've been kind of lucky actually, my things that this wasn't that much of a problem, but the amount of people I know who have a task where the results are just really inconclusive and, you know, I mean, especially if you think about like clinical stuff where it's difficult to get large samples and the categories are really messy and all this kind of stuff, if you now also have a messy parameter

I mean, you're just probably not going to find anything.

So like just knowing that, okay. have a task that actually gets a precise parameter out of this, just substantially increases the chance that your experiment is actually going to show something.[01:17:00]

Robert Wilson: yes, that's right. That's right. And I think that Yeah, and it's a pain up front right to do all these simulations and to fit the simulated data and to do Everything before you've even done an experiment but boy, can it save you a lot of time and especially I mean with clinical populations like you say I mean, these are hard populations to work with and get access to You and yeah, you, you want to be sure that you're just not wasting everybody's time with it, with a task, you know, that doesn't tell you anything, even in

Benjamin James Kuper-Smith: accept reject task for loss aversion, I think they said what did I say in the paper? I can't remember what it was, but like, you know, it doesn't write about this task in particular, but you know, if the thing I said if loss aversion, if the true value is the lambda of two, You know, for this task, what you get is roughly between 1.

7, 2. 3, then, you know, if that's the range you can expect for any participant, what is your hope for getting results when you compare patients with health, like 30 patients with 30 other controls or whatever, right?

Robert Wilson: Exactly. Exactly. Right. [01:18:00] Yeah. If the changes goes from two to five, yeah, you can do that. But 

Benjamin James Kuper-Smith: But it's probably not good.

Robert Wilson: change, it's probably not that right. Yeah. Effect sizes in psychology don't tend to be that big.

Benjamin James Kuper-Smith: Yeah.

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: Um. So to, to, I think I've sufficiently made the point of why I think the thing that I'm struggling with is actually cool. Uh,

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: um, I mean, it's just uh, yeah, implementing it and doing it properly. But uh, six is model recovery. I guess it's quite linked to parameter recovery in that sense. But yeah just what's that, how

Robert Wilson: Yes, so this is the other thing you might want to do with a model is you are modeling is you have a bunch of different models and you don't know which one. that best describes human behavior, and that can be human behavior in general, it can be the behavior of this particular participant. And so you'd like to be able to tell the different models apart.

And in the same way [01:19:00] you can do parameter recovery by simulating one model with a bunch of different parameters, you can do the same thing, simulate a bunch of different models with different parameters. You know, so this is like hundreds of thousands of simulations. You can fit All of those simulated data.

And then you can ask if I use model a to simulate my data, do I, is it best fit? Is the data from that model best fit by model a, or is it best fit by the other models? And so how much information do I have in the best case scenario when I, you know, when I run this on human data and I find. Okay, Model A was the best fitting model.

Well, is that because everybody uses Model A? Or is that because you know, for whatever reason, Model A is biased in my model selection procedure, right? And that can happen for all sorts of reasons, right? A simple reason is just You don't have enough data and model a doesn't have many [01:20:00]parameters.

And so, you know, when you have some kind of penalty for free parameters in your model comparison, like BIC or, or even you know, leave one out you know, likelihoods, right? If you do your selection with that, if you've only got one trial in the simplest case, right, you're only going to be able to, you'll always pick out the simplest model, right, the one with the fewest parameters.

The more trials you have, the more chance you have of picking out a model that has more parameters. But again, this is another one where you do these simulations and, you know, especially as you get to more and more complicated models, telling them apart. It just gets really, you know, gets harder even in the best case.

Yeah.

Benjamin James Kuper-Smith: I mean, what is the. So in a way I guess I'm because I'm I guess so rooted in experiments I see a lot of this as a way to just ensure that my experiment is really good and you know Does what I wanted to do? But in a way, of course You can just as true that you can use this to make sure that the model that you're doing is good and so I'm just [01:21:00] curious if let's say you have this task and you have five models or some, whatever, right?

So you have like a no model or something that kind of just doesn't do anything really. And then you have you know, added complexity of models or something like that. And then you find that like model four and five. always outperform the others, but you can't distinguish them really, even though they are different.

So is, when do you know that that's a problem about your task? And when do you know that actually the models just kind of do the same thing?

Robert Wilson: Yeah. Yeah. I mean, that, that's a big question. So there's, I guess there's a question, you know, I guess the question you want to try and get at is do they do the same thing under all circumstances, right? Or do they just do the same thing on your task because you've not You know, tested them in a wide enough parameter regime, for example, you know, parameters of the task now and this can be a hard one to tell.

I mean, there are definitely some cases where you have models [01:22:00] that are on your task like a rotation of each other in parameter space. And, And we've come across this in in one task, looking at sequential effects as perceptual decision making tasks, looking at perceptual effects. where you can, um, the outcomes of each trial are win or lose, it's binary outcomes and it's binary choice as well.

And because of that you can make, you can have two models that are mathematically equivalent to each other, just rotations. One, one model is a sort of win stay lose shift type choice kernel. And the other one so, so you have a win stay. So you have a, you have a choice kernel for winning and a choice kernel for losing on the last trial, right?

So, or you can have a reinforcement learning effect, which has win, stay, lose, shift in one parameter and a choice kernel, right? Those are actually in principle two different models But if you only have binary rewards Those two [01:23:00] things are just rotations. It's like a 45 degree rotation And those models, in principle, you cannot tell apart at you know, they're identical on any one subject.

Now, it's an interesting thing, if you look at individual differences, you fit those two, two parameters, you'll find The win stay lose shift version, those two parameters don't correlate with each other across the population, which maybe suggests those are the independent components, so you can, there's other things you can do after the fact but, in this case it was the experiment, and a different experiment, or looking at individual differences, maybe you can tell those models apart.

So, so that's a case where, you know, and sometimes you can see that you make a model, you think it's a different model, and then you think a bit harder about it. You're like, actually, these are mathematically identical, but then there's this, you know, this sort of funky case where there are models that are really quite similar over the parameter regime you care about.

But they're not completely different. [01:24:00] Now, one example of this is you do the prospect theory utility curve. You can do a fixed curvature for gains and losses and then a loss aversion parameter which just scales the loss domain. Or you can have separate curvatures for gains and losses, or you can have all three, right?

But if you look in a range, I think I did this in my class, if you look in a range I think minus 100 to plus 100, and your curvature, you do the curvature model versus the loss aversion model, for a curvature of 0. 5 in the gain domain so square root utility curve in the gain domain.

You can make those two curves look almost identical over that range. And in fact to the point that you would never be able to tell them apart by an experiment. But if you did a bigger range, Because the you know, scale factor on the losses is fixed, right, but this is the curvature changes with the magnitude of the reward.

Then potentially you could tell them apart in a bigger range. So that would be a case where in your experiment, these are the same models. There's no point in making a big deal of distinguishing between, is it [01:25:00] curvature or is it a scaling of loss, right? Different curvature of losses versus, you know, an overall scale, scaling for losses.

 And that would be a case where You would have to ask do I care about that question, right? Do I care about, you know, curvature versus scaling or am I okay just saying fitting one of those models and saying these people have a bigger loss aversion than these people, this group of people and depending on the question that you want to ask You may care about that or you may not care about that.

So that, so I think, yeah, it really does come down to the question that you want to ask. Like with everything, right? And with this, you know, it all comes back to what's the science question you want to ask. And yeah, in some cases you can get models where you can't tell them apart. You just don't care about the difference, right?

You know, it's not model class A, but it could be models B or C, but they're similar enough that I don't really care. 

Benjamin James Kuper-Smith: There's a, I think there's a bunch of reasons for why people don't do a lot of this modeling. One thing that maybe we should also mention is [01:26:00] that, you know, a lot of this is computationally quite heavy and, you know, So I'm not surprised that the early prospect theory papers that tried to estimate this parameters and do it because it was like the early 90s.

So, you know some of the stuff that I do on my laptop can take quite a while and this is a fairly new pretty good machine.

Robert Wilson: Yeah. Yeah

Benjamin James Kuper-Smith: But anyway, so that's one reason. The other reason is we now get into step seven, which is actually collecting data.

Robert Wilson: Yes

Benjamin James Kuper-Smith: So everything we've talked about so far, we still haven't collected any data.

and you know, that's a lot of people don't or can't spend that time. Or don't want to, but we've reached that point now. Well, actual data, what am I going to do with that?

Robert Wilson: right, right. So actually, I mean, if you have done all the steps beforehand, the actual data part's easy, right? And this is where more sort of just prosaic things come in, right? If you simulate your data to have the same structure as your actual data, you know, either you put it in CSV tables in the same way as your data, or you load it [01:27:00] into You know, whatever package you're using, Python or MATLAB in the same form, you can just use all the same functions that you used for parameter fitting for parameter recovery, just do it with your data.

And now you can ask, well first, which model fits best of the model, you know, if you have a set of models, which model is the best fitting model for each person and for the group as a whole. And then you can ask for the best fitting model or models, what are the parameter values?

And then, you know, you're starting to ask questions, you know, then it's, you know, what questions do you actually ask? Care about then, you know, are you interested in parameter differences between groups? Are you interested in which model fits best? You know to it to a particular group or in general You know, are you using it for fMRI to you know, I guess that gets in there later on, right?

Are you using it to then extract model based parameters that you [01:28:00] can correlate with some brain signal? But yeah at that point you've gotten You know, you've done what you wanted to do, which was fit the model to the data, hopefully figured out a model that fits best, hopefully got some interesting parameter differences, and now you can start thinking about making a story, so long as you can actually validate the model, right, which is the next step.

Which is to go back to simulation, and simulate behavior with the parameters that you have to show that with the parameters that you have, you get the same, you know, you get qualitatively the same behavior out. And Especially for reinforcement learning models this can be really instructive.

Because reinforcement learning models, they kind of yoke themselves to the choices of the participant during the task. And so if, they can never go off in some weird direction. Because they're always yoked to what the participant does. They're only predicting one step ahead. Whereas when you simulate them, they could end up in some weird and wonderful place and end [01:29:00] up not doing the task.

And so that, that's a really useful one, and this comes back to this, you know, earlier point about choice kernels, right, as being good good models in some cases, right, or best fitting models in some cases, but they can't do the task. This again, is another chance to show that you fit your parameters to the task, you simulate it, and you find It earns, you know, half the reward that people do in the task. That's a really that's troubling, right? That suggests you've really missed something. Or, you know, you get it and it doesn't show the same sort of. Win stay lose shift effects that, that people see that people do. Oftentimes that's not the case. You know, this is one you, where more, you know, whereas parameter recovery and model recovery, I would say are as likely not to work as they are to work.

Just from experience, I think model validation, where you sort of fit the model and then plot the [01:30:00] fit behavior, you know, sort of simple statistics of the simulated behavior. Yeah. Usually that does a pretty good job. Usually that does a pretty good job.

Benjamin James Kuper-Smith: I mean, one question I had is, if you do parameter model recovery properly. With sufficient parameter space shouldn't, I mean, your actually observed stuff should pretty much be in there or, I mean, how like, I don't know how,

Robert Wilson: it should do. I mean, I guess you could have great parameter recovery on a model that can't do the test. That's the extreme case. So a choice kernel task, right? A choice kernel model on a reinforcement learning task. If it's got no way of integrating reward, it can't do the task. But you could have great parameter recovery on simulated data from that model.

Benjamin James Kuper-Smith: No, what I mean is like you, you know, you collect the actual data. And participants you know, because you're only testing the models that you tested before or maybe you're adding one, but you know, you're, so basically my point is if you say here's the winning model and you get the distribution of parameter [01:31:00] values from your participants, ideally that distribution would be part of your initial model.

Exercise anyway, right?

Robert Wilson: yes, right,

Benjamin James Kuper-Smith: Or is that often not the case because people are weird?

Robert Wilson: I mean, I guess, yeah, I so I think I understand your question, right, in that the distribution of fit parameters, say, is very different to what you simulated for parameter recovery. Is that

Benjamin James Kuper-Smith: I mean, my question is basically from what I understand, the validation is you collect the data, you fit the parameters. So now you have, you know, a list of parameters for each parameter and each participant. And now you basically test whether those values, whether they all have good parameter recovery, right?

Or did I misunderstand

Robert Wilson: so, so there's that part. Yes, there's definitely that part, right? A validation of,

Benjamin James Kuper-Smith: Yeah. And my people, but what it's like that part should be

Robert Wilson: yes. So ideally you've simulated in the right range. And for some parameters like learning rate that are bounded between zero and one, there's no problem there. It can come, it can be an [01:32:00] issue where you just get these really un, unexpected outlier parameters.

And we got this in the explore exploit behavior with With these folks who never chose the uncertain option, they, you know, in a particular parameterization of the model, it give us really large parameter values. And in fact in at least one of those papers in the schizophrenia paper, where we found that.

We re parameterized the model to, it's the same model, but, you know, just slightly re parameterized meaning that the values that you get have a better distribution. Yeah, it's like one over the value instead of the value itself, right? The, you know, difference between temperature and inverse temperature.

So occasionally, yeah, sometimes you find that you didn't simulate in the right range. Sometimes you find, and this can be an issue, sometimes you find you can recover parameters in one range, but not another, and hopefully you're in that range when you simulate when you fit real data.

So, so that's definitely one, right? But then there's also the simulating the model itself, like with the parameters, simulating the choices of the model, and then Putting those choices [01:33:00] through you know, the simple statistics, like what's the mean number of times they choose X versus Y, you know, under some situation, you know, some circumstances that also should match up with what you get in your actual behavior.

Benjamin James Kuper-Smith: Yeah. Yeah.

Robert Wilson: A key part of it.

Benjamin James Kuper-Smith: So I guess we've got a couple of minutes and two steps left. So I guess I didn't think we were actually going to go through all of them, but I guess we could, we can do it briefly. Uh, Number nine, latent variable analysis. What is a, latent variable? Let's start there.

Robert Wilson: Yes. This is, you know, one of the things you can do with a model is. If you if it really is the cognitive process that's going on under there, then there are a bunch of variables in your model that get computed, you know. So if it's a reinforcement learning model, you've got a value on every trial for every option.

And you've got a prediction error on every trial, right? For the outcome that you see. And those are things that you don't directly observe in the experiment and potentially after you fit the model the values that the model has should [01:34:00] be close to the values that the person had when they were doing the task.

And so now that latent variable becomes something that you can you know, correlate with the brain, right? You could say, you know, put it into an fMRI analysis and say, which areas of the brain show a signal that changes over time like this value signal does, or this prediction error signal does.

It turns out that's not completely straightforward to do, and there are caveats to all of that, right? But it's Certainly something you can do and gives you, you know, again is a thing that you can use the model for to pull things out, right? One thing we've done in this head turning task is, so you're turning in the dark and then you get a flash of offset visual information, which is a prediction error.

You can, you know, because people have systematic. Error, make systematic errors in this as well. They're not perfect. You know, some people underestimate, you know, some people's path integration, they will underestimate how far they've turned. Some people overestimate. So you can fit that model and you can get a [01:35:00] subjective prediction error instead of an objective prediction error, right?

The, you know, what's the prediction error? That this subject saw, given that they tend to underestimate where they are, which is different to the actual prediction error, which was the offset of the room. And potentially that gives us a little more power to, for example you know, look at what's the neural correlates of that prediction error.

So, so that, that kind of analysis is really what we're talking about with

Benjamin James Kuper-Smith: Yeah. And I guess in, in kind of cognitive and computational neuroscience, it seems to me in large part, that is almost the goal of. having these models, right? To be able to get to these things that you can't get directly from behavior and see whether the brain does that thing.

Robert Wilson: That's it. Yeah, exactly. And then you can ask the question at the level of the brain, right? I mean, and that's the hard interpretation of these models and the strong interpretation and I think something that may be, you know, again, with prospect there, I don't think they were thinking about neural implementations when they first did this, right?

But you can ask things like that. Is there a utility signal in the brain, right? Lots of people have asked [01:36:00] that. Does it have the same curvature? Does it do this? Does it do that, right? Yeah. And the model gives you a handle on that you know, in, in a way that you can get experimentally as well, but, you know, can maybe give you a little bit more sensitivity,

Benjamin James Kuper-Smith: Okay. We've reached, I always call it steps, but it's actually called rules in the, and then you also call it sections at some point 

Robert Wilson: Yeah, I'm sorry. Yeah 

Benjamin James Kuper-Smith: I realized, so we've reached the final rule not step which is reporting your results, which sounds like a good idea. Um, So what are some um, Yeah.

Just briefly kind of the main things to pay attention to here. What, What should I report? What

Robert Wilson: think. Yeah. I mean, I think the big thing is to report as much as you can. Right. And a lot of this goes into supplementary material, like parameter recovery, model recovery.

Benjamin James Kuper-Smith: pages 30 to 55. Yeah.

Robert Wilson: stuff, right? I mean, sometimes you could do it in a method. I mean, it really depends, right? If you're using a, you know, if it's a new task and a big part of your, [01:37:00] why you're using it is parameter estimation, then maybe it goes in the main text, right?

But yeah, just reporting that model recovery parameter values and distributions of parameters, right, not just means and variances right? You know, the simulations after the fact, the model validation, and then obviously, you know, sharing the code and the data, right, is a big you know, a big component and in a way that, that can be used, right, in a way that you can just throw in you, you should be able to boot it up in, you know, a reasonable programming language and you should be able to generate the figures that were in the in the text.

Yeah, I think that's pretty much all, you know, sort of standard good science stuff

Benjamin James Kuper-Smith: Yeah. I guess, I mean, yeah, if you do all of this and then you don't share your code. I don't know.

Robert Wilson: right,

Benjamin James Kuper-Smith: Anyway. Um, But yeah, it's funny. One thing that occurred to me whilst we were talking about some of the early rules is that in a way what. What this is it's kind of a parallel function to the rise of [01:38:00] registered reports in a sense where there you know, you write your report and you say what you want to do.

And then people comment on it and they find some flaws with the design and how you analyze it and all this kind of stuff. And in a way it seems to be like, a lot of the benefits from completion modeling, not all of them, but a lot of them have a very similar role that you'd kind of get this outside feedback on your task and how well it's going to work and that kind of stuff.

And additionally you can do all this other stuff like latent variables and that kind of stuff.

Robert Wilson: Exactly. Yeah. Yeah. No, that, that is a big part. And yeah. And obviously this paper was written in that context, right, of the open science movement and reproducibility crisis, right. And sort of trying to head off something similar in the in the computational world, right. Just, you know, low quality models that, you know, you can't even tell parameters apart in in theory.

Benjamin James Kuper-Smith: Yeah. I don't know. I guess the cool thing is also that, you know, If people, again, like it sounds so trivial and obvious, but like increasing the precision of the task [01:39:00] is such a crucial thing. And, you know, if you realize it has bad parameter recovery, and then you manage to make a few changes and what values you use, and then suddenly it's much more precise.

And then, I mean, you know, that's great, especially if it becomes widely adopted.

Robert Wilson: Yes. Yeah, exactly.

Benjamin James Kuper-Smith: suddenly the entire field is just more precise because of a couple of simulations you ran basically.

Robert Wilson: Right. Exactly. Yeah, I

Benjamin James Kuper-Smith: Is there anything else you want to say about this? Otherwise I'll go to the recurring questions.

Robert Wilson: think we probably covered everything right in exhaustive detail. I'm sorry for your listeners.

Benjamin James Kuper-Smith: Well, they're here by their own choices. They could have left. So it's their own fault. They have only themselves to blame. And this also isn't the longest episode. So,

Robert Wilson: Well, we, you know, see if we can. Yeah, how long have we got to go to beat that?

Benjamin James Kuper-Smith: a while I did. So I think the longest I did was two episodes. I think it was actually two and a half hours with Toby Wise very early on one of my first episodes Not at all about his main research

Robert Wilson: Okay.

Benjamin James Kuper-Smith: but

Robert Wilson: bet that was [01:40:00] fun.

Benjamin James Kuper-Smith: Yeah, but since then I've tried to not do more than two hours because I just want to kill myself in the edit

Robert Wilson: Right.

Benjamin James Kuper-Smith: Which is yeah, it's just once it takes several days to answer.

It's what am I doing here?

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: anyway So for current questions First is what's a book or paper you think more people should read? Old, new, famous, unknown, whatever you want. Just some recommendations.

Robert Wilson: I think I, probably two so one of them I actually just tweeted about the other day after the Nobel Prize, which is David Mackay's information theory book, which is just really superb. I mean, it's not a psychology book at all. But if you really want to understand the sort of Bayesian approach it's really good and certainly.

You know, I, I basically did every problem in the first, I don't know, 10, 15 chapters. And by the end of that you really understand something about that. And it also has the later chapters like neural networks and a Bayesian, you know, he was sort of an arch [01:41:00] Bayesian. So that was, that's a beautiful book.

Um,

Benjamin James Kuper-Smith: uh, For context, I am pretty sure the last episode I released on Friday with Soledad Gonzalo Conyo. I think that was one of three books you recommended. I'm pretty

Robert Wilson: that's brilliant. Yeah, it's a great book. It's a great book. So, you know, if you weren't sold from that podcast, you should be 

Benjamin James Kuper-Smith: If I remember correctly.

Robert Wilson: Right. Okay. Thanks Um, the other one just one of these crazy books from the early days of cognitive science. This book uh, plans and the structure of behavior.

Um, And it's yeah, it was like one of the founding books of cognitive science is Miller and a couple of the folks and it's just a crazy book. It was written after they went to this Dartmouth conference on computational modeling like, you know, back in the mainframe days in 59 or 60 or something like that.

And they were just like really psyched about this idea of computational modeling of behavior and they just ran with it. And it's such a beautifully written book, [01:42:00] beautifully written, crazy. It's got a chapter on hypnosis. Like a legitimate chapter on hypnosis where they, you know, where once you've read all the rest, it like fits in with their idea.

You know, it's all about how behavior is driven by plans. And, you know, this is in, you know, pre the good old fashioned AI days where everyone thought it was going to be if then loops, right? To, you know, understand behavior, but I do feel like it's such a, I feel like it's do a comeback, this idea, you know, and and certainly in, in the context of you know, what's missing in modern AI, right?

You know, it really is this sort of system to type behavior and the plans and the structure of behavior. And it's just such a crazy book and you can, you sense the excitement that they have. for this idea of computational modeling. They have no idea, you know, clearly they went to some workshop on it.

They have no idea how to actually program a computer. You know, they, you know, I think Simon and Newell, who were actually programming computers hated it. But it's [01:43:00] just such a, an interesting idea. And, you know, it was very sort of anti behaviorist in its approach, right? So it was, you know, you know, sort of Chomsky is, is not a co author, but definitely a sort of latent factor, right, in this book, and maybe not even that latent.

I think he gets mentioned a few times. So that book, yeah, I read it after I got tenure, actually, and it was just so, so fun to to read something just so crazy. So crazy. So I recommend that one.

Benjamin James Kuper-Smith: Okay, good. Oh, you know, as maybe we should have mentioned this earlier, but as all the stuff we've mentioned, I'll put it in the description. So people then have to look for it. Second question is what's something you wish you'd learned sooner? This can be from your work life, from your private life.

I don't really mind just something where you think that's something that, I don't know, would have helped me if I'd learned a little bit sooner and maybe how you figured it out or what you did about it or. Yeah, whatever you're willing to share,

Robert Wilson: yeah, I think you know, the thing I read this question yesterday and he sent in the email and I think the big one actually is like the professional thing is control theory. I wish I'd learned [01:44:00] control theory a lot earlier. Um,

Benjamin James Kuper-Smith: Control theory. Or is that just a part of it

Robert Wilson: no literally the sort of branch of engineering control theory.

So, you know, this is, I feel like it should be a standard part of education in academia. In computational cognitive neuroscience. I'm going to push for classes, Georgia tech on it and the undergrad program. It's been a bit of been applied to, to neural systems, you know, to, to how you control the eyes, how you stand up.

motor movements, you know, it's much, you know, Daniel Wolpert's big in this area, right? That, you know, it's not like it's not there, but I do feel like it's been a missing part of my education and it's such an interesting way to view things. I mean, reinforcement learning gets at it in some, you know, it's all part of it, right?

Reinforcement learning, you can think of as being a, being highly related to control theory. But it comes at it from a different approach and I do think. This side, you know, understanding controllers and the brain is a control system, right? I mean, the [01:45:00] body is a control system, right? Whether it's temperature or, you know, or anything or standing up straight or motor control.

I do feel like that's something that kind of missed in, in my, Own education that I wished I would have gotten I mean, that's maybe not sort of kind of career Guidance that maybe you're asking for that. I mean the only other thing I can think of for private life is realizing the the ligament benefits of mild weight lifting and clearly i'm not a big guy at all, but I had a couple years ago.

I had all these shoulder issues and You know, I thought, Oh, am I going to have to go get surgery or something? I started lifting weights and just went away. It's not, you know, it's probably 96 percent of what it was. Um, You know, at its peak. So that's, you know, benefits of exercise and and resistance training.

I would definitely. I wish I'd learned that a lot sooner.

Benjamin James Kuper-Smith: So yours, your doctor just went, bro, do you even lift? And then that kind of

Robert Wilson: that was it. Yeah. Well, I went that. Yeah. I didn't even go to the doctor, but I was like, Oh, this is kind of hurting all the [01:46:00] time. And, you know, I had injured it a few years ago. You know, kids and, you know, you do all sorts of stupid things, you know, and I had injured it and it just never really got better.

But then suddenly I started lifting weights and it was instant. It was really, you know, within, you know, Within a week or two, I think just tensing things up. It's obviously not building muscle in a couple of weeks but just tensing everything up got it back into the right position. So that's the sort of private life thing

Benjamin James Kuper-Smith: Yeah. No, you mean I have it with, I've. I'm lucky in that I'm quite tall but never had knee problems or anything like that.

Robert Wilson: uh, 

Benjamin James Kuper-Smith: But, you know, since I've, I mean, I was very I know lots of people who are my height or even shorter who, you know, have had to deal with that. I never had any problem with that until I kind of started doing less sports. And then it's now I have it like when I don't know, like now the last 10 days I've barely exercised and you know, I've been moving and I've moved you know, moved from one flat to another. So there's been movement involved, but not really the proper exercise and my legs have [01:47:00] not, my knees and legs have not felt great all day today.

Robert Wilson: it is amazing how, you know, using the joint, right. You would think it would make it more painful, right. Event, you know, takes the pain away. You know, I mean, obviously, unless it's like a,

Benjamin James Kuper-Smith: are limits

Robert Wilson: injury.

Benjamin James Kuper-Smith: to this.

Robert Wilson: Yeah.

Benjamin James Kuper-Smith: Um, Anyway uh, final question. Um, so yeah, I mean, this is basically any advice for people at that kind of PhD or postdoc border. 

Robert Wilson: Yeah. Yeah. Yeah.

Benjamin James Kuper-Smith: you can take this however you want. It could just be to full PhD students, just for postdocs. I mean,

Robert Wilson: Yeah, I mean, I guess for anyone early career, I mean, I would say, you know, I say this to all my students, like you've got to focus on doing quality work, like embrace the craft of the work and do high quality work that you're going to be proud of. Especially if you stay in science, it doesn't, you know, your papers don't go away.

And you want to be able to look back on them and think, you know, This was real, what I did, you know, like maybe I only answered a really small [01:48:00] question. But, you know, I stand by the work that I did and, you know, sometimes you can be wrong and sometimes something you do might not replicate.

Right. I mean, you know, there's no shame in being wrong. Right. But but you want to know that with the data that you had and the experiment that you had, you did. You did your best work. So I do think that's a big big part of it. 

Benjamin James Kuper-Smith: You mean like also compared to doing like lots of small projects that don't really answer the question properly, or I just kind of done hastily or

Robert Wilson: yeah, I think yeah. Doing things hastily, doing things you know, there's so much pressure to publish, right? Yeah, I mean, I guess the one thing I want to say is if you're a postdoc I see, you know, like I've been there and, you know, I'm, Not young anymore, but I'm young enough to have been in a job market.

It was pretty, pretty tight and pretty horrible. And it is, you know, I, you know, I remember I've been told by senior faculty back in the day, I'll post up the best time in your life, you know, all of this, and I'm like, you know, it was maybe in the nineties when [01:49:00] the career path was, you would. If you know, if you wanted it, you would go straight into a faculty position after a postdoc.

Now it's not. It is a very stressful time if you want to be on that faculty path. And yeah kind of acknowledging that is is a thing. Yeah, sorry, I kind of went off on one there but I do think

Benjamin James Kuper-Smith: no, I mean, it's interesting

Robert Wilson: well. It's yeah, to acknowledge that It is a stressful time of life.

Everybody feels like that. And, you know, less and less now as faculty age out. But the folks who came up, you know, earlier than that, You know, a lot of them understand it now, but it's different to have lived through it, to have been a postdoc, I mean, is this the end of my academic career, right?

You know, and yeah, so I think I'm just, you know, understanding that it's not the end of the world if you leave academia, you know, and a lot of people are happier doing that as well, you know, I think that's and I often wonder, maybe I would have been happy doing that, you know, like it's it's, yeah, so.

Benjamin James Kuper-Smith: been,

Robert Wilson: What could have been? Yeah, [01:50:00] exactly. You know, and it really is just you know, I got a job offer at the right time, you know, I was definitely getting to the point where I was like, well, if I don't get an offer this round, I probably will step, you know, could go a different route. So, yeah.

Benjamin James Kuper-Smith: but then former guest, Lynn Nadel, who then hired you, or 

Robert Wilson: exactly. So, Lin, you know, exactly, you know, I got the job at U of A , and then things kind of came together from there. But yeah, I mean, you know, there's, it's, I think it's, you know, to acknowledge that it can be a stressful time. There's the hooter, the whistle again.

Benjamin James Kuper-Smith: okay. I guess that that's, that's the signal that we have to finish that. That's the, That's the bell for this, for this episode. Uh, So yeah, thank you very much.

Robert Wilson: Well, thank you. Yeah, this is really fun.