72. Nico Schuck: Replay, cognitive maps, and multivariate decoding with fMRI

Nico Schuck is Professor and head of the research group 'Mechanisms of learning and change' at the University of Hamburg, where his research focuses on the neuroscience of learning, memory, and cognitive maps. In this conversation, we discuss his work on cognitive maps and replay in Orbitofrontal Cortex and Hippocampus, decoding even brief events with fMRI, and much more.

BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith.

Support the show: https://geni.us/bjks-patreon

Timestamps
00:00: Nico's work elicits 'limited enthusiasm'
04:36: Multivariate decoding with fMRI
13:23: Start discussing Nico's paper 'Human OFC represents a cognitive map of state space'
19:39: Weird tasks in computational neuroscience
27:30: Start discussing Nico's paper ' Sequential replay of nonspatial task states in the human hippocampus'
36:45: How can the slow fMRI signal pick up on very fast neural dynamics?
43:02: What is Orbitofrontal Cortex and what does it do?
49:24: Some books and papers more people should read
55:17: Something Nico wishes he'd learnt sooner
56:40: Advice for young scientists

Podcast links

• Website: https://geni.us/bjks-pod
• Twitter: https://geni.us/bjks-pod-twt

Nico's links

• Website: https://geni.us/schuck-web
• Google Scholar: https://geni.us/schuck-scholar
• Twitter: https://geni.us/schuck-twt

Ben's links

• Website: https://geni.us/bjks-web
• Google Scholar: https://geni.us/bjks-scholar
• Twitter: https://geni.us/bjks-twt

References
Aly & Turk-Browne (2016). Attention stabilizes representations in the human hippocampus. Cerebral Cortex.
Bishop (2006). Pattern recognition and machine learning. New York: Springer.
Kaplan, Schuck & Doeller (2017). The role of mental maps in decision-making. Trends in Neurosciences.
Knudsen & Wallis (2022). Taking stock of value in the orbitofrontal cortex. Nature Reviews Neuroscience.
Moneta, Garvert, Heekeren & Schuck (2023). Task state representations in vmPFC mediate relevant and irrelevant value signals and their behavioral influence. Nature Communications.
Schuck, Cai, Wilson & Niv (2016). Human orbitofrontal cortex represents a cognitive map of state space. Neuron.
Schuck & Niv (2019). Sequential replay of nonspatial task states in the human hippocampus. Science.
Shepard (1987). Toward a universal law of generalization for psychological science. Science.
Skaggs & McNaughton (1996). Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science.
Sutton & Barto (2018). Reinforcement learning: An introduction. MIT press.
Tang, LeBel, Jain & Huth (2023). Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience.
Todd, Nystrom & Cohen(2013). Confounds in multivariate pattern analysis: theory and rule representation case study. Neuroimage.
Wilson, Takahashi, Schoenbaum & Niv (2014). Orbitofrontal cortex as a cognitive map of task space. Neuron.

Transcript

[This is an automated transcript that contains many errors]

Benjamin James Kuper-Smith: [00:00:00] Actually, I wanted to ask, uh, the first thing I wanted to ask was whether you could provide a bit more context on, uh, a line you have on your Twitter profile, which says, uh, science that causes limited enthusiasm. Enthusiasm. I think an anonymous reviewer wrote that. So I'm curious, what was the context of the limited enthusiasm? 
 
 

Nico Schuck: So this was, um, it was for a paper, the paper that came out in 20, uh, 22 from Leonard Vico in, uh, nature Communications. And I think, which has received quite well, and we have worked hard on this, like, on all of the other papers. But, uh, of course it had, uh, Had a history of being rejected before. And one of the comments, uh, from one of the reviewers I found just quite snye who said he has very little or has very limited enthusiasm. 
 
 

I think for my work in general, something like that, it sounded like this. His person just said, I don't like that person's work. And he said, he didn't say, I don't like the work. He said, I have limited enthusiasm for it. So I'm guessing it was a, a [00:01:00] Brit. Um, but yeah, you never know. I found it funny, so I was like, okay. 
 
 

That's my line up for it. 
 
 

Benjamin James Kuper-Smith: How did Leonard is a PhD student, right? How did he take that as like one of his first papers? I mean, it's because it seems like it's, I dunno, I've, I've gotten very mixed reviews for, for papers so far, and especially like the first time you get something negative, it's, it's not fantastic to to hear and that seems like it was quite a unnecessarily snarky comment. 
 
 

Nico Schuck: Yeah, exactly. I think it took a while. I mean, I usually, I tell all the students beforehand, so it's kinda, it might get rocky. You might sort of read things that feel a bit unpleasant, but it's totally fine. Uh, in the end it will get published somewhere and, uh, it happens through the best papers. And there's tons of stories of papers that are absolutely fantastic and ha, before. 
 
 

Really hard reviews and difficult stories of getting published. So I don't know. He didn't, if he had any kind of, uh, whatever, uh, strong feelings about that, he didn't let me know. He [00:02:00] was, he seemed okay by my calming 
 
 

Benjamin James Kuper-Smith: Okay. 
 
 

Nico Schuck: about all of this as normal. 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah. It's kind of funny, like the long history of very influential papers that were rejected because they were boring or not relevant or had no potential or whatever. It's kind of, and I guess it's just in general, it's difficult to judge. Well predict the future, I guess. 
 
 

Nico Schuck: Yeah, I don't know how true that is, but maybe you have also heard about like Jeff Finch, uh, making the rounds now on podcasts and, uh, in different newspapers in this context of ai, AI safety and so forth. But he's also been asked to a ton of times to tell his own story, and one of the things that he keeps saying is that initially everyone thought he's completely crazy to believe in neural networks. 
 
 

And that was a, he said, everyone fa. Thought it's a stupid idea to, to work on that when he started working on that. So it was interesting reminder, uh, if that is true, that yeah, things can turn out quite differently than they see him in the beginning. 
 
 

Benjamin James Kuper-Smith: So anytime, even if they have legitimate criticism of my papers, I can [00:03:00] think, well, 
 
 

Nico Schuck: Yeah, 
 
 

Benjamin James Kuper-Smith: but you'll see. 
 
 

Nico Schuck: Of course. There's survi. 
 
 

Benjamin James Kuper-Smith: Yeah, yeah, exactly. 
 
 

Nico Schuck: bias, right? I mean, now we, we hear about Jeff Fenton, who's been so successful, but there might be ton of others 
 
 

Benjamin James Kuper-Smith: Yeah. I mean, you don't tell the stories about a paper that people criticized and it was just bad. It's just not a great story. 
 
 

Nico Schuck: I think they don't get invited to podcasts for something like that. 
 
 

Benjamin James Kuper-Smith: Anyway, so you are, you are not particularly enthusiastic research, I guess what we're talking about that now for the next hour or so. Um, I actually had a, a question, like a general question first. Like kind of what's your, actually, your main interest? Is it decision making, learning cognitive maps, uh, rfc, hippocampus? 
 
 

Like what's the, is like one thing or is it kind of just like a bunch of things you're interested in? 
 
 

Nico Schuck: Mm. Yeah, I, uh, I would say all of them if, if that's an acceptable answer. I mean, they're deeply relation. I, I started out being interested in learning. And then understood at some point how deeply related this, uh, to decision making because decisions, of course, there's a particular [00:04:00] process of just reasoning in the moment of the decision and maybe using heuristics and so forth. 
 
 

Uh, that is not really related to, uh, learning or using particular experiences about how that. Choice that you're about to make has worn out in the past, but in many cases, I guess we get to make decisions over and over again so we learn from our mistakes and that's the the kind of thing that I'm interested in, in the hippocampus and upper frontal cortex and a bit of, uh, learning and a bit of decision making. 
 
 

I think all of these play play a role in that process. 
 
 

Benjamin James Kuper-Smith: Okay, so all of the above. Yeah. 
 
 

Nico Schuck: Mm-hmm. 
 
 

Benjamin James Kuper-Smith: Um, yeah, I think the, the. The papers of yours will focus most on today are the 2016 neuron and the 2019 science paper. Um, if I'm not mistaken, I think both of those use a very similar f MRI approach and methods approach. Um, so I thought maybe it might make sense to kind of talk about that a little bit before so we don't get, you know, so we don't have to talk too long about the methods while discussing the content of the paper. 
 
 

Um, so maybe kind of before we start talking [00:05:00] about the stupid papers, kind of. What more decoding maybe as a first, as a first question. Like what does it, how does it work? Um, 
 
 

Nico Schuck: So in all of these papers, uh, I think the backbone is that you basically look at your experiment and you usually have something like different conditions, which we'll get to in a second. And you ask a question that is kind of like, if I have a particular condition, do I see a stable pattern of activation across some boxes in a given ROI or, or socialite or something like that? 
 
 

And if that pattern that is elicited by this condition is stable. That means you can decode it. It means you can take part of the data, you train an algorithm to learn what that pattern is, and then the algorithm is applied to new data that it hasn't seen before. And if the same pattern occurs, it's gonna say, I think that's this particular condition. 
 
 

So you're trying to predict the condition from the patterns observed. And the main difference is that it's, uh, the one difference is that it's multivariate. Meaning you're not just [00:06:00] trying to leverage information of a single box, but actually of many boxes at a time and how they respond to differences in conditions or cognitive processes. 
 
 

And the other difference is actually a little bit more subtle and some people have criticized it as a weakness or like a particular statistical, um, tweak that you need to be careful about, which is. The analysis is individual, so you ask, uh, you train a classifier for every single subject, and the particular association between condition A and the pattern in one brain area can be entirely different in subject A than it is in subject B or subject C, as long as it's stable within that one subject, as long as the same condition always causes the same pattern. 
 
 

And what you average across subjects is only something that presents evidence for stability, stability of a relationship between the pattern and, and the commission. But the actual patterns can go completely in different, uh, in different directions. They can, yeah, like I said, have different directions, or in one subject, it's only a few boxes that [00:07:00] drive the classifier. 
 
 

In another subject, it's all of the boxes that drive to classify a little bit and so forth. And that's very different from the univer approach where you're actually asking a question about, Is it true that in all subjects a particular voxel reacts in the same direction, goes up, for instance, when people, whatever, exert more mental effort when they are in this one condition compared to the other. 
 
 

So you allow more flexibility about inter-individual differences in the precise relationship between the brain and the condition. 
 
 

Benjamin James Kuper-Smith: Um, I mean, you mentioned that it can be very different. I mean, from. The studies you've done, is it very different across participants, the activation pattern, or is that more like kind of theoretical concern? I mean like, I mean, so you mentioned some examples earlier. Is that something that can happen or does it actually happen very much like that? 
 
 

Nico Schuck: I think it depends on the brain area, and I would assume that it happens more in higher order brain areas, uh, like the orbitofrontal cortex than in maybe primary, uh, sensory areas to have a more, uh, homogeneous [00:08:00] organization also across, uh, populations even. So I, I would think that, uh, Yes, that is generally true, but actually now that I think about it, it's probably also different between different signals. 
 
 

So when you look at value signals in the over frontal cortex, for instance, or the V M P F C, there is some understanding about a consistency of the sign of the response, uh, with increasing value. So the, the, uh, imaging signal would usually go up with increasing, uh, value or value difference, and that's true for all of the participants. 
 
 

But if you have something like states, I'm not sure that the represent, the patterns of voxels activated for stage one is the same across participants and in the hippocampus. I would also guess it's tricky, but, um, I think maam Ali has some work on the, like the link between the multi variants effect on a universe effect. 
 
 

And I think she says that it's weak in the hippocampus. So in the hippocampus you really kind of get different answers from the two different methods, uh, more [00:09:00] so than in more cortical areas. Um, but yeah, there's probably a lot of research that I'm not aware in this. 
 
 

Benjamin James Kuper-Smith: Yeah. Um, by the way, can you, um, send me a paper like by my Ali that does that? Um, then I'll put it, 
 
 

Nico Schuck: The. 
 
 

Benjamin James Kuper-Smith: and I always put the, like descript like links and references, uh, in the description of the podcast. I'll put that there, um, in case 
 
 

Nico Schuck: Yeah, absolutely. 
 
 

Benjamin James Kuper-Smith: Um, 
 
 

Nico Schuck: good. 
 
 

Benjamin James Kuper-Smith: so you said that, that the potentially quite large differences between people that that's been a, a potential criticism of the method. 
 
 

Is it a, is it a problem or is it, you know, what's a feature or bug? That kind of thing. I mean, can't that be, I dunno. It sounds like it could also be, you know, very beneficial that you, um, 
 
 

Nico Schuck: I think it's, it is very beneficial because it does make sense, particularly in higher, uh, brain areas, that the kinds of responses that you see are, are learned and shaped through the individual histories of [00:10:00] participants rather than some kind of, uh, Evolutionary shaped structure of the visual pathways and the neurons, how they react to particular, uh, visual quantities and so forth. 
 
 

Uh, so I think it makes sense, but the criticism that some people have, uh, expressed is that it also, um, Increases the danger that you are just picking up on a bunch of confounds. So in one participant, it might be that condition, uh, A is different from B because that participant is finds condition a more effortful. 
 
 

Uh, so the whatever, there's more activity in this direction. And then another participant, they find the stimulus more attractive in condition B or whatever. So there it can be, the claim was there could be a bunch of processes subsumed by this. Predictiveness relationship that you're extracting with decoding methods, um, which is true, you can control for that. 
 
 

And I, I think the, the benefits are also 
 
 

Benjamin James Kuper-Smith: But how can you control that? It sounds like, it sounds like to me, like when you said that, it sounded to me like if you. [00:11:00] That, that's basically that kind of method is someone that you can't control for because you don't know specifically, you know, if someone's like, oh, I really like orange, therefore they, you know, remember that state or whatever. 
 
 

Um, 
 
 

Nico Schuck: well control if you have the variables right, if you have reaction times. For instance, I think that was, there was a series of papers, uh, from TAR and others who basically said reaction times are one particular concern and. You always have to make sure that's true, but there could also be attentional differences and so forth. 
 
 

But yes, if you can't measure it, then you can't control for it. Um, 
 
 

Benjamin James Kuper-Smith: Okay, so what kind of, uh, I hope this is not too broad, but like what kind of study or question maybe more does this method, Leonard, I mean, it seems to me like you need a particular kind of, Data that, you know, data structure that you're interested in? Yeah. 
 
 

Nico Schuck: I, I think you can actually use it for every study, and I think it makes sense to use it for every study. It might, in some cases pick up on the very same thing that a univer analysis would pick up, and in other cases it will pick up [00:12:00] on something different. Uh, that's the danger of just using it maybe without ever looking at u univer data. 
 
 

And as I said, it could actually have sensitivity to different kinds of signals. There might be part of the signals that are pretty similar across participants, and they elicit the same kind of magnitude response. In a particular brain area. And there might be other bits of the, uh, of the response in the brain that are a bit more idiosyncratic, but nevertheless meaningful. 
 
 

And I think there are even people who are basically try and really separate those out and treat them as different quantities. You can subtract the mean activity pattern across conditions and then basically just look at the, try to decode from the residuals, um, if that makes sense. And. It's not necessarily said that one is better than the other. 
 
 

There are probably cases of both where you find, for instance, relationships to behavior with, uh, decoding scores, but not univer activity and the other way around. I wouldn't be surprised if that is true. These are probably just [00:13:00] different aspects of the, the activity that is going on during our experiments as far as we can see. 
 
 

F M I. 
 
 

Benjamin James Kuper-Smith: Okay, so I guess that was a kind of a rough introduction to the topic. Uh, maybe we can be a bit more specific now. Kind of like. Alex, oh, not necessarily like specifically how we use that method, but I guess we can talk more about what you actually did and then it maybe becomes a bit more concrete. Um, so yeah, your 2016 paper Human Orbiter frontal cortex represents a cognitive map of state space. 
 
 

Basically, what were you trying to do? What did you do and what did you find as like mini abstract? 
 
 

Nico Schuck: So maybe I'll, I'll start with a little bit, uh, of a backstory, which started before, uh, I joined Ye Nle in Princeton. I think. Jeff Schau basically approached Yael and said that they had, um, some recordings from dopamine neurons in rodents in an experiment where they had lesioned, the ob frontal cortex, and they couldn't really make sense of them, and it looked like as if the [00:14:00] prediction error signals in the dopamine neurons, they were altered, but quite. 
 
 

In a quite subtle way. They were altered in some conditions, but not in others. But that alteration seemed reliable and not random, but it wasn't clear why is it this condition and not that other condition. And then, yeah, Al sat down with Bob Wilson and in, in a 2014 paper that came out, they basically proposed the solution that what happened in that experiment, in that data is that the animal was confused. 
 
 

During some stages of the experiment, which stage of the experiment it was in, it just wasn't sure, and therefore it had kind of blurry predictions about the reward to be expected, and that was the result that they then saw in the prediction error differences. So why was the animal confused? The argument was that the lesion of the orbital frontal cortex had kind of impaired the animal's ability to distinguish between stages of the task that are difficult to keep apart when you only rely on sensory information. 
 
 

And that's [00:15:00] what's called partially observable task states is a bit of a technical term coming from the r l literature, and the idea is basically sometimes it's quite easy. There's the green light and the red light, and it's very clear, okay, these are different situations. I maybe have different reward expectations when the light is green and when the light is red, or there's different orders when I'm just little rodent in the experiment. 
 
 

But sometimes things get a bit more complicated in the sense, for instance, that it could be that. The rat had waited a long period before receiving the reward in the last trial, and that means that in the current trial, it only had to wait a short period. So knowing that now it's in the stage where it only has to wait a short period before the kind of drop of water comes is something that is not as easily detectable from the environment in the in the moment. 
 
 

You have to use your memory. You have to kind of accumulate over time or have a sense for time, which is also something that is not easy to observe. And the idea that animals lost that ability to distinguish between these more subtle task stages that are [00:16:00] not directly signaled was basically the solution to explaining the differences in the, in the prediction era. 
 
 

So what we did in, in the paper that you asked about is when I then trying to, yeah. AL'S lab was that we asked. Are those representations indeed in the over frontal cortex? Does the over frontal cortex indeed distinguish between different states of the task, in particular when those states of the task are not, uh, signaled with any kind of sensory queue in, in the experiment? 
 
 

And so what we did is that we designed a task in which people had to make decisions, um, about stimuli that were, that were, uh, kind of ambiguous. Uh, and the kind of decision they had to make didn't really depend on what, what exactly is on the screen, but on the history of responses that they had given in the, in the previous trials. 
 
 

So it was a task in which they had to, uh, pay close attention and then basically on every trial, uh, integrate information from the past in order to know, ah, okay, I'm seeing this particular stimulus and I [00:17:00] know that now for me, it's time to pay attention to this particular aspect of the task and give this. 
 
 

Response and maybe more concretely, what happened is that people were looking at, um, pictures of faces and houses. There was uh, there was semi-transparent, there was always a face, uh, on the screen and a house they were overlaid. And we told them initially that, uh, we would give them a cue for the first trial that would tell them, okay, you have to start looking at the faces. 
 
 

And you tell us in every trial whether that face is old or young, that you see. But once you, uh, notice that there's a change in the age of the phases, so you might see the first age that's young, and the the next, uh, phase that's also young, and then the third phase might be old. That's a signal for you that now you have to pay attention to the houses on the next trial. 
 
 

There's no cue in the task. It was just you knowing that beforehand you had paid attention to the faces and there were young faces for a bunch of time, and then they changed, and that was the signal for you to switch your attention to houses. [00:18:00] And then you would continue to see the same stimuli. There weren't any different, but now you would know, okay, have to look at the houses and do a similar old young judgment, uh, in the sense of it is house modern or more, uh, non-modern architecture. 
 
 

So people were doing that task and basically always giving us a response, older, young, but in their brain, they started to make this more subtle decision, uh, distinctions between which tasks state am I in? Do I have to pay attention to the houses? Did I just notice a switch? And maybe on the next trial I have to switch my attention and these kinds of things. 
 
 

And that's what we try to decode from the upper frontal cortex. Then asking the question, Does the operat frontal cortex provide that distinction between the clinicians? Elle and, and Bob had basically hypothesized based on their interpretation of the data that I mentioned before. And, uh, that was the case, uh, pleasantly to our surprise. 
 
 

Uh, it worked out and we, when we looked at the different pieces of information that were necessary, uh, from the [00:19:00] task, they were all there in the upper frontal cortex. And we had started out initially, um, With just looking in the OS C and making sure that this is where we can decor. But even when we started to look across the entire brain, we found that it was, uh, pretty uniquely the operative frontal cortex, a medial part of the over frontal cortex where we could find those distinctions between partially observable task states. 
 
 

Benjamin James Kuper-Smith: And luckily that fits. With, uh, sh bomb, uh, 
 
 

Nico Schuck: Luckily that fit. Otherwise I might have, uh, lost my postop position. Uh, no. Yeah, I was set like that, but, uh, that was a, that was a good start, I guess. 
 
 

Benjamin James Kuper-Smith: Yeah, I mean, um, yeah, I'd like to ask a little bit about the task because it's, it seems like a slightly weird and artificial task, and it seems to me that this is for many tasks that's quite the norm, right? You have a, you have a task that if you, if you are the first time you're doing that kind of task, you're like, what am I supposed to do? 
 
 

This is super weird. Uh, you know, like seeing these superimposed images of houses and faces and then [00:20:00] deciding between them and that kind of stuff. Maybe as a first start, like why in this particular experiment, I mean housing faces has obviously a long history in cogniti of neuroscience, but um, yeah. Why did you take that kind of approach to this question? 
 
 

Nico Schuck: Yeah. Uh, so houses and faces, that was basically a plan B. That was just in case we can't find anything in the higher order. Areas. There's, there's something of a, I mean, they have been a workhorse of cognitive neuroscience for a while because they, you get very reliable responses to those in, in inferior temporal cortex in the, in the para campus, in the future face area. 
 
 

And we just wanted to make sure that we have something to look at in case the auto frontal cortex analysis. It doesn't come out and I didn't end up looking at them. Well, you can clearly see when you just ask the question, can you decode whether it's a face or a house that people are paying attention to, then you end up in these brain areas. 
 
 

But these are not the same brain areas where you can decode other information that's relevant for the task stage, such as whether it was [00:21:00] young or old. That's just somewhere else because it's visual in nature. So it was basically, uh, pragmatics. It wasn't anything speci, uh, specific about, uh, faces in houses. 
 
 

They were overlaid and semi-transparent and they looked that weird just because we wanted to get rid of, uh, any compounds in terms of eye movements and so forth, and wanted to make it really hard to basically just find any kind of shortcut to do the task based on purely visual information. And of course then you can also, with that setup, you can perfectly control the frequency of switches, the frequency of trials in which people pay attention to houses, the frequency of. 
 
 

Old images and young images and the transition probabilities between the images that people are not paying attention to. I'm, I'm that kind of person who like gets really antsy about these kinds of things to make sure there are no confounds. But it's really just due to that kind of, um, Yeah, [00:22:00] knack for detail and worry about confines that the task ended up being so artificial. 
 
 

Um, I should say though, uh, two things. So the one thing is that I think the topic in general is not, uh, in my mind, this is totally not some kind of, uh, artificial phenomenon. It only occurs under rare lab, uh, conditions. But I think it's really fundamental. I think the. Kinds of information we have to generalize all the time. 
 
 

We have to all the time make a decision about when something is rewarding or not rewarding, or a correct response or not correct response. What does that mean for all of the other things that look similar or maybe not similar to what I just got, this respon, the, the reward, uh, for, and so you have to ask yourself, which are the things that are similar and which are the things that are not similar in terms of. 
 
 

Are they giving me the same reward when I respond to them concretely? And that is not easy. Uh, even in the case that we kind of met [00:23:00] with this experiment in the sense that sometimes two things that look completely identical can give you very different responses. Very simple. Um, example would be there is an amazing coffee shop and you can go in there and you can have an amazing cup of coffee and it's wonderful. 
 
 

But if you had five cups of coffee on that day, if you go on that very same coffee shop, then you will get. Pretty bad hardware. Um, so it's not a smart idea and I think that we have to kind of contextualize and de alias, uh, sensory information all the time in order to make our decisions in real life, and the task is meant to reflect that. 
 
 

So that's, I think, one aspect. The other aspect is that I have an interest in making tasks more ecologically, uh, valid. And we have, uh, run one experiment now that we hopefully will publish it. Sometimes we still, in the analysis stage in which people, uh, play more of a, uh, video games, there's many more degrees of freedom. 
 
 

They can walk around freely across different rooms. The [00:24:00] kinds of rules that they learn are more complex than the ones that we used before. And it feels a lot more natural. It feels like an eighties, uh, computer game. And the question is very similar to what I, what I meant bef uh, what I mentioned before in terms of does the orbiter frontal cortex, uh, help people to differentiate between situations that look quite similar, but are actually meaningfully different considering the structure of the task. 
 
 

And we find the same thing even under those circumstances. We have a lot harder time in our analysis because we need to control for all the, the, the visual flow variables and so forth that come of the computer game setup. 
 
 

Benjamin James Kuper-Smith: Yeah, I know. It's, it's kind of a, how should I put it? Um, I mean, I guess it's just a, a very. Difficult topic in general that you want to, you know, have a task that's as precisely controlled as you can be. Whilst it also being natural and to some extent it seems to me, I mean, they are not exactly, um, antithesis or like putting in different directions, but they kind of are [00:25:00] there, let's say, orthogonal to each other at, at the very, at the very least, Yeah, I mean, I guess I'm very aware of like how difficult it is to have a nice controlled design and everything. 
 
 

But yeah, I do often, particularly with this kind of decision making tasks, wander like. You know, because I agree that the, that the, what it represents and what you're measuring with it is fundamental, but then you always feel like, yeah, but they can't there be like a more natural way of testing this if it is so fundamental. 
 
 

And I mean, this is not, you know, specifically about, uh, your task. This is just something I've had a few times. I mean, I know with, uh, with a two step task, I once took part as a participant in that and it took me a while to understand it and apparent lots of participants just don't really get it. Um, or it takes them ages. 
 
 

And yeah, I mean, I guess it's just like a difficulty of the, of the field, but it seems to me like particularly in kind of decision making that comes from like the reinforcement learning side. There's a lot of very weird tasks that take a long time to explain. 
 
 

Nico Schuck: Absolutely. Uh, I mean, uh, maybe, uh, I think one way of you [00:26:00] looking at it is that it's undoubtedly true that you, uh, when you do these artificial tasks, uh, that you get, uh, Very precisely defined or a more precisely defined experimental situation, and a more precisely defined interpretation of your data because you have ruled out a bunch of other factors that could have influenced your data. 
 
 

But the danger is, of course, that the truth of that finding might be very narrow in some cases, even, uh, narrowed down to that very, uh, precise setting that you set up in your experiment. And in the case of this two step task, as soon as you change even something. Minute, like the instructions or what you emphasize in the instructions or maybe the stimuli and so forth. 
 
 

Then because things interact and nomes are complex and of course they're very sensitive and pick up on all of these things, uh, then behavior might change and cognition might change. So you might have sort of found something that is true, but only true in that very sort of limited circumstance. I think, yeah, the interpretability of [00:27:00] the data is cleaner, but the generalizability there is, of course, you need to run the same task with a different setting where you change a bunch of variables in order to understand is this really a general phenomenon or is that something that only happens in this particular situation that are forced my, my, uh, undergrad participants 
 
 

Benjamin James Kuper-Smith: Yeah, but I guess it's also a situation where you kind of, not, not the only ways through, but like you have to start somewhere and figure it out. And as you said, uh, with the houses and faces, there's a. A good, a good other reasons for using that, so why not? Yeah, maybe, um, I mean, we haven't actually talked about the pa uh, the findings that much, but I thought maybe actually we could go kind of straight to the science paper, because I guess to some extent it seems to me that you were like, well, that worked. 
 
 

Let's, let's do it again, but slightly different. Uh, looking at, uh, like 1, 1, 1 step further. Um, uh, yeah. 
 
 

Nico Schuck: Yeah. So there is actually the conversation we had just now is, uh, interesting about that paper because there, I should say, that wasn't the plan with the paper, what came [00:28:00] out of it. Uh, that was more of an like accidental finding if you want. But, uh, maybe first off, so what we did in that paper was pretty much the same that we did in the paper that, uh, described before. 
 
 

So people were doing this weird task with the faces in the houses and old and young, uh, and they had to switch. And there were only, uh, I guess two changes. One was that participants learned how to draw the task inside the scanner. We even instructed them inside the scanner. Um, and then they did it over two different sessions, over two different, uh, Scanning days. 
 
 

And the second one was that we asked participants to rest in the scanner, uh, before and after each task session where their instructions were just that they can, uh, take a rest line of scanner, keep their eyes open, and the usual resting state, uh, scanning procedures. Uh, so no particular instructions, what to do during these periods. 
 
 

And so what we had originally wanted to do is to look at how these state representations [00:29:00] emerge in over frontal cortex. Uh, and there's a bit about that actually ended up in the paper, but the main finding that I made, and it was literally when I was starting to look at the data and have no idea how that idea came about. 
 
 

I must have, I might have just seen a talk about replay or something like that. But what I had basically noticed is that I can also record the tasks that from the hippocampus, just not as well than in the OB frontal cortex. And beforehand, I hadn't really seen that or paid much attention to that because if you do whole brain correction, for instance, then it doesn't really come out. 
 
 

Or maybe it was to close in the frontal cortex kind of took center stage, but there was a signal in the. Over in the, in the hippocampus that I believed in. And what I then did is that I looked at the resting state data and asked, well, that classifier that I trained on the task data that would now give me, if I give it a new set of data that it hasn't seen during training, give me a prediction [00:30:00] about which condition it believes the participant is in currently, if I apply that classifier to the rest data. 
 
 

What does it tell me? And it turned out that the classifier started to make predictions whose sequences kind of looked like as if participants were playing the task in their minds while they were lying in the scanner. And of course it was very subtly, it wasn't something that you could immediately see with your eye, but it was, uh, big enough actually the effect that without having a very kind of sophisticated approach at that time, I happened to spot that. 
 
 

I happened to see that, wow, the transition of this task, state number 13, to whatever other tasks that usually came, that seems quite frequent. That's the one that happens in the task as well, is not a random one. And then I started to look at those transition probabilities, uh, more systematically and. What came out of that is that we found that apparently during the resting periods after the task, but not before the [00:31:00] task participants show signs of sequential re reactivation of those abstract task states that they underwent while uh, they were doing the task. 
 
 

And that was first of all. There was, we couldn't believe it ourselves. Uh, yeah, I was skeptical. So was I, uh, we had a bunch of meetings with many people in Princeton back at the time who all found it quite exciting, but kept asking us to kind of do more tests. Uh, we got more data at some point. Uh, Do more sanity checks and so forth. 
 
 

So there was a period also where I, I faced limited enthusiasm, uh, that we mentioned before, but it survived I guess all of the kind of tests and we convinced ourselves that the signal is real. And what was really exciting about it is that, is were activation in the hippocampus during resting state sessions that were se substantial that few people had seen in F M I beforehand, I think. 
 
 

People had seen re reactivation in the hippocampus at that time, in the [00:32:00] hippocampus, in the sense of if people, uh, have more experiences with houses than faces, you would see more activity of house radi brain areas, uh, during resting state, um, conditions. But you didn't, you hadn't really seen the kind of the, the precise sequential structure. 
 
 

And the other thing that was, uh, exciting is that these were not. Observable things that people had seen, but these were the abstract task states. There was not a replay of, okay, this particular picture was followed by that particular picture was followed by that particular picture, but it was what participants thought about that picture when they saw it, and the picture that came afterwards in the light of these task rules that required them to think about similar looking pictures quite differently depending on whether they're looking at their faces and the houses and so forth. 
 
 

So that was, that was quite cool. And, and I think paved the way, uh, way for a bunch of other work that we're now doing that is kind of continuing to use Arama eye to look at these sequential, uh, signals in, in the [00:33:00] hippocampus in elsewhere in the brain. 
 
 

Benjamin James Kuper-Smith: Uh, maybe as brief context, um, can you probably a little bit more of like what replay is. Um, and I guess there's quite a lot of research there, just not in fm. Right in. Yeah. 
 
 

Nico Schuck: Yeah, so the, the original finding is, uh, that when you record, uh, play cells, um, off a rat while the rat walks run in a, in a mason, you see they are especially selective and so forth, but when you keep recording them, when the animal sleeps, I think the first studies were in sleep, then you find that those cells reactivate se sequentially again, in a manner that looks like as if the, the rat is kind of mentally walking along those paths that it had worked before. 
 
 

So the sequences are not random. And in the original findings, those, uh, activation sequences are actually highly compressed in time. So the, the kind of typical figure that people refer to is that it looks as, as if the animal is walking 20 times as fast in their mind as opposed to what happened in real time. 
 
 

It would only take. [00:34:00] Whatever, a hundred milliseconds, uh, to, to walk a path mentally or see the reactivation associated with a path that took two seconds to walk in real life. Something like that. And I think starting from that original discovery, people have found that there's a pretty large diversity of that phenomenon happens in sleep, but also happens during resting periods. 
 
 

Can be forward, sometimes can be backwards. Uh, there's instances when it's slower and. When it's faster, but it became pretty clear also that it's not just a random phenomenon, but it's related to memory consolidation. I think that's the strongest case to be made, uh, as of yet in the sense that re reactivating those sequences leads and is associated with stronger memory after work. 
 
 

Yes. And, uh, so what we find in, in the 2019 paper is kind of a, first of all, few people had seen something that is much closer to the animal observations in humans. Uh, and it's also broadening the phenomenon away from re reactivation of play cells to a re reactivation of really abstract [00:35:00] representations that happened in a mental space rather than in, in a physical space. 
 
 

Benjamin James Kuper-Smith: I was wondering is that, you know, nons spatial is one of the words in the title Nons spatial tile states. I mean, was that part of the, the, the special thing that you, it wasn't an animal or, I mean, or human or anyone, like walking around an arena and you being able to trace that back, but like a very, uh, as you said, highly complex and abstract abstracted from the sensory, um, Information. 
 
 

Yeah. 
 
 

Nico Schuck: Yeah. I think, I mean, part of the reason why we wanted to emphasize that because there is, uh, maybe perhaps until today, a discussion ongoing, as you might know about. What is the nature of those cognitive maps in the, in the hippocampus and the, the cognitive maps in the hippocampus originally have been observed in the context of spatial navigation experiments. 
 
 

Coming back maybe to a discussion about the, the specific tasks and the culture of tasks that we have been using. And it's just true that in the rodent literature, People use spatial tasks all the time. They have, they are [00:36:00] really well trained to find cells that are spatially selective, record from them, and then do all sorts of tasks from them. 
 
 

So the picture that had emerges, this is a spatial brain area and replay, therefore looked like it's a re reactivation of spatial trajectories, but that I think strong formulation. Uh, started to crumble at the time or was already kind of questioned by many and many people said that the cognitive map is much broader, can refer to nons spatial aspects. 
 
 

And the fact that replay, which is kind of a mental walking in your, uh, kind of mental map happens about these nons spatial abstract task states or involves these nons spatial task states, was kind of further adding to that discussion. I think that's what we were hoping to say. 
 
 

Benjamin James Kuper-Smith: Yeah, I mean, I had one other question about, um, I mean this is, you know, you, you mentioned this quite a bit in the paper also, and I think it's, I mean, cause it's worth mentioning. The question is like, how can you. I think one thing that seems [00:37:00] so, at least to me, so incredible about the paper is that you are using a very supposedly slow method that, you know, averages a large piece of time to get at something that happens very, very quickly and very, not quickly, but briefly also. 
 
 

So I'm just, I don't know, um, exactly where I'm going with this, but can, can you maybe like, um, maybe try to explain if you know, kind of how can you. It just seems surprising to me if you have like a whatever, one or two second window Yeah. Over which you average that, like something that's 50 or 80, whatever milliseconds long would have such an effect. 
 
 

You could even measure it. 
 
 

Nico Schuck: Yeah, yeah, no, that's a, that's a really good question, actually. One that, um, I, we kept asking ourselves at the time of the paper and I continued to ask myself, um, after the paper was out. And we, we've written about that, uh, since then because it does seem surprising, and I think there are two very simple intuitions that help a lot with that. 
 
 

The one is, Yes, the ball signal is slow, but it's kind of [00:38:00] slowness helps you to pick up on events that were really, really short. So even if the entire replay sequence happened within two 300 milliseconds, we have a bunch of seconds in which the signal is gonna be influenced by those replay events. In the, the bold signal is gonna be influenced by that. 
 
 

So we don't need to rush, basically say, okay, we need sequences where we need to sample every 10 milliseconds, the bold data. Um, because whatever consequence a replay event has on the bold data, whatever signature, it kind of, uh, um, kicks off. Is gonna be in the signal for like 10 or 15 seconds. So we have, that's our chance. 
 
 

The other question is, what kind of response does it cause when you have these like really fast activation sequences in neural activation patterns and what we are observing is are, are both signals, uh, not neural signals directly. And I think the worry that people had or the intuition that people had beforehand, including myself, [00:39:00] is that if it's the very same box that would be triggered by those events, Then you have a big problem because the voxel in response to event one starts to rise, and then if the other event comes on and causes the voxel, whatever, to rise even more or maybe decrease a little bit, that is very difficult because the interference processes that happen within the voxel and the slow response, it has to like one event. But if it would be true that in the ideal case, voxel number one, response to event number one starts to rise, voxel number two responds to event number two, at least a little bit more so than voxel one and starts to rise a tiny bit later. Then you have something to can you can pick up on. And those are activation differences between the two Vs. 
 
 

And. Combined with the first fact that I mentioned, maybe those activation difference are gonna stick around in the signal for 10 seconds or so. That is basically, I think, the fortunate combination that we are picking up on in the sense that [00:40:00] there are, there's different events are carried by different boxes and they translated, uh, the temple difference eventually translates itself into a kind of activation difference and that activation difference. 
 
 

Signature of replay, if you want, is gonna be present in the signal for, for a couple of seconds so we even when we have a tr of only two seconds, we get to get to see it. 
 
 

Benjamin James Kuper-Smith: Yeah. 
 
 

Nico Schuck: Does that make sense? I mean, 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah. I mean, do you, I'm curious, like, does that, did that then change like how you design your experiments for now on, because you realize you can pick up on more stuff than you thought you could or? 
 
 

Nico Schuck: Yeah, I think, um, I mean one thing that we're doing in a lot of experiments now is basically to put in pauses, uh, of longer periods where nothing happens. So people are in the task and they do some conditions, and then we just. Stop it and there is whatever fixation cross on the screen for 15 or 20 seconds, and those 15 or 20 seconds that are clean of any ongoing visual input are the window where we [00:41:00] basically can look more precisely at the aftermath of whatever happened in response to the the former. 
 
 

Um, Former stimuli that might involve, for instance, spontaneous forward sequences starting from the stimulus you have seen last. Um, I think that's one particular consequence. Um, we have collected data on the question whether it is worthwhile to sample faster. Uh, As I mentioned before, so in fmi, typically the, uh, tr, which is the sampling frequency, is about two seconds. 
 
 

And I'm not super convinced it will pay off, but we, we are about to look into that in more detail. Um, and it's possible that, uh, this is, uh, this could be an important factor as well. 
 
 

Benjamin James Kuper-Smith: Hmm. Yeah, it's really interesting. Uh, yeah, it's kind of fun how like, yeah. It's just how much more you can do with FM r i than even I was taught to my masters, like in, there's like lots of neuroscience and it seems like even since then, this not just raggedly coming out where you're like, [00:42:00] oh, we could do this also. 
 
 

Nico Schuck: Yeah, absolutely. I mean, I shouldn't, I should say that even though I, I work in the field and I kind of made a career out of trying to do things of, uh, fm I that are hard to do. I wasn't quite sure at some point how much of a future it really has and whether we haven't just reached a ceiling here and we just have to wait for another, fundamentally different technology. 
 
 

But, uh, well recently, I mean, there is, uh, yeah, Alex Hu's very impressive work on semantic decoding, uh, that came out in, uh, in nature earlier this year. Uh, There are a number of other groups who do really impressive stuff of f from my eyes. So I'm, I'm, I think I might have a future now working with the same machine. 
 
 

I don't have to retrain and learn how to work with a new kind of machine that's yet to be invented. 
 
 

Benjamin James Kuper-Smith: That's nice that you have a future. That's good. 
 
 

Nico Schuck: That that will be good. I mean, no one 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah. I mean, this is a hypothesis of this one. 
 
 

Nico Schuck: the illusion that I have a, I have a future. 
 
 

Benjamin James Kuper-Smith: Yeah. You have. Yeah. It's vague optimism. Um, 
 
 

Nico Schuck: [00:43:00] Okay. 
 
 

Benjamin James Kuper-Smith: Yeah, maybe to, uh, to, to slightly broaden kind of the, the conversation and kind of what, you know, taking both of those studies together and like all the stuff we've discussed. 
 
 

And maybe a very, very simple question, uh, what does Orbi or Orbital frontal cortex do? 
 
 

Nico Schuck: Um, so, 
 
 

Benjamin James Kuper-Smith: I'd say one a brief question first, is uh, orbital frontal cortex the same as a vent media or prefrontal cortex, or is there a difference between the two or, cuz I think y yeah, they seem to be sometimes used kind of synonymously. I. 
 
 

Nico Schuck: Yeah. Uh, so no, they're not the same. Uh, but they overlap and it's where the kind of the, the confusion and the terminology comes from. So over frontal cortex is the kind of the most vent bits of your frontal cortex. It just sits above your eyes and the. So the thing to know there is it's the whole kind of ventral surface, meaning goes all the way from the medial parts up to the kind of more lateral parts.[00:44:00]  
 
 

And much of the animal work actually focus on the, the more lateral parts, but it includes a medial part. And the medial part is actually stretches a little bit dorsally and is pretty connected to all of the dorsal, the medial parts that we see. More dorsally that would be whatever variously referred to as. 
 
 

VM P F C, then going into dm, pfc, maybe even ACC and so forth. That's what some people have called, I think the orbi, frontal medial network or so. So there is a continuity. There is a bit of the medial part that is just part of O ffc. Classically, anatomically speaking, there is maybe a bit of the VMP C definition that doesn't exist anatomically, but it's more the functional definition. 
 
 

That you would say, well, anatomically that's just outside of the bo the borders of OB frontal cortex. But if you look at connectivity patterns, it seems to be functionally the same thing. So yes, it's a bit confusing. There are other things that we shouldn't get into because they're very confusing about the homologies [00:45:00] between rodents and, and humans and the medial and the lateral parts and so forth. 
 
 

Very complicated. People can get upset about that. Um, I wanted to, uh, stick with. O C because that's how we started out. But we actually changed to, uh, speak, talk more about V M P F C because we end up finding effects in the, in the medial area. And maybe that is a terminology that's just better understood by the field and speaks to more people, um, than the terminology of. 
 
 

Orbit frontal cortex were, what do you really mean? At least in terms of our findings, is an i medial orbit, frontal cortex. And that's, I should say, at least in humans, because many of the findings from Jeff Schon lab who records in rodents come from lateral ooc. Um, but, uh, whether those are not medial now after a bunch of million years of evolution, it's not, not quite clear. 
 
 

Benjamin James Kuper-Smith: Yeah, but is it, uh, I mean, it seems like there's. I mean, there was one quite [00:46:00] cool review paper that I think it came out last year by Joni Wallace and someone else. I think I've literally said that sentence before in the podcast. I should look up who this other person is. Um, uh, that I think talked about like the hippocampus FC connection and kind of this question like how is it related? 
 
 

In terms of like a, lots of the o c stuff in Rhodes is also with value, um, that comes out with economic value and that kind of stuff. And then, but also the cognitive map stuff. So I'm just, I mean, it seems like the stuff you do is quite related to that. So I'm just curious like kind of what's your, uh, view on this from having done quite a bit of work on this? 
 
 

Nico Schuck: exactly. So, so, um, first of all, yes, the connection to the hippocampus of the o c is actually more strongly so in the medial parts of o c, uh, as far as I know, at least in primates. Uh, so let's kind of get another reason to maybe why the medial parts of the o FFC is where we are finding, uh, things and the medial part of the O FFC or VM pfc. 
 
 

Then how it's called net literature is also where people find values and I think. [00:47:00] In in combination. My interpretation of what the Order Funnel Cortex does is, of course, it's not just one thing, but it seems to be very clearly related in decision making, in value-based decision making in particular. And there is not just one thing that needs to be done, so to speak, computationally if you wanna solve a value-based decision making task. 
 
 

But there are these kind of at least two different categories. One is you need to store and retrieve values of. How, basically your expectations, how good is it to do a choice in response to that? Particular situation or stimulus. And the other one is you need to have a mental map or like a set of states that tell you basically how to sort keep apart or merge these different reward experiences that you have across your sensory space in the sense of you might get reward in response to when you eat some, uh, vanilla ice cream in, uh, ice cream shop a And the question is, does that. 
 
 

[00:48:00] Generalized to all the other flavors in that same ice cream shop, or does it just generalize to other vanilla ice creams in other shops? So you need to, when you learn about rewards, you need to know what things are equivalent in terms of how rewarding they are or how. Uh, how I can apply my current learning experience to those other sort of related or possibly related things. 
 
 

And that's what the cognitive map does. So they are really, really related to each other. It's not just forming expectations, calculating prediction errors, which is what the value-based typical value literature is focused on, but it's also. Precisely to update with those experiences, um, and how to generalize. 
 
 

And I think both of these functions are in the orbi frontal cortex. Uh, and they interact with each other. And, uh, yes, we don't have time to get into that, but there's a very, uh, recent paper, uh, from near Monetta and well me, uh, where we go into that question, uh, and look at values and states at the same time in the over frontal corts. 
 
 

So, [00:49:00] uh, yes, I think it's basically. Yeah, representing the map and the values, and they are very, very closely linked to each other. 
 
 

Benjamin James Kuper-Smith: Yeah, yeah, yeah. So I mean, I'm trying now to have my. To have some questions that repeat from, from one episode to the next. So I'd like to add those kind of recurring questions now, uh, at the end of the episode. Uh, the first is, um, yeah. Do you have any, do you, do you know of a paper or a book or, I dunno, some sort of document of some sort that, uh, I dunno, more people should read or that's something that's been forgotten or, I dunno, just a personal favorite paper of yours or whatever. 
 
 

Something like that. 
 
 

Nico Schuck: Yeah. So there are two books that have really kind of influenced my, uh, whole thinking have been very, I influential, two technical books. One is, uh, one is Sudden and Borrow, uh, reinforcement Learning. The other one is the machine learning book. Uh, by. Uh, Bishop by Chris Bishop. Uh, I think it was only in combination with my knowledge about neuroscience and psychology that [00:50:00] they kind of really helped me to think about some things, but those were like the few books that I have maybe written such a level of detail in those two books. 
 
 

Um, I would say so. I would recommend them. In terms of a paper, I hap doesn't happen very much. Uh, but I happen to have just read one of the kind of old classics, uh, namely this, uh, old paper from Shepherd, uh, about, uh, universal Laws of Generalization. Uh, that's from the, uh, mid eighties or something like that. 
 
 

And I think that's, uh, it's a really interesting, uh, article. Um, I wouldn't say it's beautifully written. I actually found it sometimes. Difficult to follow and a bit obstru, but uh, the ideas behind it are, I think, are really, really striking and very, I think he was ahead of his time and they are super relevant even today. 
 
 

And I think the major thing that he says there is that instead of looking for kind of regularities in behavior that are [00:51:00] universal, Across some stimulus space. So how do people generalize when you go from frequency 100 hertz to frequency 120 hertz of frequency 130 hertz. We should recognize that people might have different internal mental representations about how similar those frequencies are, and when you have figured that out, then. 
 
 

The generalization is actually very irregular. It might be very, uh, kind of lawlike. That's the claim he makes across different participants. But he places the emphasis on kind of the internal map of people and variability therein on the psychological space, how he calls it, uh, and says, when we have understood. 
 
 

What the particular psychological space of a person is within that space, things might behave, uh, start to behave more regularly and even more homo homogeneously across different participants. And I think that general insight is, uh, is absolutely important, uh, and significant [00:52:00] even today as we start looking more at the hippocampus and over frontal cortex as possible neuroscience, understanding metrics. 
 
 

That could reflect those mental spaces, uh, grid cells and so forth. Um, and yeah, I think that's a copa that we recommend reading. 
 
 

Benjamin James Kuper-Smith: Sounds very cool. Yeah, I haven't read it. I mean I've, I've, I've obviously heard the name, but I, but I think I have to read it now. Uh, yeah, as before I put, I'll put, I'll find the references and put them in the, uh, description. Uh, just brief question, uh, I guess you, I don't actually know what your background is, but from the readings you said you liked, it sounds like you didn't study computer science, but you are still very influenced by it now. 
 
 

Cause both of the books are basically computer science, right? 
 
 

Nico Schuck: Yeah. Yeah. I did study a little bit of computer science, so I had a bit of a life crisis in, uh, at the end of my undergrad in psychology. Was really, uh, unsatisfied with how difficult it was to quantify things. And then I. Teach, uh, one year exchange, uh, in Toronto to do machine learning. Uh, and that was a academically, an extremely [00:53:00] intense year because it was, uh, complet completely out of my depth, uh, sitting with all these computer science students and trying to kind of solve, solve all of these assignments. 
 
 

Uh, and it was, it was fortunately enough, Jeff Entran, who was basically his class I, I was visiting and doing the. Doing all the exams, but, uh, it was hard, but I survived. It has really influenced me, but I think only in the combination with kind of the neuroscience psychology bits that I had already brought to the table. 
 
 

Um, 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah, I, yeah, I know the, the sense of feeling out of your depth with computer scientists. I once wanted to take a, Was it unsupervised learning at uc, the Gatsby Center And I sec. I mean, the thing is I just didn't know any of the, not like the mathematical notation of anything. So I was just so lost at as ended up not taking the course. 
 
 

Cause I was like, I have no idea what any of this means. I think I might fail this course. This is probably not, I think there's other cool stuff I could do here. I don't have to do this. Um, but how did you. 
 
 

Nico Schuck: it was difficult. [00:54:00] I guess the what, what helped me, and this is that I had moved to Canada. I had a whole year ahead of me, and I had only signed up for these like two machine learning courses. So I was like, man, I have this whole year ahead of me. And somehow I found it really interesting. And I have to say that was also about Jeff Hinton and his style, because. 
 
 

The way he talks or talk to in those lectures, and I see that when I see interviews with him today is very clear. It sounds very simple and incredibly logical as if it's like the most obvious thing and he just walks through neural networks and whatever he is talking about in a language, we're like, yeah, that makes a lot of sense. 
 
 

And then you see the assignments and it's all math and all, uh, coding. I'm like, what was, was the relationship between his kind of beautiful, uh, very intuitive description that he has given before and, and, and most difficult mathematical derivations. But that motivated me a lot to kind of try and understand that. 
 
 

And then I didn't have, uh, much other [00:55:00] things to do. So I was, uh, yeah, 
 
 

Benjamin James Kuper-Smith: Yeah, I guess the commitment of moving to another continent helps you 
 
 

Nico Schuck: Yeah, exactly. It was also cold in dark in Toronto for the most part. So 
 
 

Benjamin James Kuper-Smith: Yeah. 
 
 

Nico Schuck: good conditions. 
 
 

Benjamin James Kuper-Smith: Um, that sounds like a pretty cool time. Um, the second question is, uh, yeah, I mean, I dunno about you, but I tend to repeat mistakes and I'm curious. Anything you wish you'd learn sooner? This can be whatever you want. Uh, academic, private, I dunno. 
 
 

Nico Schuck: Um, I think. I'm still learning this, but it's to focus my energies. Uh, sometimes involves, uh, saying no and, uh, involves drilling down on a particular subject, uh, that you are really, uh, interested in and just keep at it rather than. To continue to kind of do new [00:56:00] things and participate in that, and participate in this and so forth. 
 
 

Um, I think there's also, I, I like kind of being out and about and kind of think about different kinds of topics, but, uh, I think it's, it's also good to kind of leverage the things that you have. Built on before and yeah, continue that work. So with the science paper 2019, I'm very glad that we continue to do even like methodological investigations, trying to answer some of the questions that he occur to us, like, how's it even possible to do that signal? 
 
 

And I think it's great that I, instead of moving on, I was like, okay, let's wait a moment. Let's try and do a few studies and figure out how does that work. And I'm, I'm really glad I did that because I think I learned a lot from that. 
 
 

Benjamin James Kuper-Smith: Okay, cool. Um, if I'm not mistaken for anyone wondering whether they're starting to hear voices, I think that's your kid in the background, right? 
 
 

Nico Schuck: Oh yeah, 
 
 

Benjamin James Kuper-Smith: Yeah. Yeah. It's not a problem. I just in case anyone, sometimes. As I listen to podcasts and you suddenly think like, go, what's going on? I'm having hallucinations. 
 
 

No, that just happened in real life. You're not hallucinating. Um, and uh, last question. [00:57:00] Um, yeah, any advice for, let's say, uh, someone, uh, yeah, 
 
 

Nico Schuck: should can give some advice. 
 
 

Benjamin James Kuper-Smith: yeah. A advice for, I dunno, uh, scientists like me who are, you know, in some sense starting out their, their career. 
 
 

Nico Schuck: Yeah. Mm. I think, uh, the most important thing for me until today, and I see also a Latin sort of younger people, is to be clear on the big picture before you, like, dive in into an experiment to make sure you can. Summarize in few words why is it interesting and what, what's the question? What is really behind that? 
 
 

Before you dive into the details of the experiment and the analysis, because we spend a lot of time, most of the time I spent with my PhD students in postdocs and in other projects on, on the details, on designing the experiments, on the logistics. And like there are so many things that need clarification, picking the stimuli and so forth.[00:58:00]  
 
 

But it's really important before you start to basically. Know what you want with that experiment, uh, and why you want it. Um, and I think that's, that's the thing I would advise people otherwise, I mean, I still think it's exciting times in, in psychology and neuroscience. I mean, academia is, can be hard. 
 
 

It's difficult. I think many people have realized that and has problems. But the field is just, it's, it's really cause a cool time I think to be in it. Uh, because. Things are moving, uh, increasingly faster. What's happening in AI is obviously, uh, pretty stunning and it might have consequences, uh, for us, how we understand our data, how we understand intelligence in biology and beyond. 
 
 

And I think it's, uh, it's an interesting time to work in that film. 
 
 

Benjamin James Kuper-Smith: Cool. Well on that positive note, thank you very much. 
 
 

Nico Schuck: Cool. Yeah. Thank you for, for having me.