Everything we see is a mash-up of the brain’s last 15s of visual info

February 11, 2022 4:08 AM

The human brain, rather than analysing the world as a series of snapshots, perceives any given moment as the average of what we saw in the past 15 seconds. That means that, in effect, your brain is a like a time machine, living 'in the past' to allow it perceive a stable environment and handle everyday life. The Conversation, via a subscriber only Future Crunch newsletter. Full research article (no paywall).
posted by ellieBOA (42 comments total)

Oooohhh this is so cool.
posted by rebent at 5:37 AM on February 11, 2022 [2 favorites]

Interesting, and perhaps the explanation for why the jar of instant coffee occasionally refuses to emerge from all the other clutter on the kitchen bench until I've stared at the bench for a good ten seconds, or why finding the right set of little ms flabdablet's socks sometimes requires the full power of a Mum's Look.
posted by flabdablet at 5:55 AM on February 11, 2022 [4 favorites]

I'd heard of* (and noticed**) saccadic masking, so knew our brain time traveled our vision, but had no idea it was longer than just the microseconds of a saccade.

*-Thanks, Peter Watts for Blindsight
**-the blinking do not walk sign... if you look at it and away a couple times you'll sometimes notice it stays lit(or unlit) for longer than it should...that's just your dumb brain lying to you!
posted by Grither at 6:10 AM on February 11, 2022 [12 favorites]

This really makes professional sports even more impressive. I mean, keep this in mind while watching, say, world-class badminton players at work, where 15 seconds might be 20 exchanges, or the Isle Of Man TT motorcycle races where in 15 seconds you've driven ... maybe a kilometer and a half.
posted by mhoye at 6:22 AM on February 11, 2022 [3 favorites]

Watch The Changing Room Illusion for serious headfuckery on this theme.
posted by seanmpuckett at 6:24 AM on February 11, 2022 [17 favorites]

Well, our brains are optimized to keep us from getting eaten by tigers, so things that move fast are prioritized for attention. Slow moving shit is much less important. So that's why tigers move slow until they move REALLY FAST.
posted by seanmpuckett at 6:26 AM on February 11, 2022 [13 favorites]

The other nasty trick the brain's visual processing system plays on us is to fill in most of our world with something like a Photoshop cloning brush, giving the impression that all parts of our visual field have contributed much more detail than most ever actually do.

This is why finding the tiny screw that falls off the bench and lands on the carpet as you're trying to fix that iPhone requires getting down on hands and knees and conducting an absolutely methodical square inch by square inch grid-pattern scan. Letting your eyes saccade around at random until you think you've got a completely detailed view of what's down there just doesn't cut the mustard.

See also: sleight of hand and the basketball pass counting exercise.
posted by flabdablet at 6:26 AM on February 11, 2022 [16 favorites]

that's why tigers move slow until they move REALLY FAST

and why drop bears stay out of our usual line of sight altogether until they're ON US from above.
posted by flabdablet at 6:28 AM on February 11, 2022 [12 favorites]

And why it's so hard to parse anything that's utterly unexpected. I once saw a bird caught in a rat trap. It felt as if my brain got stuck. One part kept insisting "rat!" but my eyes were sending me escher-like bird-puzzle images. Like a cubist painting. Until it suddenly clicked into focus and I was able to interpret what I was seeing.
posted by Zumbador at 6:32 AM on February 11, 2022 [15 favorites]

It felt as if my brain got stuck. One part kept insisting "rat!" but my eyes were sending me escher-like bird-puzzle images. Like a cubist painting. Until it suddenly clicked into focus and I was able to interpret what I was seeing.

This puts an interesting twist on biblical descriptions of angels that I'm going to have to think about.
posted by mhoye at 6:41 AM on February 11, 2022 [5 favorites]

This really makes professional sports even more impressive. I mean, keep this in mind while watching, say, world-class badminton players at work, where 15 seconds might be 20 exchanges, or the Isle Of Man TT motorcycle races where in 15 seconds you've driven ... maybe a kilometer and a half.

Well, that is a clue that this isn't the whole story about visual perception. If we never only perceived an average of the last 15 seconds, how does anyone hit a baseball? The last 15 seconds were mostly the pitcher scratching himself.
posted by thelonius at 6:47 AM on February 11, 2022 [5 favorites]

Thelonius, I think what the 15 second period refers to is the 'image stabilization' around the core focus of priority, even if that focus is changing moment to moment.
posted by ananci at 6:50 AM on February 11, 2022 [4 favorites]

basketball pass counting exercise

[golf clap]
posted by Reasonably Everything Happens at 7:01 AM on February 11, 2022 [4 favorites]

**-the blinking do not walk sign... if you look at it and away a couple times you'll sometimes notice it stays lit(or unlit) for longer than it should...that's just your dumb brain lying to you!

If you've got a clock or watch with a tick-tick-tick seconds hand as opposed to a smooooooth one, that works too.
posted by GCU Sweet and Full of Grace at 7:11 AM on February 11, 2022 [2 favorites]

When the creatures moved, I heard the sound of their wings, like the roar of rushing waters, like the voice of the Almighty, like the tumult of an army. When they stood still, they lowered their wings.
mhoye what an amazing passage.
I remember somewhere in the Bible, angels are described as having six wings that are covered in eyes. Book of Revelations, maybe?
posted by Zumbador at 7:18 AM on February 11, 2022

I'm having a hard time understanding this point. If someone throws a baseball at me to catch, I'm really seeing where the ball was 15 seconds ago? But it seems like I'm able to follow the ball and successfully catch it in real time.
posted by star gentle uterus at 7:25 AM on February 11, 2022 [4 favorites]

Blindsight.
posted by snwod at 7:28 AM on February 11, 2022 [1 favorite]

Hmmm. This is very interesting but I think they're overstating the case in an attempt to give a simplified summary of their findings, in a way that ends up confusing things more than it illuminates. star gentle uterus's question is exactly the right one I think. I know there are several actual vision researchers on Metafilter who might weigh in, but I did my early training in a vision lab, and I'll try to give a bit of detailed criticism later this afternoon if I can. I should say I think the research looks good, and the illusion they've discovered is very compelling! I just have some concerns with their interpretation/presentation of the broader implications for how we experience vision (and sensation more generally).
posted by biogeo at 7:59 AM on February 11, 2022 [6 favorites]

I'm having a hard time understanding this point. If someone throws a baseball at me to catch, I'm really seeing where the ball was 15 seconds ago?

No, you're seeing it move in close to real time. Your model of where the ball is will be maybe milliseconds to tens of milliseconds out of date, and your model of where it's going will be pretty much dead-on, which is why people can catch balls.

But your internal model of the world through which the ball is moving is a cobbled-together patchwork, many pieces of which will be several seconds out of date. And changes to stuff you're not specifically paying attention to, even if they occur well inside your visual field, won't be perceived at all unless they persist for more than several seconds. It's dangerous to play baseball in drop bear country.
posted by flabdablet at 8:22 AM on February 11, 2022 [12 favorites]

I do sometimes do animation which means I work in discrete increments as short as one-third of a second. This creation-of-illusion is both aided by this fifteen second lag and sorta puts the lie to it. I guess I'm not surprised. Just another paradox involving the brain
posted by philip-random at 8:25 AM on February 11, 2022

It's dangerous to play baseball in drop bear country.

Words to live by. :D
posted by snwod at 8:25 AM on February 11, 2022

It's dangerous to play baseball in drop bear country.

A few years ago somebody pointed out that one major challenge of natural language processing is that something like 90% of all human sentences uttered, typed or spoken, are being uttered for the first time.

Metafilter reminds me of that a lot.
posted by mhoye at 8:27 AM on February 11, 2022 [14 favorites]

> and why drop bears stay out of our usual line of sight altogether until they're ON US from above.

That's why you carry repellent!
posted by genpfault at 8:30 AM on February 11, 2022 [1 favorite]

Neurons spike, they don't just sit at elevated voltages continuously. In mathematical neuroscience it's sort of a ground assumption that the way those spikes are integrated into a coherent gestalt is literally through integration, i.e. the mathematical process of summing a changing quantity over time. For this to make sense, there must be a time span involved; why not peg it at 15s?

Based on the linked article, I think it's still an open question of exactly what form of time averaging is used. There are lots of different ones; you might weight all past times (back to a certain point) equally, or weight more recent ones more highly, or other more esoteric schemes. I wonder if this experiment could be modified to try to figure out what that averaging function looks like!
posted by dbx at 8:31 AM on February 11, 2022 [1 favorite]

Letting your eyes saccade around at random until you think you've got a completely detailed view of what's down there just doesn't cut the mustard.
My oldest child has always had some strange sort of pattern matching ability that lets him find items out of the background very quickly. He used to pick dozens of four leaf clovers at a time, and I'll never forget when someone lost a tooth in a pile of straw, and he simply looked down and picked it up.
posted by Flight Hardware, do not touch at 8:34 AM on February 11, 2022 [7 favorites]

A few years ago somebody pointed out that one major challenge of natural language processing is that something like 90% of all human sentences uttered, typed or spoken, are being uttered for the first time.

That doesn't seem right. More than 10% definitely have to be the same rote pleasantries that are said millions of times per day.

There have to be more "howzit goin'?"s and "not bad, yourself?"s and the like.

Maybe there's more variety in writing than in conversation, but even in the corpus of texts between friends or partners, you'll see as many "Good morning!" and "I love you" as brand new sentences.

That said, even if 10% of sentences are brand new, that's a big hurdle for AI.
posted by explosion at 8:37 AM on February 11, 2022 [2 favorites]

So "Be Here Now" means "Be Here in a More or Less 15-second Time Frame."

JK. This is about vision, the sensory framework with which humans are particularly obsessed. In general, our sense of the present--unlike our expansive and flawed views of the past and the future--is just more or less the present moment. A little more, really.

In meditative states, especially when the sense of self dissolves, "being here now" is literally possible, though.
posted by kozad at 8:48 AM on February 11, 2022

He used to pick dozens of four leaf clovers at a time, and I'll never forget when someone lost a tooth in a pile of straw, and he simply looked down and picked it up.

I once found someone's contact lens on a mown lawn from 10 feet away. I was 14 at the time and it is a feat that would not be possible for me today. Now I wonder if it was as much a matter of having a fresh brain as it was having acute eyesight.
posted by Horkus at 8:48 AM on February 11, 2022 [1 favorite]

In meditative states, especially when the sense of self dissolves, "being here now" is literally possible, though.

way back when in my early psychedelic adventures, somebody put forth the hypothesis that what hallucinations really were, was reality -- that normal consciousness always involves an incremental pause (maybe one-thirtieth of second) which is the time it takes for the nervous system to process what the eye sees. So even without this fifteen second mash-up, we're always incrementally living in the past. And what psychedelics do is speed up the nervous system. You go from one-thirtieth of a second in the past to maybe one-hundredth. You go from a normal Las Vegas hotel lobby in 1971 to something like ...
posted by philip-random at 8:59 AM on February 11, 2022 [1 favorite]

Huh! that explains a trick I use to find a jar of spices that's right in front of me, but I can't see it. I just 'reset' by closing my eyes and then blink a few times, and lo, it was right there all this time.
posted by dhruva at 9:23 AM on February 11, 2022

When I was a kid, I had an optometrist appointment. One test he did involved a large black cloth rectangle on the wall with a small white spot in the center. He held a long black wand with a small white ball on the end. He told me to stare at the white spot while moved he the white ball around. I was to tell him when the white ball disappeared. So we went through the test. Then he told me to stare at the white spot, then close my eyes, I was to open them without moving my eyes when he said to. Ok. So he said open them and tell me what you see. I see your hand pressed to the cloth. Look at my hand. It was his hand. Ok. Close your eyes. Open them. What do you see? Your hand pressed on the cloth. Look at my hand. His wristwatch was strapped around the back of his hand. I did not see that. The test was a way to find the blind spot. He put his hand there and he told me that the tips of his fingers and his wrist were still visible. I saw that. My brain filled in the rest. The second test, my brain didn’t know about the watch, so I just saw a whole hand. This experience has stayed with me, making we aware that what I see has a component of being just my brain filling in the blanks.
posted by njohnson23 at 9:33 AM on February 11, 2022 [7 favorites]

In meditative states, especially when the sense of self dissolves, "being here now" is literally possible, though.

Also in non-meditative states. It's largely a question of how willing one is to accept the brain's apparently default offer to identify that sense of self as oneself.

Learning to file that offer under "convenient oversimplifications, reasoning for the purposes of" allows one to understand and accept, even in completely ordinary states of consciousness, that being here now is in fact completely unavoidable. From which it follows that any sense of being somewhere or somewhen other than here and now must always be to some degree counterfactual and therefore illusory.

The sense of self doesn't actually need to be dissolved with meditation nor rendered bizarre with psychedelics in order to do this trick, though contemplating experiences with unusual states of consciousness can be a real help in knocking it off its customary though ultimately unjustifiable conceptual pedestal. It's completely feasible to retain a sense of self and recognize it for the convenient oversimplification that it actually is.

Brains are astonishingly capable bits of kit, but if they didn't apply frankly staggering amounts of lossy compression to the process of building our world models, no such model could ever possibly fit.
posted by flabdablet at 10:04 AM on February 11, 2022 [7 favorites]

Veritasium has a good video on this phenomenon: What Exactly Is The Present?
posted by lock robster at 11:01 AM on February 11, 2022 [2 favorites]

A certain drug taught me that vision is quite overwhelming without the usual filters that the brain uses to turn it into a coherent scene. It took 10 seconds or so, but I was eventually able to make sense of it all without the benefit of edge detection and similar functions.

My friend's schizophrenia taught me that the sense of self is another of those filters. I was never able to take enough drugs to get rid of that one entirely, despite some heroic attempts on a few occasions.
posted by wierdo at 11:05 AM on February 11, 2022 [1 favorite]

Metafilter: reminds me of that a lot
posted by Greg_Ace at 11:14 AM on February 11, 2022

I did manage to take enough. And it was surprising to me that heroic doses of psychedelics many times ended up taking me to places I had been before, either by accident or by conscious efforts.

The advantage of the psychedelics is that one can more or less plan when and where the experience will take place. It is kind of annoying to suddenly be here and now in the middle of a chemistry final (true story. maybe with more practice I could be here and now and pass a chemistry final, but at the time I really needed to be 15 minutes in the past when I knew all the formulas and where the play of the dappled light on my desk and the feeling of the way my tongue rests awkwardly in my mouth was not of exactly equal relevance as the test in front of me)

I found the article confusing and interesting, and it resonates with other stuff I've read and experienced.

As a kid I was really good at noticing details and finding needles in haystacks, but terrible at seeing the big green box in the pantry just in front of my eyes that my mother asked me to fetch.

Now I know that I had (have) atypical visual perception, and that not everyone has to learn the same "tricks" I did to be able to do a bunch of normal stuff, but also that not everyone has my superpowers {/}.

I can easily focus on the tiniest of details, notice stuff that most people don't, but I can easily lose track of the "big picture", which if I understood correctly, is what gets averaged over 15 seconds.

On the positive side, I can do things like spot a Campanotus queen ant across 6 lanes of busy traffic while riding my bike at 30 kph.

On the negative side, if I am doing something and people or objects change position around me, I will bump into them and get a jump scare. Not the normal one where you are so focused in something that you lose track of everything around you, a different one where I believe I am keeping track of everything around me, but evidently I am not updating the model.

I've accidentally elbowed too many people and spilled too many cups of coffee.

My trick to deal with rapidly changing environments is to consciously move my eyes around the room which, as flabdablet pointed out, does not cut the mustard. And I think it does not cut the mustard because the eyes will jump to salient stuff in the room (the tigers and the juicy ripe fruit) and maybe update that, but not everything in between.

Step 2 is the trick. Just after the random jump I look a little bit above, to the left, below and to the right of whatever my eyes randomly focused on. And it is in these little spaces that would have been ignored if I let my eyes do whatever they thought they were doing that I notice the interesting and important details. This is also great for finding needles in haystacks. I am pretty good at bug and mushroom hunting using this trick.

Reading this article, maybe this trick I have is causing the model to update at the normal 15 second interval, maybe I am even able to update it faster than that.
posted by Dr. Curare at 12:11 PM on February 11, 2022 [5 favorites]

Speaking of drugs, a hit of salvia divinorum once made whatever I as looking at into a sort of vertical kaleidoscope where one portion of the visual scene was repeated like a vertical strip as the sections just piled up in front of me. Lasted a few seconds, but what the hell was that? The brain and its chemistry is a never ending playground of mystery.
posted by njohnson23 at 12:12 PM on February 11, 2022 [1 favorite]

Well I'm a bit short of time, so I'll just make a few quick notes.

The authors of this work are looking at a very specific visual task: assessing the features of faces. This choice of task has important implications for thinking about the study. For one thing, faces are very complex visual features, and are processed in regions that are many synapses removed from the retina. There is a lot of integration of information that goes into processing a face, compared to, say, a featureless blob, or an edge. Our perceptions of faces are necessarily more "in the past" than could be the case for other, simpler stimuli, albeit only by a few tens of milliseconds. For another thing, our brains possess highly specialized systems for processing faces, and facts or quirks about face processing don't necessarily generalize to other types of percept.

Furthermore, the aspects of faces that they looked at are essentially stable in the real world. The exact details of how they appear can be highly unstable, with changing expression, changing angle, changing lighting, etc., but the features that allow us to classify a face by age or gender do not actually change dynamically in real life. This combination of facts (unstable low-level perceptions, stable underlying features) mean that under normal conditions, a healthy brain should, in fact, "average" information over time to produce the perceived face: you don't want a passing shadow to suddenly cause someone to look like an entirely different person. Other environmental features don't necessarily have this same property: clocks change meaningfully every second, and it would be quite maladaptive to "average" the perception of a clock over fifteen seconds. In fact, the more dynamic aspects of faces (expression) would also be maladaptive to average in this way, and we don't seem to: we're exquisitely sensitive to even very brief, very small muscle twitches around the eye or mouth, for example.

I could go on, but I'm in a hurry and apparently lots of people on Metafilter hate long comments anyway. In short, my take on this is that when you start closely examining vision (or any other aspect of human perception), you realize that although our experience of it is usually unified, it in fact consists of many parallel, interacting, but also somewhat independent subsystems, that don't all obey the same rules. The article uses both the terms "averaging" and "mash-up" to describe our visual perception over time. I'd argue that while "mash-up" is nontechnical and somewhat vague, it's also probably a more accurate description than "averaging": the complexity of the information acquired over time is still there and used in different ways by different subsystems of our visual processing.

TL;DR: your brain doesn't average the perception of a moving baseball over 15 seconds, and I think that's probably pretty obvious to all of us. If someone seems to suggest otherwise, they're probably oversimplifying to make a point, and maybe didn't make the point they meant to all that well.
posted by biogeo at 12:59 PM on February 11, 2022 [18 favorites]

WRT to 90% of sentences being new, you can test this in a search engine. Pick a part of a sentence from a book or article or paper and search for it as an exact match. Once you get to around a seven word segment you will find very few matches. This is useful when you are suspicious that something is plagiarised.
posted by bifurcated at 5:14 AM on February 12, 2022

Well, our brains are optimized to keep us from getting eaten by tigers, so things that move fast are prioritized for attention. Slow moving shit is much less important. So that's why tigers move slow until they move REALLY FAST.

Speaking from experience, this feels like some kind of weird equivalent of those times when small talk suddenly goes wrong. I'm standing here agreeing with you about the weather and neither of us actually care, why is this happening?
posted by I'm always feeling, Blue at 12:13 AM on February 13, 2022 [1 favorite]

Make the biting stop!
posted by I'm always feeling, Blue at 12:14 AM on February 13, 2022

a hit of salvia divinorum once made whatever I as looking at into a sort of vertical kaleidoscope where one portion of the visual scene was repeated like a vertical strip as the sections just piled up in front of me. Lasted a few seconds, but what the hell was that?

There does seem to be something specific going on with salvia and the visual processing of verticality. I got a thing where half my visual field looked pretty much normal but the other half became extremely weird in undescribable ways, with a sharp, distinct, ruler-straight vertical edge between the two halves. No bleed whatsoever across that edge; the sharpness of it was actually the weirdest part.
posted by flabdablet at 4:38 AM on February 13, 2022