Image inpainting for irregular holes using partial convolutions.
April 27, 2018 1:14 AM Subscribe
Inpainting. Rese rchers fr m NVI A introd ce a deep lea ning me hod that ca edit imag s or recon ruct a corrup ed image that has holes or is mi ing pixels.
That was interesting (and scary as hell) for some of the scenes, but it all went to shit once Borgnine showed up.
posted by Samizdata at 1:32 AM on April 27, 2018 [5 favorites]
posted by Samizdata at 1:32 AM on April 27, 2018 [5 favorites]
What if they train the network on photos of libraries, but then inpaint on photos of faces?
posted by sixohsix at 1:40 AM on April 27, 2018
posted by sixohsix at 1:40 AM on April 27, 2018
> sixohsix:
"What if they train the network on photos of libraries, but then inpaint on photos of faces?"
Then we go down that Google DeepDream rabbit hole. Madness ensues.
posted by Samizdata at 2:42 AM on April 27, 2018 [1 favorite]
"What if they train the network on photos of libraries, but then inpaint on photos of faces?"
Then we go down that Google DeepDream rabbit hole. Madness ensues.
posted by Samizdata at 2:42 AM on April 27, 2018 [1 favorite]
For me the link went to a video of ML self taught running stick figures, which is disturbing in it's own way.
posted by sammyo at 2:46 AM on April 27, 2018
posted by sammyo at 2:46 AM on April 27, 2018
> Inpainting. Reseorchers fram NVINIA introduce a deep leavning mejhod that cam edit imagas or reconrruct a corrupjed image that has holes or is mixxing pixels.Previously, sort of
posted by farlukar at 3:28 AM on April 27, 2018
This has been making waves in photography circles for a few days now; it's powerful stuff. This type of tool has been around for a little while—Photoshop has a "content-aware fill" tool that is very powerful and makes repairing flawed photos very easy indeed, but what NVIDIA has here looks like some next-level stuff. The results these tools generate aren't perfect, but they frequently get you close enough that completing the touch-up is a quick and easy job rather than the laborious and time-consuming process that it used to be.
It really starts to ask the question of what a photograph actually is, though. None of these tools can reconstruct what actually was there at the time of the shot; what they're doing instead is filling in what the AI thinks might have been there, based on its experience of what photos are supposed to look like. The results can be very convincing, but they don't reflect anything that ever actually existed in the world. They're not anchored to reality in any way, and to me that gets at the root of what photography is: on some level, a photograph is an image that was taken from reality. At least, I think so.
This is an old debate, and in some situations it simply doesn't matter—if you're a commercial photographer for instance, all that matters is that the image looks good and serves its purpose. And of course this kind of doctoring has been possible for quite a while now. But the proliferation of these powerful tools makes it absolutely trivial, and that's going to be tempting for a lot of people.
This stuff is going to be on phones soon, if it isn't already. Very soon it won't just be commercial photographs that blur the distinction between reality and fantasy, but the everyday snaps that we all see on social media, posted by people we know. If you think people's social media personas have a tendency to be false and performative now, just you wait. It's about to get crazy.
posted by Anticipation Of A New Lover's Arrival, The at 4:05 AM on April 27, 2018 [6 favorites]
It really starts to ask the question of what a photograph actually is, though. None of these tools can reconstruct what actually was there at the time of the shot; what they're doing instead is filling in what the AI thinks might have been there, based on its experience of what photos are supposed to look like. The results can be very convincing, but they don't reflect anything that ever actually existed in the world. They're not anchored to reality in any way, and to me that gets at the root of what photography is: on some level, a photograph is an image that was taken from reality. At least, I think so.
This is an old debate, and in some situations it simply doesn't matter—if you're a commercial photographer for instance, all that matters is that the image looks good and serves its purpose. And of course this kind of doctoring has been possible for quite a while now. But the proliferation of these powerful tools makes it absolutely trivial, and that's going to be tempting for a lot of people.
This stuff is going to be on phones soon, if it isn't already. Very soon it won't just be commercial photographs that blur the distinction between reality and fantasy, but the everyday snaps that we all see on social media, posted by people we know. If you think people's social media personas have a tendency to be false and performative now, just you wait. It's about to get crazy.
posted by Anticipation Of A New Lover's Arrival, The at 4:05 AM on April 27, 2018 [6 favorites]
Is there some sort of library or image set-up that the tool is referencing/working from? I'm asking because, when it converted the relatively plain, non-makeup eyes of the female to full-on eye makeup, that seemed to me that it had to be referencing a pre-set image of the full makeup eyes. Otherwise, one has to ask "why does the tool assume to add full makeup to the eyes?"
posted by Thorzdad at 5:18 AM on April 27, 2018 [1 favorite]
posted by Thorzdad at 5:18 AM on April 27, 2018 [1 favorite]
> This stuff is going to be on phones soon, if it isn't already.
The shift has been happening gradually for years. Selfies apps have been continually adding features that smooth out the face, change skin tone, provide anime eyes, and put the subject in different locations, in addition to more obvious stickers and decorations. So a machine-learning driven method of content-aware fill (or even live-action CAF) is going change things a little more, but in the user domain I don't think it's going to be received as a dramatic new feature.
Which doesn't mean the ethics of using these things in photographic imagery is equally self-adjusting. I think the old questions of truthfulness and verifiability are as important as they've ever been. But I also think the debate over distinctions between casual and social photography vs. documentary photography are going to never go away either.
posted by ardgedee at 5:21 AM on April 27, 2018 [1 favorite]
The shift has been happening gradually for years. Selfies apps have been continually adding features that smooth out the face, change skin tone, provide anime eyes, and put the subject in different locations, in addition to more obvious stickers and decorations. So a machine-learning driven method of content-aware fill (or even live-action CAF) is going change things a little more, but in the user domain I don't think it's going to be received as a dramatic new feature.
Which doesn't mean the ethics of using these things in photographic imagery is equally self-adjusting. I think the old questions of truthfulness and verifiability are as important as they've ever been. But I also think the debate over distinctions between casual and social photography vs. documentary photography are going to never go away either.
posted by ardgedee at 5:21 AM on April 27, 2018 [1 favorite]
Leave Ersnt Borgnine face alone!
posted by SageLeVoid at 6:42 AM on April 27, 2018 [4 favorites]
posted by SageLeVoid at 6:42 AM on April 27, 2018 [4 favorites]
None of these tools can reconstruct what actually was there at the time of the shot; what they're doing instead is filling in what the AI thinks might have been there, based on its experience of what photos are supposed to look like.
To be fair, anybody who has ever spent any time looking for a dropped iPhone internal screw on a patterned carpet already knows that this is exactly what the human visual processing system does as well.
posted by flabdablet at 6:49 AM on April 27, 2018 [5 favorites]
To be fair, anybody who has ever spent any time looking for a dropped iPhone internal screw on a patterned carpet already knows that this is exactly what the human visual processing system does as well.
posted by flabdablet at 6:49 AM on April 27, 2018 [5 favorites]
Why do I get the feeling this video was cooked up about four weeks ago by some underlings at NVIDIA, who will soon be laughing their way to the unemployment office?
posted by Sys Rq at 7:07 AM on April 27, 2018
posted by Sys Rq at 7:07 AM on April 27, 2018
Alright, so my 'next' video was one from Google teaching an AI to virtually walk... which I hope made it here last year when this came out. However, I'm pretty sure Monty Python can claim some intellectual property infringement as some of this can be found in prior work as The Ministry of Silly Walks.
posted by Nanukthedog at 7:33 AM on April 27, 2018 [2 favorites]
posted by Nanukthedog at 7:33 AM on April 27, 2018 [2 favorites]
The video starts out using what looks just like Photoshop's content-aware fill, which seems like the clone tool on steroids—it's just sampling the region around the erased area and filling in from that.
When it gets to faces, it seems clear that there's some kind of facial recognition happening, so that when they paint out an eye, the software knows "there should be an eye here," etc. Unless I'm mistaken, it's using the same replacement eyes for the female model and for Ernest Borgnine, which suggests to me that the facial recognition is very basic—just proof-of-concept stuff. The idea of using object recognition in general to fill in the gaps in a photo is really interesting.
There's also a lot of metadata in many of our photos now, and more that could be derived. Software can get the location and time from any smartphone photo, and based on that could figure out the weather, sun's position, etc. It could sample other photos from the same location and make guesses about how to fill in this one. (Microsoft has done some work along those lines, IIRC.) Or you could tell it "make this photo look like I shot it at 4:00 PM."
posted by adamrice at 7:38 AM on April 27, 2018
When it gets to faces, it seems clear that there's some kind of facial recognition happening, so that when they paint out an eye, the software knows "there should be an eye here," etc. Unless I'm mistaken, it's using the same replacement eyes for the female model and for Ernest Borgnine, which suggests to me that the facial recognition is very basic—just proof-of-concept stuff. The idea of using object recognition in general to fill in the gaps in a photo is really interesting.
There's also a lot of metadata in many of our photos now, and more that could be derived. Software can get the location and time from any smartphone photo, and based on that could figure out the weather, sun's position, etc. It could sample other photos from the same location and make guesses about how to fill in this one. (Microsoft has done some work along those lines, IIRC.) Or you could tell it "make this photo look like I shot it at 4:00 PM."
posted by adamrice at 7:38 AM on April 27, 2018
why hello consensus reality
y u look at me liek that
i no hurt you
dont cry
posted by lalochezia at 8:02 AM on April 27, 2018 [3 favorites]
y u look at me liek that
i no hurt you
dont cry
posted by lalochezia at 8:02 AM on April 27, 2018 [3 favorites]
Yeah, I wonder about those eyes, like, if you erase one eye does it try to match the other eye? According to what sorts of attributes? Does it give everyone blue eyes or will it change that based on skin colour?
posted by RobotHero at 2:01 PM on April 27, 2018
posted by RobotHero at 2:01 PM on April 27, 2018
This stuff scares the shit out of me.
AI like this is more or less playing adversarially with our ability to discern real from fake. Like, the less we can distinguish that an AI altered an image, the more successful it is deemed. The AI is not just learning patterns in images. It is learning the human perceptual system, as well as how humans are persuaded.
I do not believe people's skepticism will evolve to suit these technologies. Audiovisual evidence could, however, become subject to arbitrary FUD-based denials. Throw in some advances in natural language processing, and we could have AIs that rephrase articles to slip bullshit arguments past us, make us react with a particular emotion, etc.
I don't see how this doesn't become very bad, very fast.
posted by andrewpcone at 2:47 PM on April 27, 2018 [2 favorites]
AI like this is more or less playing adversarially with our ability to discern real from fake. Like, the less we can distinguish that an AI altered an image, the more successful it is deemed. The AI is not just learning patterns in images. It is learning the human perceptual system, as well as how humans are persuaded.
I do not believe people's skepticism will evolve to suit these technologies. Audiovisual evidence could, however, become subject to arbitrary FUD-based denials. Throw in some advances in natural language processing, and we could have AIs that rephrase articles to slip bullshit arguments past us, make us react with a particular emotion, etc.
I don't see how this doesn't become very bad, very fast.
posted by andrewpcone at 2:47 PM on April 27, 2018 [2 favorites]
Why do I get the feeling this video was cooked up about four weeks ago by some underlings at NVIDIA, who will soon be laughing their way to the unemployment office?
I have no idea. It's useful work, not connected to video gaming but requiring what nVidia is good at, and Deep Learning might be the only realm easier to get a job with skills in than even Security.
posted by effugas at 12:49 AM on April 28, 2018
I have no idea. It's useful work, not connected to video gaming but requiring what nVidia is good at, and Deep Learning might be the only realm easier to get a job with skills in than even Security.
posted by effugas at 12:49 AM on April 28, 2018
> effugas:
"Why do I get the feeling this video was cooked up about four weeks ago by some underlings at NVIDIA, who will soon be laughing their way to the unemployment office?
I have no idea. It's useful work, not connected to video gaming but requiring what nVidia is good at, and Deep Learning might be the only realm easier to get a job with skills in than even Security."
Actually NVidia has been pushing their architecture hard for AI processing, and, in fact, has a special GPU made for just that, as well as some custom software and it's CUDA processing platform is well documented and pretty well supported. So, making videos like this help cement NVidia as an AI platform in people's minds, as opposed to "just gaming" devices.
posted by Samizdata at 7:34 AM on April 28, 2018
"Why do I get the feeling this video was cooked up about four weeks ago by some underlings at NVIDIA, who will soon be laughing their way to the unemployment office?
I have no idea. It's useful work, not connected to video gaming but requiring what nVidia is good at, and Deep Learning might be the only realm easier to get a job with skills in than even Security."
Actually NVidia has been pushing their architecture hard for AI processing, and, in fact, has a special GPU made for just that, as well as some custom software and it's CUDA processing platform is well documented and pretty well supported. So, making videos like this help cement NVidia as an AI platform in people's minds, as opposed to "just gaming" devices.
posted by Samizdata at 7:34 AM on April 28, 2018
« Older “I would want to get out, if I was in a fish tank... | From blue to red Newer »
This thread has been archived and is closed to new comments
posted by yaymukund at 1:23 AM on April 27, 2018 [9 favorites]