Over the last few months, there’s been an explosion of AI-powered tools for photographers. While there were already a few AI features in Photoshop, the most recent release introduced Generative Fill, which is on another level. Today, I’ll be taking a look at Generative Fill – from its potential to its limitations.
Table of Contents
What Is Generative Fill?
Generative Fill is a quintessential example of AI (artificial intelligence) imagery. It’s using machine learning to generate entirely new images, pulling from an expansive dataset based around Adobe Stock images and openly licensed work.
There are essentially two different implementations of Generative Fill in Photoshop’s Beta at the moment. One of them (which I expect to be more useful to photographers) basically functions as “content aware fill on steroids.” It’s capable of filling in huge areas, all without falling into the trap of repeating patterns that typify other fill or healing processes. Note how smoothly it removed this car despite the complex background:
That’s a really impressive result, considering the complex patterns, challenging lighting, and areas without any source material. Photoshop essentially dreamed up what the most likely background would be like, and it filled in the missing area really well.
But generative fill can do much more than that – and this is where it starts to be less useful for pure photography and perhaps more useful for commercial/advertising work. Namely, Generative Fill can add entirely new elements to your scene, based on a short text description of keywords that you write. This functionality, commonly referred to as “text-to-image,” generates entirely new content based on your keywords.
You can see below that it can create a white pickup truck in the same area of the image, if I type “white truck” after making my selection. The truck that Photoshop generated definitely has some of the weirdness you’ll see in AI-generated photos, but it’s close enough to a real-looking truck that it would only take a minute or two of manual fixes to make it look pretty seamless. The crazy part is that Photoshop even generated the shadow of the truck correctly, as well as the wide-angle (slightly stretched) perspective of my lens! The truck has some weird-looking parts, but it doesn’t feel like a cardboard cutout stuck onto the image. It feels like part of the scene.
How is Photoshop doing this? In short, it uses context from the other areas of your image to generate its best approximation of the subject. That includes all sorts of complex things: perspective, reflections, lighting, and the appearance of shadows.
Even so, the tool is not without its limitations. Generative Fill is still in beta, and it’s also inherently a bit random. One limitation right now is that the generated area is limited to a maximum of about 1000 pixels long. If your selected area is bigger than that, the generated material is upsampled and stretched to cover the gap, which can make it look blurry and out of place. You can work around this by making multiple, smaller selections, however.
Another limitation with Generative Fill is its reliance on cloud processing. Because the tool runs on Adobe’s servers, you need to have an active internet connection when using it, and there’s a delay in processing speed. It’s still fast compared to some AI image generators, and it even provides 3 variations each time, but this isn’t the near-zero processing Photoshop users might expect from other tools.
While Generative Fill’s text-to-image functionality might be the more eye-catching aspect that captures headlines, for many photographers, the content-aware filling side of things will probably be more useful. Many times before, I’ve faced a daunting amount of work cloning and spot healing unwanted distractions in a difficult photo. When I’ve revisited those pictures with this tool, I was amazed at how it eliminated the distraction with one click and a sloppy selection.
Replacing Content Aware Fill and Heal Brushes
In combination with the new Remove Tool that Photoshop also added (which effectively covers smaller areas), Generative Fill is now high on my list of Photoshop tools when I need to remove large or complex distractions.
It’s honestly so effective that I think photographers will need to reckon with their personal ethics and artistic responsibility when it comes to creating images for contests, client work, and even just personal photos. The ease and degree to which you can alter a scene is really impressive, and a bit spooky.
The ethical aspects will make a good article itself. But to put things into context, let’s look at some more examples of how dramatically this tool can change a photo. Here’s one that amazed me:
What would you do if you wanted to remove all the cars from this photo? Using previous tools in Photoshop, it would be a very time-consuming task. Just look at the wide variety of textures around the cars – you would have to rebuild plants, trees, the road, walls, and so on. While it’s totally doable, the effort would be pretty massive. Yet with Generative Fill, it took 5 selections and just a single run of generative fill for each car, totaling maybe 2 minutes of work.
Here’s the result – pretty incredible at web resolution, and even when zooming into the full-res photo, essentially flawless:
A lot of photographers will find themselves removing telephone poles or streetlights against complex backgrounds, so I wanted to try that next. Here’s an example of a photo that would take a lot of manual editing to remove the light pole, considering the fine patterns in the glass:
Again, this would have been doable with Photoshop’s previous tools, but it would have taken some time. Generative Fill got it very close to “optimal” right away, though, even with a loose selection that I made:
Canvas Extensions
Another thing that works surprisingly well is the ability of Generative fill to expand outside the frame. Maybe you’ve already seen people experiment with this feature by expanding classical paintings or even album covers to give humorous “context” to the original pieces. For photographers, the more useful situation would be to fill in some extra canvas on any side of the photo.
Again, the ethical questions arise that will leave many photographers outright refusing to use this tool. But if you’ve ever dealt with a picky client who suddenly changes their mind about something way too late to fix it, this could be a lifesaver. What if you shot a horizontal photo, but they suddenly switch to requesting a vertical photo? Generative Fill can have your back:
This application of Generative Fill works best, at least so far, for web-resolution images. Zooming in extensively will usually show a few areas that don’t look totally right. It also depends on the image – sometimes, the canvas extensions look great. Other times, Photoshop will hallucinate some unwanted new objects in the generated areas. And that brings me to the next point I want to make: this type of technology isn’t always perfect
Challenging Situations for Generative Fill
If you play around with Photoshop’s beta for Generative Fill, you’ll soon realize that there are some things that it just doesn’t get right. Most of these limitations involve generating new content from scratch, but some of them also apply to spot-healing type applications. The biggest area where it struggles? Text.
Generating text is a major limitation of AI imagery in general. The software knows how to create text-looking gibberish, and sometimes it generates real letters of the alphabet in an unintelligible order – but if you want the AI content to actually say something, you’re almost entirely out of luck.
A related area where Generative Fill struggles is if you need to generate a logo or anything like that. I’d imagine this is a tough issue from both a copyright perspective and a training data perspective. The more obscure you go, the worse time you’ll have. For example, you can get Generative Fill to make a reasonable facsimile of an American flag on a flagpole, but flags of other countries can range from decent replicas to complete misses.
Remember the truck that it generated in my earlier example? That’s another limitation of Generative Fill – it was definitely a “truck-shaped object,” and it looks fine out of the corner of your eye. But close inspection reveals a lot of issues, like differently-sized tires, strange door handle positioning, and no license plate.
That’s often the case when generating objects from scratch in Photoshop. More complex subjects tend to have more issues, but even something simple like a bench doesn’t look quite right upon close inspection:
The software also has trouble combining multiple concepts at once. For example, if I ask Generative Fill to generate “dog sitting on a park bench,” it gives me the following illustration:
Pretty awful!
Granted, you can improve matters by splitting this prompt into two concepts – first generating a bench, then a dog on that bench. It still looks iffy, but at least it’s better:
However, that goes to show that it will still take some time before the “generative” side of things works as well as the “fill” part. As photographers, that’s probably how we’d want it anyway – but for commercial applications, it may still take a while before you can get totally convincing results for any variety of complex prompts.
The Future
It’s still very early days for this type of tool. Adobe specifically has it limited to the beta branch of Photoshop, and has disclaimed it for commercial use. But a lot of questions remain about Generative Fill’s future (and other content-generation tools like it).
One open question is just how copyright is handled when you use this tool in an image. Will you be able to copyright the work if 5% is AI generated? What about 80%? Different locales are also considering regulation and disclosure around AI content – this could affect commercial photographers heavily. I suspect that Adobe will be able to navigate copyright-related issues better than most AI companies, thanks to their large stock library, but it’s hard to know the full implications just yet.
For artistic purposes, there’s inevitably going to be a massive divide in the art community as a whole, and photography specifically. If you can generate an amazing sunset on demand, does that devalue the photographer’s work in returning to a location multiple times to get the perfect light? What if the photo is mostly real, but a tool like Generative Fill was used to extend the canvas on one side by an inch or two?
Then there’s a question of how Adobe is going to price this. Server time isn’t free, particularly for the high-powered GPU compute necessary to run these models, and Adobe hasn’t shied away from moving to pricing models unpopular with the community. It remains to be seen if this tool will be hidden behind a paywall one day or not. The tech still has ways to go before I’d pay for “credits,” but it may get there before long.
I think Pandora’s Box has already been opened, and AI-generative tools like this one will only get better. We can expect it to work at higher resolutions in the future, with better handling of details and more natural image generation. Even in the current form, however, this is a very powerful tool for photographers – especially if you find yourself spending a lot of time spot healing for your type of work. It’s remarkably effective at eliminating distractions that would be too time-consuming to just clone out.
Have you tried out Generative Fill yet? Do you see it playing a significant role in your editing workflow in the future, or are you anti-AI? I’m curious to hear your thoughts in the comments below.