Advanced Layer Mask Creation in Photoshop

One of my readers, Lee, left a comment on the Dark Architecture post asking how I created the layer mask to separate the sky from the building. Good question Lee, and I didn't get into it there because it's a process in itself. I probably spent more time trying to get the mask right on that photo than anything else.

In my post a couple of weeks ago called Digital Photo Editing With Layer Masks in Photoshop, I covered the basics of masking and touched on the advanced stuff. If you haven't read it, now's a good time. Now, I'd like to get into the advanced techniques for creating masks right from the photo that's being processed. I probably won't cover every single method for doing this, but I'll at least explain the one that I frequently use.

Why should you learn how to create advanced layer masks? I must say, I don't use this on every single photo I process, but it's awfully nice to have the ability when the need arises. I use this technique for photos that need different levels of editing on distinct parts of the image. For instance, you might be toying around with a curves adjustment and find a setting that makes the sky look outstanding. Unfortunately, it makes the other subjects in your photo too contrasty or dark. This is a case where separation of the sky from the rest of the photo would be beneficial.

So you've identified the areas you want to separate from each other (1). Before you do this, make sure you've done all your touch-ups so you don't start working with the wrong image. Now convert your image to LAB color (Image >> Mode >> Lab Color). Go into the channels, select the Lightness Channel, and duplicate it to a new document (2) (Right Click >> Duplicate Channel…). The lightness channel is going to contain the best quality information, but you can use one of the RGB/CMYK channels just the same. Now delete the LAB conversion in your history palette of the original image to go back to RGB without doing another conversion.

In your new image, convert from Multichannel to Grayscale (Image >> Mode >> Grayscale). Now you should have one layer and one channel that look exactly the same. From this image, we want to darken the darks and lighten the lights to the point that we have just a black & white stencil of one region in the photo. Cool, now add a levels adjustment layer to the image. You'll want to bring the black point up (3) and the white point down (4) until you have a mask. It's okay to have some off-spots in either the white or black region, because you can go back over them with a pencil (5). This particular image required a black point around 147 and a white point around 229, plus a little penciling-in to darken some of the grays inside the building.

This levels adjustment is the most important step in the process. The goal is to bring the black point up just far enough to get a good solid outline around your subject. Same with the white, bring it down just far enough to whiten around the boundaries. If you take either adjustment too far, you'll end up converting the remaining mid-tones into blacks or whites. This will produce a mask that's too sharp and either too bulky or skimpy for the original image. This matters because the mask will cause your image adjustments to give off halos if it's not created just right.

The mask is ready to go back home, but it's a good idea to keep this document open in case you need to make some adjustments to the mask or the levels adjustment layer. Just duplicate the channel (not layer) back into the original document (Right Click >> Duplicate Channel…). This channel will be available in every layer as an “alpha” channel, or extra channel. You can use it to create layer masks for specific adjustment layers such as levels or curves.

To do this, pick your mask channel and load it as a selection (Select >> Load Selection…), or you can select it's inverse if need be. In the Channels palette, click on the RGB channel to bring back the original image. Now, with your selection, when you apply a new adjustment layer it will automatically be masked off based on your selection. You can also apply the mask to existing adjustment layers by duplicating the channel into the mask channel.

That's it. It's pretty simple, but there are a lot of variations on this you can use to create masks for isolating certain tones or subjects within the image. You can also fine tune the mask by applying things like blur, sharpening, burning, dodging, etc. To see how I applied this mask to the photo, see my post titled Dark Architecture.

Also, a big thank you to Lee for asking the question. If any of you have other questions about how I do something, or just questions in general, don't hesitate to ask. Chances are, the question will result in a whole post dedicated to the subject. And there's also a good chance that you're not the only one who has the question. Ask, ask, ask.