Dealing with Photo Storage Pains

Over the holidays, I finally decided to start a thorough clean-up of my Lightroom catalog and images. Many years of shooting resulted in accumulation of almost 400 thousand images and videos in original formats, and I got tired of trying to keep my data backed up and synced between different storage solutions. I glanced at some of the photos in my Lightroom catalogs and realized that between personal and commercial work I have done over the years, most of the images I have been keeping are likely never going to be touched again. So what is the point of keeping them in the first place? It was time for a massive purge…

All photographers start their digital camera journeys with lots of camera clicks, and I was no exception. The first few years were filled with the following:

• “Experimental” images of pretty much anything around the house
• Images of the same subject and scene using different camera settings
• Poorly composed images without clear subject(s) or themes
• Similar / duplicate family photos
• Photos of subjects in and out of focus (after acquiring the first prime lens)
• Many similar images of birds and mammals (after acquiring the first telephoto lens)
• Many close-up images of flowers and insects (after acquiring the first macro lens)

It was also pretty funny to discover the time I got into HDR photography. Anyone remember early versions of Photomatix? I remember that at times I would feed Photomatix JPEG images with different brightness from the same RAW image to produce that popular “HDR look”. Yup, single-image HDR photos were a real thing, even though many of us wouldn’t admit it :)

People talk about the “HDR hole“, but the “panorama hole” turned out to be much worse for me, since I ended up doing panoramas of anything that looked pretty at the time. Below was my sequence, in this order:

• Horizontal panoramas (whoa, I can do this?)
• Unstitchable panoramas (nodal point and parallax discovery)
• Horizontal panoramas in vertical orientation (more resolution baby!)
• Vertical panoramas in horizontal orientation (tall prints anyone?)
• Ridiculously wide panoramas (not sure why, but will do it anyway)
• Multi-row panoramas (first panoramic head)
• HDR panoramas (Photomatix + PTGui craze)
• HDR multi-row panoramas (all of the above, time for computer upgrade)

I had them all – hundreds and hundreds of awful panoramas:

The worst offender was a panoramic scene where I somehow managed to capture over 500 multi-row HDR images with a telephoto lens. The software and my computer at the time could not even process that large of a pano, let alone that many images, so I had to extract smaller size JPEG photos from the original RAWs in order to stitch it all together. At the end, it was a terrible image anyway, but for some reason I kept it all – perhaps at the time I thought that it could be useful someday? I still don’t know.

Anyone remember a “pro” tip back in the early DSLR days, where we would shoot a burst of photos to reduce the effect of “mirror slap” in order to get sharper images? Found plenty of those. What about post-processed images of the same photos in color, as well as in black and white just in case? Check. Images of bricks and walls for lens sharpness? Another check. Images of the same subject zoomed in and out for no reason? Yup. Wide open and stopped all the way down? Plenty of those. Vertical and horizontal shots of the same subject? Absolutely. “Creamy” bokeh highlights? Oh yeah! Exposed to the left, middle and right? You betcha. With and without filters? Certainly. On-camera and off-camera flash? Yes to that one too.

I could go on and on, and it looks like the cycle repeated with more camera gear purchases, newer software and newly-acquired tricks.

All of the above, coupled with plenty of terrible imagery meant that the clean up process for my early work would result in over 95% reduction of my image storage and Lightroom catalogs. That’s a great start!

However, the problem turned out to be much harder than I thought – trying to go through hundreds of thousands of images in Lightroom and deleting them is no easy feat, especially when images are not starred or tagged properly. Lightroom was never good for fast image culling, so I figured it would be much faster to use FastRawViewer to delete images, then go back to Lightroom and re-sync my photo catalogs to remove all the missing files. But how do you go about doing this for 400K images? Sadly, one year at a time, and one folder at a time. While I was happy that I kept my images properly organized in Lightroom catalogs, the sheer volume of images potentially meant weeks of clean-up work.

At the end, I still decided to go through the pain and follow this process, no matter how long it takes:

1. Back up all images to a large external drive before clean-up (about 16 TB of total data).
2. Consolidate all yearly Lightroom catalogs into a single mega-catalog, but keep images in file system organized by year. This makes it easy to see the big picture.
3. Save all Lightroom metadata into sidecar files for FastRawViewer (FRV) to be able to access starred and marked up images.
4. Set up FRV to dump rejected images into a dedicated folder, since FRV does not actually delete any data.
5. Go through each year, one folder at a time with FRV as quickly as possible. Start by rejecting all easily identified bad images (out of focus, bad exposure, duplicates, etc), unwanted HDR and panoramic images, and other photos that will never be used. Leave any images in doubt.
6. Delete the “rejected” folder if it resides within the main “Photos” folder.
7. Re-sync the Lightroom catalog to remove all missing images.
8. Repeat #5-#7, but this time go through images slower to only keep the strongest of the bunch.
9. Optimize the final Lightroom catalog, and rename / reorganize images, if necessary.

My biggest savings are expected from commercial work (headshots, weddings, corporate photography, etc) that Lola and I did for about 10 years from 2009 to 2019. Most of those images will be permanently removed from working image and Lightroom catalogs. Even though I don’t expect any of our past clients to request images in the future, I do want to keep the strongest images as part of the portfolio.

If you want to avoid image clutter and the resulting pains in the future, it is always a good idea to properly cull images after every trip and photo shoot, then purge the rest. I wish I did that from the start.

More to come…