Data loss is a very painful experience that unfortunately many of us go through at some point of our lives. In my workshops, lectures and this website, I spend quite a bit of time advocating the need for a well-established workflow that incorporates solid backup strategies to prevent data loss. And during this process, I came across many different backup routines practiced by other photographers, some of which I found to be downright scary. You have probably heard of horror stories of professional photographers losing their life’s work, or wedding photographers losing images of weddings that they were not able to deliver to their clients yet. It sure happens, and it usually happens at the worst possible time too! It is one thing when you lose your personal data / photos and totally another when you are dealing with a client who paid you money. I cannot imagine how one could even handle a situation with lost wedding photos, as it would be impossible to recreate those precious moments. Sadly, for many of us, it seems like data loss has to take place in order for us to seriously consider a solid backup strategy and workflow. But it does not have to! In this article, I will walk you through two scenarios for establishing a good photography backup workflow: a low-cost and painless workflow for hobbyists, and a much more serious workflow for enthusiasts and professionals. For the second scenario, I will reveal my own backup strategy.
Any time you deal with data, there is always a potential for partial or complete loss and it is not a matter of “if”, it is a matter of “when”. You might think your data sits safe on your computer and your external drive, but what if you lose both? Have you thought of a possibility of burglary, fire, flood or other natural disasters? What do you have in place today to protect your data from such catastrophes? If you have never thought of such questions, it might be a good time to revisit your workflow and make necessary adjustments, before it is too late.
1) Start with a Solid Workflow
Before you think about a backup strategy, it is a good idea to revisit your current workflow process. Where do you import your images? Where do you store your images and how? Is there a single place where your data resides, or is it scattered in different locations? These are all very important questions to ask and if you do not have the answers or you are confused about the process, this is where you need to start. Personally, I like taking a simplistic approach to file management. I always import my images to a single folder and store them there, which makes it easy to locate, backup and potentially restore images. I organize images with a two-tiered approach: both on file system level and on software level (Lightroom), a process that I dubbed “hybrid file management”, which you can read about below. It is a very solid and proven approach and I am happy to say that many photographers have successfully adopted it, using it actively as part of their workflow. Let’s discuss it in detail.
If you have not read my article on photography workflow, now is the time to do it, as you need to understand what workflow stands for and what it is comprised of.
Here are my recommendations on setting up a workflow process:
- Do not reinvent the wheel – use best practices
- Do not oversimplify
- Do not overcomplicate
- Do not dismiss important workflow components
- Take time to establish a good, solid workflow
- Once solid workflow is established, stick to the same process for consistency and efficiency
- If new technology is available or there is a faster/better way to do something, make your workflow flexible to potentially modify it as needed
- Look at all recommended options before committing to a certain workflow process (again refer to point #1)
2) File Management and Organization
A major part of your workflow starts with the transfer of images from your camera to your computer. Whether you copy images, or import them via software, you need to decide where and how you will be storing those images. And this is one of the most critical steps, because it will define your backup strategy going forward. While some photographers prefer importing images first and organizing them later, I strongly oppose such practice, because it can lead to inconsistencies and other problems in the long run. My preference is to organize images at the time of import and if I never have a chance to revisit my files, it is not a big deal, as images were properly named and stored in the first place. This leads us to the importance of file management and organization – a topic that we have written about a number of times in the past due to its importance.
Another article that you should give a read is on organizing photos in Lightroom. Whether you use Lightroom or any other software, I recommend that you check this article out, as I discuss the process of organizing and storing images in detail. In short, I store all images in a single folder called “Photos”, which is located in the root folder of my storage drive, and I group photos in folders by event name. A very simple and yet effective method of storing images that works for myself and thousands of other photographers. But how do you actually do it? Let’s take a look at two different methods – manual file management and file management via software.
2.1) Manual File Management
A manual file management is the most simplistic approach to file management and organization, since everything is done manually and it is something that you are already familiar with. You start by copying images to a designated folder, then you decide on the folder structure and where those files will reside in the file system and finally, you name the files to your liking. You do this on the file system level and the process is really no different than the process of transferring other files from external media. I detailed this process in my “how to organize pictures” article.
Manual file management is simple, because it is easy to understand. Those who do not use Lightroom, Bridge or other software with file management capabilities often end up using this process and sometimes even quite effectively. Since you would know exactly where files are going, it also simplifies the process of backing up images.
However, there are many disadvantages to manual file management. First, it is quite inefficient, since traditional file systems are not designed and optimized for easy file management and indexing. Your computer might have the capability to preview your RAW images, but it surely has no way of reading the proprietary EXIF data / metadata in order to be able to filter through specific criteria, like the type of gear that was used, or the date when images were captured. Your operating system cannot go through this data and index it, to make it easily searchable for you. In addition, there is a big potential for file duplication, as your operating system has no clue about what photos you are copying – it just does what you want it to do. Even if you do efficiently manage files on the file system level, how would you be able to locate a specific photo without going through tens, hundreds or even thousands of images? That’s when file management via imaging software comes into play.
2.2) File Management via Software
Whether you use Lightroom or any other file management software, the advantages of software that is capable of going through images and extracting information are numerous. First, being a specialized tool for photography needs, such software can be very efficient in scanning through and gathering of valuable metadata, then indexing it for searching. Software can reduce and potentially even eliminate duplication, which can save space and improve the efficiency of the backup process. Once indexed, images can be easily searchable in many different ways. For example, you can use Lightroom’s library filter to see what lenses you use the most, which is really neat. You could filter through specific metadata such as lens used, camera used and even find images captured at specific apertures and shutter speeds.
But just like any other process, file management with software certainly does have its own disadvantages. It is more complex to understand. It requires the creation and potential maintenance of a database catalog, which adds more steps to the backup process. If done improperly, it can result in files residing in different partitions / drives, folders, etc, making it difficult to identify file locations. And lastly, it is often impossible to share data and access it from multiple computers, making such process potentially difficult to implement in workgroup environments.
2.3) Hybrid File Management
But what if you were to combine the best of the two worlds to create the most efficient way to organize your photos? I came up with a name for such a process and I called it “hybrid file management”. I personally use this method and after tens of thousands of images stored over a period of 8+ years, with very positive feedback from many other photographers, I would call this the ultimate way to organize images. In short, you get all the advantages of both manual file management and file management via software. If executed right, you end up with very few disadvantages to worry about. The reason why hybrid file management works the best, is because you organize images both on file system level and software level. If you ever lose your software catalog / database, you have enough information on file level to be able to easily rebuild that database from scratch. Since files are already properly named and sorted, you don’t have to worry about doing it again via software. You also do not lock yourself into using one particular software package – you could easily move to other software in the future with this approach.
So how does hybrid file management work? Below are some of the steps, which you can extract from the same article I pointed out above on organizing images in Lightroom:
- First, decide on a master folder to store your photos, such as “Photos” or “Pictures”. Ideally, put it in a root folder of your storage drive, as previously explained.
- Come up with a proper and consistent way to name files, so that you have enough information on the file system level to be able to quickly locate files. Ideally, each of your file names should contain the date (year, month and day) and the name of the event or sub-event, followed by a sequence number.
- Import images using software such as Lightroom and create folder structure during the import process. Images should be renamed at the time of import.
- Do not heavily rely on keywords and other software-based indexing criteria to reduce software dependency. If you need to reorganize or rename images, do that through software to avoid potential mismatches between the database and the file system. Software such as Lightroom performs moving and renaming of images on the file system level and automatically remaps images in the database after the completion of the process. Use the provided tools to mass-rename images when necessary.
- After completing the editing process, force the software to write changes to each image (this will write changes to DNG files or force the creation of sidecar / XMP files), so that you do not lose your changes if your database gets corrupted or lost.
These steps are discussed in detail in the below video, where I demonstrate the file management capabilities of Lightroom:
Once you come up with a solid workflow and you organize your images, it is time to think of a backup strategy.
3) The Simple Approach: Hobbyists
What is the “bare minimum”, low-cost approach for performing backups if you are a hobbyist? Beginner photographers cringe when looking at complex backup strategies of enthusiasts and pros, because the process can be over-the-head complex and very expensive. So if you are just starting out or if photography is your small hobby at the moment, I recommend a different, simpler and more cost effective approach for backups. No, I won’t suggest keeping your images on your memory card as a backup – that is not good enough even for a beginner. You want to make sure that you are protected against two main potential problems: hard drive failure and loss / theft / natural disaster. Always keep two backups at the minimum in two separate physical locations. The second part is very important. Do not just rely on keeping one single backup or multiple backups in a single location. This is something you should not skip on and it should become a good habit in the future. Hard drives fail and having another backup will certainly give you a peace of mind. But it won’t protect you from theft, flood, fire or other natural disasters. So while buying that external drive for backups, make it two in order to have two copies of the same data. Establish an offsite backup routine, depending on how often you shoot and how important your pictures are for you – have a weekly, monthly or quarterly process of updating your offsite backups to keep that data somewhat fresh.
The good news is, hard drives are very cheap nowadays. You can buy a small, portable, large capacity hard drive for less than $100. Just buy two of those and store the second drive somewhere safe, preferably away from where you live. If you do not want to pay for storage services, perhaps you could ask your parents or your relatives to store that drive for you. If it is too much of a hassle, just get a storage box in your local post office or bank and make it a routine to drop off the updated backups.
As for the backup process itself, you can do it manually at zero cost, which would require copying and overwriting files each time, or you could pay $20-50 to buy backup software that will backup your photos automatically. There are many different software and hardware solutions out there to accomplish automated file backups. Apple offers Time Capsue and there are literally hundreds, if not thousands of backup software suites. I personally used to love Acronis software for backups and used it for many years. You can backup and restore not only individual files, but also entire volumes / partitions, if needed.
So here is the quick summary:
- At the minimum, buy two external drives for backups
- Backup to one drive on a continuous basis either using automated software tools, or manual file copy
- Backup to the second drive as often as you will be taking it offsite. You can rotate the two drives, but you have to make sure that you are not just doing incremental / differential backups – you have to have full backups for rotation
If you follow this process, you will have your images stored in three places: your computer, an external drive connected to your computer and another external drive that is sitting in an offsite location.
What about Cloud Storage?
With cloud storage getting incredibly cheap and potentially even free, you might wonder if it is worth keeping your backups on the cloud. Companies like Dropbox, Copy, Amazon, Google and many others have been offering free limited storage or practically unlimited storage if you subscribe to their services. For example, if you are an Amazon Prime member, you will get unlimited storage for your photos without extra cost. And you can upload not only JPEG images, but also RAW files as well (provided that RAW files are from Nikon, Canon or Sony cameras, as detailed on this page). Sounds great, doesn’t it? Why wouldn’t you skip on the second external drive, since the “Cloud” can be considered an offsite backup? Well, it all depends on how much you shoot and what your upload bandwidth is like. Unless you live in an area where providers can supply you with a boatload of upload bandwidth, trying to upload your entire collection of photos might take a very long time. I would start by estimating the total storage that your photos are currently consuming. If it is reasonable and your bandwidth can keep up without angering your ISP, then go right ahead. It will save you the hassle of going offsite for backups.
In case your computer hard drive dies, you have a backup of your files on an external drive. If both fail, you still either have the offsite backup, or a backup in the cloud depending on which route you take. With this method, you won’t risk losing everything you have, as the likelihood of all three failing or disappearing on you is extremely small.
4) Solid Backup Strategy: Serious Amateurs and Pros
If you are a serious amateur or a pro with a large collection of photos, creating a solid backup strategy is your responsibility, so you should take it very seriously. If you are a pro, it is a part of your job, it is what you are paid to do, even if you think you are not paid enough. I cannot imagine what this photographer went through, trying to explain to 20 different customers that her photos were stolen from her car:
Leaving 20 photo shoots in a single location like a car is not something one should do. This is a serious problem in the workflow process and if the photographer followed best practices, she would have had copies of every shoot on her home computer or a single external drive at the minimum. People can be nice and understanding, but some might start a lawsuit and bring your business and you down in no time! I can’t blame clients suing photographers for such poor practices. You cannot repeat a wedding. You cannot bring back a deceased person. Those memories would be lost forever all thanks to a poor workflow and no backups. Don’t make the same mistake – re-evaluate your current workflow processes and make sure that you back up after every single photo shoot.
Obviously, for a serious amateur, semi-pro or a pro, a simple manual backup process with two external drives, or one external drive + cloud storage would not work for a number of reasons:
- A single, cheap external drive is not only unreliable, but also too slow to handle thousands of images from every shoot
- External drives are often limited in capacity, which limits how much data can be backed up to them
- One should seriously evaluate cloud storage reliability, bandwidth capacity and simulate failure to see how much time a restore would take to complete. For critical shoots and time-sensitive jobs / clients, cloud storage might not be an option
A more serious approach should be taken with establishing a good backup strategy, which involves fast, reliable storage and images stored in multiple locations. You want to follow the 3-2-1 backup rule, which states that three copies of data should be kept (1 primary and 2 backups), with files kept on two different media types and one backup being offsite. So far that’s the only option that I talked about for a reason – it is the best backup strategy and it really works.
Now if you don’t have a lot of content and you have a lot of time on your hands, you could adopt the above-mentioned workflow for hobbyists and still be fine. However, if you cannot afford to be down and your business heavily depends on your ability to quickly process images, you should look into more serious storage solutions.
First, start off with your editing machine. Does it have enough redundancy built-in to keep you going if a primary drive fails? While a lot of photographers prefer editing from a laptop with a single hard drive, I would explore the possibility of adding a secondary drive in a mirror / RAID 1 configuration. This way, no matter where you are, you won’t have to worry about dealing with repairs and restoring your work – the data is duplicated on two drives. If one of the drives fails, you can continue working and you can replace the failed drive at your convenience. However, this does not mean that you should think of the second drive as a backup, it is NOT (see more on this below)!
If you have a dedicated workstation, ideally, you should have a separate volume dedicated for photos, which is hosted on two drives in a mirror configuration (protected with RAID 1). I go through such configuration in my ultimate PC for photography guide. In such configuration, if one of the drives fail, you will have the exact copy of your photos on the other drive. So if you experience a failure, you just replace the failed drive and the rebuild process will be very fast. In comparison, a failed RAID 5 volume might take several days to complete and during this process, everything will slow down to a crawl.
RAID is not a form of Backup!
Whether you run RAID 1 (mirroring), RAID 5 (striping) or any other type of redundant RAID, never make an assumption that your data is backed up and you do not need to back it up elsewhere. RAID does not replace backup. When RAID volumes need to be rebuilt, especially if it is a RAID 5 volume, there is always a chance that another drive might fail during the rebuild process. In fact, most people buy drives for a RAID volume in the same time-frame, which increases the chance of more than one drive dying at the same time. RAID is great to keep your storage running and keep you working, but you should always have at least two more backups, as explained above.
What about external storage? Remember, for a 3-2-1 backup, you will need at least two more backups on different media, with one being an off-site backup. Which means that aside from your working machine, you will need a primary backup device and a secondary backup device that will be taken off-site. I recommend that you size your primary backup device with a good potential for future growth. I personally have over 6 TB of images alone and if I add all the video footage, that’s another 2-3 TB of data. So for me, a small storage box with a couple of drives simply won’t cut it. After going through a number of different storage solutions, I ended up choosing the Synology DS1815+ for my current needs. It is a very powerful device that hosts a total of 8 drives (5 TB each) and gives me roughly 35 TB of total storage. I put everything I have on the DS1815+ and it still has plenty of storage left for me to use.
The way I would size your storage needs is by looking at each year of images. How fast is your data growing each year? If you have a lot of data and you don’t want to invest in a big storage box, one solution would be to start archiving your work. John Bosley uses this method and it works for him – he archives his old images to external drives and mirrors the data. He keeps two mirrored drives in a closet and another drive in an off-site location. He does this for each year. To mount and dismount drives when he needs access to them, he has a little accessory device, which is connected via USB 3.0 to his machine. I personally prefer all of my data to reside in one place, because I find myself accessing older images often and I do not want to store so many hard drives. There is no right or wrong – you can go with either way, as long as you are not forgetting about 3-2-1.
As for off-site backup, there are plenty of different options, including options for cloud storage. I have been backing up to an external drive that hosts two 6 TB drives in RAID 0 configuration to give me a total of 12 TB of storage. Data is backed up automatically and at the end of each month I visit a local USPS office to swap out the external drives. It is a hassle, but it works. The main reason for going with a PO Box was my terrible ISP, which capped me at 768 Kbps upload speed. I recently moved to a different location and I have a much bigger upload speed, so I am actively looking into CrashPlan or other services that offer sending the initial backup through a hard drive, as I am not looking forward to uploading 5+ TB of data.
To recap, here is my recommended backup process for serious amateurs, semi-pros and pros:
- Dedicate two separate hard drives for images on your editing machine. Put them in RAID 1 configuration for redundancy.
- Get an external storage array with 4+ drives to make it your primary backup volume.
- Either use another external storage for off-site backups, or subscribe to a solid cloud solution tailored for photographers.
- Use automated software for backups to replicate data in order to improve speed and efficiency of backups.
5) My Backup Workflow
If you are interested in how I back up my data, here is the entire process:
- Primary Editing Machine: Two 1 TB SSD drives in RAID 1 for performance dedicated for image editing. Only images that need to be edited are copied / imported to this drive due to the small size of this SSD volume, which is typically images from the current year. Data is backed up automatically via Synology Cloud Station client.
- Primary Backup Storage: Synology DS1815+ with 8x 5TB 7200 RPM drives in RAID 5 configuration. The volume is mapped on my primary PC as a drive and all current photos are stored there. Older images are accessed directly through the network (Images in Lightroom are mapped to the network volume), as shown below:
- Secondary Backup Storage: DroboPro with 8x 2TB 7200 RPM drives (my older backup storage) in BeyondRAID configuration. Data is manually replicated from the Synology DS1815+ every month.
- Tertiary Backup Storage: ioSafe 214 (fireproof and waterproof) with two 5 TB NAS drives in RAID 0 configuration. Data is replicated with DS1815+ via Synology Cloud Station client.
- WD My Book Duo 12 TB: two drives in rotation for off-site backup (drop off in a PO Box). Data is replicated with DS1815+ via Synology Cloud Station client. Soon to change to cloud backup for convenience.
6) Summary
The thought of potentially losing data is very scary, especially when your work depends on it. If you have been slacking on establishing a solid workflow process with a working and proven backup strategy, don’t wait until you lose data – act now and save yourself a lot of time and money in the future. It might require a bit of effort and some investment, but it is totally worth it!
Please keep in mind that the above-described processes work for me and many others, but they might not be ideal in every case. Take time to evaluate your backup needs and come up with a solution that you can actually simulate for a potential disaster. You do not want to end up with something that does not work, or you are too lazy to properly execute. Make it a strict and a routine practice to not only back up your photos, but also check on those backups, making sure that they are taking place successfully.
Hope you found the information in this article useful. I would love to find out what your current photography backup solution is – please leave feedback in the comments section below! If you need advice, please feel free to ask and I will do my best to answer your questions.
I import my photos to a temp file in LR. The file is subdivided into various files. I import by using a date. Once I have looked at the photos and delete the ones I don’t want I transfer them to a permanent file. When I have the ones processed I want I back them up to a separate hard drive.
I have an offsite back up at Backblaze. The initial backup took a long while but now it is done automatically. It is quite simple and inexpensive.
Can you use a USPS PO box to store one or 2 portable drives?
The loss of images, be they our present ‘data’ or the previous generations’ negatives and slides, or whatever is coming in the future, is a horrible thing to happen.
In the case of hobbyists like myself, precious family images of children growing up, parents no longer with us, and family celebrations, along with our other hobby interests, such as landscape, travel and macro, among other things, can be lost in an instant, through theft, fire, a mistaken deletion, a mechanical failure, it is all too easy.
My hardest adjustment to digital involved this very topic, alongside getting to grips with how digital cameras work best.
It took me a while but, I ended up with a method. Each ‘event’ gets a name, such as ‘Helens wedding 2008’ and the Nef files are copied onto the primary editing computer, into a so named folder, under a main ‘photograph’ folder.
The original images are left on the SDHC card and another card put into the camera there and then.
Edits of the originals are saved in this way, first, the originals will be c:/photographs/Helens wedding 2008/Nefs. the edits will be in c:/photographs/Helens wedding 2008/PSD and / or c:/photographs/Helens wedding 2008/DXO, and the finished images in c:/photographs/Helens wedding 2008/TIF and c:/photographs/Helens wedding 2008/Jpeg.
In this way, originals, edits, transitional versions, and everything associated with the event are all contained in one folder called c:/photographs/Helens wedding 2008.
Depending on the importance of the event – a wedding 10, a casual hours shooting 2, a back up of the entire folder is taken by copying to a primary external backup drive, either immediatelly (10) or within a few days (2).
I have a second external backup drive, which I employ approx once a month, a little more in summer.
One drive is stored on the ground floor in the dining room, the other, in my bedroom, and the originals, on the computer in my study. At the time the images are copied to the second external hard drive, the original SDHC is deemed free to reformat and use once again.
It took me a while to get to this stage, but for my needs, it has worked fine for over three years, and only once have I needed to restore a backup, when I accidentaly deleted a folder, so it was nice to know it was saved off the computer somewhere.
Recently, I have had to upgrade my computer so I have been in the process of migrating everything onto that, and resuming the backup from the new machine, which took some time, but now I am up and running, as they say.
The needs of the Pro are a little more demanding, but this system, followed more rigidly than a hobbyist needs to, should serve most photographers very well.
Interesting topic though, and had me perplexed for a while, a few years back, until I thought it through and got the above system running.
Thanks Nasim :-)
Unless you have a house fire, anyway. There’s a reason one backup needs to be offsite. :)
If you already have Amazon Prime, it’s pretty easy to back things up to that. Just very, very slow.
Thanks for the inputs Nasim! Always good to know what the pros are doing!
I am a hobbyist but take backups very seriously. Not just photos, but also my lifetime’s collection of non-itunes digital music and important documents.
I have a 4TB seagate USB 3 drive ($100) plugged into my Netgear router at home, turning it into a poor man’s NAS and backup software (Syncback Free) backs up my photos and music to that on a schedule. I also have a Backblaze could subscription ($5 per month or $50 per year) that backs up the hard drive with photos, music and docs to the cloud.
Apart from restoring after a crash, Backblaze also gives me online access to all my files, which is neat if I have to get to something while I am travelling. I can restore from Backblaze by downloading a zip (not ideal) or getting the whole dump in a USB drive ($$$).
I have a had a few crashes over the last decade and this setup has helped me to not loose anything.
Costs me $50 a year for peace of mind!
What happens to someone’s archive of RAW files when the photographer dies? all these images may be lost. the next generation may lack the skills or the know how to sift through tens or hundreds of thousand images.
Now we can find an old photo album of some deceased family member. But what is going to happen to the digital generation? WIll only the shared images on social media be preserved?
Great question! I would suggest writing explicit instructions for your wishes about the disposition of your digital files in your will. It would be crucial that this be written in absolute layperson’s language and assume that they might have no specific photographic or file-organization knowledge.
Instructions for viewing the stored raw files so they could evaluate what might be important would be a good first step.
I need to do this for my 130,000-image library! It would obviously be too daunting for non-expert photographers to even think about dealing with it.
I have never lost a photo, and I have never deleted a photo from a card. Thus, I have never accidentally deleted any photos from cards as I never do it.
For each shoot I do the following (for every card used) using folders named the same (with dates) on all drives:
1 – I copy my photos from the card to my primary drive (one of 4 4TB local drives).
2 – I copy my photos from the card to an external network drive.
3 – I upload my photos from the card to my website (no public access).
4 – I move my photos from my card to an external USB drive that is kept in my fire rated safe.
Because my last step is a MOVE instead of a COPY, it copies each file one at a time, verifies it is copied, and then deletes it from the card. This is only done after I have 3 copies already. The card is then empty, and can be formatted without risk of deleting a photo. Since I only format blank cards, I can never lose a photo doing it. There is never a risk when you format an empty card.
I have heard many horror stories of people deleting photos on cards, or formatting cards with photos on them that they thought they had copied. This workflow will prevent this ever happening. If I am traveling, each day I copy the day’s photos to an external portable device, and I upload the same photos to my website, but keep them on the card until I return home.
This method is inexpensive, and protects against fire, theft, disk crashes, and accidental erasure. Although my best deterrent for theft is my 2 German Shepherds. I’ve never had a break in.
USB drives are very cheap these days, so you can easily have multiple copies on multiple drives stored in multiple locations. In addition to the multiple copies I also do Ghost image backups of my primary drives on a regular basis onto a separate “Ghost only” drive that is also in my safe with the other drives.
I never use utilities to import my photos. Manually I can control exactly what happens, and I don’t have to worry about wrong settings or a utility deleting photos from anywhere. Again I have seen a lot of people lose photos because they didn’t understand what the “import tool” was doing.
If you have many drives in storage (like I do) make sure you take them out about every 6 months and run a disk surface checking utility like Norton against them to check for faulty sectors. You will notice when the disk is starting to age and replace it with a newer one.
A good workflow technique will ensure you never lose a photo.
Very interesting article – thanks. But can I just point out a typo? AFAIK each photographer has only one life – so, “losing their lives’ work” should be “losing their life’s work”. HTH. Be my guest if you want to delete this comment ;-)
Martin, thank you for letting me know – I fixed the typo!
Well Done Article
I would hope the group takes notice, I have lost a trip to South America once and never herd the end of it from the wife
I’m certainly agree with you, it’s not IF it’s WHEN
I was the Director of Engineering at a very large design house and Lose of data was NOT an option, Engineering data is way costlier than losing wedding pictures (but that would be sad for the newly married) I carried this back up philosophy over into my Photography world.
On trips I back up CF once or twice a day to Laptop (HP Spectre x360 – 13-4102dx great i7 Laptop/Tablet) with a 500GB SSD, And 2 more copies 1 each on two different 2TB Western Digital HD’s. Then I will reformat. IF I’m really concerned (IF My Wife really likes the shots and gets nervous) Ill set the cameras to capture the same pictures on BOTH Camera Disks at the same time, 1DX has (2- CF) and Mark5D III has (1-CF AND 1 SD card).
At Home
I have a ASUS x99-deluxe U3.1 Motherboard with 12 SATA Hard Drive ports. I use a Cooler Master case HAF922 that allows for front panel access to install/remove 2 Hard Drives without Opening the case. Easy as installing a cd into your player. So I have 4- 6TB Hard drive running in raid (2 at Raid1) and 4- 4TB single back up.
I use the front Hard drive access to back up off site drives, 1 complete back up is in the safety deposit box. And another is hidden away from the PC
You did forget to mention the Main reason NOT to have all your data backed up on the same platform. LIGHTING. I live in Florida and A Lightning hit could take out ALL my data at once, regardless of how many copies I have on the system. Yes, you should also have a UPS of the system, if anything it will help keep those BIG Raid Hard Drives in sync during power glitches. Takes a Long time to rebuild 6TB drives. I have been using CyberPower UPS’s and they have been working ok.
Thanks for the Great article and reminder to Back up often
Your article was great. Folks who are not actively working to keep their data safe should pay close attention.
My wife and I both need access to the images, so we use a server to house the files. I’m running a Linux server that has two 2TB drives using an LVM striping format. From there, I have an RSYNC script that runs nightly and makes backups to a couple of 4TB drives striped together (8TB total) and mounted in the front of the server. Each week, I rotate those backup drives out and store them off site, swapping for two more 4TB drives. In this case, off site means that I store them in my car. But the point is that they’re not where the main data is.
Since the data drives are physically present in the server and therefore susceptible to lightning strikes and so on, I of course have the server running on a UPS. Also, the drives are not mounted in the Linux system until just before the backup is run, then unmounted when the backup is complete.
To backup my Lightroom catalogs, my script mounts the Lightroom catalogs (a folder on each of our desktops) to a directory on the server just prior to making the nightly backups so they’re included in the backup.
The cool thing about using RSYNC is that it can (and I do) place files that have either changed or been deleted into a separate directory for archiving. I keep one directory that is a mirror of the actual data, then a separate directory for each day of the week containing anything that was changed or deleted. By keeping the one week’s archive, I can recover anything I deleted or screwed up for at least a week afterwords. Actually, for up to two weeks since one backup is “off site” every other week. And for what it’s worth, I have had to go back and recover files that were mistakenly deleted.
I also make periodic disk images of the drives containing the OS for each of my desktops and the server; the thought being that if a hard drive crashes containing the OS, I can minimize the down time to bring things back.
I started this process without knowing anything at all about running a Linux server, but determined to have a reliable method to backup my stuff. Over the years, it’s proven very useful, and I feel reasonably confident that my data is as safe as I can make it. Of course, having said that, I’m always tweaking the system to make it work better, faster, safer, and so on.
Any tips for the wayward noob, that through trial and error, has created triplicates of photos, in various drives across my PC.
…and now ive lost track of what was the backup, and what was the original.
Can anyone recommend a software that can scrape selected drives and advise on these duplicates? i have googled this, but would like to know from this community.
Here’s what I would do:
First, it sounds like you’re not real familiar with your computer system, and probably don’t use it very frequently. I would spend some time and get more familiar with the computer, paying particular attention to where things are usually stored. If you don’t work with the computer very often, jot down some notes.
Next, get yourself an external drive, and figure out how to identify it on the system when it’s plugged into the computer.
Actually, I would read Nasim’s article above again, and adopt some of his ideas about making backups. That would mean buying two external drives and rotating them to somewhere off site. And again, if you don’t do this very often, take notes about the process you use to make the backups.
As for the existing triplicate files, I wouldn’t worry about them too much. Find a set that seems to be most complete and herd them up into one place. As you get more familiar with your computer, you will no doubt discover the “triplicates” in time and can deal with them on your own terms.
This is an old thread but…I use CloneSpy (http://www.clonespy.com/). It’s a very powerful and useful tool. It’s free so please consider donating to the author. Thanks.
How do you guard against corrupt files in backup. How do you detect these and avoid backing them up/over writing good ones and not noticing them until it is too late.
Tanks
I can’t speak for all backup software, but in the case of RSYNC, it uses an algorithm that compares the original file against an existing backup file. If the two files are different, it makes a copy of the original to the backup.
There are two possible outcomes here in terms of possible file corruption:
1 – The backup was corrupted and now it’s corrected since it’s made to be just like the original.
2 – The original is corrupted and now they’re both corrupted since the backup is, again, just like the original.
One thing RSYNC can do, and I assume many other backup programs can do as well, is make a copy of the old backup and place it somewhere else. Actually, RSYNC moves the old backup file to the new location, so if it was good in the first place, it’s still good.
I have no idea how to detect if a file is corrupted without trying to load it. Since I have over 100,000 images in my active database and a few thousand more on archived drives, I wouldn’t be surprised if a couple are indeed corrupted. Having said that, I do have to say that since I stopped storing my backups on CD and DVD way back when, I haven’t encountered a file that’s corrupted. Not once.
In my experience, I’m way more likely to foul up a file myself by my own antics than I am to encounter a file that’s been corrupted for some reason on the system.
One thing I recommend with backups is that you stay away from programs and routines that are proprietary, and especially programs that encrypt or otherwise make the backups unreadable from outside sources. Also, I stay away from programs that compress the data to save space. I want my data in plain sight and accessible from a variety of sources. The way I see it, hard drives are cheap, so extra space is not a problem. And, when I have a catastrophic failure and need to recover data, I certainly don’t want to be fighting a proprietary system trying to recover my stuff.
Sha
I have used different SW in the past and find it slow and painful to use.
I have a simple way of keeping track of all my backups. I religiously Name all folders with the same format ie: “2013-04-05 New Zealand, 2014-04-06 Amazon……”
As long as the date is in front of the text you can sort on the files on you PC. All the NEW folders will be at the bottom. You can very easily know the differences between your Main system AND all your backups. I use “Downloader Pro” from this Co. http://www.breezesys.com/ it will detect your CF or SD card, create the folder (with your preferences and location) and down load your pictures. I have been doing this for years now and it works for me. You just HAVE to keep the beginning naming structure the same so your PC sorts will show what’s there in the same order each time.
Hope this helps
Hi, Great article. Very similar to what my current workflow looks like. Just want to share a minor difference that may be helpful. I prefix the event folders with the month, except the family one.
File structure as per your article…… C:\Photos\YYYY\event (ex. C:\Photos\2015\Singapore trip
My file structure C:\Photos\YYYY\MM-Event (ex. C:\Photos\2015\11-Singapore Trip
This way the events are chronologically sorted within the year, obviously this only make sense from the file system perspective.
I hope my five cents helps.
hi, dear nasim,
for d 7000 and d750 with best nikkor 50 mm f1.8D or nikkor 50 mm f1.8G lens?, the difference between these lenses?
I’ve been using cloud backup for four years or so and it’s wonderful. I probably don’t have anywhere near your volume of data to deal with as you do, and it took something like three months before everything got uploaded (we could have sped it up by backing everything onto a 2TB hard disk and mailing it to them). Since then our machines are simply backed up whenever we check. We’ve had file losses (both from OS/hardware failures and accidental deletion) and been able to recover specific files very easily.
What I don’t care for in your workflow is the manual element. This stuff can all be automated very easily with software like Chronosync or Crashplan (its software is free and you can use it to backup from one computer to another). So by all means use RAID1 as primary storage. Another option is not to erase memory cards until the photos on them are fully backed up — probably more reliable too (memory cards are pretty cheap — aside from XQD). But from that point on everything else can and should be automatic and effortless.
Regarding cloud backup: “If it is reasonable and your bandwidth can keep up without angering your ISP”
While upload speed is certainly an issue for many people, most home ISPs only care about downloads for bandwidth caps. This is, in part, to allow for cloud based backups.
I looked into a product called Carbonite, as a cloud backup solution, but soon discovered severe limitations. This product runs in the background on your computer and backs up whatever folders you specify to their cloud while you are working on other things. First of all, the speeds are based on getting full access to your bandwidth so any time you are using the internet the performance will plummet. That includes watching Netflix, streaming videos, etc.
I checked the speed that they offered for uploads and downloads, and determined that it would take 12 months for my computer to back up just the RAW photos/videos on my desktop. That means leaving my computer running and connected to the internet for 12 months straight. And if I lost these photos, it would take 9 months to restore them. Can you wait months for lost photos? What happens if you are 6 months into your backup and you need to restore one that it hasn’t even backed up yet?
And these times don’t take into consideration your new photos from shoots done in that year …
I was sure my math was wrong so I e-mailed them and they e-mailed back that my calculations were correct. They said their services are not for people with lots of data like raw photos. And I pay a premium for faster downloads and faster uploads than most people have.
Any time you are looking into services on the cloud, make sure you know the size of your data, your upload speed, your download speed, and the speed of the cloud provider. The latter will likely be slower than your ISP but use the slowest speed. Then get working on your calculator to see how long one backup would take. For the cost of this service I can buy multiple external drives every year, have complete backups on the same day, and without restricting bandwidth month after month after month. If your ISP is not unlimited, you will need to calculate the surcharge for your massive data usage as well. It may end up costing you more in data charges than the service itself. Much more.
If you have a router with USB ports on it, then you can turn an ordinary USB3 cheap external drive into a network drive. My router has 2 ports on it where I keep 2 USB drives plugged in, that any computer on our network can access. It is also a quick way to copy data from one computer to another one.
Steven
I also looked into various cloud options and came up with the same conclusion. Its great for kids taking a few cell phone photos. But when you want to back up over 5TB of pictures/video/data , the cloud does not work well.
I posted earlier. My PC allows for external drives to be install and removed as easy as a cd inset and remove. These disks are for my off site back ups. And it only takes a few hours.
Again I posted earlier what I have and do
Someone please inform me if I am missing out on the Cloud product. I need app 5.5TB of storage
If you’re in the US, check out Amazon Snowball:
https://aws.amazon.com/importexport/
I realized that I have Free unlimited picture storage on Amazon Cloud with my Prime membership. So I’m giving it a go. So far a got almost 1TB uploaded 4.5TB to go. I have been pecking away at it and letting it upload in the background. Ill still keep my PC, and two off site copies. The Cloud will be as a last resort to recover. Once you have the main/bulk uploaded the incremental uploads should go fast. I have not tried to “recover/Download” any pictures yet, but the site says “You can download up to 5 GB or 1,000 files at a time. Please select fewer items or use the desktop app to download more files.” This recover could be quite painful. Hope I never have to use it.
Thanks for the advice on Snowball
Yea, the recovery is always a problem, as that could take a few weeks in itself.
Personally, I don’t use additional drives to back up, I just archive everything online in addition to a local copy. If I lose my local copy, it will be relatively quick to redownload just what I need (I figured I would never need all my data back at the same time).
Fortunately, I have access to a decent connection, so a full redownload will only take 4 or 5 days.
If you have a few terabytes of data, and are wanting to use cloud storage, a number of the service providers offer physical drive import services as well.
Amazon has Snowball, which is a 50TB device they ship to you, you transfer content, and then ship it back.
If I recall, Google allows you to send hard drives in for import.
This will at least alleviate the initial import hassle.
Has anyone used M discs for backup?
A word of caution when running SSDs in RAID-1 configuration: Make sure you check if your RAID controller supports the TRIM command.
Otherwise you may risk degrading your SSDs very quickly!
RAID functionality built into the Intel Z97 chipset for instance only supports TRIM for RAID-0.
I would love your opinion on something I do to never delete photos I have not backed up, and reformat my camera cards with confidence that they have been backed up, since my camera tells me no pictures are in the file when I reinsert the card.
I change the file name ON THE CARD to the permanent file name, before I back it up.
Since I may have more than one event on a card, I just go by year/mo/day/ +hour if I have more than one card that day. I can then organize by event in LR.
Thanks for your time and great article, Kim
Good article, it definitely jump started my backup planning process.
One thing that may be macro to the topic but is arguably paramount is that you must edit, edit and EDIT some more. Do you really need 25 different shoots the door of that church in Italy? Do you need 200 landscapes of that plain in Africa you visited? I doubt it. Sure they are all in focus and exposed correctly but go ahead a pick on or two and throw the rest away. No one will ever notice it’s gone. Monks spend weeks working on sand mandala only to blow it all away. While I’m not suggesting you blow all your photos away. I do think we are entering the realm of digital hoarding and, as one person pointed out, when you’re dead and gone who the heck is going to go through your 900,000 photos? Even if you’re the next Ansel Adams and Edward Weston you probably already have too many photos. Don’t get another hard drive, hit the delete button.
Excellent topic. Always timely. My current backup solution is a NAS, synology DS416slim 4 WD Reds 2.5 inch 1 TB and using a standard raid 5 confiduration so i get less than 3TB of storage there. then that automatically backs up to one of 2 3TB WD red 3.5 inch every night. the second 4TB is always in the fireproof safe, i swap them out frequently. Nothing off site and no cloud storage usage yet. I am covered regarding a flood, and possibly a small fire but a thorough thief or a large (very hot) fire could wipe me out. So I am toying with the idea of locking up one of the 4TBs at work but i dont know if i trust the nighttime cleaning crew.
I will eventually run out of space and am researching building my own FreeNAS server, i have a first pass at a parts list, would love to hear some feedback on this dream storage rig.
http://pcpartpicker.com/list/ws3sLD
Thanks.
I am wondering if you can give any advice please?
My daughter and son in law used a ‘professional’ photographer for their wedding in April, paid £1850 ,they have chased and chased, as after a month he advised was having problems with his server. he has today advised that the photos could not be retrieved from the server. (he has a few from his 2nd photographer but these are side facing shots). They are devastated and we will do anything to try get them retrieved. In his emails few weeks ago he said the problem is beyond raid? Is there any advice or suggestions you could give us please/
Regards
My question and my method are quite simple.
May photo storage drive, input and output is one one external hard drive (USB 3, 2 TB.) (I do not use my system drive for photographic storage) It is about half full, and will last for another five years before it is full.
I want to copy this drive to two other external hard drives – you would probably say three – with automatic backups so I do not have to replace the whole disk every time I backup. A rotation system would work.
I have not yet found the right software for this automated backup – software seems aimed at backing up system drives rather than external hard drives with data on them. It is also often too complex for my simple method.
Any suggestions?
F.
Thank you for publishing this, it’s very helpful.
I was wondering if you could clarify one bit. You wrote “Backup to the second drive as often as you will be taking it offsite. You can rotate the two drives, but you have to make sure that you are not just doing incremental / differential backups – you have to have full backups for rotation”
When mapping out a backup strategy, I was considering a scheme which had 2 drives that rotate, each containing a backup created via Time Machine. Does that violate this principle?
I’m a (headshot) pro of 18 years. I have a workflow with a hole or two on the back up side and I’m looking for the easiest way to correct to best practices. My workstation has a Drobo 5D where all my work photos are stored. Then I burn DVDs of every shoot and take them and store them offsite at my home. What’s the best/esiest thing to add or change to follow best practices. Thanks anyone in advance.
Since a DVD lifespan can be as short as 5 years, and can be easily scratched, I would change that to an external hard disk. It will also speed up your process.
My workflow is as follows for every card I have shot:
Note that the folder name I decide on is identical on every drive.
Copy the card to a local disk
Copy the card to an external disk
I now have 3 copies.
Then I Move the card to a network disk (this copies, verifies and then removes the photos one at a time from the card)
Using this workflow you can never delete a photo by mistake, since you never “delete” any.
Plus you are doing a move after you have 3 copies already.
I also have a web space that I copy to but that is overkill for some.
The external disks I keep in my safe.
The web space is my offsite backup.
I don’t use any mirroring or RAID – I used to but lost massive amounts of photos on one failure.
Fortunately I had other standalone external drive copies …
Now I control every copy. If an external drive starts to go … with normal wear it will lose a sector at some point. I just run my repair for that sector, and copy the missing photo from another copy. If I see several failures I would just copy the entire drive to a new drive and chuck it.
I have never lost a photo, and never deleted one by mistake. With the large external drives being so cheap now, it makes multiple backups quite easy and affordable.
In case of fire, I have my web copies and safe copies.
In case of theft, they would have to get past my 2 German Shepherds and internet video cameras.
And I would still have my web and safe copies.
Thank you for the article. I have shot slide film my whole life….I am now 54 and last year bought a Canon 5D Mark III. I am overwhelmed with the technical stuff! I am trying to figure out backup first….before I move on to getting my files organized….which are a mess. MY QUESTION: I use only a lap top (MAC) with a flash drive. I have come to the conclusion I want to keep everything on an external hard drive. I now have it all on a 2TB (which is enough for now) drive and that external drive is all backed up to the cloud (Crash Plan). I want to now make two ongoing COPIES of this external drive I work on (in addition to the cloud which backs up the external drive). I am confused when you say RAID is not back up….but you recommend the Synology backup which appears to be RAID??? How can I simultaneously make two identical backups of this one external hard drive I am using as my main drive…..and perhaps be able to take one of those backups to a different location (retrieve it when I want to make the two identical backups again)? Any help would be appreciated.
RAID allows redundancy of data stored on the array. The disks in a RAID are not backups for each other because although some disks might be mirrored, the entire storage scheme is managed by either a hardware or software controller. This means that all data into and out of the array flows through the controller and if that fails, the entire array can become corrupted. Remember, too, that the data on the RAID will be updated in real-time, so if you accidentally delete files every instance of that file in the array will also be deleted. The purpose of RAID is to strap multiple drives together for either speed (RAID 0), convenience and reliability (RAID 1 and 6), or both (RAID 10). RAID can be used at any point in the file management workflow in the place of a single disk.
A backup is a complete copy of your working drive or RAID at some point in time to another drive or RAID where the data can then be stored in another physical location. Backups can be quite useful when done incrementally or differentially so that if you lose a picture or file on your main working drive, you can retrieve it from a prior date. The largest drives currently available that are close to being reasonably priced are 6 to 8tb in size. If you need more storage than that or want faster external drive performance, then RAID would become an option. Since you already have an external working drive that works for you, I’d suggest you grab a couple of 3-6TB USB 3.0 external drives (depending on budget) as backup drives to your 2TB to allow for a backup of both your photos through Crashplan (https://support.code42.com/CrashPlan/4/Backup/Backing_Up_External_Hard_Drives) and your Mac’s SSD through Time Machine. Note that if you go above 4TB with the externals, they will most likely come with a separate power adapter rather than pulling power from the USB port (if that bothers you).
For around 50 a year I use Amazon Cloud to store all my RAW images.
Keep 2TB drives per year of pictures that I can put on my exchengeable Sata desk drive and keep my light room database running with the pictures of the last 6 month.
As soon as I download something is backed-up to the cloud.
Thanks so much for your articles, they are very helpful. I am an eager, amateur looking to get started and organized properly. When you upload your photos to Lightroom, how do you then differentiate between the ones you have edited and the ones you are yet to edit? Thanks again, all this information is much appreciated.