Over the past few months, I have been testing the new generation NAS storage array devices with 10 Gigabit (Gbit) network interfaces to see how they perform when compared to 1 Gbit network. While I am pretty happy with a standard 1 Gbit NAS setup for photo storage, any time I have to deal with stacking large panoramas, saving large TIFF files with several layers in Photoshop or doing any video work, my performance starts to go down the drain due to the 1 Gbit network bottleneck. As a result, whenever I deal with such projects, I have been using my local SSD storage for working on files and after I am done, transferring images and video back to the NAS storage for future access. So when I found out about 10 Gbit options on new storage arrays from QNAP and Synology, I decided to try them out in a productive environment. In this article, I will go over my current 10 Gbit Ethernet setup with a QNAP TVS-882T and compare its throughput to 1 Gbit setup when copying both RAW images and video, then list out the costs associated with implementing 10 Gbit Ethernet to see if is a viable solution for working professionals.
Table of Contents
1) The Pros and Cons of 10 Gbit Ethernet
As I have already pointed out in my Storage Considerations for Photography article, one of the biggest benefits of a Direct-Attached Storage (DAS) compared to Network-Attached Storage (NAS) is performance. Since DAS does not have the same latency and network throughput issues as NAS, as long as DAS is connected over a fast interface such as Thunderbolt or USB 3.1, one can get incredible read and write speeds. However, a DAS does not work well for a multi-user setup and does not provide all the benefits of a dedicated NAS, so those of us who need those features or have to work in multi-user environments have no other option but to go for a NAS, which is where we hit the network bottleneck.
That’s where 10 Gbit Ethernet (10 GbE) comes to the rescue. Compared to 1 GbE, 10 GbE has 10 times the network throughput potential, so it is as fast as USB 3.1 and first generation Thunderbolt, which is 1250 MB (megabytes) per second of throughput compared to 125 MB/sec limit of 1 GbE! That’s a lot of data one could move from a storage unit, so high performance is the prime advantage of 10 GbE.
With the growth in popularity of 10 Gigabit Ethernet (10 GbE) solutions based on SFP+ (optical fiber) or RJ45 (copper), we are now seeing this technology (which was once only available for large enterprises) becoming more widely available for small to medium-size businesses at much lower costs than before, and with time, the technology will eventually make it to consumer-grade products, desktop PCs and laptops as well. While one might think that they would be an early adopter of 10 GbE, that is certainly not the case – 10 GbE has been already available for many years and it has been thoroughly tested to work in enterprise environments, so there are no concerns about its future or its reliability.
However, due to lack of consumer-grade 10 GbE solutions on the market at the moment, implementing a 10 GbE network is still cost-prohibitive (as pointed out further down in this article), so its use is going to be currently limited to larger photography businesses that require shared storage for a multi-user environment.
2) 10 GbE vs 1 GbE Performance Comparison
While 10 GbE should be able to yield up to 1250 MB/sec throughput on paper, that number is practically impossible to reach due to a number of different factors. Transferring one large file is going to take less time than transferring many small files, because sequential read and write operations are always going to be faster. Therefore, in order to assess the potential of 10 GbE in a more realistic environment, we have to test transfer of both RAW and JPEG images, along with a separate transfer of large video files.
Let’s first take a look at the comparison of 10 GbE vs 1 GbE when transferring different images. For this test, I took a sample Photo folder containing a total of 688 files (a mix of RAW, JPEG, TIFF and XMP files), roughly 13.5 GB in size and copied the files 3 times to a NAS device (write operation) in order to yield the best transfer time. I used the “robocopy” tool to do this, since it shows the total amount of time it takes for the files to get transferred:
As you can see, the difference between 10 GbE and 1 GbE is pretty big, but not 10 times big. At its best, it took 33 seconds to transfer a total of 13.5 GB over 10 GbE (roughly 427 MB/sec), whereas the same amount of data was transferred over 1 GbE in 139 seconds (roughly 104 MB/sec). Still, that’s about 4.2x the amount of time you would be saving by transferring images over 10 Gigabit network.
Let’s now take a look at the performance differences using CrystalDiskMark:
For this test, I ran CrystalDiskMark three times on each interface and measured the results, then took the best result out of the three. As you can see, we can see a different picture here. We are talking about up to 1166 MB/sec read and up to 733 MB/sec write performance compared to a maximum of 118.4 MB/sec read and write time. We can see that in the second test, the bottleneck was the network, which explains why both read and write numbers never went past 118.4 MB/sec. That’s a pretty incredible difference between the two. For sequential reads, that’s roughly 10x difference in performance, whereas for sequential writes, it will depend on the maximum write speed your NAS can deliver. In this particular case, I saw a 6.2x difference, which is still huge.
So depending on the workload and what you are trying to transfer, expect to see anywhere from 4 to 10 times the performance when copying files over 10 GbE compared to 1 GbE. That’s a significant difference in performance.
3) The Cost of Implementing 10 GbE vs 1 GbE
Let’s take a look at how much money one would need to invest in a 10 GbE network infrastructure when compared to 1 GbE. Since most people are already familiar with 1GBASE-T using RJ45 connectors (standard CAT-5e or CAT-6 patch cable), it would make sense at look at the cost of implementing 10 GbE using 10GBASE-T for backwards compatibility and other reasons (in simple language, the connectors would look the same and the switches would have similar inputs as your existing ones).
3.1) Cable Runs – CAT-5e, CAT-6 or CAT-6a?
The good thing about running 10 GbE network, is that as long as your cables are not too long, you don’t need to upgrade your cabling to the newest CAT-6a or CAT-7 standards. So if you have a relatively small home / office with Ethernet cable runs not exceeding 45 meters, you can run a 10 GbE network using standard CAT-5e UTP cables! And if you have CAT-6 cables, you could push the length of cables to 55 meters. Anything longer than that will result in lots of errors and inconsistent performance, so you will need to run CAT-6a or better cables in those situations.
Unless you are planning to put your NAS device far away from your desktops, you should not need to worry about upgrading your cable infrastructure – your existing cables should be good enough. For this reason, I am not going to include the cost of CAT-6a / CAT-7 cabling as part of the cost.
3.2) 10 GbE Ethernet Switch
The biggest cost of implementing 10 GbE network is the switch and sadly, there are no cheap solutions on the market at the moment. If you want to take advantage of your existing cabling, my personal choice for a 10 GbE switch is the NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Switch (Model XS708E-200NES), which sells for $699:
The good news is, NetGear is no longer the only company making these switches. Buffalo is also now competing in the same space and they have an even cheaper 8-Port 10 Gigabit switch that is currently selling for $542 on Amazon:
I expect prices on 10 GbE switches to continue to come down in the future, but at the moment, they are still rather expensive for an average photographer.
3.3) 10 GbE Network Card
Lastly, for every PC that needs access to 10 GbE network, you will need to make sure that it has a 10 GbE network adapter to be able to take advantage of the speed. While many 10 GbE network adapters are rather expensive, I personally went for a third party adapter that has the Intel X540 chip. The cheapest one on Amazon is from a company called 10Gtek, which supplies network cards based on this chip with one or two ports. I personally went for the X540-T1 model that costs $190, but you can also get the dual port X540-T2 for $210:
These cards work really well and they have been solid for the past 6 months on both of my computers where they are installed.
If you are willing to go the eBay route (eBay Partner Program affiliate link), search for “Intel X540-T1” or “Intel 540-T2” and you will find plenty of options from companies in China. I was able to find Intel X540-T1 adapters as cheap as $85 (eBay Partner Program affiliate link), which is significantly cheaper than what one can find on Amazon.
4) Total Cost
So let’s now calculate the totals for a 10 GbE infrastructure:
- 10 GbE Switch: $542
- 10 GbE Network Card: $85
For a total of $627 you could get one machine connected to a 10 GbE switch. If you already spent the money to get a 10 GbE SAN, this actually does not sound too bad in comparison.
Now the big question is, would a 10 GbE infrastructure be worth the investment for photography? That all depends on your needs. For me personally, moving up to a 10 GbE infrastructure was worth it, because I work in a multi-user environment where I need fast access to my data, especially when it comes to editing videos. Thanks to the 10 GbE network, I now use a NAS as a local drive and host all of my photos there, and my NAS can provide amazing speeds for me that even a local SSD drive could not provide. All of my photos and videos now reside in one location.
For most photographers who are starting out, it does not make sense to spend a lot of money on a NAS, let alone on 10 GbE network. However, if you are a busy working photographer and you want to future proof your investment, a solid NAS with 10 GbE might be worth looking at.
Let me know if you have any questions regarding any of the above!
It’s not just the cost of the switch you have to keep in mind for a home officer or something intimate they are noisy large as they have fans compared to 1 GB switches that are fanless and expensive and don’t make any noise. And don’t forget about the weirdness of the version 7 cables being not quite standard
Hey Nasim, great work here – far too few are aware of how straightforward it can be to implement a 10GBase-T network at home. Thought you might be interested to know that it can be accomplished significantly cheaper, if you just have two devices that need 10GbE capability, you can create a secondary private network by direct attaching the NAS or server to the client device and eschewing the switch entirely. Downsides here are that you’ll need to manually assign IP addresses and it won’t be quite as fluid as using a switch. The slightly more elegant (and costly) solution for two devices would be using a mixed speed switch like the GS110MX (~$175, 8 ports total with 2 10GbE RJ45 ports) or the GS810EMX (about the same price, goes as low as $150 on sale and has 10 ports total).
If you’ve got a need more more than 2 clients, basically the only cost effective option besides what you’ve listed above is the MikroTik CRS305-1G-4S+IN, which is a 5 port SFP+ switch that’s only $135 and has excellent performance, and is compatible with most 10GbE copper transceivers (around $35/port) if you prefer to use existing Ethernet cabling over fiber & DAC cables.
tl;dr if you’ve just got two clients, you can do this entire setup for like $200-250 total with a NAS that supports a cheap PCIe 10GbE card ($150 for a TB3 10G adapter if you have a laptop, $100 for a PCIe adapter for a desktop and $100 for the NAS or server adapter). 40GbE Ethernet is around the corner and I hope 10GbE makes it to the mainstream soon, and articles like this only help!
You’re saying a single user doesn’t need a switch? That makes it so much easier for a 10gbe home network.
Have read your great articles with interest, total newbie question, if I have a computer with a 10g card can I not run a cable directly into that? I am talking a single computer setup
Robert, you absolutely can! A direct connection will allow direct communication between your computer and the storage unit, similar to a DAS. The nice thing is, you can utilize the additional Ethernet ports on the NAS to connect to your network, so that you can use it from other devices as well.
Hi Nasim,
Today I work of a SMB share form a Linux server and using Lightroom for editing. As I like to keep all in one place, I have the images imported into Lightroom from the mapped drive. This is rather slow, although I think my network is not 100% busy when importing/editing. Did you see any speed improvement here as well? or is the delay caused by all the layers and Lightroom being ugly slow on network drives still an issue?
Many thanks for your feedback.
roman
Roman, no, Lightroom works very fast with the 10 GbE connection. I cannot see much difference between a network drive and a local drive…
I always appreciate these articles. Also, I am heartened to see that the 10gb infrastructure is not out of reach expense wise. I figured I would need to upgrade my cabling, but happily due to relatively short distances, that would not be necessary. So for around $800 I could upgrade. It is certainly unnecessary right now, but more affordable then I realized. Thanks for the info.
Hoping that 10G reaches consumer levels, and it is getting close. I run a Synology 412+ as my main unit and although tuned for performance, as the files get larger so do the loading times. At 4yrs old it will be time to replace, and doing that and upgrading the switch will help.
With network performance improvements we are also reaching the need for SSD drives of larger capacity.
I want higher capacity SSD’s at better prices more then I want network upgrades at this point. It makes such a big difference to be able to edit files locally on an SSD.
Great information, Nasim, thank you!
Nasim. Moving data around is OK but the link speed between the two devices is only part of the story. The speed you can transfer a file also depends on lots of other factors. I work on large computer systems, and when I go to configure a new system. I take into account, processor speed, bus speed (The physical path that carries around the data), drive I/O speed, and if I am using external storage, the drive I/O of the external storage. These are all measurable and if you are looking for transferring data, necessary. So when you get a speed increase in your interface are you taking into account all the other variables?
Is it possible that 10 GbE will be replaced by Infiniband in foreseeable future? With 8K video and over 100mp stills even faster connections will be required.
Cray CX1-LC “deskside” computer had Infiniband in 2009, and prices started at 12000. By 2019 it might be possible to include in desktop….
Andrey, that’s a good question. It is hard to say how technology will trickle down to the consumer market. With content only increasing in size due to much higher resolution, by the time 10 GbE gets into every computer, more speed might be already needed. Thunderbolt 3 is already at 40 Gbit / sec and the latest generation Ethernet is also matching that speed. The biggest problem today is infrastructure – every new home in the USA is still wired with outdated copper CAT-5e / CAT-6 cables. CAT-6a and CAT-7 are very expensive to run and you cannot even find those cables in any local stores (I had to special order mine online). However, Ethernet over RJ45 is very popular at consumer level and I really do not see any other technologies including Infiniband replacing that. In my opinion, Infiniband lost its market traction a few years back and the latest generation Ethernet is again taking over. Ethernet is simpler to implement and everyone gets it, especially if it is going to be backwards-compatible (and so far it has been).
My bet is with 10 GbE hitting the consumer market and not any other networking technology. By the time 10 GbE becomes popular, we will be talking about “affordable” 40 GbE+ infrastructures :) And by the way, in between, we should see 2.5-5 GbE hitting the mass market, thanks to this new standard. They say we will be able to get up to 5 GbE utilizing old CAT-5e infrastructures!
Wow, that’s a great news! I would like to see 5 GbE in my home without big investments.
For photos local SSD plus cheap NAS storage/ backup works OK for me. NAS is very useful in many ways for a family where more than one person uses computer.
Dear Nasim, thank you for sharing your experiences in this article. You mentioned you work in a multi-user environment but you do not mention how the other users (or machines) are connected to the network. They stay with 1GB/s? Also, working with video files (especially if they go up to 4k-resolution) is a complete different story than editing photographs (36-50MP). I wonder too if working in a multi-user environment and having to (or more) people accessing the NAS would result in a severe performance degradation (I think so). The hardware of NAS is not built to handle multiple parallel requests with that kind of transfer speed. Furthermore, if you work with a laptop, there is a hard time to get 10GbE for it. And you probably want to use wireless for convenience.
So how to have fast editing and secure storage at the same time? I solved it with a constant backup (mirroring) from a fast local SSD to a NAS. The SSD is just some kind of a fast local cache. What I do is:
Copy the data from the memory card (or the device) to a local SSD, sort out what I want to keep, store the data in my folder structure, process the data and let the backup software do the rest. The mirroring (done with rsync, NOT deleting the remote date, just adding new data) runs in the background.
Of course, when 10GbE gets cheaper (it will be) and NAS’ do handle parallel requests well, your solution is more convenient. And if you work with (very) large video files and multiple tracks there is no way around it (in a multi-user environment).
Jan, sorry that I missed to point out the details about my particular setup. In my multi-user environment, I have a total of 3 desktop machines (2x PCs and 1x Mac) and laptops that frequently access the shared storage. The two desktop machines are connected over 10 GbE, while the Mac and laptops are connected over 1 GbE. Even with a single 10 GbE connection, I am able to get plenty of speed across all the machines on the network. When working with video content, I mostly need access from the two desktops that use Premiere Pro. You are absolutely correct about 4K video – it is far more demanding than regular 1080p content, which is why it helps a lot to have the 10 GbE connection. Perhaps there would be a problem connecting over 2 users, but so far with 2x desktop machines over 10 GbE, I have not had any hiccups – the QNAP NAS has been performing very well. Laptops are a different story and I rarely need fast access with those, so wireless works out fairly well for some files. If I need to work on any photos though, wireless connection is junk – I often end up copying content to an external M.2 NVMe drive and using that instead.
Your solution to put content to SSD and work with it, while automatically duplicating it over the network via rsync is a solid solution. That’s what I used to do (although manually) before I fully switched to NAS.
Thank you Nasim for giving more details. With your demand of performance and the layout given it makes perfect sense to switch to 10GbE. I guess we will hear sometimes in the future how your work flow changed because of the speed advantages and how it enhanced. The instant accessibility of one’s data is a never ending dream because all speed improvements gets eaten up by the exponentially growing size of the data. I very well remember my first token ring network or Apple’s Localtalk and the long path to today’s technology. The bottleneck was and still is the I/O performance of the x86 hardware. Today’s PCIe delivers a maximum of 16GB/s using all(!) lanes. The real world maximum is between 2GB/s to 8GB/s. I/O operations per second (IOPS) and latency are the other main limiting factors. So a 10GbE network is much more than actually goes through the PC hardware. One might try channel bonding or teaming first to get higher network throughput using existing (and capable) 1GbE hardware. Of course with the falling prices for 10GbE this is the correct investment for the future.
Thanks for your wonderful and very inspiring site and to all the authors and especially to you.