Battle of the Mirrorless – Part 1 (Low Light Performance)

I have spent a considerable amount of time working with 7 different mirrorless cameras from Sony, Canon, Nikon and Olympus. I apologize for not being able to provide periodic updates on these cameras. I have come up with new ways to measure digital camera sensor performance, so it took me a long time to do it in a way that I believe will be more accurate and objective compared to my previous methods. Not only will you be seeing crops of sensor performance in a controlled environment, but I will also provide some numbers to quantify performance in colors and dynamic range. As I have already mentioned before, I will be measuring dynamic range myself going forward without having to rely on other websites for the data. It will be interesting to see how my data compares to other sites like DxOMark. I am not planning to do anything super intensive and I bet my measurements will not be without issues and errors, but I believe it is something worth trying. Hopefully it will give a different perspective to testing sensors.

Here is the first test that shows the low light performance of the following mirrorless cameras: Nikon 1 J2, Canon EOS-M, Sony NEX-F3, Sony NEX-5R/NEX-6, Sony NEX-7 and Olympus OM-D EM-5. Since these cameras all look excellent at ISO levels between 100 and 800, I decided to only show ISO performance at 1600 and above. Take a look!

Nikon 1 J2

Nikon 1 J2 ISO 1600 Nikon 1 J2 ISO 3200

Nikon 1 J2 ISO 6400

Canon EOS-M

Canon EOS M ISO 1600 Canon EOS M ISO 3200

Canon EOS M ISO 6400 Canon EOS M ISO 12800

Sony NEX-F3

Sony NEX-F3 ISO 1600 Sony NEX-F3 ISO 3200

Sony NEX-F3 ISO 6400 Sony NEX-F3 ISO 12800

Sony NEX-F3 ISO 16000

Sony NEX-5R and NEX-6

Sony NEX-6 ISO 1600 Sony NEX-6 ISO 3200

Sony NEX-6 ISO 6400 Sony NEX-6 ISO 12800

Sony NEX-6 ISO 25600

Sony NEX-7

Sony NEX-7 ISO 1600 Sony NEX-7 ISO 3200

Sony NEX-7 ISO 6400 Sony NEX-7 ISO 12800

Sony NEX-7 ISO 16000

Olympus OM-D E-M5

Olympus OM-D E-M5 ISO 1600 Olympus OM-D E-M5 ISO 3200

Olympus OM-D E-M5 ISO 6400 Olympus OM-D E-M5 ISO 12800

Olympus OM-D E-M5 ISO 25600

Quick analysis of the above noise tests (from best to worst):

  1. ISO 1600: Sony NEX-7, Sony NEX-5R/NEX-6, Sony NEX-F3, Canon EOS M, Olympus OM-D E-M5, Nikon 1 J2. Sony NEX-7 stands out with excellent detail due to more megapixels. All other Sony cameras look very similar to Canon EOS M. Olympus also looks very good and Nikon is the worst in the group with the lowest resolution sensor. Colors look best on the Canon EOS M and Sony NEX cameras.
  2. ISO 3200: Sony NEX-7, Sony NEX-5R/NEX-6, Sony NEX-F3, Canon EOS M, Olympus OM-D E-M5, Nikon 1 J2. The Nikon J2 clearly looks the worst in the group and the rest of the cameras again behave similarly as at ISO 1600, with Sony leading the game in terms of noise levels and details. Canon looks a little cleaner in the shadow area.
  3. ISO 6400: Sony NEX-5R/NEX-6, Sony NEX-F3, Canon EOS M, Sony NEX-7, Olympus OM-D E-M5, Nikon 1 J2. Lots of noise, loss of color and loss of dynamic range on all cameras. The Sony NEX-7 gets plenty of false red colors in the shadows, making NEX-5R/NEX-6 and F3 look better in comparison. The Olympus OM-D E-M5 loses lots of detail and the J2 again looks the worst.
  4. ISO 12800: All look really bad as a group. The Sony NEX-5R/NEX-6 and the Olympus OM-D E-M5 seem to be a little cleaner in the shadows.
  5. ISO 16000 and above: Sony NEX-7 is the only camera that shows a little more detail and better colors due to its high resolution sensor. All look extremely bad though, unusable for any sort of photography.

Looks like the Sony NEX-7, NEX-5R/NEX-6 sensors handle noise levels the best in low light conditions. The Canon EOS M also looks great and has superb colors. Olympus is not bad either, but it loses a little more detail at ISO levels above ISO 1600. The Nikon 1 J2 obviously looks the worst, thanks to its smaller, lowest resolution sensor.

Don’t jump to conclusions quite yet, because these tests only show one aspect of camera performance. Colors, dynamic range, camera features, AF performance, lens selection and other criteria are also quite important, which I will cover in detail in the upcoming reviews.

Nikon D800E

And just for fun, I added the Nikon D800E performance to show you where today’s mirrorless cameras are compared to one of the best DSLRs on the market today:
Nikon D800E ISO 1600 Nikon D800E ISO 3200

Nikon D800E ISO 6400 Nikon D800E ISO 12800

Nikon D800E ISO 25600

There is a solid 2 stop difference between the D800E and the best Sony NEX series cameras, which is roughly what we should expect when comparing a full-frame sensor to a cropped sensor. The D800E images look very clean and extremely detailed in comparison, thanks to its massive megapixel count.

In Part 2, which I will post tomorrow, I will show the dynamic range performance of all of these cameras.

Comments

  1. 1
    ) Thomas Stirr
    December 14, 2012 at 4:57 am

    Hi Nasim….any chance you can add the Panasonic GH3 to this review? It features a ‘pro’ body i.e. weather sealed etc. and is the camera of choice for professionals shooting a lot of video. It has an new 16 MP sensor (rumored to be a Sony) with enhanced low light performance and much better dynamic range. At about $1,300 for the body only it is expensive….but has the specs to justify the price.

    • December 14, 2012 at 3:43 pm

      Thomas, absolutely – it will be included in part 2 of this! :)

      • 10
        ) Thomas Stirr
        December 14, 2012 at 4:01 pm

        Hi Nasim…..many thanks! I thoroughly enjoy your reviews and find them incredibly useful. Your blog is one of the best anywhere on the internet. Keep up the outstanding work!

      • 12
        ) Stefan
        December 14, 2012 at 8:33 pm

        Can I suggest one more camera? Samsung NX20.
        From what I read it seems is a very capable camera and the lenses are amazingly sharp

        • 14
          ) Raf
          December 15, 2012 at 3:30 am

          I also second on testing NX20 or N200

      • 32
        ) Nishant
        October 1, 2013 at 2:50 pm

        Hey Nasim, I guess you never got around to doing that Part 2. It’d be great to see a shootout similar to this with cams launched after you did this piece. Thanaks

  2. 2
    ) Wally Kilburg
    December 14, 2012 at 6:09 am

    No Fuji cameras included? Not invited to the battle because……?

    • 3
      ) Ertan
      December 14, 2012 at 6:28 am

      Are these JPEG or RAW?

      • 4
        ) Ertan
        December 14, 2012 at 6:28 am

        Sorry, wrong place.

    • December 14, 2012 at 3:43 pm

      Because the EX-1 was not available when I got these :) EX-1 + X-Pro1 will be included in part 2!

      • 11
        ) Sudharshun
        December 14, 2012 at 8:06 pm

        I am eargerly waiting to see XE-1 ‘s performance review

    • 29
      ) Sanjeev
      December 17, 2012 at 6:09 am

      Fuji X100 ?

  3. 5
    ) Yooshik
    December 14, 2012 at 2:17 pm

    Hi, where is the Fuji x-pro, or xe-1? From what I’ve seen, the x-trans sensor looked to be the mirrorless sensor to beat for low light. Will you be comparing it in the future?

    • 6
      ) Yooshik
      December 14, 2012 at 2:20 pm

      Nevermind, I saw you’ll include in round 2. Looking forward to the results. Thank you for all your work.

    • December 14, 2012 at 3:44 pm

      Yooshik, it is coming in round 2 as you have already noticed :)

  4. 13
    ) Anu
    December 15, 2012 at 1:57 am

    Nothing on the methodology.

    • December 15, 2012 at 11:47 pm

      Anu, please see my responses to David below. Thanks!

  5. 15
    ) David B
    December 15, 2012 at 8:25 am

    Nasim you say nothing about your methodology in your article. your conclusion of 2 stops in Nikon D800E vs Mirrorless is again based on ‘NORMALIZING” (meaning downconverting resolution of 36MP to some smaller number and then comparing?)? We know how DXO does it by downconverting everything to 8MP, but you did not mention ANYTHING in your article here about methodology. just wrote that you figured out how to compare things, but said nothing about what exactly you are doing. That is kind of odd.

    • December 15, 2012 at 7:08 pm

      David, I first wanted to write about the testing methodology, but I decided to keep it for later, perhaps in a separate article. Because it could dwell into a completely different debate and I did not want it to go that far.

      Either way, in my opinion, there is only one way to truly evaluate sensor performance – by downsampling. Why would it be fare to look at 36 MP and 10 MP images at 100% view, if both print completely differently on the same size paper? Noise looks better when printed small and it is a fact. Hence, all of the above images are down-sampled to the lowest resolution sensor (Nikon 1 J2, 10 MP).

      Let me know if you have any questions. I will add the exact testing methodology later this weekend.

      • 18
        ) David B
        December 15, 2012 at 8:48 pm

        You answered the question, so you downsampled everything to 10MP. You should have mentioned it in the article, because everyone who has followed your blog/site for a while, seemed to already participated in a debate on that issue before. it seems that everyone is entrenched in their opinions: you have yours and nobody is going to change it, and some other people have others, and you are probably not going to change it.

        I should just mention that the most popular photography site – dpreview.com DOES NOT downsample when comparing in their high iso comparison tool, so those who want to compare that way are welcome to look in dpreview’s review of any camera Those who subscribe to yours (and DXO) idea of downsample everything to one number can do it this way too. I understand your logic, but I am in the other camp.

        • December 15, 2012 at 11:46 pm

          David, I will definitely mention it in the article, similar to how I mention it in my reviews. Now when it comes to opinion, I would gladly change mine if you can give me a good argument on why I am wrong. If I show everything at 100%, the lower resolution camera that has the best pixel level performance will shine. While excellent cameras like Sony NEX-7 and D800 will look quite average. Here are my reasons why all comparisons should be down-sampled:
          1) When one shows images for the web, they are always down-sampled. For example, when I take a 36 MP image, I never publish it on my site at its native resolution. I downsample to 1024 pixels wide, which significantly reduces noise and increases sharpness. Most people are going to be doing the same thing when publishing photos. Hence, pixel level performance is irrelevant. If I take a 36 MP image with high pixel level noise and a 12 MP image with low pixel level noise, both will look similar in terms of noise at resolutions up to 12 MP (as long as the image processing pipeline is the same).
          2) The same thing with printing – most people will print smaller than the native resolution of the camera, which also decreases noise and sharpens images. Again, pixel level performance is not as important with small prints.
          3) When people do decide to print very large, then a high resolution sensor is going to give more options. Hence, higher resolution sensors are preferable for printing work.

          If we only look at pixel level performance of all sensors, then sensors with the largest pixels are always going to win, making high-resolution sensors look very unattractive. Also, the physical size of the sensor in that case will not make ANY difference. Both the D7000 and the D800 have the same pixel pitch and similar pixel level performance. If I only look at images at 100%, then my conclusion will be that the D800 is no better than the D7000. But that’s not true and will never be true. Having more than twice the area and a lot more resolution on the D800 allows me to yield sharp images with much less noise if I take the D800 image and down-sample it to the D7000’s resolution.

          This is why I embraced this particular method of testing sensor performance. This is not about dxomark vs dpreview. I do mention DxOMark in my reviews quite a bit, because their testing methodology is very similar to mine. They down-sample to 8 MP, I down-sample to the lowest resolution camera that I am testing. In fact, once I start quantifying sensor performance in numbers (which is what I am currently working on), I will have to pick a particular resolution to down-sample/up-sample all cameras to. Otherwise, those numbers will be completely inconsistent.

          I hope what I said makes sense – please let me know if you have any questions :)
          Also, thank you for questioning my testing methodology, I am glad someone has interest in it!

          • 22
            ) David B
            December 16, 2012 at 8:34 am

            as I stated, many large established sites as dpreview.com or imaging-resource.com DO NOT downsample, even though, imaging-resource for example discusses the issue in their testing methodology page. After all, by downsizing you are introducing yet another variable, depending on how the particular program that you choose performs the task. Who knows what is going on it and how it deals with different types of RAW files that exists. Never mind the fact that many people say that certain manufacturers “cook” their RAW files to begin with by still adding some noise reduction without telling us, etc. And now you added another step by downsizing.

            Also when I for example work on my files on Lightroom, or look at them, I look at them at 100%. I don’t downsample them at 3.6 times and then look at them.

            Bottom line, it would be much easier to avoid this discussion and it would be VERY EASY FOR YOU to just post BOTH, comparisons at FULL SIZES at 100%, following by comparison after you do your downsampling , for example of 36PM by 3.6 times and then comparing it to 16MP downsized by 1.6 size. By doing both Which is easy for you since I am sure you still have the full size files, you can demonstrate the things. For example, Jeff Keller of dcresource.com does it all the time with his JPEGS high ISO samples, when he takes the RAW file at the end and converts it and cleans it, demonstrating that RAW is always better. But he still shows you the JPEGS. maybe not a great example, but If I were you, I would show both. You can show at 100% and then do your downsampling, do a discclaimer that you are doing downsampling with Lightroom and perhaps people using other programs to downsample might have different results (after all, who knows what algorithms are used to downsample by different program mfrs?).

            • December 16, 2012 at 1:52 pm

              David, large sites like DPReview do not downsample, because they started out without downsampling in the beginning and they cannot just scrap their entire database and start all over. The downsampling process works very similarly across Adobe products, which is what I and most photographers use for the process. To make the process more realistic and more relevant to most photographers, I import images into Lightroom, process them like I would process normal files and downsample using Lightroom’s built-in algorithm. I import with default settings, which apply nothing to the image at all, so there is no extra processing of any kind. All RAW files get treated exactly the same way. The downsampling algorithm works EXACTLY the same way no matter what RAW file you apply it to. As far as the program is concerned, you are downsampling a TIFF image, it does not care if it is RAW. As for manufacturing “cooking” their RAW files, which is a very normal thing to see today, there is nothing you can do about it. And the downsampling won’t make it look better or worse. If a RAW file is cooked by the camera, no software will reverse it or change it.

              Downsampling is not a new term, but unfortunately, a lot of people do not understand what it means just because the word “sampling” is grossly misunderstood. Downsampling is exactly the same as resizing to a smaller image. When you export out of Lightroom at a smaller resolution, you are downsampling. That’s what everybody does when exporting images for the web and even for print. And you looking at an image in Lightroom at 100% is a check to make sure that the image is sharp. In the back of your head, you know that it will look better when you reduce its size. If you had a 60 MP camera, you would still be looking at it at 100% in Lightroom. And at pixel level, high ISO, the image would probably look nasty, given the small pixel size. But are you going to judge the performance of the camera at pixel level like that every time? If yes, then you do not need a high resolution camera. I can tell you, that Nikon can easily make a 50+ MP camera today with ISO performance that will match, if not surpass the Nikon D800. At pixel level, it will look terrible, especially when you go higher than ISO 400. The moment you downsample images though, it will look amazing. So if we use the logic of not downsampling, then you are not looking at the real picture. You are looking at individual pixels, which will always look bad when they are small. And given the fact that all cameras are increasing in resolution every year, we might as well make a conclusion that cameras are getting worse. Which they are obviously not :)

              Now in terms of doing both comparisons. Count the number of images on this page. Now double that and double the amount of effort that goes into it. On top of that, how would you be able to compare the images if they are of completely different size? You look at the above image at 10 MP and then at the same image at 36 MP, which is much larger. How are you going to evaluate the performance of both cameras? Trust me, I have done it before and people jump to wrong conclusions. See my Nikon 1 V1 Review. When I posted my original comparison of the different cameras at 100% pixel performance, it was the folks at DPReview that got very mad about seeing that kind of comparison. The Nikon 1 V1 looked excellent compared to the Sony NEX-5N and the Olympus. Which in real life when you downsample, does not look nearly as good. So I had to go back and the second comparison with downsampled images, which showed the real performance. And even then, many readers were looking at the pixel level more than downsampled images, again, making wrong conclusions. After this, I figured out that most people do not understand the difference of actual versus downsampled pixels. Therefore, I decided to only provide the correct way to compare sensor performance, which is by downsampling.

              The same goes with dynamic range. I am currently working on a dynamic range test/comparison between different sensors, including the D800E. How do you think I am doing the comparison? Again, by downsampling first. If I look at dynamic range difference between the say Sony NEX-5R and the Sony NEX-7, the former will have better dynamic range at the pixel level – just because its pixels are bigger and it can store more information. But in reality, the NEX-7 is better when downsampled. It will have less noise, more tones, colors, etc. The blacks and the whites stand out better when they are downsampled as well. DxOMark does exactly the same thing, which is the method I agree with. If I don’t down-sample images for measuring dynamic range, then the D800 will probably look no better than the Nikon 1 J2. Is that the reality? No :)

          • 28
            ) Abhijth
            December 17, 2012 at 2:21 am

            So, you mean you downsample the pics from camera A and B to the resolution of camera C.
            that’s ok. But what about pics from camera C? You consider those pics as shot, directly from the camera?

            You must take into consideration that the pics from A and B have now been subject to downsampling (that is, some sort of editing by the way) which may have introduced artifacts in it. so, it is not fair comparing those files to the ones you get from C (either Jpeg, or raw converted to jpeg).

            So, please consider downsampling all files to a certain resolution, i.e, the same way DX0mark does.

        • 21
          ) Ed
          December 16, 2012 at 5:00 am

          How can you NOT downsample??

          So if I set my D800 to take small JPEGs it will be a better camera than when I shoot Hi-res JPEG or RAW because a small JPEG viewed at 100% will be cleaner than a large one???

          How does that make sense?

  6. 16
    ) Heidi
    December 15, 2012 at 2:33 pm

    I have read almost all your for the beginner, but still I am not catching something….. Can you please take a look this picture which I took today: http://2.bp.blogspot.com/-RKFYK0rkPeY/UMxuJTybAeI/AAAAAAAAApI/hVC4F7lkeZo/s1600/PC157863.JPG

    There was almost no light, so I used my tripod. I also haven’t done anything to the picture with photoshop/photoscape/etc…..
    My camera is Olympus PEN Mini E-PM1 with 14-42mm/F 3.5-5.6 lens.
    Details of the photo:
    Focal lenght: 20 mm (distance of the kitchen table was about 1.5 meters)
    Shutter speed: 1.6”
    Aperture: F11
    ISO: 320
    The targeting point was in the center of the photo.

    I have a decoration blog, so I am photographing a lot of rooms and so on. But almost always I have this same problem: walls are “falling” and something is blur.
    Well, this picture is actually very good comparing the photos I have taken! Wohoooo…. There is SOMETHING (like the table) quite sharp.

    It’s annoying because I don’t know where is the problem! Usually I take the room pictures with F8 or F11. I thought maybe the lighting was too dark and that’s why I didn’t get sharp picture (it really was gloomy lighting), BUT I have the same problem even the lighting is OK.
    Huh….. Can you figure, where is the problem?

    • 23
      ) HomoSapiensWannaBe
      December 16, 2012 at 8:47 am

      Heidi,
      The photo example you gave actually doesn’t have bad sharpness. It’s not tack sharp, but it’s not bad. Are you doing any post-processing sharpening and/or contrast enhancement?

      With the 4/3 format, you don’t have to use F11 to get a lot of depth of field. The next time you take a photo like this, try using F5.6 and 8.0. You are going beyond the diffraction limits of the sensor and making the overall image slighty softer by stopping down to F11.

      Do you turn off Image Stabilization when you use a tripod? Many, possibly MOST, systems give sharper images without IS when on a tripod.

      Do you use a self-timer to avoid shaking the camera?

      ALL of these could be factors taking away a bit of sharpness.

      Do some experimentation and have fun!

      • 24
        ) Heidi
        December 16, 2012 at 10:50 am

        Thanks!!! I will try your tips :)

  7. 26
    ) Brent
    December 16, 2012 at 2:44 pm

    Hi Nasim,

    I read that there can be a difference between the ISO a manufacturer claims and the actual ISO. I think one review claimed that one of the cameras was off by a factor of two (for example, they were claiming ISO 3200 but the real ISO was only 1600).

    Can you look through the exposure information and tell if any of the ISO ratings are significantly different?

    • December 16, 2012 at 4:41 pm

      Brent, all the above images were shot at exactly the same shutter speed, aperture and ISO. As you can see, there is very little difference between cameras in terms of exposure…

  8. 30
    ) Adnan Khan
    January 6, 2013 at 1:23 pm

    I’m surprised not see Pentax K-01 in this “battle” :)

  9. 31
    ) Jason
    February 7, 2013 at 1:47 pm

    Nasim:

    Looking for a mirrorless camera to take photos of my newborn, high speed video of my golf swing, and some lanscape golf course photos. I have no experience, but I am looking for more than your typical point and shoot. Casio has great reviews for its high speed capabilities. But, Casio doesn’t have great representaion in the USA. Any assistance or suggestions would be greatly appreciated.

    Jason

Comment Policy: Although our team at Photography Life encourages all readers to actively participate in discussions, we reserve the right to delete / modify any content that does not comply with our Code of Conduct, or do not meet the high editorial standards of the published material.

Leave a Comment