It reminds me of Goldoni’s “Servant of Two Masters“; only masters are now more than two and quite often they are not only capricious but they do not know what they want. First, any comparison is open to critics because even in a well-equipped lab it is impossible to repeat the shooting conditions from a year ago, or even from a day before while shooting to compare a newer model to an older one; the criteria for necessary accuracy is not set, or not made public, or not recognized by the community. Second, one single body in the testing opens the door for sample variation questions; and once again tolerances are not brought to the light. Third, using different lenses for different mounts does not help leveling the field. Using lens adapters to shoot with the same lens is often suggested, but it opens another can of worms: adapter alignment problems and different amounts of internal flare added by different adapters skew the results.
To level the field in the instrumental sense we need to surrender the idea of producing photographic images for such tests (and thus visual image comparison becomes impossible, hardly a welcome sacrifice for most of the photographers) in favor of direct projection of flat fields, slanted edges, grids, and spectrums right onto the sensor, without any lens on the camera. This allows for a bench-quality stable light source and aligned mounts; and we can now test more than just one camera body sample with minimal additional effort. As a result we can come up with a bunch of numbers, like color fidelity and metameric failure under different light sources, read noise, pattern noise, quantum efficiency, well depth, digital vignetting, and resolution at different contrast levels. This, of course is exciting – but, and it is a big but, do we know how to interpret these numbers, and how these numbers relate to photographic quality in real life? That is much open to discussion, to say the least.
Now, what happens if the tests are about images, and not just about numbers. If one is shooting JPEGs, comparing out-of-camera JPEGs with default settings is telling, apart from that some aficionados will always vocally accuse the lab in using “less than optimal settings” on their favorite brand or model.
With those who shoot raw and are unwilling to download and process the test raw files it is even more complicated. If somebody is using Adobe CameraRaw or Lightroom, they insist on comparing with the help of one of those, and than again, somebody will come to say “sub-optimal settings, wrong exposure, different lenses, different light, and you need a custom DCP profile”. And of course it is not a given that an Adobe converter interprets raw files from different cameras equally well.
Another problem here is that for those of us who do not use Adobe converters such comparison is meaningless. Adobe converters are not the only ones in use, about ten others are actively present on the market. What should a tester do, master each and every converter and convert with every single one? Slim chance.
Whatever the lab does, claims of sub-optimal raw conversion are universal, especially with a third-party raw converter. It is impossible to avoid being accused in “not doing justice” because of conversion settings, and staying at defaults does not mean everybody agree. Cross-converter comparisons are even more problematic, with maybe just one exception: defaults in Nikon Capture vs. defaults in Canon DPP.
Comparing always leaves somebody grossly unhappy, especially if it does not favor a bias. But the best camera in fact is the one you have with you, ready to shoot. Learning photography and shooting discipline goes a long way towards a “better camera”, and offers much more satisfaction, usually ;)
Mastering new ways of shooting also goes a long mile towards “a better camera” – less noise, more dynamic range, better landscapes and macro. It all starts with a tripod, and a good one. Focus stacking, panoramas, blending HDR exposures (just to mention few obvious techniques) help to get more from the camera. Next, a filter system – like how many times folks shoot a sunset and complain of the dynamic range while a simple cyan filter would provide them 1+ stop better exposure, holding down the red channel and allowing more light to be accumulated in green and blue channels, resulting in much less noise because the individual channels are now aligned. Polarizing filters are often used (and abused) to make skies deeper blue, but in fact they are very useful when shooting foliage and greenery, allowing to get rid of unpleasant specular reflections which look like blown-out spots and ruin the natural continuity of the image. The key is to study photography, and not just compare one camera to another…
Every tester makes an honest effort when it comes to organizing and taking the shot. However it is up to the reader to download raw file samples, process those in their favorite raw converter and draw conclusions…
Iliah: Couldn’t agree more with everything you’ve said here, and that is a stunning, serene, peaceful panorama.
Thank you, Rishi.
Thanks Iliah for your article on the multiplicity of variables involved in testing. Insights from knowledgeable persons like yourself are always invaluable (and thanks to Nasim for ‘having you here’).
Your final comment “Every tester makes an honest effort when it comes to organizing and
taking the shot. However it is up to the reader to download raw file
samples, process those in their favorite raw converter and draw
conclusions…” begs the question of whether one chooses the camera to match one’s favourite PP software or the PP software to suit the camera?
On a similar vein, in your experience, is the camera manufacturer’s software (in general) a better choice than a third party’s (assuming this is even sensible to ask based on your article)?
So basically you are arguing that we can’t compare any camera/sensor to another because we can’t fundamentally control the quality of the profiling done for that camera in the RAW convertor needed to extract the images. You’re overthinking the problem, my friend. If we can see the same things reproduced in conversions coming out of 6 or 7 separate RAW convertors, then that concordance is evidence in itself.
Just because we don’t have certainty over all the variables doesn’t mean we can’t make conclusions on a trend.
I’m saying that no comparison can satisfy everybody, if you read what I wrote. For example, what you are suggesting can’t satisfy me.
Ok, good to know. At least I’ll know never to debate cameras sensors with you as no data post RAW conversion will ever be acceptable. :D
Right. I also can’t debate the vocal of Caruso based on Turovski’s imitation.
I must admit to falling down a bit on the filters. I stopped using some of the more specialized ones and instead “abuse” polarizers and a variable ND grad. It is time to break out some of the old filters or purchase some new ones!
I couldn’t agree more about your comment on tripods. If you want great shots, put your camera in front of something interesting (sometimes you might need a plane ticket to find that place….) and put your camera on top of a sturdy tripod / ball head / gimbal etc. and you are most of the way there!
One thing you left out is lighting. I get so much more out of my cameras with use of additional lighting. To me, being a bit of a strobist is required to get consistent results. I think this is so important that I won’t purchase camera systems with inadequate flash support. When choosing a smaller system, I was undecided between Fuji and Olympus and chose Olympus (despite the smaller sensor) because of their full featured flash support. This is also a big reason I have stuck with Nikon for all these years. Their implementation of flash (CLS) is first rate and easy to set up. I frequently use flash manually, but in a pinch, CLS works great without the use of radio poppers (slightly longer setup) and with less experimentation / chimping / calculating.
Regardless, just a great article!
Well, to leave light out is a capital crime in photography ;) The reason I did not mention the light is that a discussion of upping ISO against allowing more light is for another article.
Yeah, slightly different topic, but you can get more stops with light as well. No sin in leaving it out because it doesn’t fit your precise point.
Again, a great article and advice.
> you can get more stops with light as well
I fully agree, adding, re-distributing, and shaping light is one of the most important skills for a photographer.
Ilija you wrote this: ‘The key is to study photography, and not just compare one camera to another…’
In my opinion this shall be motto for every photographer in the World!! Thank you! And no one will do it (study) for you but you!
Iliah, I found your comment regarding the use of a cyan filter for sunsets interesting and useful. I had written the use of coloured filters off as a relic of the analogue era, but it seems I was wrong to do this. Is there a handy reference on the use of coloured filters that you would recommend?
We have three colour channels on the sensors, are something we call Red, Green, and Blue. You can think of it as if you are taking 3 exposures simultaneously.
If the histogram of raw data is showing the green channel is clipped first (most of the cases for daylight and studio strobes), the filter to use is 30% or 40% magenta. Magenta, being opposite to green, holds the green channels back and balances the per-channel exposures.
So, it is good to have 6 filters for 3 main and 3 complimentary colours: Red, Green, Blue will compensate for 2 channels, while Cyan, Magenta, and Yellow will compensate for one channel clipping.
Iliah, in the words of a well known UK phrase, you have thrown a spanner in the works! In fact, you’ve thrown the whole toolkit, plus the toolchest! :)
I feel very strongly about following several review / tests on anything that reaches my equipment purchasing shortlist. I use the philosophy that if six reviews / tests on that item produce five good and one average result… then it is worth the risk (loss of money) of making that purchase.
Indeed, I have changed my camera body twice in five years to gain the higher resolution sensor that came with the ‘upgrade’ whilst being aware that more pixels on a given sensor area, could easily lead to more noise. Reading around the available tests, each time I was keen to upgrade, revealed my fears were largely unfounded. The negatives I uncovered were pointing at the use of high ISO settings, and since I am a (mostly) landscape and macro shooter, looking for high quality images anyway, I don’t think I veer beyond iso 800 ever, so noise was not, in my case, going to spoil the party.
The other thing that happened to me, while making this journey to higher resolutions, was the discovery of better software. I got hold of DXO Optics Pro, and though it at first was a devil to understand, after I mastered it, the resulting images were mostly better than I could get in ACR and Photoshop. For those who don’t already know, DXO Optics Pro is the MASTER of cleaning up noise, so, with that software available, my iso ceiling is now much higher than iso 800 if I find myself ever needing to push it.
The one thing I find a little hard to understand is photographers who get hold of a relatively expensive DSLR such as the Nikon D810 or D750, in order to capture the best image quality, and then they whack up the iso setting to 6400 or even much higher, and then moan about noise in the image. Surely it were better that they buy a ‘lesser’ camera body, and spend the cash on better, and faster glass?
My experience with that software led me to totally trust the review opinions of DXO Mark, who, for those who don’t follow them online, apply a slightly more interesting approach to testing. When they get a new body, they test it with just about every lens on the market, not only the marque lenses (Nikkor lenses on Nikon bodies) but also with independant lens manufacturers such as Sigma, Tamron etc.
What this produces is the ability to find your camera body on the site, and then see ‘how it does’ with lens a, lens b, lens c.
Of course, this is imperfect, as your article points out. There will be batch variations potentially affecting all results, but from my experience of the software they produce, I am inclined to trust their results as being more scientific than the oftentimes subjective reviews we all see, in print, and online.
It is an imperfect world, and the D600 sensor getting sprayed with oil was a good example of how things can go awry, but within that imperfection, we have to be guided by something, so whenever I am thinking of a purchase involving a camera body, or a lens, DXO Mark is always one place I check.
As you rightly say, the right technique, filter, tripod, mirror lock up, and proper understanding of the medium is not only a pre-requisite, but an essential part of capturing the image you dream of.
Thanks for a thought provoking insight into your own ideas, very interesting reading it.
I am a DxO Optics Pro user as well, but my opinion of their noise removal is not as high as yours. I feel that a properly used Lightroom will do a better job, but I really don’t like Lightroom and only use it when I have no real alternative. Optics Pro does just fine, though, for most of my work.If it had exposure mods through “dodge and burn” brushes it would be way cool, which is mostly what I use LR for..
Hi Shane,
I fear you missed my main point. Most good photographers try to capture the best image possible before they ever get near a computer. In that light, it were better that a lesser body be employed and better glass purchased, for the same money.
I still fail to understand why people buy a DSLR for thousands of dollars, and then twist the iso knob to maximum, which, effectively, is saying to the camera… ‘please perform at the worst you possibly can’.
It’s akin to a race driver telling his pit crew to put on the worst, most slippery tyres on his car, before he starts the race, to cripple the performance.
I can see there are certain situations where high iso can be useful, and also some areas of special interest photography where that may be true, but the majority of us, I am sure, would prefer to get the best image possible at the shooting stage, and not have to be so over-dependant on software to fix the problems later.
On the Optics Pro thing, have you got version 9 or the latest 10? I know version 8 does not have such a good noise reduction algorithm, nor does it have the new ‘PRIME’ option within the noise reduction. I would be astonished if anything Adobe provides, can outperform the noise reduction of Optics Pro, since this feature, above all, is their mantra.
That said there is no one complete software solution now. If I have any issues with noise, which happens rarely, I run the image through Optics Pro to sort that out, and then usually export a Tiff, and work on that in photoshop (if indeed, any further adjustment is needed which was not available in Optics Pro).
The other great thing about Optics Pro is the continued additon of camera bodies and lenses to the downloadable free profiles. If I, or you, change cameras, we dont have to sit on our hands waiting for adobe to issue a new version of ACR, as DXO provide these free, and in a very timely manner.
Hope my point on original, in camera, image quality optimisation, is well taken :)
“I still fail to understand why people buy a DSLR for thousands of dollars, and then twist the iso knob to maximum, which, effectively, is saying to the camera… ‘please perform at the worst you possibly can’.” Strike!!!
Reviews are not perfect, of course, but that’s not what I’m trying to say here. I’m saying it is impossible to make a review that satisfies everybody. I’m also trying to explain what are the challenges while reviewing cameras and suggest that readers not just look at the rendered samples, but download and study the raw files when those are available.
The other point is, to follow my grandfathers’s adage, first 5 years in the profession you discuss camera bodies, next five years you discuss lenses, and only then you switch to discussing what is really important – tripods :) In other words, there is so much more to it than just cameras and sensors getting so little attention.
And finally, in the words of one of my mentors, the best investment into the photography is an airplane ticket – to visit new places, to see new people, and to shoot.