It reminds me of Goldoni’s “Servant of Two Masters“; only masters are now more than two and quite often they are not only capricious but they do not know what they want. First, any comparison is open to critics because even in a well-equipped lab it is impossible to repeat the shooting conditions from a year ago, or even from a day before while shooting to compare a newer model to an older one; the criteria for necessary accuracy is not set, or not made public, or not recognized by the community. Second, one single body in the testing opens the door for sample variation questions; and once again tolerances are not brought to the light. Third, using different lenses for different mounts does not help leveling the field. Using lens adapters to shoot with the same lens is often suggested, but it opens another can of worms: adapter alignment problems and different amounts of internal flare added by different adapters skew the results.
To level the field in the instrumental sense we need to surrender the idea of producing photographic images for such tests (and thus visual image comparison becomes impossible, hardly a welcome sacrifice for most of the photographers) in favor of direct projection of flat fields, slanted edges, grids, and spectrums right onto the sensor, without any lens on the camera. This allows for a bench-quality stable light source and aligned mounts; and we can now test more than just one camera body sample with minimal additional effort. As a result we can come up with a bunch of numbers, like color fidelity and metameric failure under different light sources, read noise, pattern noise, quantum efficiency, well depth, digital vignetting, and resolution at different contrast levels. This, of course is exciting – but, and it is a big but, do we know how to interpret these numbers, and how these numbers relate to photographic quality in real life? That is much open to discussion, to say the least.
Now, what happens if the tests are about images, and not just about numbers. If one is shooting JPEGs, comparing out-of-camera JPEGs with default settings is telling, apart from that some aficionados will always vocally accuse the lab in using “less than optimal settings” on their favorite brand or model.
With those who shoot raw and are unwilling to download and process the test raw files it is even more complicated. If somebody is using Adobe CameraRaw or Lightroom, they insist on comparing with the help of one of those, and than again, somebody will come to say “sub-optimal settings, wrong exposure, different lenses, different light, and you need a custom DCP profile”. And of course it is not a given that an Adobe converter interprets raw files from different cameras equally well.
Another problem here is that for those of us who do not use Adobe converters such comparison is meaningless. Adobe converters are not the only ones in use, about ten others are actively present on the market. What should a tester do, master each and every converter and convert with every single one? Slim chance.
Whatever the lab does, claims of sub-optimal raw conversion are universal, especially with a third-party raw converter. It is impossible to avoid being accused in “not doing justice” because of conversion settings, and staying at defaults does not mean everybody agree. Cross-converter comparisons are even more problematic, with maybe just one exception: defaults in Nikon Capture vs. defaults in Canon DPP.
Comparing always leaves somebody grossly unhappy, especially if it does not favor a bias. But the best camera in fact is the one you have with you, ready to shoot. Learning photography and shooting discipline goes a long way towards a “better camera”, and offers much more satisfaction, usually ;)
Mastering new ways of shooting also goes a long mile towards “a better camera” – less noise, more dynamic range, better landscapes and macro. It all starts with a tripod, and a good one. Focus stacking, panoramas, blending HDR exposures (just to mention few obvious techniques) help to get more from the camera. Next, a filter system – like how many times folks shoot a sunset and complain of the dynamic range while a simple cyan filter would provide them 1+ stop better exposure, holding down the red channel and allowing more light to be accumulated in green and blue channels, resulting in much less noise because the individual channels are now aligned. Polarizing filters are often used (and abused) to make skies deeper blue, but in fact they are very useful when shooting foliage and greenery, allowing to get rid of unpleasant specular reflections which look like blown-out spots and ruin the natural continuity of the image. The key is to study photography, and not just compare one camera to another…
Every tester makes an honest effort when it comes to organizing and taking the shot. However it is up to the reader to download raw file samples, process those in their favorite raw converter and draw conclusions…