How We Test Cameras
See How We Test Other Products:
Here at Reviewed, we've been testing cameras and camcorders for over a decade. In that time we've refined our methods again and again, working with many partners and inventing several of our own tests from scratch to produce the most accurate reviews of digital cameras around.
Like all of our product reviews at Reviewed, we apply the scientific method to tests cameras in the most objective, repeatable way possible. Every camera that is reviewed in our labs goes through the exact same process with the same equipment, ensuring the greatest degree of accuracy possible. We also shoot with every camera out in the field to get a feel for how it will perform out in the real world.
All of our cameras are tested in our Cambridge, MA labs, in a consistent, climate-controlled room. In accordance with our ethics policy, we also believe it's important to be transparent with both our methodology and the equipment we use. Though our scoring model is proprietary for the purposes of preserving a level playing field amongst all manufacturers, read on for an outline of all of our testing procedures and equipment.
Because we want to be able to keep up with the newest technology, we partially assess the camera as the sum of its parts and features. How many ISO settings does it have? Does it have a cold shoe or a hot shoe? Does it have an electronic viewfinder (EVF)? Can the camera illustrate overexposed sections on a preview with flickering zebra stripes? Can it shoot in 4K video? Does it have dual card slots? We award points only to the features that we believe are most useful to the photographer; this avoids “feature creep”, which can occur when manufacturers decide to throw everything but the kitchen sink into a camera, whether they are useful features or not.
We rigorously test these cameras using four objective assessments because while a product can have any number of impressive features, the camera has to be able to walk the walk (i.e. actually take high quality images and video).
Noise and Noise Reduction
To test a camera’s noise rendering and reduction algorithms, we employ the industry standard X-Rite ColorChecker Classic 24-patch chart lit with standard D50 4700K lighting at 1500 lux. The ColorChecker chart has 24 patches in total: 18 color and six monochrome. We shoot at every ISO through the camera's entire ISO range, starting with the default noise reduction setting. We repeat this test for the “low” noise reduction setting, as well as the noise reduction setting for RAW images to get a solid representation of the camera's ability to render the color patches (of known color value) as evenly and with as little pixel noise as possible. We run the images at each ISO and noise reduction setting through Imatest (see below) to determine the actual noise present in each image. The less noise present in the image, the higher the score will be.
To assess the quality of the noise reduction algorithm, we shoot a custom-made version of what is commonly known as a "spilled coins" chart, which is filled with occluding circles. Cameras with high-quality noise reduction software can faithfully reproduce the fine details of each of the individual colored circles, as well as clearly delineate a circle’s edges, regardless of the surrounding color or circle arrangement. The monochrome gradation on the right side of the chart ensures the noise reduction does not distort simple, discrete shapes or introduce any unwanted issues with exposure throughout a scene.
For this test we employ the state-of-the-art in dynamic range testing, the Xyla-21, generously provided for our testing by DSC Labs. Unlike dynamic range tests that interpolate results taken from multiple reflective charts, the Xyla-21 is a backlit chart with a density scale that covers a full 20 stops of dynamic range; the 21st patch is over 1:1,000,000 darker than the first patch.
With the Xyla-21, we are able to get an exact picture of an imaging system's dynamic range capabilities from a single photo. The Xyla's careful calibration ensures precise results that are highly repeatable from camera to camera, with a total dynamic range that far exceeds the capabilities of even the best cameras on the market today. We measure dynamic range by capturing photos at the base ISO speed in RAW; that RAW file is then converted to a 16-bit TIFF file using the "Zeroed" preset in Adobe Camera RAW. For cameras that cannot shoot in RAW, we capture the same data with a JPEG with noise reduction tuned to its lowest possible setting. Running the image through Imatest tells us how many patches of light the image sensor can capture while still maintaining a 1:1 (or better) signal-to-noise ratio. The more tones that a camera can faithfully represent, the higher the score will be.
We test all cameras for their ability to accurately diagnose the color temperature of a scene and compensate for it. For this test, we use the X-Rite Judge II, which gives us a controlled environment for testing cameras in a repeatable, objective manner. We shoot an X-Rite ColorChecker chart under two color temperatures: 5500K (daylight) and 3700K (compact white fluorescent), for both the automatic and a custom white balance setting. Test images are processed using the ColorCheck module in Imatest and color temperature errors are reported in kelvins. In general, the lower the color temperature error, the higher the score will be.
Because cameras face stiff competition from smartphones, they have to be able to shoot high quality video, in addition to beautifully balanced images. With video sharpness, we shoot video of the CamAlign Mega Trumpet 12 chart from DSC Labs in bright light (1500 lux), using the best compression and frame rate mode available on the camera.
We record this chart while moving the camera in the vertical and horizontal planes, looking at the sharpness both with the camera standing still and in motion. We view the videos on a standard reference display in our labs when possible, looking for the point at which resolution (measured in line pairs per picture height) is at its highest. We also note any extreme examples of moire or false color, which contribute to the tone of our review, but don't factor directly into the video sharpness scoring. The higher the line density that can be resolved on video, the higher the video sharpness score.
While we can (and often do) test these cameras until the cows come home, at the end of the day, if a camera is ungainly or impossible to navigate, we don’t want to recommend it to our readers. To get at this, our testers address the more amorphous questions that have to do with their personal experience with a camera: how did it handle? Was it too heavy? Was it absolutely impossible to find anything in the menu? Were the controls intuitive and logically laid out? Did the product break or fail at any point during testing?
We answer these questions and more so that we can provide our readers with the most complete picture of each and every camera; we do the nitty-gritty testing and investigation, and our readers can enjoy the fruits of our labor.
If you want to read our most recent camera reviews, be sure to check out our library of camera reviews. To see some of the cameras we love the most, navigate over to our The Best Right Now articles, where we talk about all types of cameras, from the Best Cameras Under $1000 to the Best Instant Film Cameras.
Our 7th-generation testing facilities are located in Central Square in Cambridge, MA where we test all manner of personal electronics, televisions, displays, home appliances, and a variety of other consumer products. Our camera lab is a climate-controlled room where temperature is kept at 70°F, within roughly a two-degree tolerance, and normal humidity levels. The lab is light-sealed and lined with black Duvetyne flame retardant fabric to prevent reflections, glare, and outside light from polluting the test conditions.
Our bright light testing is conducted using an array of 12 50-watt black-backed SoLux MR16 4700K D50 bulbs with a 36 degree spread, which are purchased retail from Solux and placed on a track lighting system. We also utilize two LitePanels Micro adjustable LED lighting arrays, which we use to evenly illuminate test targets for low light testing.
We regularly keep tabs on the color temperature and lux levels in our labs utilizing Spectra Professional IV-A digital exposure meters. Per Spectra's description, these light meters measure brightness with f-stops from f/0.35 to f/128 and photographic illuminance from near total darkness (-0.1 foot-candles or 1 lux) to a full 100,000 lux.
To ensure accurate testing we use a number of color neutral reference patches for white balance. Though all of our targets are color neutral, certain cameras seem to perform better with certain targets. Some of these targets are made in-house, but we also frequently make use of the CamWhite provided by DSC Labs to ensure that our measurements are accurate.
All of our image analysis is conducted on a computer running the latest version of Imatest, at Matlab module designed by Norman Koren. As stated above, we also use the Adobe Camera RAW plug-in (as part of the latest build of Adobe Lightroom, when available) for developing RAW files where necessary.
For more information on our labs feel free to reach out to our Editor-in-Chief, TJ Donegan, available at email@example.com. For a complete breakdown of our strict ethics policy, please head here.
To develop our testing methodology we are constantly looking across the industry at the current trade standards, the methods of our colleagues, and the enterprising work done by other sites and testing outfits. Though we've never hesitated to develop our own methodologies and equipment, we're also grateful for the assistance of several partners who have helped us in developing one of the most complete, rigorous, and scientific testing models in the industry. We'd like to offer special thanks to the following groups who have been essential in helping us achieve our goal.
Imatest & Norman Koren
Imatest LLC, headquartered in Boulder, Colo. and founded by Norman Koren, is the world’s leading provider of software for testing digital image quality — in still shots or video, and in visible light or infrared. Imatest delivers software and consulting services to support product design, configuration, and production of digital imaging systems across a wide range of industries, including mobile imaging, medical imaging, aerospace, and defense.
David and Sue Corley started DSC in the basement of their home in 1962, and moved into the custom 15,000 ft DSC labs building in 1989. Originally award-winning film producers, DSC has evolved to become a world leader in test chart production – DSC’s flagship, ChromaDuMonde, is often referred to as “the Hollywood standard” test chart.