How We Test Digital Cameras
See How We Test Other Products:
Here at Reviewed.com we've been testing cameras and camcorders for over a decade. In that time we've refined our methods again and again, working with several partners and inventing several of our own tests from scratch to produce the most accurate reviews of digital cameras around.
Like all of our product reviews at Reviewed.com, we apply the scientific method to tests cameras in the most objective, repeatable way possible. Every camera that is reviewed in our labs goes through the exact same process with the same equipment, ensuring the greatest degree of accuracy possible. We also shoot with every camera out in the field to get a feel for how it will perform out in the real world.
All of our cameras are tested in our Cambridge, MA labs, in a consistent, climate-controlled room. In accordance with our ethics policy we also believe it's important to be transparent with both our methodology and the equipment we use. Though our scoring model is proprietary for the purposes of preserving a level playing field amongst all manufacturers, below you can find an outline of all of our testing procedures and equipment.
Color Accuracy & Noise
For both our color accuracy and noise testing we employ the industry standard X-Rite ColorChecker Classic 24-patch chart lit with standard D50 4700K lighting at 1500 lux. We compare the known values of each of the 24 patches (18 color and six monochrome) to each camera's JPEG output to measure accuracy.
Color error is measured using Imatest's ColorChecker module, with fixed tolerance thresholds for both exposure and white balance accuracy ensuring accurate results. We score based upon the ∆C00 (saturation corrected) difference between the reference and output values, shot using every color mode available on the camera. Our scoring model weighs both the color accuracy and the saturation output, penalizing cameras for both incorrect colors and photos that are heavily over- or under-saturated.
We use the same setup for our noise testing, except we shoot in the most accurate color mode at every ISO through the camera's entire ISO range. We repeat this test in every noise reduction setting as well as RAW to get a full picture of every camera's performance through all of its sensitivity settings. For reference we also shoot a custom-made version of what is commonly known as a "spilled coins" chart filled with occluding circles in order to see both the noise levels and the impact of the camera's noise reduction system
For this test we employ the state-of-the-art in dynamic range testing, the Xyla-21, generously provided for our testing by DSC Labs. Unlike dynamic range tests that interpolate results taken from multiple reflective charts, the Xyla-21 is a backlit chart with a density scale that covers a full 20 stops of dynamic range; the 21st patch is over 1:1,000,000 darker than the first patch.
With the Xyla-21 we are able to get an exact picture of an imaging system's dynamic range capabilities taken from a single photo. The Xyla's careful calibration ensures precise results that are highly repeatable from camera to camera, with a total dynamic range that far exceeds the capabilities of even the best cameras on the market today. We measure dynamic range by capturing photos at every ISO speed available in RAW, converted to a 16-bit TIFF file using the "Zeroed" preset in Adobe Camera RAW. For cameras that provide high ISO speeds in JPEG only we capture photos for illustrative and informative purposes only, with only RAW results used for scoring. The Xyla can also be used for analyzing dynamic range in video, though we presently do not score that.
For scoring we look at the amount of patches that the camera is capable of presenting, from the brightest to the darkest. We report results in two ways, cutting off our measurement at the point at which the signal-to-noise ratio drops below 1:1 and 10:1. Though an SNR of 1:1 makes sense from an engineering standpoint, it's only the point at which you can be consistently sure that the output image is at least 50% signal. We score based off of the higher 10:1 threshold, however, because this is better representative of the point at which shadow detail is no longer rescuable for photographic purposes.
We test all cameras for their ability to accurate diagnose the color temperature of a scene and compensate for it. For this test we use the X-Rite Judge II, which gives us a controlled environment for testing cameras in a repeatable, objective manner. We shoot an X-Rite ColorChecker under three color temperatures: 5500K (daylight), 3700K (compact white fluorescent), and 2800K (incandescent lighting). Test images are processed using the ColorCheck module in Imatest and color temperature errors are reported in kelvins.
Resolution, Chromatic Aberration, and Distortion
To test the quality of the lenses used by cameras we are reviewing, we employ a 7x11 SFRPlus chart as produced by Imatest, lit with the 4700K D50 bulbs to an even 1500 lux, ±5%. Though Imatest's site has a more in-depth explanation of the benefits of the SFRPlus chart, we use this particular version because it offers us a complete picture of lens quality across the entire frame.
For evaluation and scoring we take both JPEG and RAW (where available) photos of this chart at a variety of aperture settings and throughout the focal range. These images are processed using Imatest's SFRPlus module, from which we score resolution, chromatic aberration, and distortion for the imaging system.
Resolution testing is scored using a weighted average that emphasizes center performance based off of the resolved line widths per picture height at a contrast of MTF50. Again, the benefits of using MTF50 and how it corresponds to the human visual system is better explained by Imatest's creator Norman Koren in this article.
For continuous shooting we test each camera at its fastest available continuous capture speed for which full resolution, full quality JPEG shooting is available. To test continuous shooting speeds we capture a series of photos of a rolling stopwatch on a computer screen at Online Stopwatch, repeating the test multiple times and using only results for which we have a clear timestamp down to a thousandth of a second.
In addition to measuring each camera's continuous shooting abilities we also measure the capacity to see how many shots can actually be captured at the maximum speed. We repeat this test in JPEG, RAW, JPEG+RAW, and any other modes that may be available, including lower-res modes, using the fastest full-resolution results available. We also evaluate any specialty modes and modes for which continuous shooting is paired with continuous exposure and autofocus, though the fastest mode is the one we use for scoring.
Reviewed.com was originally founded as CamcorderInfo.com (and eventually DigitalCameraInfo.com), which quickly became one of the premier destinations for unbiased reviews of camcorders on the web. As such, we have a highly developed system for testing video quality that has adapted over time to the changing quality and features available in the consumer video market. We take great pride in not only our quality of video testing today, but in our legacy of reviewing video cameras for the past 15 years.
Though you can (soon) visit our Camcorder-specific testing page, we'll briefly address the tests that we use to evaluate video quality on still cameras, most of which use the same equipment and methodology.
We measure video sharpness in both bright light (1500 lux) and low light (60 lux) using all of the available full-resolution compression and framerate modes available on the camera. Though we test all of the available modes, we only use the best-performing mode for scoring purposes.
Resolution in video is currently measured using a CamAlign MultiBurst Squarewave chart as provided to us by DSC Labs. We record this chart while moving the camera in both a vertical and horizontal plane, looking at the sharpness both with the camera standing still and in motion. We view the videos on a standard reference display in our labs when possible, looking for the point at which resolution (measured in line pairs per picture height) is at its highest. We also note any extreme examples of moire or false color, which contribute to the tone of our review, but don't factor directly into the scoring where they don't specifically impact resolution.
Low Light Sensitivity
For our low light sensitivity video testing we utilize a ChromaDuMonde 24+4-patch chart with a cavity black center point. This chart is available through DSC Labs and provides a number of references for image quality, color rendition, as well as both a peak white patch and a true black reference. We capture this chart using the best available video mode with auto ISO and the maximum possible aperture. The chart is lit at a low lux level (usually around 30 lux), which is lowered until the chart's center white patches no longer hit 50IRE on a waveform monitor. This light level is recorded and scored.
For motion video quality we capture our standard video motion rig at all of the available full-resolution video modes (and 4K, where available), lit to at least 1,000 lux. We review this footage on a reference display in our labs and compare it against other cameras we've tested previously. We score based on four critical factors that give us a complete picture of the quality of both the sensor and the underlying compression: ghosting, subject trailing, frequency interference, and the smoothness of motion (lack of stuttering/rolling shutter). We also often shoot this same rig in low light with a single offset incandescent bulb for reference and analysis, but this is not used for scoring.
Our 7th-generation testing facilities are located in Central Square in Cambridge, MA where we test all manner of personal electronics, televisions, displays, home appliances, and a variety of other consumer products. Our camera lab is a climate controlled room where temperature is kept at 70 degrees, within roughly a two-degree tolerance, and normal humidity levels. The lab is light-sealed and lined with black Duvetyne flame retardant fabric to prevent reflections, glare, and outside light from polluting the test conditions.
Our bright light testing is conducted using an array of 12 50-watt black-backed SoLux MR16 4700K D50 bulbs with a 36 degree spread, which are purchased retail from Solux and placed on a track lighting system. We also utilize two LitePanels Micro adjustable LED lighting arrays, which we use to evenly illuminate test targets for low light testing. Embedded lights in our still life and video motion rig are also used for lighting purposes, though these are for illustrative purposes only and do not affect scoring.
We regularly keep tabs on the color temperature and lux levels in our labs utilizing Spectra Professional IV-A digital exposure meters. Per Spectra's description, these light meters measure brightness with f-stops from f/0.35 to f/128 and photographic illuminance from near total darkness (-0.1 foot-candles or 1 lux) to a full 100,000 lux.
To ensure accurate testing we use a number of color neutral reference patches for white balance. Though all of our targets are color neutral, certain cameras seem to perform better with certain targets. Some of these targets are made in-house, but we also frequently make use of the CamWhite provided by DSC Labs to ensure that our measurements are accurate.
All of our testing is conducted on a Windows PC running the latest build of Imatest, designed by Norman Koren. As stated above we also use the Adobe Camera RAW plug-in (as part of the latest build of Adobe Lightroom, when available) for developing RAW files where necessary. We also employ other pieces of software for renaming and storing files, though these have no impact on scoring and may be changed as necessary.
For more information on our labs feel free to reach out to our Editor-in-Chief, TJ Donegan, available at TJ at Reviewed.com. For a complete breakdown of our strict ethics policy, please head here.
To develop our testing methodology we are constantly looking across the industry at the current trade standards, the methods of our colleagues, and the enterprising work done by other sites and testing outfits. Though we've never hesitated to develop our own methodologies and equipment where we had to, we're also grateful for the assistance of several partners who have helped us in developing one of the most complete, rigorous, and scientific testing models in the industry. We'd like to offer special thanks to the following groups who have been essential in helping us achieve our goal.
Imatest & Norman Koren
Imatest LLC, headquartered in Boulder, Colo. and founded by Norman Koren, is the world’s leading provider of software for testing digital image quality — in still shots or video, and in visible light or infrared. Imatest delivers software and consulting services to support product design, configuration, and production of digital imaging systems across a wide range of industries, including mobile imaging, medical imaging, aerospace, and defense.
David and Sue Corley started DSC in the basement of their home in 1962, moving into the custom 15,000 ft DSC labs building in 1989. Originally award-winning film producers, DSC has evolved to become a world leader in test chart production – DSC’s flagship, ChromaDuMonde, is often referred to as “the Hollywood standard” test chart.