How We Test Digital Cameras

Like all of the sites in the Reviewed.com family, our results are based on scientific, in-depth tests that analyze the performance of cameras in great depth. To achieve this, we find the best way to evaluate how well cameras perform, working with industry experts.

Many other reviewers rely on their own perceptions, judging color and resolution by eye (the so-called “golden eye” approach), but this can be biased by the preconceptions and prejudices of the observer, as well as the vagaries of the human eye. Our scientific tests have no such biases, and produce precise, objective results that can used for comparison between cameras, even if the cameras were not tested at the same time.

Many other reviewers also don’t reveal the details of how their tests are carried out, but we provide full details of our testing procedures, so both readers and manufacturers can understand how we reach our conclusions. Combined with our strict ethics policy, this means that every camera gets the same chance to prove its worth on a level playing field.

Equipment

All of our tests are carried out in our lab facilities in Cambridge, MA, using the same set of professional test equipment, which is detailed below.

Imatest SFRPlus chart

Sfrplus chart

Designed by Imatest, the SFRPlus chart features a number of angled squares to allow for the testing of sharpness and chromatic aberration at a wide range of points across the image. The bars at the top and bottom are used for testing the distortion of the lens. For testing purposes, we have also attached an X-Rite ColorCheck and Kodak Grey Scale chart to this chart setup.

X-Rite ColorCheck Chart

Colorchecker

Designed by Gretag Macbeth (who were purchased by X-Rite in 2006), the ColorChecker chart features 24 patches that cover a very wide range of colors, including 6 shades of grey. It is used in our color and noise tests, in both bright light and low light.

Kodak Grey Scale

Kodak grey scale

The Kodak Grey Scale (also known as the Q-13 or Q-14 chart) is an industry standard chart that features 20 patches of decreasing density. It is used in our dynamic range test.

Solux MR16 Bulbs

Our primary lighting source for testing is a set of 6 Solux MR16 bulbs, running on two horizontally mounted light strips. This allows us to light the large SFRPlus chart evenly, and the Solux bulbs are known for their stability and constant light level. They also have a very even color spectrum, so there are no chromaticity peaks that could affect color reproduction. We use the 4700k versions, and the lights are regularly checked and replaced as required.

LitePanels Micro

For our low-light testing, we use the LitePanels Micro. This LED light source allows us to evenly illuminate the ColorChecker chart that we use, and to vary the light level by altering the output of the LEDs. Unlike other light sources, varying this light level does not alter the color temperature or other characteristics of the light; the color temperature and chromaticities remain constant to a very low light level.

X-Rite Judge II

The Judge II is a viewing booth designed for evaluating how colors appear under different light sources, including daylight, fluorescent and incandescent light. Each of the light sources is very closely calibrated to produce light with the same characteristics as the light source it simulates. We use this in our white balance testing.

Spectra Professional IV-A Digital Exposure Meter

This is the tool we use to determine light values, measured in lux, to ensure that our charts are appropriately and evenly lit.

Zaber Linear Slide & Rotary Stand

To test image stabilization, we use a T-LSR-75A linear slide and a T-RS60 rotary stand, produced by the Canadian company Zaber. We mount the cameras and camcorders on these stands using a standard tripod mount. These devices allow us to apply shake to cameras and camcorders in a precisely controlled way, meaning that we can mimic human hand shake without the unpredictable nature of real humans. We use a custom script to control these devices to produce the required levels of movement to accurately mimic human hand shake.

Zaber1
Our image stabilization testing system

Video Testing Equipment

All digital SLR cameras that offer video recording are tested and analyzed by Camcorderinfo.com for their video quality. The equipment and methodologies used are identical to those used for all camcorders that Camcorderinfo.com reviews.

DSC Labs 28R Chroma DuMonde Chip Chart

The chart was designed by DSC Labs, used for evaluating color accuracy in both bright and low light. Its black & white resolution trumpets can be used as a quick reference, but are not used for the final scoring.

Dsc chromadumonde250

DSC Labs CamAlign MultiBurst Test Pattern Chart

This is the chart used to test video sharpness.

Dsc multiburst250

Moving Still Life Scene

Our custom-built 'moving still life scene' includes a number of elements: brightly-colored pipe cleaners, a moving train, a waterfall, a wrist watch, and two moving discs – one black & white and one colored. The discs are attached to Lego Mindstorm NXT motors, programmed to spin at fixed speeds.

Trainstation250

Westinghouse LVM-37w3 HDTV

We use this television as a quick-reference monitor for aligning shots. It's also the standard monitor for watching the motion test footage and evaluating resolution. However, the TV is not used for determining color accuracy.

Lighting

For bright light testing, we use two Lowell RIFA-Lite softboxes loaded with 500W GE Quartzline Halogen bulbs with a color temperature of 3150 K. In low light, we use two Litepanels LP-Micro LED arrays with a color temperature of 5600 K.

Software

Imatest

ImatestImatest is our primary tool for analyzing images. We use it for  in-depth analysis of color accuracy, noise, and resolution. Reviewed.com has worked extensively with Norman Koren, the developer of Imatest, to understand and customize the analysis to meet our specific needs. Imatest produces a wide variety of results and graphs which we use in our analysis of camera performance.

Video Software

Final Cut Express

Final Cut Express is the main software used to import and analyze video, as long as the file formats are compatible. Because new formats are routinely introduced, there's no single NLE (non-linear editing) package that can handle them all. Because of this, the editorial staff may use other NLEs when necessary.

DV Rack 2.0 HD

Only one function of this software is used: the waveform monitor, which is the primary tool for our Low Light Sensitivity test. DV Rack has since been purchased by Adobe. It is now marketed as Adobe OnLocation.

The Tests

Color Accuracy
For SLRs and point-and-shoots

Our color accuracy test is, as the name implies, designed to scientifically measure the precision of each camera's color reproduction. To test this, we illuminate the ColorChecker chart to an even 3000 lux, perform a custom white balance, then take photos in every color mode that the camera offers. These photos are then analyzed with Imatest, measuring the difference between the color on the chart and the color that the camera captured for every mode. We then determine which color mode is the most accurate, and the scoring for this section is based on the accuracy of this mode; other modes do not influence the score. The accuracy is determined using the CIEDE2000 color difference formula, which more accurately mimics the sensitivity of the human eye than the older color accuracy formulae that we previously used (and which many other review sites continue to use). To help you visualize how colors are captured by the cameras, we also feature swatches of the captured colors next to the ideal color for up to five of the color modes that the camera offers, and for the four cameras that we use for comparison purposes.

Long Exposure
For SLRs only

For our long exposure test, we illuminate the ColorChecker chart at a light level of 20 lux, then take exposures of 1, 5, 10, 15 and 30 seconds length in the most accurate color mode (as determined in the color test). We then repeat the process with long exposure noise reduction on, and analyze the images in Imatest, evaluating image noise levels and color accuracy. Our score for this section is based on both; if a camera has low noise and good color accuracy across all tested shutter speeds, it will get a high score.

White Balance
For SLRs only

We test two aspects of white balance; the performance of the camera in auto mode, and the performance of the camera using the custom white balance feature it offers. In both cases, we take photos of the ColorChecker chart using three light sources in our X-Rite Judge II lighting booth; daylight, incandescent and fluorescent. For the auto white balance test, the camera is set to auto mode and the chart is photographed. For the custom white balance test, we use the custom white balance process that the manufacturer recommends on an 18% grey card. This process varies, but usually involves taking a photo of a white or grey object, then allowing the camera to process the image, derive the required white balance setting and use this for color balancing images.

When the images are captured, we analyze them with Imatest, which produces a figure for the white balance error in degrees kelvin. The higher the color temperature error, the lower the camera's white balance score. In both tests, the ideal is that the camera will be able to accurately judge the color temperature of the light source and compensate for it, leading to a lower white balance error and more accurate whites and colors. Our scoring is based on the average color error for all three light sources. In previous versions of our reviews, we tested the white balance presets. We have eliminated this test.

Resolution

For SLRs

To test resolution, we illuminate an SFRPlus chart at an even 3000 lux, then take photos of the chart at three zoom settings; the widest, the midpoint and the telephoto end of the zoom range. At each of these points, we also take shots at the widest, mid and smallest aperture. Each of these photos is then processed in Imatest, which uses the slanted edges of the blocks on the SFRPlus chart to analyze the sharpness and chromatic aberration of the image. This analysis is done at nine places on the image: at the center, four places at the corners of the image and four points in between. We analyze the sharpness and chromatic aberration figures for both horizontal and vertical orientations at all of these places, as shown below; the red areas are those that Imatest analyzes.

Sfr rois

This produces a large number of results that provide us with a good overall picture of how well the camera performs across the image. When this is combined with the images we take across both the zoom and aperture range, we get a detailed overview of the performance of the camera across the entire range of zoom and aperture settings that it offers, as well as how the sharpness and aberration change across the image at each point.

All of these figures are used to produce the overall resolution score. To create this score, we factor in the highest sharpness (which is usually at the center of the image) and how much the sharpness falls off at the other points. This means that, to earn a high score, a camera must have good sharpness and keep it across the image; if it falls off significantly at the edges, the camera will get a lower score. To evaluate chromatic aberration; we look at the lowest aberration (again, this is usually at the center of the image) and how much it increases at the edges. So, to get a high score, a camera should have low aberration across the entire image.

We also factor in how the sharpness and aberration hold up across the zoom and aperture range; to earn a high score, a camera must have high sharpness and low aberration across both ranges.

Imatest uses the thick black bars at the bottom and top of the image to work out how much the image is distorted.  For SLRs, we do not score on this factor.


For point-and-shoots

To test resolution, we illuminate an SFRPlus chart at an even 3000 lux, then take photos of the chart at 3 zoom settings; the widest, the midpoint and the telephoto end of the zoom range. At each of these points, we also take a single shot, letting the camera decide the aperture. Each of these photos is then processed in Imatest, which uses the slanted edges of the blocks on the SFRPlus chart to analyze the sharpness and chromatic aberration of the image. This analysis is done at nine places on the image: at the center, four places at the corners of the image and four points in between. We analyze the sharpness and chromatic aberration figures for both horizontal and vertical orientations at all of these places, as shown below; the red areas are those that Imatest analyzes.

Sfr rois

This produces a large number of results that provide us with a good overall picture of how well the camera performs across the image. When this is combined with the images we take across both the zoom and aperture range, we get a detailed overview of the performance of the camera across the entire range of zoom and aperture settings that it offers, as well as how the sharpness and aberration change across the image at each point.

All of these figures are used to produce the overall resolution score. To create this score, we factor in the highest sharpness (which is usually at the center of the image) and how much the sharpness falls off at the other points. This means that, to earn a high score, a camera must have good sharpness and keep it across the image; if it falls off significantly at the edges, the camera will get a lower score. To evaluate chromatic aberration; we look at the lowest aberration (again, this is usually at the center of the image) and how much it increases at the edges. So, to get a high score, a camera should have low aberration across the entire image.

We also factor in how the sharpness and aberration hold up across the zoom and aperture range; to earn a high score, a camera must have high sharpness and low aberration across both ranges. We also take into account if the camera over-sharpens the image dramatically or not. Sharpening is used to make the edges of objects look crisp, but if it's applied too heavily, they will look jagged, and have discoloration around them. We penalize the resolution score if the camera over-sharpens.

Imatest uses the thick black bars at the bottom and top of the image to work out how much the image is distorted. Since the lens can't be changed on a point-and-shoot camera, this becomes an important factor in the overall score (unlike our SLR reviews, which do not score for distortion).

Dynamic Range
For SLRs only

To test dynamic range, we take a number of photos of the Kodak Q-13 chart at a number of exposure levels, then process these through the dynamic range test in Imatest, which analyzes the images to find the maximum overall range that the camera is capable of capturing. We also test the dynamic range at all of the ISO levels that the camera offers, except those that the manufacturer does not assign a specific ISO value to (such as the Lo and Hi modes of some Nikon cameras).

The wider the better is the rule with dynamic range, so our scoring is based on both the widest dynamic range that the camera can capture, as well as how the dynamic range holds up as the ISO increases. So, to get a high score, a camera should have a wide dynamic range at low ISO levels, and also maintain this as much as possible at higher ISO levels.

Image Stabilization

Human beings are not very good at holding things still; their hands shake, and this shake means that the camera you are holding moves slightly. And this means blurry pictures, especially if you are shooting at a slow shutter speed, where the camera has more time to move. So, most modern cameras include a feature that tries to get rid of this shake, either by increasing the shutter speed or by detecting the shake and moving a part of the lens or the sensor to compensate.

Zaber2
Our image stabilization testing system

For SLRs

We test this feature by shaking the camera in a precise, controlled way using our Zaber linear slide and rotary stand and shooting a large number of images at a number of different shutter speeds, ranging from 1/500 of a second down to 1/8 of a second. We shoot with the Image Stabilization off, with it on and with any different modes that the camera offers. We then run all of the images through Imatest, analyzing the sharpness of the image, and do a complex statistical analysis of the results that tells us what the difference between stabilization being on and off is. From this, we can determine how well the image stabilization works, and how much improvement you are likely to see in photos.

We then plot the results on a graph to show how the stabilization works across the range of shutter speeds; some stabilization systems are more effective at certain shutter speeds, and some actually make things worse.

Horizontal: Sharpness Vs Shutter Speed

Canon 5dmkii is low horiz

Vertical: Sharpness Vs Shutter Speed

Canon 5dmkii is low vert

The percentage on the Y axis of the sample graph above is the sharpness (the measured MTF50) of the images, with 100% being the maximum sharpness that the camera managed in this test. For more information on MTF50, see here.  So, if a camera achieved a maximum MTF50 of 1500, that would be the maximum on the graph. If, in one of the tests, we measured the sharpness at 1/125 of a second at 750, that would be 50%. We use a percentage so the values can be compared across cameras.

The next graph that we prepare shows the sharpness improvement that the image stabilization system made as a percentage. If our test camera had a sharpness of 500 with image stabilization off and a sharpness of 750 with it on, that would mean that the image stabilization gave a 50% improvement in sharpness. If both tests produced a sharpness of 500, the improvement would be 0%.

The scoring for this test is based on how much improvement the image stabilization adds across the entire range of shutter speeds; a camera that produces sharper images across the entire shutter speed range would score higher than one that only had an improvement at one shutter speed, or one where the stabilization made it worse. 

For point-and-shoots

Due to the lack of manual control on many point-and-shoot cameras, we test this feature in a more limited scope. We illuminate a chart to approximately 100 lux. If the camera allows manual shutter speed control, we set it to 1/30 second. If not, we adjust the lighting to get an exposure as close to 1/30 of a second as possible. We shoot with Image Stabilization off, with it on, and then run all of the images through Imatest, analyzing the sharpness of the image, and do a complex statistical analysis of the results that tells us what the difference between stabilization being on and off is. From this, we can determine how well the image stabilization works, and how much improvement you are likely to see in photos.

Noise

Noise is the annoying stuff that ruins images; the same static that you see on a detuned TV sneaks into your image and blocks detail and affects the colors. We test noise by photographing the ColorChecker chart, and analyzing the images in Imatest.

For SLRs

These tests produce accurate readings for the noise in the images, with figures for noise in the red, green, blue and yellow color channels, plus the luminance noise (the noise in pure grey parts of the images). We repeat this test at all of the ISO levels that the camera offers, and with all levels of high ISO noise reduction, and analyze how the noise increases as the ISO level increases. For the noise score, we factor in both the noise level at low ISO levels, and how much this increases as the ISO level increases. To get a high score, the camera must have low noise, and keep the noise low as the ISO increases.

For point-and-shoots

For point-and-shoot cameras we shoot at every available full-resolution ISO under 3000 and 60 lux of illumination, to see if noise varies widely between normal and low-light performance. The score for this section is based on both light levels, across a wide range of ISOs that are found on all models. The camera needs to keep low image noise across all ISOs to perform well in this section.

The Video Tests

All digital SLRs that offer video recording are tested and analyzed by the staff of Camcorderinfo.com, under exactly the same conditions as camcorder reviews. A full description of all the test procedures is below.

For testing point-and-shoot cameras, we run fewer tests, but use the same procedures. In these situations, only Color (Bright Light) and Video Sharpness are tested and scored on.

Color (Bright Light) Camcorderinfo logo brought250

Dsc chromadumonde250 To determine the video mode's color accuracy in bright light we shoot the Chroma DuMonde chart, lit at an even 3000 lux. The camera is set in fixed position behind a small opening in a Duvetyn curtain. Cameras are shot in auto mode with a manual white balance, when possible. The highest possible quality mode is the standard. Depending on the complexity of the product, additional shooting modes may be tested and described in the body of the review.

Video footage is imported to a computer, where a number of high-quality bitmap frame grabs are generated. Final Cut Pro is used in the process, unless the file formats prove incompatible. We then import the bitmap images into Imatest and run the Multicharts module. Imatest scans all 28 color patches on the Chroma DuMonde chart, analyzing the colors produced by the camera against the known color values. The CIEDE2000 color error formulas are applied, calculating the color error. The better the performance, the smaller the color error.

The chart shown in the review is referred to as the color error map. It shows the difference between the known value of each color patch (the square) and what the camera produced (the circle). Longer tails indicates a greater error. The direction of the error is also significant, indicating under and oversaturation, and the camera's tendencies to warm or cool the image.

The final score for the the Color (Bright Light) section is based on the average color error under auto mode with a manual white balance.

Noise (Bright Light) Camcorderinfo logo brought250

To determine video mode noise in bright light, we shoot an X-Rite ColorChecker chart (lit at an even 3000 lux). The cameras are in auto mode with a manual white balance. Currently, we do not factor in manual sensitivity/ISO settings or noise reduction settings. They may be tested ad hoc, but are not factored into the score.

To better illustrate our findings, we publish 100% crops of the Chroma DuMonde chart. These crops of the Chroma DuMonde are not used in the scoring. However, we find that this chart can be more illustrative than the X-Rite ColorChecker chart in regards to how the human eye might perceive noise.

After the footage is shot, it is transferred to a computer and bitmap frame grabs are created. We import those bitmaps to Imatest, where the noise scores are calculated. The noise performance is an average of the four channels: red, green, blue, and luminance. Smaller noise percentages indicate better performance.

The final score for the Noise (Bright Light) section is based on the noise average of all four channels, according to the performance in auto mode after a manual white balance.

Motion Camcorderinfo logo brought250

Trainstation250 It's a difficult thing to define the quality and characteristics of video motion. Unlike color, noise, and sharpness, there is no standardized metric. (Conversely, there are methods of quantifying motion performance on a television, but these methods only analyze the playback itself, and not the source material.)

To tackle the problem, we shoot a custom-designed 'moving still life scene,' sometimes referred to as 'the Train Station' in our reviews. The tableau contains a handful of simple and complex types of motion. Each element moves at a fixed speed, allowing for direct cross-comparisons between models. We shoot at every available frame rate that the camera offers.

The video clips are then imported to a computer and readied for upload to YouTube. To obtain the best results, we use QuickTime, converting the clips to 1280 x 720 in the .MOV format at an 8000kbps bit rate. Clearly, these re-compressed files do not match the quality of the original, unconverted video. Unfortunately, our traffic would simply be too much to bear if we hosted uncompressed clips for download. That plan is being considered as a possibility for the future.

The final score for Motion is based on qualities of trailing, artifacting, smoothness, and high frequency detail.

Video Sharpness Camcorderinfo logo brought250

Dsc multiburst250 Actual video sharpness is never the same number that manufacturers advertise. Cameras recording at '1920 x 1080' do not actually capture one thousand nine-hundred and twenty horizontal lines of information. That number is just the size of the 'container' that the camera outputs (also known as the resolution).

To get a more accurate assessment of video sharpness, we shoot a DSC Labs Multiburst chart at an even 3000 lux. The camera is placed in a fixed position, aimed to align with the chart's 16:9 guideframes. The camera is then slowly panned left and right for approximately 30 seconds. The camera is reset, then tilted slowly up and down for about 30 seconds.

We test sharpness with the camcorder in motion, rather than a static shot, for a simple reason. How often do your video clips contain no motion? Motion is inherent in the nature of video. It's not the method that manufacturers would prefer, but we think it makes sense.

Once the shooting is done, the camera is connected to an HDTV with the best available connection (typically component out or HDMI). The playback footage is examined several times, looking for the point at which the lines on the chart blur and become indistinguishable. The left/right and the up/down clips are used to determine the horizontal and vertical sharpness, respectively.

The final score for the Video Sharpness section is based on the horizontal and vertical sharpness, based on shooting in auto mode at the highest resolution frame rate available. Other frame rates may be examined, but do not factor into the final score.

Color (Low Light) Camcorderinfo logo brought250

Dsc chromadumonde250 Low light color accuracy in video mode is tested much the same as bright light color test, with a few changes. We light the Chroma DuMonde chart at an even 60 lux. At this light level you could read a book, but not for long periods with any comfort.

We take a manual white balance reading, then shoot in auto mode. Any auto gain/ISO adjustments are left in the 'on' position. In most instances, there is no way to turn it off, which is the reason for making this the standard. If a camera offers additional gain/ISO settings, we may examine them, but they are not factored into the final score. Also, we may test in multiple frame rates, if available, but scores are based on the highest resolution setting.

Video footage is imported to a computer, then bitmap frame grabs are generated and run through the Imatest software. Imatest identifies the difference between the known value and what the camera actually produced for each of the 28 color patches. Smaller color errors mean better performance.

The final score for Color (Low Light) in video mode is based on the average color error.

Noise (Low Light) Camcorderinfo logo brought250

The low light noise in video mode is tested in a similar manner to how we test bright light noise. The X-Rite ColorChecker chart is lit at an even 60 lux. Cameras are shot in auto mode in the highest available resolution setting, with a manual white balance. If alternate frame rates or sensitivity settings are available, we may shoot in those modes.

The video footage is imported to computer, then bitmap frame grabs are generated and run through Imatest for analysis. The smaller the noise percentage, the better the performance.

The final score for the Noise (Low Light) section takes the average noise for all four channels.

Sensitivity (Low Light) Camcorderinfo logo brought250

The low light sensitivity examines how much light a camera's video mode needs to produce a decently exposed image. That threshold is determined according to the camera's outputted exposure levels, measured in IRE levels. The IRE range extends from 7.5 IRE at the darkest levels up to 100 IRE for the brightest exposure (at least for those countries on the NTSC standards).

The test begins with the Chroma DuMonde chart lit to approximately 60-80 lux. The camera's signal is outputted to a waveform monitor, providing real-time data about the camera's IRE levels. We begin recording, then slowly dim the light pointed at the chart, constantly checking the lux levels with a light meter. The light is steadily reduced until the waveform monitor indicates a maximum exposure of 50 IRE from the camera's signal. 50 IRE was chosen as our standard because it's the mid-point of the IRE range, and the point at which a camera can capture a fair amount of detail in the chart. The picture quality would generally be darker than ideal, but good enough. Anything exposed darker than this might be considered too compromised.

In order to remove the chance of possible data degradation when testing live (due to poor quality connections, etc.), we transfer the recorded footage to a computer and run the test again on the original footage.

The final score is based on by the lux level the camera needed to produce a maximum exposure of 50 IRE. The test is shot using the highest resolution available on the camera. Other settings and frame rates may be tested, but are not factored into the final score.