Lenses and Optics

Notes on Lens and Camera Variation

Published October 2, 2011

A funny thing happened when I opened Lensrentals and started getting 6 or 10 copies of each lens: I found out they weren’t all the same. Not quite. And each of those copies behaved a bit different on different cameras. I wrote a couple of articles about this: This Lens is Soft and Other Myths talked about the fact that autofocus microadjustment would eliminate a lot, but not all, of the camera-to-camera variation for a give lens. This Lens is Soft and Other Facts talked about the inevitable variation in mass producing any product including cameras and lenses: that there must be some real difference between any two copies of the same lens or camera.

A lot of experienced photographers and reviewers noted the same things and while we all talked about it, it was difficult to use words and descriptions to demonstrate the issue.

And Then Came Imatest

We’ve always had a staff of excellent technicians that optically test every camera and lens between every rental. But optical testing has limitations: it’s done by humans and involves judgement calls. So after we moved and had sufficient room, I spent a couple of months investigating, buying, and setting up a computerized system to allow us to test more accurately. We decided the Imatest package best met our needs and I’ve spent most of the last two months setting up and calibrating our system (Thank you to the folks at Imatest and SLRGear.com for their invaluable help).

It has already proven successful for us, as it is more sensitive and reproducible than human inspection. We now find some lenses that aren’t quite right, but that were perhaps close enough to slip past optical inspection. Plus the computer doesn’t get headaches and eyestrain from looking at images for 8 to 10 hours a day.

Computerized testing has also give me an opportunity to demonstrate the amount of variation between different copies of lenses and cameras. We have dozens (in in some cases dozens of dozens) of copies of each lens and camera. While we don’t perform the multiple, critically exact measurements that a lens reviewer does on a single copy, performing the basic tests we do on multiple copies demonstrates variation pretty well.

Lens-to-Lens Variation

We know from experience that if we mount multiple copies of a given lens on one camera, each one is a bit different. One lens may front focus a bit, another back focus. One may seem a bit sharper close up, another is a bit sharper at infinity. But most are perfectly acceptable (meaning the variation between different copies is a lot smaller than the variation you’re likely to detect in a print). I can tell you that, but showing you is more effective.

Here’s a good illustration, a run of 3 different 100mm lenses, all of which are known to be quite sharp: the original Canon 100mm f/2.8 Macro, the newer Canon 100mm f/2.8 IS L Macro, and the Zeiss ZE 100mm Makro. The charts shows the highest resolution (at the center of the lens) across the horizontal axis, and the weighted average resolution of the entire lens on the vertical axis, measured in line pairs / image height. All were taken on the same camera body and the best of several measurements for each lens copy is the one graphed.

Resolution of multiple copies of several 100mm lenses

It’s pretty obvious from the image there is variation among the different copies of each lens type. I chose this focal length because there was a bad lens in this group, so you can see how different a bad lens looks compared to the normal variation of good lenses. As an aside, the bad lens didn’t look nearly as bad as you would think: if I posted a small JPG taken with it, you couldn’t tell the difference between it and the others. Blown up to 50% in Photoshop, though, the difference was readily apparent.

My point, though, is while the Canon 100mm f/2.8 IS L lens is a bit sharper than the other two on average, not every copy is. If someone was doing a careful comparative review there’s a fair chance they could get a copy that wasn’t any sharper than the other two lenses. I think this explains why two careful reviewers may have slightly different opinions on a given lens. (Not, as I see all too often claimed on various forums, because one of them is being paid by one company or another. Every reviewer I know is meticulously honest.)

Autofocus Variation

We all know camera autofocus isn’t quite as exact as we wish. (Personally, after investigating how autofocus works for this article, I’m amazed that it’s as good as it is, but I still complain about it as much as you do.) But when I started setting up our testing, I was hoping we could use autofocus to at least screen lenses initially. The results were rather interesting. Below is the same type of graph for a set of Canon 85mm f/1.8 lenses I tested using autofocus. Notice I again included a bad copy as a control.


Test run of a dozen Canon 85mm f1.8 lenses (and one known soft copy) using autofocus

(For those of you who are out there thinking “I want one of those top 3 copies, not one of the other ones”, and I know some of you are, keep reading.)

Then I selected one copy that had average results (Copy 7), mounted it to the test camera, and took 12 consecutive autofocus shots with it. Between each shot I’d either manually turn the focus ring to one extreme or the other, or turn the camera off and on, but nothing else was moved. (By the way, for testing the camera is rigidly mounted to a tripod head, mirror lock up used, etc.)

In the graph below, overlaid on the original graph, the dark blue diamond shapes are the 12 autofocus results from one lens on one camera. Then I took 6 more shots, using live view 10x manual focus instead of autofocus, again spinning the focus dial between each shot. The MF shots are the green diamonds. I should also mention that when I take multiple shots without refocusing the results are nearly identical – that would be a dozen blue triangles all touching each other. What you’re seeing is not a variation in the testing setup, it’s variation in the focus.

Copy 7, repeatedly autofocused (blue diamonds) and manually focused (green triangles)

It’s pretty obvious that the spread of sharpness of one lens focused many times is pretty similar to the spread of sharpness of all the different copies tested once each. It’s also obvious that live view manual focus was more accurate and reproducible than autofocus. Of course, that’s with 10X live view, a still target, and a nice star chart to focus on and all the time in the world to focus correctly. No surprise there, we’ve always known live view focusing was more accurate than autofocus.

One aside on the autofocus topic: Because it would be much quicker for testing, I tried the manual versus autofocus comparison on a number of lenses. I won’t bore you with 10 more charts but what I found was that older lens designs (like the 85 f/1.8 above) and third party lenses had more autofocus variation. Newer lens designs, like the 100mm IS L had less autofocus variation (on 5DII bodies, at least – this might not apply to other bodies).

Oh, and back to the people who wanted one of the top 3 copies: when I tested two of those repeatedly, I never again got numbers quite as good as those first numbers shown on the graph. The repeated images (including manual focus) were more towards the center of the range, although they did stay in the top half of the range, at least on this camera, which provides me an exceptionally skillful segue into the next section. (My old English professor would be proud. Not of my writing skills, but simply that I used segue in a sentence.)

Camera to Camera Variation

Well, we’ve looked at different lenses on one camera body, but what happens if we use one lens and change camera bodies? I had a great chance to test that when we got a shipment of a dozen new Canon 5D Mark II cameras in. First, I tested a batch of Canon 70-200 f2.8 IS II lenses on one camera, using 3 trials of live view focusing on each. The best results for each lens are shown as green triangles.

Then I took one of those lenses (mounted to the testing bench by its tripod ring) and repeated the series on 11 of the new camera bodies. The blue diamonds and red boxes this time each represent a different camera on the same lens. (4 test shots were taken with each camera, and while the best is used, each camera’s four shots were almost identical.) Obviously the same lens on a different body behaves a little differently.

A group of Canon 70-200 f2.8 IS II lenses tested on one body (green triangles) and one of those lenses tested on 11 brand new Canon 5DII bodies (red squares and blue diamonds).

I separated the cameras into two sets because we received cameras from two different serial number series on this day. I don’t know that conclusions are warranted from this small number, but I found the difference intriguing. And maybe worth some further investigation.


Notice I don’t say conclusion, because this little post isn’t intended to conclude anything. It simply serves as an illustration showing visually what we all (or at least most of us) already know:

  • Put different copies of the same lens on a single camera and each will vary a bit in resolution.
  • Put different copies of the same camera on a single lens and each will vary a bit in resolution.
  • Truly bad lenses aren’t a little softer, they are way softer.
  • Autofocus isn’t as accurate as live view focus, at least when the camera has not been autofocus microadjusted to the lens.

All of this needs to be put in perspective, however. If you go back to the first two charts, you’ll notice the bad copies are far different than the shotgun pattern shown by all the good copies. And when we looked at those two bad copies, we had to look fairly carefully (looking at 50% jpgs on the monitor) to see they were bad.

The variation among “good copies” could probably be detected by some pixel peeping. For example if you examined the images shot by the best and worst Canon 100 f2.8 IS L lenses you could probably see a bit of difference if you looked at the images side-by-side (the images I took on my test camera). But if I handed you the two lenses and you put them on your camera, they’d behave slightly differently and the results would be different.

So for those of you who spend your time worried about getting “the sharpest possible lens”, unfortunately sharpness is rather a fuzzy concept.

Roger Cicala


October, 2011



Matt’s comment made me realize I hadn’t talked about one obvious variable in this little post: how much of the variation is caused by the fact that these are rental lenses that have been used? The answer (at least for Canon prime lenses) is not much, if at all. For example the graph below compares a set of brand new Canon 35mm f/1.4 lenses tested the day we received them (red boxes) to a set taken off of the rental shelves (blue diamonds).

Comparison of stock 35mm f/1.4 lenses with new-from-box copies

Please note I make this statement only for Canon prime lenses. Zooms are more complex and I see at least one zoom lens that doesn’t seem to be aging well, but until I get more complete numbers to confirm what I think I’m seeing I won’t say more. I see no reason to expect other brands to be different, but at this point we’ve only been able to test Canon lenses (these tests are pretty time consuming and we have a lot of lenses).


Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • Arnout

    Hi Roger,

    I know this is an older post already, but I stumbled upon on of your history posts, and have been binge reading your blog since. I have a question about how you measure your lenses. You say: “All were taken on the same camera body and the best of several measurements for each lens copy is the one graphed.”. I was wondering why you decided to use only the best measurement? From doing experiments at school, my fist instinct would have been to perform like 10 measurements per lens, throw out any outliers, and take the average and standard deviation. That way I think you can be more sure that the variation you see between lenses is caused by differences between those lenses and not because of measuring errors. Have you tried this? How does the standard deviation in the measurement of one lens compare to the variation between lenses?

  • Rainer

    Generally speaking correct. When I spoke to the supplier who is responsible for warranty issues on a Lens he was only too eager to point me to read all this. he debated with me that it may be my camera instead of my Lens causing a substantial back focus. I was not there to debate or argue a point. I merely wanted to drop a bad copy lens off to have a substantial back focus issue fixed on this Lens. The Sigma 150-500mm just was not cutting it. Heck! I know all this stuff about slight variations from Camera to Lens and vice versa. I have been around long enough, got 8 other Lenses with only slight variations as to be expected.
    It beats the crap out of me when someone attempts to override commons sense with some tech jargon and try to tell you that a bad Lens is not the problem but the Camera is. I had this Lens on auto fine tune to minus 20 and it was still slightly back focussing. That is clearly a bad Lens and not a Camera. There is still a clear distinction between small focus variation versus a bad copy. I just got a bad copy and I am hoping it will come back fixed as it is the second time. I knew I should not have purchased a Sigma Lens!! Always felt they are too soft for me but it is the only reasonable priced lens at a 500mm range. Needless to say if all fails I will try and get my money back and buy something more decent even if it costs me double….

  • It’s actually a great and useful piece of information. I am happy that you just shared this useful information with us. Please stay us informed like this. Thank you for sharing.

  • I’d just like to let u know how much I learnt from your blog Dugg you.Hope 2 be back fast for some more good stuff

  • Roger Cicala


    It depends what the finished photo is (difference is readily apparent in a large print, hardly apparent on a web jpg.). But it’s about the stop of light and the ability to blur the out of focus areas mostly. Increasing vision is increasingly expensive – an f/1.2 lens is a very special, and expensive thing. But most photographers don’t need it.

  • Great article, I am sure you can answer my question on lenses.
    I am looking for a prime lenese for my Canon – about 85mm.
    I note from the regular data that the same FL lenes range from £650 to over £1200, the latter being the L series.

    Q. Why the difference in price and does it really make a difference that can be seen in the finished photo or is iot just about a stop of speed?


  • Scott

    Great article, Roger.

    Trying to get a sense of how much variability is represented by the cloud in your first graph, I have a question:
    Suppose I’m walking around my city in daylight, taking actual pictures. I mostly have the lens stopped down a bit, and I pay attention to using appropriate shutter speeds. Is there enough difference between the best and worst results in the cloud (not including the failed lens) to be consistently visible in (say) an 11 x 14 print?

  • Hendrik

    I second Jack C on
    “interested to see a similarly in-depth analysis of how Contrast Detect AF compares with Phase Detect AF in terms of accuracy and consistency of results”.

  • If you want to evaluate the sharpness of your own lenses, or you wish to calibrate the AF fine tuning of your DSLR body, you should try MTF Mapper


    This utility is totally free, and source code is provided. Or you could buy Imatest for $300-$5000 🙂

    Please read the user documentation thoroughly, though.

    I would appreciate any feedback, and welcome all discussion!

  • Roger Cicala

    Hi John,

    The weighted average uses Center point x 1, 6 mid points (which include the top and bottom center edges) x 0.75, and corners and lateral edges x 0.5. On bad corner does drag the weighted average down significantly, but more importantly we hardly ever see one bad corner: usually it’s a bad side (or top/bottom) and occasionally it’s contralateral corners both bad (certain kinds of tilt can cause it) and in such cases the weighted average is awful. But as far as our testing we flunk far more on the weighted average than on center sharpness. Not surprising, really, when you think about how much more a tilt or decentering will affect the outer area of the lens.


  • John Angulo

    For these plots, how do you compute the average resolution of the entire lens? You call it a “weighted average” – how is it weighted? In a simple unweighted average of multiple measurements across the image, a fuzzy corner due to decentering might not have much influence on the result, even if some of the measurements capture it. But a fuzzy corner or side is just what makes the difference between good and bad copies for critical purposes. Would it be possible to plot the lowest measured resolution on the vertical axis, whevever on the image that occurred? It might be interesting to see the scatter among different copies in this case.

  • Roger Cicala

    Thomas, we do check that and all of the zooms (and a few primes) vary a bit in the actual, versus stated, focal length. For example the Canon 24-70 zoom is actually 25mm to 67mm, the 70-200 is 74mm to 195mm, and the Sigma 50-500 actually is a 55mm to 470mm. I think the rule of thumb is they should be within plus or minus 5% but I’m not certain that’s always true.

  • Thomas Andre

    This isn’t the same issue, but is related. I have a Tamron 18-270 pzd and Canon 55-250 lens (for digital smaller sensors). At 18mm the Tamron matches, in size of image, my 18mm-55mm Canon kit lens (for a 20D). At full extension, however, the Canon at 250mm produces a more magnifed image than the Tamron at 270 mm. Have you every check the magnification accuracy?

  • dan

    If your new to nature photography–remember this; there’s a reason they give you a nice box on big canon lenses. That’s where they stay most of the time. Buy a long prime and a tripod and get outside and take pictures.

    Too many photographers scour the internet looking for a technical edge,
    and really don’t take many pictures. I used to be one of those boobs.

  • dan

    I personally feel we have been duped by Canon L lenses. Almost every
    good nature photograph I have seen has been using a tripod.
    I question the need for IS over any lens over 300mm, and the only reason the 300mm 2.8 is is so sought after is because Bob Atkins published his unscientific study for years. Canon should be paying Bob Atkins royalties.

    I don’t blame Bob, but his gullible readers are another thing. Maybe
    it’s human nature. The 300mm 2.8 seemed like the perfect lens.
    It used to be reasonably priced–not anymore.

    consumers won’t buy the most expensive lense, or the least expensive-
    it’s usually the one in the middle. The 300mm 2.8 was that lens–until
    the internet.

    Oh yea, people B and H is Not the only store in the world.

    When I saw the price of the 300mm 2.8 zoom from $4200.00 to
    close t0 8 grand; I started to boycott the company. It’s pure greed.

  • Roger Cicala


    I don’t have a blanket answer, but you are providing me a nice intro to an article I’m working on now: looking at how various lenses perform in relation to their age. It’s interesting because the one lens we see deteriorating over time is the 24-70 f2.8: the zoom linkage / seals tend to wear out. I assume (but don’t know for certain) it has to do with a heavy internal barrel that causes wear and tear with long=term use.

  • Robert S


    Thanks for a helpful article. How often should one send their lenses in for service? Recently, I noticed softness in some images and thought I needed to buy primes. I sent my 24-70 mm zoom and 5D Mii back to Canon and they came back noticably improved. Canon’s note that they repaired the mechanical linkage reminded me, as several writers above did, to follow the camera’s maintenance schedule?

Follow on Feedly