Equipment

Sigma 24-35mm f/2 DG HSM Art MTF Curves and Sample Variation

By
Published September 10, 2015

For years now, I’ve been telling people to always take the results of tests done on one copy of a lens with a serious grain of salt. Not matter how carefully it was tested, it’s still just one copy. Others will be worse. Others will be better.

Because I’m so aware of copy variation, I (in theory at least) test at least 10 copies of a lens and average those results before I present them. I do that because it gives a much better idea of what the average copy will be like.

Sometimes, though, it seems the true purpose of my life is to serve as a warning to others. I just can’t pass up an opportunity, don’t listen to myself, give in to temptation, and test just one copy of a lens. I justify it to myself because sometimes one copy is all I’m going to get, at least for a good while.

So today I’m going to use myself as an example of why testing one copy of a lens is just a bad idea. While doing so I’ll eat some crow and also apologize to Sigma. Because a few weeks ago I published the MTF charts on one copy of the Sigma 24-35 f/2 DG HSM Art zoom that I was loaned for a day and published the results. After that Lensrentals got multiple copies in stock and I did my usual testing of 10 copies. It is very apparent that that first copy was the worst of the bunch. The other copies, each and every one, were better.

Sigma 24-35 f/2.0 MTF

I’m going to put our original results from one single copy we tested earlier on the left, the average of 10 other copies on the right, so that you can see how different the average results are compared to the original single test lens.

Roger Cicala, Olaf Optical Testing, 2015
Roger Cicala, Olaf Optical Testing, 2015
Roger Cicala, Olaf Optical Testing, 2015

 

Again, I want to emphasize that the MTF charts on the right are our official average results from testing 10 copies of the lens. I just put the original single test lens results on the left up so you can see the difference.

This gives an excellent example of why single-lens test results can be deceiving. If you notice, the single lens had excellent center resolution, a bit better than average.  But off-axis (away from center) the test lens had a LOT of astigmatism, much worse than the average lens. In fact, it was worse than any of the 10 copies we ran for this test.

The average MTF curves for the Sigma zoom actually do (as Sigma claims) compare very well with prime lenses. Below are comparisons between the Sigma 24-35 f/2 and the Canon 24mm f/2.8 and 35mm f/2 IS lenses. The zoom is certainly right there with the two Canon primes.

Roger Cicala, Olaf Optical Testing, 2015
Roger Cicala, Olaf Optical Testing, 2015

 

 Copy-to-Copy Variation

Now that we have our 10-copy testing done we can generate consistency numbers and variation graphs for the Sigma lens. These are really quite good, comparing very favorably to what we saw from Canon and Nikon 24-70 f/2.8 zooms.

 

Roger Cicala, Olaf Optical Testing, 2015
Roger Cicala, Olaf Optical Testing, 2015
Roger Cicala, Olaf Optical Testing, 2015

 

Conclusion

Looking at the results of the Sigma zoom now that we have tested a number of copies, it’s very obvious Sigma has done it again. They’ve broken new ground and made a zoom that is a full stop wider than other zooms, yet is optically on a par with similar aperture prime lenses. And they’ve maintained excellent copy-to-copy consistency while doing it. Superbly done, yet again, Sigma.

 

Roger Cicala, Brandon Dube and Aaron Closz

Lensrentals.com

September, 2015

 

Addendum: I know many of you want to know if I have an explanation as to why the first lens was the worst lens. I can’t say for certain, it was loaned to us for just one day, but I would guess that it may have gotten knocked around during use or shipping. The problem it had was subtle. Center resolution was still excellent and there were no clear signs of decentering (we checked carefully on some pretty expensive equipment as well as the usual test targets). It’s a good example of why it’s so difficult to decide, using just one copy, if a lens is as good as it can be or not. Even out of sorts, the Sigma was still excellent, about as good as we’d expect an excellent zoom to be. It just wasn’t as good as the other copies, which are better than we expect an excellent zoom to be.

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Tweet
Posted in Equipment
  • iKonOkLasT

    Some of these plots need to be revised in light of later reviews. Variance plots on a scale from 0 to 2? The results of that single lens should have created much greater variance bands than plotted here.

  • I don’t know if this will be seen since it’s an old post, but it would be useful if you added a link to this post in the original single copy analysis from July 2015–just in case people are searching and end up at the old post rather than this corrected one.

  • Carl

    Perhaps you can update your original post with a link to this one? It’s a shame as many, including myself, shied away from purchasing the Sigma because of the atypical astigmatism.

  • Roger Cicala

    CarVac, they have them, but I think they’re just backed up a bit.

  • CarVac

    These still haven’t been put up on The-Digital-Picture yet for comparison.

  • Brandon

    Frank, the astigmatism of the early sample isn’t caused by a tilt or decenter so to speak, as it was pretty highly symmetrical. A spacing error front-to-back can hugely affect the astigmatism, which is what I suspect happened to that sample. Perhaps that will be a consistent issue with this model, but I doubt it given the track record of the other “Art” series lenses.

    It probably was just an extreme outlier.

    Brandon

  • Frank Kolwicz

    The difference between that first example and the 10 brand new(?) ones may speak more to the durability of the lenses than anything else. I’ll be watching for reports on how they hold up to a year’s worth of rental abuse.

  • Roger Cicala

    EL we buy from a number of retailers, but basically off the shelf stock. We don’t buy lenses directly from manufacturers, although we do buy other things directly.

    Anyway, while a manufacturer might cherry pick a few lenses to go to reviewers, I’m pretty comfortable that none has the capabilities of cherry picking 100 copies to go to a lens rental house.

  • EL

    Roger,

    Do you buy lenses directly from manufacturers or from commercial retailers? I’m somewhat curious, given how important your data and reviews are to influencers within the camera community, whether lens manufacturers might start “binning” lenses, i.e., selecting the best copies to ship to you, in order to get better reviews from you. That would be no bad thing for your business or your customers, but it might skew the data more favorably.

  • Roger Cicala

    Car Vac, that’s a good idea. I’m forwarding that.

  • Roger Cicala

    Andrew, we’ve considered much of that. With some high-end lenses we do include an MTF chart with the lens at LA, but MTF testing is expensive and for most lenses it hasn’t made any difference in sales (when we’ve listed lenses both with and without MTF tests it hasn’t mattered much in how quickly they sell – people are much more worried about if there’s a scratch on the front element or how much dust is inside 🙂

  • Andrew

    Do you have plans to sell pre-tested lenses with MTF results for that particular copy included in the package? A lot of people would be interested, I’m sure.

    Another service could include sending to LensAutority client’s own camera body to find best matching lenses, or vice versa.

    Like “OK, we tested your 5Dsr with 20 samples of 24mm f/1.4, would you prefer better center peak performance or better sharpness across the frame? Do you plan to use it mostly for interiors, landscape or astrophotography? Results for 5 best copies attached to the email, please pick which one will be yours.”

  • CarVac

    Wow, what a lens. Fascinating that the first copy happened to be exceptionally bad. If it was loaned to you, then clearly somebody else used it (for a review?) too and might have the wrong impression of the lens.

    One question as an aside: would you be able to link to relevant blog posts from the individual lens pages?

  • brandon sundt

    dear randy,
    I think you skipped the part where Roger says that the charts on the right are an average of ten lenses, and how the lens copy variation numbers are good.

  • Randy

    Thanks, Roger but I wouldn’t be too quick to disregard your first impressions. I know Sigma is supposed to have turned over a new leaf with these Art lenses but as someone who used to sell Sigma/Tamron/Tokina/Vivitar lenses, Sigma variation was pretty extreme. Lens design and optical quality is one thing; consistency is something else. Since most of us can’t easily compare multiple lenses it’s nice to know a brand is really on top of quality control.

  • Roger Cicala

    Joshua, by our protocols any lens off from the average by that much would have been removed from service.

  • Roger Cicala

    Chris, I can’t say. It was a while ago and we didn’t even write the serial number down, just Copy1 on the test files. I would guess that it probably had more to do with that particular lens having made the rounds from place to place before it got to us. We see our own lenses get knocked out of sorts by shipping every once in a while, for sure.

  • Roger Cicala

    Andre, they would have definitely been worse, but I didn’t plug it into the calculations.

  • For years now, I’ve been begging you guys to do multiple copy reviews so I don’t have to take results of tests with a serious grain of salt. Thanks again!

  • Joshua

    I’m curious what would have become of the the first, poor-testing loaner lens if it had been one of the lenses procured for normal service. Would it have been included in the rental pool?

  • Chris Knight

    I am curious if the loaner was an early production unit (low serial number). If you happen to have pics of the lens from when you had the loaner, check the serial number and check it against your batch of 10.

  • Thanks Roger and Brandon for following up on the original report. If you had put the bad sample in with the rest, how much would the consistency scores have changed?

Follow on Feedly