The Great 50mm Shootout

Every so often the Universe realizes I’m getting a little cocky and sends someone to ask me a simple question I can’t answer.  It happened again the other day. Michael Plumridge and Peter Lik asked Tyler what the sharpest 50mm lens was. They were shooting on Red Epics with adapters so neither brand, mount, nor price mattered. But they needed to know right away. Tyler told them he’d ask me and give them the answer in a few minutes.

But I didn’t have a clue. I have tons of data on 50mm lenses for Nikon and Canon cameras, but all of that was shot at widest aperture, which is great for finding lenses with problems, but not great for finding which is sharpest at f/4. And we had no factual data for Leica mount lenses because Rangefinder cameras are very difficult to test using programs like Imatest. But when someone like Peter Lik wants an answer and tells you Jim Jannard is curious, too, you get them an answer. So we tested every kind of 50mm lens we had available and got a pretty clear answer by the end of the day.

As always, my first thought when presented with a bunch of work to do is “what’s in it for me”? Turns out not much. Tyler got a beautiful signed picture that hangs in his office so I can see it every time I walk by. I got a nice thank you email. But, I figured if I did all that work, I’d at least  make a blog post out of it so I didn’t have to do a bunch of research over the holiday weekend. Wait, I mean, I was certain other people would be interested in what I found, so I thought I’d share the results.

What This Is, and Is NOT

This is the MTF50, measured by Imatest, of a lot of 50mm lenses to determine their relative sharpness. (Actually we’re determining resolution and acutance, but since using that word immediately loses half of all readers, I’ll stick with sharpness.) The working distance for our 50mm Imatest setup is about 15 feet, so actually we’re only measuring sharpness at that distance. The lenses might perform somewhat differently at infinity, or very close up.

It is not an attempt to determine the best lens. No real-world photographs are taken. Bokeh is not analyzed and compared. Color rendering, autofocus accuracy, chromatic aberration, build quality, flare resistance, and a dozen other real-world considerations aren’t considered at all. I also want to emphasize this little test is of one copy of each lens. My preference would be to test a dozen or more copies of each lens and average the results. But there wasn’t time to do that kind of testing.

Still, it’s an interesting test. Some of these lenses have never (to the best of my knowledge) had MTF data released before. It also gave us a chance to compare SLR lenses mounted on Canon and Nikon cameras with M mount lenses on an M9.

The Methods

We used our standard Imatest  testing protocol for the SLR cameras: SLR cameras were manually focused using Live View and a Seimen’s star chart. Several exposures were made with each lens at each aperture and the best result at each aperture used. M9s were focused with the camera’s rangefinder and then several focus bracketed shots taken. As with the SLRs, we used the best result at each aperture.

Two measurements were recorded: the MTF 50 at the center (peak resolution) and the weighted average of center, midrange, and corner sharpness. MTF 50 results are presented as Line Pairs / Image Height from completely unsharpened RAW images. (We’re starting a pool about how many emails I get asking “what do the numbers mean” because no one ever reads this stuff. The over-under is 32 if you want in.)

We used 3 cameras for this exercise, all of them test cameras that we have used extensively at Lensrentals: a Canon 5D Mk II, a Nikon D3x, and a Leica M9. From extensive (a few thousand) previous tests, we knew that the Canon and Nikon cameras have identical resolution as far as our testing can tell. (The D3x has more pixels and in theory should resolve a bit better, but if it does we can’t detect it.) We had no idea about how the M9 would compare: it has fewer pixels but its lack of an AA filter should compensate for that to some degree.

The lenses tested were as follows:

Canon: 50mm f/1.2; 50mm f/1.4; 45mm TS-E; 50mm f1.0 (that is correct, the 50mm f1.0)

Cooke: 50mm Panchro T2.8 (Cinema lens)

Leica: 50mm f/0.95 Noctilux; 50mm f/1.4 Summilux ASPH; 50mm f2.0 Summicron

Nikon: 50mm f1.4 G; 50mm f1.2 AIS (another classic); 45mm PC-E

Schneider: 50mm f/2.8 PCTS; 50mm Cine Xenar T2.0 (cinema lens)

Sigma: 50mm f/1.4

Voigtlander: 50mm f1.1 Nokton

Zeiss: 50mm f/1.4 (ZE/ZF); 5omm f/2.0 Makro planar (ZE/ZF); 50mm f/1.5 Sonnar (M mount); 50mm f/2.0 Planar (M mount)

Interpreting the Results

Sample variation

Whenever we look at numerical results from Imatest or similar testing there’s a tendency to get all absolute: lens A resolved 600 LP/IH and lens B resolved 570, therefore lens A is the sharper lens. In the image below I’ve graphed the average (mean) MTF 50 (the vertical axis is weighted average for the entire lens, the horizontal axis is spot MTF from the center of the lens) for a number of 50mm lenses we have tested extensively. Note these results are all at each lenses “wide open” aperture.

Looking at the graph the Zeiss f/2 Makro is obviously sharper, but it’s being shot at f/2, so that’s not a fair comparison. Just looking at these single numbers it’s tempting to draw conclusions like “the Sigma 50 has higher center resolution than the Nikon”, and “both the Sigma and Nikon at f/1.4 have higher resolution than the Canon at f/1.2”. Reality is not that simple. Below is a similar chart, but this time I’ll plot the actual data points for each copy tested. Reality is a lot messier than the single averaged number show above.

The takeaway message is when you are presented one data point, either in this article or in a lens review, it’s just that: one data point. In the top graph, the Sigma MTF50 is 620/510 for peak/average while the Nikon is 580/530 (rounded to the nearest 10 because that’s about as accurate as Imatest numbers are). But claiming the Sigma is really sharper in the center, or the Nikon has a better average sharpness is a stretch. There’s a bit of a tendency that way, but if I hand you one copy of one and one copy of the other you wouldn’t necessarily get that result. On the other hand, I’d be pretty comfortable that if I hand you either a Nikon or the Sigma to compare to the Zeiss 50mm f/1.4 you would notice a difference.

We’ve found that a difference of 75 LP/IH in peak resolution generally means there’s a real difference between the lenses. A difference of 150 certainly means there is difference and it’s usually quite obvious. For average (rather than peak) MTF 50 the range is smaller and we consider differences of 50 and 100 LP/IH signifiant. But this is a generalized rule-of-thumb we use in-house. It’s not a law of physics or anything.

How I graphed the results

I tried to graph the results to reflect this. The lens with highest MTF50 at each aperture has a dark box around it’s results and is colored red. Any lens within 75 LP/IH (for center MTF) and 50 LP/IH (for weighted average) of this number is also colored red, indicating there probably is no real-world difference between them.

Lenses between 75 and 150 LP/IH lower than the best result for peak MTF50 (between 50 and 100 LP/IH for average MTF50) are colored yellow. There is probably a real difference in sharpness between those lenses and the best lens, but not a large difference. Lenses that are more than 150 LP/IH for peak and 100 LP/IH for average MTF lower than the best result are colored blue. The difference between those lenses and the best lens should be readily apparent.

I should make it very clear that the colors are only pertinent within each table. That is why, for example, the Sigma 50mm f/1.4 rates a red box for weighted average at f/1.4 in Nikon mount, but an orange box in Canon mount: it’s basically as sharp as the best Nikon mount lens at those settings, but not quite as sharp as the Canon 50mm lenses.

The Results

The Canon Mount Lenses

There was not much difference in center sharpness between the two Canon lenses, the Sigma, and the Zeiss Makro Planar. As expected, the Canon 50mm f/1.0 wasn’t sharp (everyone knows it’s soft). I knew the Zeiss 50mm f/1.4 wasn’t very sharp wide open, but I was pleasantly surprised at how well it did by f/2.8.

I was rather surprised at the weighted average results, though. I had expected the Zeiss 50mm f/2.0 to be the best in the corners and edges. I shoot Canon with 50mm primes a lot and am always complaining about soft corners with the Canon f/1.2 and bragging on the corners of the Zeiss Makro-Planar. This test made me realize a couple of things. First, I usually shoot the Canon at f/1.2, so I’m mentally comparing f/1.2 images to f/2.0 images. Not a fair comparison. Second, when I shoot the Zeiss lens I use LiveView focusing which is a lot more accurate than autofocus. I tend to shoot the Canon on autofocus because I can. Just one more case of “what I know isn’t so”.

From a pure resolution standpoint, the two Canon lenses are the best corner to corner (on a test chart at 15 feet – obviously things can be different at other distances and with real world objects). As far as center resolution the Canon and Sigma are even at wide apertures, with the Zeiss their equal at f/2.0 (for the f/2.0) and f/2.8 for the f/1.4. The Schneider and Canon tilt-shift lenses didn’t compare as well: they just start coming into their own at f/4 and if we had extended the test range up to f/5.6 or f/8 they probably would have caught the others.

The Nikon Mount Lenses

The Nikon tests turned out about as I expected. (Unfortunately we no longer had any 50mm f/1.4 D lenses, which we thought were sharper in the center than the G). The Sigma and Zeiss f/2.0 were a bit better in the center, the Nikon G a bit better in average sharpness. The G has the reputation as one of the weaker Nikon prime lenses because it’s center sharpness isn’t great, but perhaps its designers gave up a bit of center sharpness to keep the corners sharp. The Nikon 45 PC-E, just like the Canon, really comes into its own about f/5.6 and doesn’t fare well at these apertures. (BTW – we only took the apertures to f/4 because we rarely see a wide aperture prime get significantly shaper past f/4.) To be honest, I had expected more from the f/1.2 AIS and had thought it might be the best lens in this group, but for this copy, at least, it wasn’t so.

 

The Cine Lenses

Just because they were possible mounts to the Red Epic, and because we had no idea as to how sharp they might be, we tested a couple of $10,000 Cine lenses. (Before you ask, we didn’t test Zeiss CP.2 lenses because they are optically identical to the ones already tested above.) Since these are PL mount lenses we had to test them on a Hot Rod modified Canon 7D, the highest resolution camera we have in PL mount. Obviously that should give these lenses an advantage: they’re only covering a crop frame rather than a full frame.

Even with that advantage, they didn’t really resolve as well as the photo lenses. Cinema lenses have never been designed for absolute resolution, they have other priorities. So for all of you thinking about buying a set of Cooke Panchros for still photography, I’ve just saved you $50,000. Makes it worth reading my blog, doesn’t it?

The M-mount lenses

This was the part of the testing about which I was totally clueless., but also the part about which I was most excited. M-mount lenses are about to explode since they’re going to be shot on Sony NEX-7s and RED cameras a lot in the near future. And while we think they’re excellent lenses it’s difficult to find actual test results with them.

I will mention (for those of you thinking of doing this at home) that performing Imatest on a Rangefinder camera is, at the very least, time consuming and frustrating. Lots of patience  and focus bracketing are necessary. It took longer to do each of the M mount lenses than it took to do all of the Canon and Nikon mounts. An Eye-Fi card would have been a good idea since we had to unmount the camera from the tripod to remove the card for every image run.  Of course, we thought of that right after we finished testing.

In one way the results weren’t shocking. I expected the Leica f/1.4 ASPH would be the sharpest, but I was shocked at how amazingly good it was. I was certain the $10,000 Leica 0.95 would not be  as sharp — you can’t get that wide aperture without some other compromises — but was impressed with how much better it performs than the Canon 50mm f/1.0. The Voigtlander was a pleasant surprise (but then I had low expectations) and while the Zeiss lenses seem a bit disappointing it’s largely because of the fast company they are keeping here. Their results, at f2.8 and f/4 at least, compare well with the SLR lenses.

Overall Comparison

There are some limitations to my attempt to answer the original question of course. Until we can test on a common body, we don’t know exactly how much difference being mounted to an M9 makes compared to being mounted to a 5DII or D3x (we do know the latter two are equivalent). The results would be at least slightly different at different focusing distances than the 15 feet we used in our setup. And of course, the limitation of only testing one copy of each lens means the results for any one of the lenses might be slightly better or worse with a different copy.

But if we keep those limitations in mind the results are interesting. I’ve regrouped the best lenses from each category above and put them in one table for a simple comparison. I’ve also recolored the results of this table according to the rules set out originally: the highest resolution at each aperture being marked, etc. It’s obviously a bit of a stretch, but this was the original assignment, finding the 50mm lens with highest resolution.

If you are shooting at f/1.4, as most people usually are with a 50mm lens, there are a lot of good choices. The two Canons, Leica Summilux and the Sigma are about the same in the center. The Sigma may be a bit softer in the corners, but if you’re worried about corner sharpness why are you shooting a 50mm at f/1.4? At smaller apertures the Leica lenses get better than the rest, but for practical purposes, even the blue lenses in the table above are very sharp. There’s not a bad lens in the bunch.

In Conclusion

I will note again that this is strictly a resolution test, not a best overall lens test. Every test like this will ignite the fanboys who feel their favorite lens was slighted. Before all the accusations begin let me note that my favorite 50mm lens (and this test certainly won’t change my mind) is the Zeiss 50mm f/2.0 Makro, which didn’t fare nearly as well as I expected.

I was very curious about how well the M9 would resolve. I would not have been shocked if it couldn’t resolve quite as well as the Canon and Nikon since it has fewer pixels. In that case we might have seen all of the lenses reach a similar peak MTF after which the M9 wasn’t capable of further resolution. That’s obviously not the case and I’m left assuming the lack of AA filter more than makes up for the fewer pixels. But that’s just an assumption. It may be that the M mount lenses are just that much better than the SLR lenses.

I should mention that both the Canon and Nikon cameras are able to resolve MTF50 numbers in the 1100 – 1200 range (we’ve gotten those numbers with several lenses, but obviously not wide aperture 50mm lenses) so this test is not limited by sensor resolution.

It will be interesting to see how all of these lenses compare when we get Sony NEX-7s in stock. Then we’ll be able to shoot EOS, Nikon, and M mount lenses on the same camera. That should let us make a better comparison, at least in the center, but we won’t be able to compare extreme corners since it is a crop sensor.

Or at least it will make an interesting comparison if I ever get time to do something like this again.

 

Roger Cicala

Lensrentals.com

January, 2012