Equipment

Why We’re Going to Start Testing Cinema Lenses. And Why We Haven’t Before.

 

Here we go again. This isn’t my first time pissing off the status quo; I’ve been doing it for years. We spent the last few years developing better metrology (lens testing) for photography lenses. The reason was relatively simple; photography cameras with higher resolutions began to see flaws that the photography company’s (and our existing) metrology couldn’t see. Test charts and lens test projectors just didn’t cut it anymore.

CineLens MTF Testing LensRentals

We don’t keep our methods secret. We write some up on this blog and publish others in peer-reviewed journals. Of course, the industry isn’t always enthusiastic about it. Their existing methods said their lenses were fine, and change is expensive. But as more and more users complained about defects they could see in their images that the manufacturers couldn’t see in their testing we had engineers from most of the major manufacturers visit to see what we’re doing. Over the last few years, most have started improving their metrology. A few are enthusiastic about it, most a bit grudging, and some just repeat ‘it isn’t necessary’ over and over.

During this time we’ve tested many thousands of photo lenses and maybe a dozen video lenses. Why? Well, because there was no reason to MTF test video lenses. So why start now? Because pretty soon there will be a reason to. So before I start rolling out MTF charts of video lenses I thought I’d put up a bit of a post about our ‘why now.’

 

Why We Haven’t Bothered

Lenses for video and cinematography are different than photo lenses in some distinct ways, but many of you will be surprised at some of the ways they aren’t different.

Cinematographers are very concerned about the ‘look’ of a lens, its color rendering, focusing accuracy, lack of focus breathing, parfocal zooming, how it handles flare and a dozen other things. They are interested in having a ‘sharp’ lens, but sharp is a vague term and very different in video and photo work.

Video is shot on much lower resolutions sensors than a photo, and there is less concern about the highest resolution. Photographers, with their higher resolution cameras, worry as much about absolute resolution as they do about contrast. In testing, fine resolution is referred to as high frequency. Test charts or lens projectors are perfectly great at low-frequency testing. There’s no need to use fancy optical machines for that.

Why We Are Bothering Now

The key here is that if you shoot on a high-resolution sensor (assuming a good lens), you see fine detail that was invisible before. Two very close points look like one point at low resolution but become two points in the image as the resolution increases.

If you quadruple the number of pixels on a sensor, you theoretically double the resolution of your image; you can see the detail that wasn’t in the low-resolution image. (This assumes you have a good lens in front of the sensor. If it’s a not-so-good lens, resolution improves some, but not as much.)

Photographers in recent years have jumped from 16-megapixel sensors to 36 and even 50 megapixels. Not quite a doubling of resolution, but enough to allow the cameras to see weaknesses in lenses that weren’t apparent before. Mainstream video has moved from 720p (about 1.3 megapixels) to 1080p (about 2.3 megapixels) to 4k (8-9 megapixels) in recent years, so there’s already been a resolution doubling although the resolution is still much lower than for photography.

But now, the higher-end video is 6k (about 19 megapixels) or even 8k (33 megapixels). Video resolution is nearly doubling again, in a very short time. When photo cameras reached 32 megapixels, photographers started seeing flaws in their lenses that the manufacturers didn’t see on their existing testing. I expect the same thing is about to happen with video.

Now 72,934 cinematographers are all going to assume that those more expensive video lenses aren’t going to have the problems that cheaper photo lenses have because they haven’t seen it (yet). And 643 places that test video lenses on charts and projectors are going to tell them they don’t need to worry about it because their tests show the lenses are perfect (so far). And then, one day soon, a cinematographer somewhere is going to look at his high-resolution footage, shot with a lens he’s used many times before at lower resolution, and scream because one side of the image is blurrier than the other.

If you want to stop reading here and go ahead and comment that I’m completely wrong because you just know I must be you may proceed. If not, you can read further, and I’ll give you some introductions to the Cinema lens tests we’ll be publishing in the next few weeks and months.

 

MTF Full-Field Display Examples

We’ve started using these because they provide a really intuitive way to look at the MTF of a lens. In previous posts, we’ve used the frequency of 30 line pairs /mm because that’s a good compromise between high and low resolutions.

For those of you who don’t speak MTF, only a high-resolution sensor can differentiate high frequencies, like 50 lp/mm. If you’re shooting a 3.2-megapixel image, you don’t see these. If you’re shooting a 30-megapixel image, you definitely can. A high MTF, the blue shades in the graphs below, mean things have high contrast. Low MTF, the red shades in the graphs below, is just a gray smear.

For the examples below I’m going to use a nice Cinema lens that many people have shot with, the Zeiss 35mm T2.1 CP.2 lens. This first set of graphs shows how the average of 10 copies of this lens performs at different frequencies.

10 lp/mm

At low frequencies, the lens is very good. The dark blue means the MTF is very high, and it stays that way from one side to the other.

Olaf Optical Testing, 2017

 

20 lp/mm

At a slightly higher frequency, things are still good, but you can see that the lens is a little softer away from centers. If you’ve shot with this lens, even at 1080p you’ve noticed this.

Olaf Optical Testing, 2017

 

30 lp/mm

This is our usual testing frequency, and things look as you’d expect, MTF is lower even in the center now. How would this matter? In theory, if you’re shooting at 1080, you probably would see the same resolution in most decent 35mm lenses, but at 4k, you will start to notice some are better or worse with fine detail. I’m not making any judgment about the CP.2 (yet), just showing that there is room to be better or worse than this lens at higher frequencies.

Olaf Optical Testing, 2017

 

40 lp/mm

Olaf Optical Testing, 2017

 

50 lp/mm

At higher frequencies we’re getting a lot of orange now, meaning the lens is just barely handling fine details. Red would be low enough that fine details would be lost if you’re shooting very high resolution video. That may or may not matter a bit to you, just pointing it out.

Olaf Optical Testing, 2017

 

So That Didn’t Matter Much at All, but This Does.

The images above were averages of 10 copies, and behaved very nicely, just like the lens manufacturer’s data does. But what if we look at individual copies of that lens? Well, that’s where it gets interesting.

Here are nine copies (because nine fit nicely in the frame) of the 35mm T2.1 CP.2 tested at 10 lp/mm. They all look great and perform nearly identically; you certainly couldn’t see any difference between them shooting 1080p video. (BTW – the green check is just from our software letting us know these files have been backed up.)

Olaf Optical Testing, 2017

 

Here are the same nine lenses but this time, shown at 30 lp/mm. At this higher frequency, it’s apparent the lenses are not all identical, although all are acceptable. But on a couple, the very edge is getting unacceptably soft (red areas) and in one (upper right) the center isn’t quite as sharp as the others. Looking at the images above, this difference isn’t very apparent at the 10 lp/mm frequency (these are the same lenses in the same order).

Olaf Optical Testing, 2017

 

If we look at them at 50 lp/mm we see there’s even more variation.

Olaf Optical Testing, 2017

 

Does this variation at higher frequencies matter at all? If you’re shooting 1080p video, absolutely not, you can’t see it. If you’re shooting 4k, you probably still wouldn’t notice it. But at 6K or 8K you certainly could.

So What Does It Mean?

Mostly this post is just an introduction to the topic so that when I start showing Cine lens tests, you understand why I’m showing these higher frequencies and whether those frequencies are important to you. And also to explain why I’m saying you may start seeing things you’ve never seen before when you start shooting 6k and 8k.

And as people transition into higher resolution video, the differences in lenses will become more important. We’ll be doing a lot of comparisons in future posts, but to give you an idea, the Canon CN-E, Zeiss CP.2, and Sigma Cine 35mm T1.5 have nearly identical performance at low frequencies on average.

Left to right: Zeiss, Canon, Sigma 35mm Cine lenses at 10 lp/mm; Olaf Optical Testing, 2017

 

But begin to look a bit different at higher frequencies.

Left to right: Zeiss, Canon, Sigma 35mm Cine lenses at 30 lp/mm; Olaf Optical Testing, 2017

 

Moreover, as we showed above, variation between lenses is greater at higher frequencies and will be more noticeable on higher resolution sensors.

I just wanted to show why, as we start publishing Cine lens MTF charts, we’ll be emphasizing how the lenses differ at different testing frequencies. Depending on the resolution you’re shooting at it may make a huge difference to you, or none at all.

 

Roger Cicala, Aaron Closz, and Brandon Dube

Lensrentals.com

July, 2017

 

. . . .

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Equipment
  • Illya Friedman

    Thanks for posting this Roger. You did a very good job of explaining the realities of what’s going on with the practical performance of cinema lenses relative to the sensors/formats they were designed for.

    Years ago I worked with a partner and a Zeiss K8/K9 to do similar MTF testing with the intention to publish some average empirical findings on popular cinema lenses particularly comparing one focal length and another of the exact same make. Manufacturers who used to publish this data, no longer do so.

    We found a fair bit of variation with some manufacturers, others less so. This isn’t exactly a revelation with optics, you can see some MTF differences on a decent projector in an A/B configuration. We were also comparing NM of torque required to rotate the focus/iris rings some companies had way too much variation from lens to lens.

    The community of professional DPs (as a whole) tend to be more “artist” than “plumber.” I’ve found few who really want to know how the sausage is made. As you know, when explaing line pairs, field, tag, sag, eyes start glazing over.

    At the time, we spoke with some rental companies and overall they were really not in favor of this information being “out there” (as the largest demographic of cine lens owners) they have a vested interest in keeping their gear working, but they were less concerned about a particular lens loosing favorability, but they really didn’t like the idea have being forced to educate clients and squash disinformation in order to clear up any misunderstanding as to what this data means in the real world.

    Having spent a long time working in cinema rentals, I understand their concerns and as much as I love full transparency, their concerns of misinterpretation of data are 100% valid.

    Ultimately my partner and I decided that the labor on our part just wasn’t worth it. Especially because such a small community that actually would be appreciate the effort.

    I will also say that with the growing demand for lenses that haven’t been made in decades, and (perhaps more disheartening), modern lenses being modified to perform worse, I am uncertain how many cine lens users want a consistenly high performing lens.

    I really think the real value of doing what you are proposing would be that it could add value to lesser known, and less popular lenses that are really well constructed and also reveal how poor some “rehousings” truly are.

    When you do go down this path, how many stops of MTF data will you include? It would be great if you could include at least 3.

    Back when, we found that some level of lens to lens variations could shrink considerably with as little as closing down 1/2 stop from max aperture, I feel it would be interesting if you checked this as well.

    Personally, I can’t wait to see what you publish, and also what kind of comments are made as a result.

    Illya Friedman
    President
    Hot Rod Cameras
    Burbank, California

  • Thomas Achermann

    as always, a very informative article!
    And this is, by far, the absolute BEST Clippy tip EVER!!! 🙂
    “…trying to comment without reading the article first…”
    🙂

  • Terry

    Thanks you so much for the tests and sharing!

  • Brandon Dube

    If a lens is designed for a curved sensor, we will just do a through-focus MTF FFD, fit out the X and Y linear field tilts (pose error of the lens w.r.t. the instrument), then use an optimizer to fit for the radius of curvature of the petzval focal surface and evaluate along that.

    The test will go from on the order of 5-10 min to 1-2 hours, but I do not see a straightforward way to do it otherwise, and the volumetric data is useful for a lot of other things as well.

  • Brandon Dube

    (first pp.)
    The data we gather comes as a set of coordinates — image height, azimuth, spatial frequency, and tan/sag, with a value. Image height is the most interesting for this question, and ranges from 0~20mm, inclusive, for our measurements. When you take ih * cos(az) and ih * sin(az) to map polar back to cartesian points, a lot of these fall outside the area of the sensor, and are not shown in the FFDs, by simple convention. When we condense the data to a plot ranging from 0~20, we do the azimuthal average over all points, not just those that fit in the sensor area. You can see in all of the FFDs that the corners look clipped off a small amount — this is because we do not have data for that location.

    (third pp.)
    The earliest I would expect that Olaf opens the floodgates on a consumer test service would be later this year, but possibly early 2018 too. It depends how quickly our internal tooling reaches the point it needs to. Roger may have a better estimate.

  • Brandon Dube

    A Full-Field Display is over the full field of view. The X and Y axes are the width and height of the sensor, and the MTF is encoded in color. the lines are contours, and drawn where the MTF crosses a boundary (e.g. 0.3). If it dips below, then comes back above 0.3, you will see two lines.

    Does this help?

  • Till Ulen

    What backup software do you use that draws these green checkmarks over files? 🙂

  • KWNJr

    I have read of some movies being done at 8K and 120p (how?) and read some reviews saying they were too real !?

  • Ted Miller

    I just read the peer-reviewed journal article you linked to, and a question came to mind, based on the diagrams in that article. What do you use as the right-hand side of your typical MTF diagram (of which I find none in this blog entry, but as shown in Figure 5 in the peer reviewed article)? During years of reading the blog, I was under the impression that the “edge” measurement was to the corner of a full-frame 35mm frame. However, Figure 4 in the article makes me question that, because none of the plotted lines are long enough to go to the corner of the frame. Is the edge of the measurement the edge of the horizontal frame, with MTF falling off even more in the corners? A clarification would be appreciated by some of us pixel-peepers.

    P.S. Your blog articles do have an effect on your business. I bought my last camera from you because I knew it would be in used-but-well-inpsected condition. I plan to buy my next lens from you, because I hope it will be one of the mid-priced super-telephoto zooms, and my 3 options are all subject to considerable sample-to-sample variation coming from the factory. I know you catch the lemons and send them back (or fix them).

    P.P.S. You have mentioned from time to time that at some point in the future you would probably offer a “test my lens” service. Any timeline or pricing ballpark for that service?

  • KWNJr

    With the current rumors about camera makers hinting at making lenses with a curved plane of focus (and curved sensors) are you thinking ahead as to how you will test them? esp. if each mfg. decides on a different curvature – in addition to better performance/simplifying design/reducing cost/etc. – making it harder for independet lens makers to produce lenses that easily fit more than one brand of camera?

  • KWNJr

    In your chart: Zeiss Full-Field Display @50 lp/mm – I see at least three lines labeled 0.3 – is this a typo – or how should I interpret them?
    Thank you for all the time and work you put into these tests. I learn a lot from them.

  • David Bateman

    I never liked that windows paper clip help assistant. Best to change it to the Einstein looking one right away.
    I agree 8k is coming quickly, seems like 2020 is a good target, so will be interesting if sigma and new fuji lenses are better than the old dogs.

  • Brandon Dube

    No, sorry.

  • Van Forsman

    Do you know of any good sources for understanding focus breathing or reviewers?

  • emp11, the problem becomes one of sample size. I can test a set if someone loans them to me (and people do a lot), but I’m always uncomfortable with how much difference there will be between one 30 year old set and another. Still, it would be fun to do.

    I tested a couple of 1880 era Petzval and Cooke lenses. It was actually kind of interesting – the Petzval made you wonder how anyone ever got an image with it, the Cooke was actually pretty good.

  • emp11

    the techie in me loves this stuff. For a long time i always worked on getting the numbers perfect. exposure to +-0.1 ,proper lighting ratios etc. and i had to use a sharp lens. now i’m more into the “look” of the light and shadow. And i’m falling in love with old derelict lenses. If you can please test some of the old cine favorite’s. Also how would some of these matched sets fare?

  • Stanislaw that is correct. That’s why we’ll also be printing field curvatures for the lenses we test, which we think will be particularly valuable for Cine lenses.

  • Stanislaw Zolczynski

    You focus the lens in the middle of frame, I suppose. Is the field curvature influencing edge resolution or not? I mean if one focused non-flatfield lens at edges then situation could be bit reversed, sharper edges softer centre.

  • I agree, but the % difference is much smaller at 10lp where MTF is ranging from mid 80% to mid 90%. There are a lot of 30% and up differences in the higher frequency.

  • pest

    It is already visible on a mobile phone screen for 10 lpmm. Lens two (extreme left edge), lens three (middle), lens four (right edge), lens five (left edge), lens seven (right edge). Much lighter blues.

    But lens 9 at 50lpmm is the big surprise.

    Nice article, good job done, thanks.

  • Santiago Flores

    I’ve read in this blog that Canon may have change something in their production line because there is less copy-to-copy variation. I think they where talking about new lenses as 50 1.8 stm

Follow on Feedly