Going Inside Beamr’s Frame-Level Content-Adaptive Rate Control for Video Coding
When it comes to video, the tradeoff between quality and bitrate is an ongoing dance. Content producers want to maximize quality for viewers, while storage and delivery costs drive the need to reduce bitrate as much as possible. Content-adaptive encoding addresses this challenge, by striving to reach the “optimal” bitrate for each unique piece of content, be it a full clip or a single scene. Our CABR technology takes it a step further by adapting the encoding at the frame level. CABR is a closed-loop content-adaptive rate control mechanism enabling video encoders to lower the bitrate of their encode, while simultaneously preserving the perceptual quality of the higher bitrate encode. As a low-complexity solution, CABR also works for live or real-time encoding.
All Eyes are on Video
According to Grand View Research, the global video streaming market is expected to grow at a CAGR of 19.6% from 2019 to 2025. This shift, fueled by the increasing popularity of direct-to-consumer streaming services such as Netflix, Amazon and Hulu, the growth of video on social media networks and user-generated video platforms such as Facebook and YouTube, and other applications like online education & video surveillance, has all eyes on video workflows. Therefore, efficient video encoding, in terms of encoding and delivery costs, and meeting the viewer’s rising quality expectations, are at the forefront of video service provider’s minds. Beamr’s CABR solution can reduce bitrates without compromising quality while keeping a low computational overhead to enhance video services.
Comparing Content-Adaptive Encoding Solutions
Instead of using fixed encoding parameters, content-adaptive encoding configures the video encoder according to the content of the video clip to reach the optimal tradeoff between bitrate and quality. Various content-adaptive encoding techniques have been used in the past to provide a better user experience with reduced delivery costs. Some of them have been entirely manual, where encoding parameters are hand-tuned for each content category and sometimes, like in the case of high-volume Blu-ray titles, at the scene level. Manual content-adaptive techniques are restricted in the sense that they can’t be scaled, and they don’t provide granularity lower than the scene level.
Other techniques, such as those used by YouTube and Netflix, use “brute force” encoding of each title by applying a wide range of encoding parameters, and then by employing rate-distortion models or machine learning techniques, try to select the best parameters for each title or scene. This approach requires a lot of CPU resources since many full encodes are performed on each title, at different resolutions and bitrates. Such techniques are suitable for diverse content libraries that are limited in size, such as premium content including TV series and movies. These methods do not apply well to vast repositories of videos such as user-generated content, and are not applicable to live encoding.
Beamr’s CABR solution is different from the techniques described above in that it works in a closed-loop and adapts the encoding per frame. The video encoder first encodes a frame using a configuration based on its regular rate control mechanism, resulting in an initial encode. Then, Beamr’s CABR rate control instructs the encoder to encode the same frame again with various values of encoding parameters, creating candidate encodes. Using a patented perceptual quality measure, each candidate encode is compared with the initial encode, and then the best candidate is selected and placed in the output stream. The best candidate is the one that has the lowest bitrate but still has the same perceptual quality as the initial encode.
Taking Advantage of Beamr’s CABR Rate Control
In order for Beamr’s CABR technology to encode video to the minimal bitrate and still retain the perceptual quality of a higher bitrate encode, it compresses each video frame to the maximum extent that provides the same visual quality when the video is viewed in motion. Figure 1 shows a block diagram of an encoding solution which incorporates CABR technology.
Figure 1 – A block diagram of the CABR encoding solution
An integrated CABR encoding solution consists of a video encoder and the CABR rate control engine. The CABR engine is comprised of the CABR control module responsible for managing the optimization process and a module which evaluates video quality.
As seen in Figure 2, the CABR encoding process consists of multiple steps. Some of these steps are performed once for each encoding session, some are performed once for each frame, and some are performed for each iteration of candidate frame encoding. When starting a content-adaptive encoding session, the CABR engine and the encoder are initialized. At this stage, we set system-level parameters such as the maximum number of iterations per frame. Then, for each frame, the encoder rate control module selects the frame types by applying its internal logic.
Figure 2. A block diagram of a video encoder incorporating Content Adaptive Bit-Rate encoding.
The encoder provides the CABR engine with each original input frame for pre-analysis within the quality measure calculator. The encoder performs an initial encode of the frame, using its own logic for bit allocation, motion estimation, mode selections, Quantization Parameters (QPs), etc. After encoding the frame, the encoder provides the CABR engine with the reconstructed frame corresponding to this initially encoded frame, along with some side information – such as the frame size in bits and the QP selected for each MacroBlock or Coding Tree Unit (CTU).
In each iteration, the CABR control module first decides if the frame should be re-encoded at all. This is done, for example, according to the frame type, the bit consumption of the frame, the quality of previous frames or iterations, and according to the maximum number of iterations set for the frame. In some cases, the CABR control module may decide not to re-encode a frame at all – in that case, the initial encoded frame becomes the output frame, and the encoder continues to the next frame. When the CABR control module decides to re-encode, the CABR engine provides the encoder with modified encoding parameters, for example, a proposed average QP for the frame, or the difference from the QP used for the initial encode. Note that the QP or delta QP values are an average value, and QP modulation for each encoding block can still be performed by the encoder. In more sophisticated implementations a QP map of value per encoding block may be provided, as well as additional encoder configuration parameters.
The encoder performs a re-encode of the frame with the modified parameters. Note that this re-encode is not a full encode, since it can utilize many encoding decisions from the initial encode. In fact, the encoder may perform only re-quantization of the frame, reusing all previous motion vectors and mode decisions. Then, the encoder provides the CABR engine with the reconstructed re-encoded frame, which becomes one of the candidate frames. The quality measure module then calculates the quality of the candidate re-encoded frame relative to the initially encoded frame, and this quality score, along with the bit consumption reported by the encoder is provided to the CABR control module, which again determines if the frame should be re-encoded. When that is the case, the CABR control module sets the encoding parameters for the next iteration, and the above process is repeated. If the control module decides that the search for the optimal frame parameters is complete, it indicates which frame, among all previously encoded versions of this frame, should be used in the output video stream. Note that the encoder rate control module receives its feedback from the initial encode of the current frame, and in this way the initial encode of the next frames (which determines the target quality of the bitstream) is not affected.
The CABR engine can operate in either a serial iterative approach or a parallel approach. In the serial approach, the results from previous iterations can be used to select the QP value for the next iteration. In the parallel approach, all candidate QP values are provided simultaneously and encodes are done in parallel – which reduces latency.
Integrating the CABR Engine with Software & Hardware Encoders
Beamr has integrated the CABR engine into its AVC software encoder, Beamr 4, and into its HEVC software encoder, Beamr 5. However, the CABR engine can be integrated with any software or hardware video encoder, supporting any block-based video standard such as MPEG-2, AVC, HEVC, EVC, VVC, VP9, and AV1.
To integrate the CABR engine with a video encoder, the encoder should support several requirements. First and foremost, the encoder should be able to re-encode an input frame (that has already been encoded) with several different encoding parameters (such as QP values), and save the “state” of each of these encodes, including the initial encode. The reason for saving the state is that when the CABR control module selects one of the candidate frame encodes (or the initial encode) as the one to use in the output stream, the encoder’s state should correspond to the state it was right after encoding that candidate frame. Encoders that support multi-threaded operation and hardware encoders typically have this capability, since each frame encode is performed by a stateless unit.
Second, the encoder should support an interface to provide the reconstructed frame and the per block QP and bit consumption information for the encoded frame. To improve compute performance, we also recommend that the encoder supports a partial re-encode mode, where information related to motion estimation, partitioning and mode decisions found in the initial encode can be re-used for re-encoding without being computed again, and only the quantization and entropy coding stages are repeated for each candidate encode. This results in a minimal encoding efficiency drop for the optimized encoding result, with significant speed-up compared to full re-encode. As described above, we recommend that the encoder will use the initial encoded data (QPs, compressed size, etc.) for its Rate Control state update. However, the selected frame and accompanying data must be used for reference frames and other reference data, such as temporal MV predictors, as it is the only data available in the bitstream for decoding.
When integrating with hardware encoders that support parallel encoding with no increase in latency, we recommend using the parallel search approach where multiple QP values per frame are evaluated simultaneously. If the hardware encoder can perform parallel partial encodes (for example, re-quantization and entropy coding only), while all parallel encodes use the analysis stage of the initial encode, such as motion estimation and mode decisions, better CPU performance will be achieved.
Sample Results
Below, we provide two sample results of the CABR engine, when integrated with Beamr 5, Beamr’s HEVC software encoder, each illustrating different aspects of CABR.
For the first example, we encoded various 4K 24 FPS source clips to a target bitrate of 10 Mbps. Sample frames from each of the clips can be seen in Figure 3. The clips vary in their content complexity: “Crowd Run” has very high complexity since it has great detail and very significant motion of the runners. “StEM” has medium complexity, with some video compression challenges such as different lighting conditions and reasonably high film grain. Finally, a promotional clip of JPEGmini by Beamr has low complexity due to relatively low motion and simple scenes.
Figure 3. Sample frames from the test clips. top: crowd-run, bottom left: StEM bottom right: JPEGmini.
We encoded 500 frames from each clip to a target bitrate of 10 Mbps, using the VBR mode of the Beamr 5 HEVC encoder, which performs regular encoding, and using the CABR mode, which creates a lower bit-rate, perceptually identical stream. For the high complexity clip “Crowd Run,” where providing excellent quality at such an aggressive bitrate is very challenging, CABR reduced the bitrate by only 3%. For the intermediate complexity clip “StEM,” bitrate savings were higher and reached 17%. For the lowest complexity clip “JPEGmini,” CABR reduced the bitrate by a staggering 45%, while still obtaining excellent quality which matches the quality of the 10 Mbps VBR encode. This extensive range of bitrate reduction percentage demonstrates the fully automatic content-adaptive nature of CABR-enhanced encoder, which reaches a different final bitrate, according to the content complexity.
The second example uses a 500 frame 1080p 24 FPS clip from the well-known “Tears Of Steel” movie by the Blender open movie project. The same clip was encoded using the VBR and CABR modes of the Beamr 5 HEVC software encoder, with three target bitrates: 1.5, 3 and 5 Mbps. Savings, in this case, were 13% for the lowest bitrate resulting in a 1.4 Mbps encode, 44% for the intermediate bitrate resulting in an encode of 1.8 Mbps, and 62% for the highest bitrate, resulting in a 2 Mbps encode. Figures 4 and 5 show sample frames from the encoded clips with VBR encoding on the left vs. CABR encoding on the right. The top two images are from encodes to a bitrate of 5 Mbps, while the bottom two were taken from the 1.5 Mbps encodes. As can be seen here, both 5 Mbps target encodes preserve the details, such as the texture of the bottom lip or the two hairs on the forehead above the right eye, while in the lower bitrate encodes these details are somewhat blurred. This is the reason that when starting from different target bitrates, CABR does not converge to the same bitrate. We also see, however, that the more generous the initial encoding, generally the more savings can be obtained. This example shows that CABR adapts not only to the content complexity, but also to the quality of the target encode, and preserves perceptual quality in motion while offering significant savings.
Figure 4. A sample from the “Tears of Steel” 1080p 24 FPS encode to 5 Mbps (top) and 1.5 Mbps (bottom), encoded in VBR mode (left) and CABR mode (right)
Figure 5. Closer view of the face in Figure 4, showing detail of lips and forehead from the encode to 5 Mbps (top) and 1.5 Mbps (bottom), encoded in VBR mode (left) and CABR mode (right).
By now, most of us have seen the data and know that online video consumption is soaring at a rate that is historically unrivaled. It’s no surprise that in the crux of the streaming era, so many companies are looking to innovate and figure out how to make their workflows or customers workflows better, less expensive, and faster.
In Episode 4 of The Video Insiders, we caught up with streaming veteran Tim Siglin to discuss HEVC implementation trends that counter previous assumptions, notable 2018 streaming events, and what’s coming in 2019. Tune in to hear The Video Insiders cover top-of-mind topics:
Want to join the conversation? Reach out to TheVideoInsiders@beamr.com
TRANSCRIPTION (lightly edited to improve readability only)
Mark Donnigan: 00:00 On today’s episode, the Video Insiders sit down with an industry luminary who shares results of a codec implementation study, while discussing notable streaming events that took place in 2018 and what’s on the horizon for 2019. Stay tuned. You don’t want to miss receiving the inside scoop on all this and more.
Announcer: 00:22 The Video Insiders is the show that makes sense of all that is happening in the world of online video, as seen through the eyes of a second generation Kodak nerd and a marketing guy who knows what I frames and macroblocks are. Here are your hosts, Mark Donnigan and Dror Gill.
Mark Donnigan: 00:40 Welcome, everyone. I am Mark Donnigan, and I want to say how honored Dror and I are to have you with us. Before I introduce this very special guest and episode, I want to give a shout of thanks for all of the support that we’re receiving. It’s really been amazing.
Mark Donnigan: 00:59 In the first 48 hours, we received 180 downloads. It’s pretty amazing.
Dror Gill: 01:06 Yeah. Yeah, it is. The industry is not that large, and I think it’s really an amazing number that they’re already listening to the show from the start before the word of mouth starts coming out, and people spread the news and things like that. We really appreciate it. So, if it’s you that is listening, thank you very much.
Mark Donnigan: 01:29 We really do aim for this to be an agenda-free zone. I guess we can put it that way. Obviously, this show is sponsored by Beamr, and we have a certain point of view on things, but the point is, we observed there wasn’t a good place to find out what’s going on in the industry and sort of get unbiased, or maybe it’s better to say unfiltered, information. That’s what we aim to do in every episode.
Mark Donnigan: 01:57 In this one, we’re going to do just that. We have someone who you can definitely trust to know what’s really happening in the streaming video space, and I know he has some juicy insights to share with us. So, without further ado, let’s bring on Tim Siglin.
Tim Siglin: 02:15 Hey, guys. Thank you for having me today and I will definitely try to be either as unfiltered or unbiased as possible.
Mark Donnigan: 02:21 Why don’t you give us a highlight reel, so to speak, of what you’ve done in the industry and, even more specifically, what are you working on today?
Tim Siglin: 02:31 Sure. I have been in streaming now for a little over 20 years. In fact, when Eric Schumacher-Rasmussen came on as the editor at StreamingMedia.com, he said, “You seemed to be one of the few people who were there in the early days.” It’s true. I actually had the honor of writing the 10-year anniversary of Streaming Media articles for the magazine, and then did the 20-year last year.
Tim Siglin: 02:57 My background was Motion Picture production and then I got into video conferencing. As part of video conferencing, we were trying to figure out how to include hundreds of people in a video conference, but not need necessarily have them have two-way feedback. That’s where streaming sort of caught my eye, because, ultimately, for video conferencing we maybe needed 10 subject matter experts who would talk back and forth, and together a hundred, then it went to thousands, and now hundreds of thousands. You can listen in and use something like chat or polling to provide feedback.
Tim Siglin: 03:31 For me, the industry went from the early revolutionary days of “Hey, let’s change everything. Let’s get rid of TV. Let’s do broadcast across IP.” That was the mantra in the early days. Now, of course, where we are is sort of, I would say, two-thirds of the way there, and we can talk a little bit about that later. The reality is that the old mediums are actually morphing to allow themselves to do heap, which is good, to compete with over the top.
Tim Siglin: 04:01 Ultimately, what I think we’ll find, especially when we get to pure IP broadcast with ATSC 3.0 and some other things for over-the-air, is that we will have more mediums to consume on rather than fewer. I remember the early format ways and of course we’re going to talk some in this episode about some of the newer codec like HEVC. Ultimately, it seems like the industry goes through the cycles of player wars, format wars, browser wars, operating system wars, and we hit brief periods of stability which we’ve done with AVC or H.264 over the last probably eight years.
Tim Siglin: 04:46 Then somebody wants to stir the pot, figure out how to either do it better, less expensively, faster. We go back into a cycle of trying to decide what the next big thing will be. In terms of what I’m working on now, because I’ve been in the industry for almost 21 years. Last year, I helped start a not-for-profit called Help Me Stream, which focuses on working with NGOs in emerging economies, trying to help them actually get into the streaming game to get their critical messages out.
Tim Siglin: 05:18 That might be emerging economies like African economies, South America, and just the idea that we in the first world have streaming down cold, but there are a lot of messages that need to get out in emerging economies and emerging markets that they don’t necessarily have the expertise to do. My work is to tie experts here with need there and figure out which technologies and services would be the most appropriate and most cost effective.
Tim Siglin: 05:48 The other thing I’m working on here, just briefly, is we’re getting ready for the Streaming Media Sourcebook, the 2019 sourcebook. I’m having to step back for the next 15 days, take a really wide look at the industry and figure out what the state of affairs are.
Dror Gill: 06:06 That’s wonderful. I think because this is exactly the right point, is one you end and the other one begins, kind of to summarize where we’ve been in 2018, what is the state of the industry and the fact that you’re doing that for the sourcebook, I think, ties in very nicely with our desire to hear from you an overview of what were the major milestones or advancements that were made in the streaming industry in 2018, and then looking into next year.
Dror Gill: 06:39 Obviously, the move to IP, getting stronger and stronger, now the third phase after analog and digital, now we have broadcast over IP. It’s interesting what you said about broadcasters not giving up the first with the pure OTT content providers. They have a huge business. They need to keep their subscribers and lower their churn and keep people from cutting the cord, so to speak.
Dror Gill: 07:04 The telcos and the cable companies still need to provide the infrastructure for Internet on top of which the over-the-top providers and their content, but they still need to have more offering and television and VLD content in order to keep their subscribers. It’s very interesting to hear how they’re doing it and how they are upgrading themselves to the era of IP.
Tim Siglin: 07:30 I think, Dror, you hit a really major point, which is we, the heavy lift … I just finished an article in ATSC 3.0 where I talk about using 2019 to prepare for 2020 when that will go live in the U.S. The heavy lift was the analog to digital conversion. The slightly easier lift is the conversion from digital to IP, but it still requires significant infrastructure upgrade and even transmission equipment to be able to do it correctly for the over-the-year broadcasters and cable.
Dror Gill: 08:07 That’s right. I think on the other hand, there is one big advantage to broadcast, even broadcast over-the-air. That is the ability to actually broadcast, the ability to reach millions, tens of millions, hundreds of millions of people over a single channel that everybody is receiving. Whereas, because of historic reasons and legacy reasons in IP, we are limited, still, when you broadcast to the end user to doing everything over unicast. When you do this, it creates a tremendous load on your network. You need to manage your CDNs.
Dror Gill: 08:46 I think we’ve witnessed in 2018 on one hand very large events being streamed to our record audience. But, on the other hand, some of them really failed in terms of user experience. It wasn’t what they expected because of the high volume of users, and because more and more people have discovered the ability to stream things over IP to their televisions and mobile devices. Can you share with us some of the experience that you have, some of the things that you’re hearing about in terms of these big events where they had failures and what were the reasons for those failures?
Tim Siglin: 09:30 I want to reiterate the point you made on the OTA broadcast. It’s almost as if you have read the advanced copy of my article, which I know you haven’t because it’s only gone to the editor.
Dror Gill: 09:42 I don’t have any inside information. I have to say, even though we are the Video Insiders.
Mark Donnigan: 09:47 We are the Video Insiders. That’s right.
Mark Donnigan: 09:49 But no inside information here.
Dror Gill: 09:51 No inside information. I did not steal that copy.
Tim Siglin: 09:55 What I point out in that article, Dror, I think which will come out in January shortly after CES is basically this. We have done a good job in the streaming industry, the OTT space of pushing the traditional mediums to upgrade themselves. One of the things as you say with OTA, that ability to do essentially a multicast from a tower wirelessly is a really, really good thing, because to get us to scale, and I think about things like the World Cup, the Olympics and even the presidential funeral that’s happened here in December, there are large-scale events that we in the OTT space just can’t handle, if you’re talking about having to build the capacity.
Tim Siglin: 10:39 The irony is, one good ATSC transmission tower could hit as many people as we could handle essentially globally with the unicast (OTT) model. If you look at things like that and then you look at things like EMBMS in the mobile world, where there is that attempt to do essentially a multicast, and it goes to points like the World Cup. I think one of the horror stories in the World Cup was in Australia. There was a mobile provider named Optus who won the rights to actually do half of the World Cup preliminary games. In the first several days, they were so overwhelmed by the number of users who wanted to watch and were watching, as you say, in a unicast model that they ended up having to go back to the company they had bid against who had the other half of the preliminaries and ask them to carry those on traditional television.
Tim Siglin: 11:41 The CEO admitted that it was such a spectacular failure that it damaged the brand of the mobile provider. Instead of the name Optus being used, everybody was referring to it as “Floptus.” You don’t want your brand being known as the butt of jokes for an event that only happens once every four years that you have a number of devotees in your market. And heaven forbid, it had been the World Cup for cricket, there would have been riots in the street in Sydney and Melbourne. Thank goodness it was Australia with soccer as opposed to Australia with cricket.
Tim Siglin: 12:18 It brings home the point that we talk about scale, but it’s really hard to get to scale in a unicast environment. The other event, this one happened, I believe, in late 2017, was the Mayweather fight that was a large pay-per-view event that was streamed. It turned out the problem there wasn’t as much the streams as it was the authentication servers were overwhelmed in the first five minutes of the fight. So, with authentication gone, it took down the ability to actually watch the stream.
Tim Siglin: 12:53 For us, it’s not just about the video portion of it, it’s actually about the total ecosystem and who you’re delivering to, whether you’re going to force caps into place because you know you can’t go beyond a certain capacity, or whether you’re going to have to partner up with traditional media like cable service providers or over-the-air broadcasters.
Mark Donnigan: 13:14 It’s a really good point, Tim. In the World Cup, the coverage that I saw, it was more of, I’d almost say or use the phrase, dashed expectations. Consumers, they were able to watch it. In most cases, I think it played smoothly. In other words, the video was there, but HDR signaling didn’t work or didn’t work right. Then it looked odd on some televisions or …
Tim Siglin: 13:43 20 frames a second instead of 60 frames a second.
Mark Donnigan: 13:48 Exactly. What’s interesting to me is that, what I see is, the consumer, they’re not of course walking around thinking as we are, like frame rate and color space and resolution. They are getting increasingly sensitive to where they can look at video now and say, “That’s good video,” or “That doesn’t look right to me.” I know we were talking before we started recording about this latest Tom Cruise public service announcement, which is just super fascinating, because it …
Tim Siglin: 14:24 To hear him say motion interpolation.
Mark Donnigan: 14:26 Yeah. Maybe we should tell the audience, for those, since it literally just came out I think today, even. But you want to tell the audience what Tom Cruise is saying?
Tim Siglin: 14:38 Essentially, Tom Cruise was on the set of Top Gun, as they’re shooting Top Gun. Another gentleman did a brief PSA for about a minute asking people to turn off motion interpolation on their televisions, which motion interpolation essentially takes a 24-frame per second and converts it to 30 frames per second by adding phantom frames in the middle. Because Mission Impossible: Fallout is just being released for streaming, Cruise was concerned and obviously others were concerned that some of the scenes would not look nearly as good with motion interpolation turned on.
Tim Siglin: 15:17 I think, Mark, we ought to go to a PSA model, asking for very particular things like, “How do you turn HDR on? How do you …” Those types of things, because those get attention in a way that you and I or a video engineer can’t get that attention.
Dror Gill: 15:33 How do you know if what you’re getting is actually 4K or interpolate HD, for example?
Tim Siglin: 15:38 Especially in our part of the industry, because we will call something OTT 4K streaming. That may mean that it fits in a 4K frame, but it doesn’t necessarily mean it’s that number of pixels being delivered.
Dror Gill: 15:52 It can also mean that the top layer in your adaptive bit rate stream is 4K, but then if you don’t have enough bandwidth, you’re actually getting the HD layer or even lower.
Dror Gill: 16:02 Even though it is a 4K broadcast and it is 4K content. Sometimes, you can be disappointed by that fact as well.
Mark Donnigan: 16:11 I have to give a very, very funny story directly related, and this happened probably, I don’t know, maybe, at least 18 months ago, maybe two years ago. I’m sitting on an airplane next to this guy. It’s the usual five-minute, get acquainted before we both turn on our computers. Anyway, when someone asks, “What do you do?” I generally just say, “I work for a video software company,” because how do you explain digital encoding? Most people just sort of stop at that, and don’t really ask more.
Mark Donnigan: 16:44 But this guy is like, “Oh, really?” He said, “So, I just bought a 4K TV and I love it.” He was raving about his new Samsung TV. Of course, he figured I’m a video guy. I would appreciate that. I said, “Hey.” “So, you must subscribe to Netflix.” “Yes. Yes, of course,” he says. I said, “What do you think of the Netflix quality? It looks great, doesn’t it?”
Mark Donnigan: 17:10 He sort of hem and hawed. He’s like, “Well, it really … I mean, yeah. Yeah, it looks great, but it’s not quite … I’m just not sure.” Then, I said, “I’m going to ask you two questions. First of all, are you subscribed to the 4K plan?” He was. Then I said, “How fast is your Internet at home.” He’s like, “I just have the minimum. I don’t know. I think it’s the 20 megabit package,” or whatever it was. I don’t remember the numbers.
Mark Donnigan: 17:38 I said, “There’s this thing.” And I gave him like a 30-second primer on adaptive bit rate, and I said, “It is possible, I have no idea of your situation, that you might be watching the HD version.” Anyway, he’s like, “Hah, that’s interesting.” I connect with the guy on LinkedIn. Three days later, I get this message. He says, “I just upgraded my Internet. I now have 4K on my TV. It looks awesome.”
Mark Donnigan: 18:04 On one hand, the whole situation was not surprising and, yet, how many thousands, tens of thousands, maybe millions of people are in the exact same boat? They’ve got this beautiful TV. It could be because they’re running some low-end router in the house. It could be they truly have a low end bandwidth package. There could be a lot of reasons why they’re not getting the bandwidth. They’re so excited about their 4K TV. They’re paying Netflix to get the top layer, the best quality, and they’re not even seeing it. It’s such a pity.
Tim Siglin: 18:37 I had a TSA agent asked me that same question, Mark, when I came through customs. I’m like, “Sure. I’ll stand here and answer that question for you.” The router was actually what I suggested that he upgrade, because he said his router was like this (old unit).
Mark Donnigan: 18:53 In a lot of homes, it’s a router that’s 15 years old and it just isn’t (up to the task).
Tim Siglin: 18:58 But it brings out the point that even as we’re talking about newer codecs and better quality, even if we get a lower sweet spot in terms of 4K content (streaming bandwidth), or as we found in the survey that we worked on together, that using HEVC for 1080p or 720p, if the routers, if the software in the chain is not updated, the delivery quality will suffer in a way that people who have a tuned television and seen the consistent quality aren’t certain what to do to fix when they use an over-the-top service.
Tim Siglin: 19:34 I think this is a key for 2019. As we prepare for ATSC 3.0 on over-the-air broadcast where people will be able to see pristine 4K, it will actually force those of us in the OTT space to up our game to make sure that we’re figuring out how to deliver across these multiple steps in a process that we don’t break.
Dror Gill: 19:54 You really see ATSC 3.0 as a game-changer in 2019?
Tim Siglin: 19:59 What I see it as is the response from the broadcast industry to, A) say that they’re still relevant, which I think is a good political move. And, B) it provides the scale you were talking about, Dror. See, I think what it does is it at least puts us in the OTT space on notice that there will be in certain first world countries a really decent quality delivery free of charge with commercials over the air.
Tim Siglin: 20:31 It takes me back to the early days of video compression when, if you had a good class-one engineer and an analog NTSC transmission system, they could give you really good quality if your TV was tuned correctly. It only meant having to tune your TV. It didn’t mean having to tune your router or having to tune your cable modem, having to tune your settings on your TV. I think that’s where the game-changer may be, is that those tuner cards, which will send HDR signaling and things like that with the actual transmission, are going to make it much easier for the consumer to consume quality in a free scenario. I think that part of it is a potential game-changer.
Mark Donnigan: 21:19 That’s interesting. Tim, we worked together earlier this year on a survey, an industry survey that I think it would be really, really interesting to listeners to talk about. Shall we pivot into that? Maybe you can share some of the findings there.
Tim Siglin: 21:38 Why don’t you take the lead on why Beamr wanted to do that? Then I’ll follow up with some of the points that we got out of it.
Mark Donnigan: 21:46 Obviously, we are a codec developer. It’s important for us to always be addressing the market the way that the market wants to be addressed, meaning that we’re developing technologies and solutions and standards that’s going to be adopted. Clearly, there has been, especially if we rewind a year ago or even 18 months ago, AV1 was just recently launched. There were still questions about VP9.
Mark Donnigan: 22:19 Obviously, H264 AVC is the standard, used everywhere. We felt, “Let’s go out to the industry. Let’s really find out what the attitudes are, what the thinking is, what’s going on ‘behind closed doors’ and find out what are people doing.” Are they building workflows for these new advanced codecs? How are they going to build those workflows? That was the impetus, if you will, for it.
Mark Donnigan: 22:49 We are very happy, Tim, to work with you on that and of course Streaming Media assisted us with promoting it. That was the reason we did it. I know there were some findings that were pretty predictable, shall we say, no surprises, but there were some things that I think were maybe a little more surprising. So, maybe if you like to share some of those.
Tim Siglin: 23:12 Yeah. I’ll hit the highlights on that. Let me say too that one of the things that I really like about this particular survey, there was another survey that had gone on right around that time that essentially was, “Are you going to adopt HEVC?” What we took the approach on with this survey was to say, “Okay. Those of you who’ve already adopted HEVC, what are the lessons that we can learn from that?”
Tim Siglin: 23:36 We didn’t exclude those who were looking at AV1 or some of the other codes, even VP9, but we wanted to know those people who used HEVC. Were they using it in pilot projects? Were they thinking about it? Were they using it in actual production? What we found in the survey is that AVC, or H.264, was still clearly dominant in the industry, but that the ramp-up to HEVC was moving along much faster than at least I … I believed. Mark, I told you when we started the survey question creation, which was about a year ago and then launched it in early 2018, I expected we wouldn’t see a whole lot of people using HEVC in production.
Tim Siglin: 24:23 I was pleasantly surprised to say that I was wrong. In fact, I think you mentioned in our recent Streaming Media West interview that there was a statistic you gave about the number of households that could consume HEVC. Was it north of 50%?
Mark Donnigan: 24:40 Yeah, it’s more than 50%. What’s interesting about that number is that that actually came from a very large MSO. Of course, they have a very good understanding of what devices are on their network. They found that there was at least one device in at least 50% of their homes that could receive and decode, playback, HEVC. That’s about as real world as you can get.
Tim Siglin: 25:06 What was fascinating to me too in this study was, we asked open-ended questions, which is what I’ve done in the research projects for the last 25 years both the video conferencing and streaming. One of the questions we asked was, “Do you see HEVC as only a 4K solution or do you see it as an option for lower resolutions?” It turned out overwhelmingly, people said, “We not only see it for 4K. We see it for high-frame rate (HFR) 1080p, standard frame rate 1080p, with some HDR.”
Tim Siglin: 25:40 Not a majority, but a large number of respondents said they would even see it as a benefit at 720p. What that tells me is, because we had a large number of engineers, video engineers, and we also have people in business development who answer these questions, what it tells me is that companies know as we scale because of the unicast problem that Dror pointed out in the beginning that scaling with a codec that consumes more bandwidth is a good way to lose money, kind of like the joke that the way a rich man can lose money really fast is to invest in an airline.
Tim Siglin: 26:19 If indeed you get scale with AVC, you could find yourself with a really large bill. That look at HEVC is being not just for 4K, HDR, or high frame rate in the future, but also for 1080p with some HDR and high frame rate. It tells me that the codec itself or the promise of the codec itself was actually really good. What was even more fascinating to me was the number of companies that had AVC pipelines that were actually looking to integrate HEVC into those same production pipe.
Tim Siglin: 26:55 It was much easier from a process standpoint to integrate HEVC into an AVC pipeline, so in other words, H265 into H264 pipeline than it was to go out of house and look at something like AV1 or VP9, because the work that was done on HEVC builds on the benefits that were already in place in AVC. Of course, you got Apple who has HLS, HTTP Live Streaming, and a huge ecosystem in terms of iPhones and iPads, laptops and desktops supporting HEVC not just as a standard for video delivery, but also with the HEIC or HEIF image format, now having all of their devices shoot images using HEVC instead of JPEG. That in and of itself drives forward adoption of HEVC. I think you told me since that survey came out, probably now seven months ago, you all have continued to see the model of all-in HEVC adoption.
Dror Gill: 28:03 This is what we promote all the time. It’s kind of a movement. Are you all in HEVC or are you doing it just for 4K, just where you have to do it? We really believe in all-in HEVC. Actually, this week, I had an interesting discussion with one of our customers who is using our optimization product for VOD content, to reduce bit-rate of H.264 (streams). He said, “I want to have a product. I want to have a solution for reducing bit-rates on our live channels.”
Dror Gill: 28:32 So, I asked them, “Okay. Why don’t you just switch your codec to HEVC?” He said, “No, I can’t do that.” I said, “Why not?” He said, “You know compatibility and things like that.” I asked, “Okay. What are you using? What are you delivering to?” He said, “We have our own set-top boxes (STB), IP set-top boxes which we give out to our customers. Well, these are pretty new.” So, they support HEVC. I’m okay there. “Then we have an Apple TV app.” “Okay, Apple TV has a 4K version. So, it supports HEVC. All of the latest Apple TV devices have HEVC. That’s fine.” “Then we have smartphone apps, smart TV apps for Android TV and for the LG platform.”
Dror Gill: 29:15 Obviously, TV’s support 4K. So, I’m okay there. With delivering to mobile devices, all the high-end devices already support HEVC. He was making this estimate that around 50 to 60% of his viewers are using devices that are HEVC capable. Suddenly, he’s thinking, “Yeah, I can do that. I can go all in HEVC. I will continue, of course, to support H.264 for all of the devices that don’t support HEVC. But if I can save 50% of the bandwidth to 50 to 60% of my customers, that’s a very big savings.”
Mark Donnigan: 29:48 What’s interesting about this conversation, Dror, is first of all I’m pretty certain that the operator you’re talking with is different than the operator that I shared, found the exact same thing. This is a consistent theme, is that pretty much in developed parts of the world, it really is true that 50% or more of the users can today receive HEVC. This number is only growing. It’s not like it’s static It is just growing. Next year, I don’t know if that number will be 60% or 70%, but it’s going to be even bigger.
Mark Donnigan: 30:27 What’s fascinating is that, again, we’ve said earlier, that the consumer is getting just more aware of quality, and they’re getting more aware of when they’re being underserved. For operators who are serving to lowest common denominator, which is to say, AVC works across all my devices, and it’s true. AVC works on all the high-end devices equally well, but you’re under-serving a large and growing number of your users.
Mark Donnigan: 31:01 If your competitors are doing the same, then I guess you could say … well, “Who are they going to switch to?” But there are some fast-moving leaders in the space who are either planning or they’re shortly going to be offering better quality. They’re going to be extending HEVC into lower bit rates or lower resolutions, that is, and therefore lower bit rates, and the consumers are going to begin to see like, “Well, wait a second. This service over here that my friend has or we have another subscription in the household, how come the video looks better?” They just begin to migrate there. I think it’s really important when we have these sorts of conversations to connect to this idea that don’t underserve your consumer in an effort to be something to everybody.
Tim Siglin: 31:57 I would add two other quick things to that, Mark. One is, we’ve always had this conversation in the industry about the three-legged stool of speed, quality and bandwidth in terms of the encoding.
Tim Siglin: 32:09 Two of those are part of the consumer equation, which is quality and bandwidth. Then, oftentimes, we’ve had to make the decision between quality and bandwidth. If the argument is ostensibly that HEVC as it stands right now, had a couple years of optimization, can get us to about, let’s say, 40%. Let’s not even say 50%. For equivalent quality, it can get us to 40% bandwidth reduction. Why wouldn’t you switch over to something like that?
Tim Siglin: 32:39 Then the second part, and I have to put a plugin for what Eric Schumacher-Rasmussen and the Streaming Media team did at Streaming Media West by having Roger Pantos come and speak, Roger Pantos being of course the inventor of HLS, and I’m not a huge fan of HLS, just because of the latency issues. But he pointed out in his presentation, his tutorial around HLS that you can put two different codecs in a manifest file. There is absolutely no reason that an OTT provider could not provide both HEVC and AVC within the same manifest file and then allow the consumer device to choose.
Tim Siglin: 33:22 When Dror mentioned the company who has the OTT boxes that they give away, they could easily set a flag in those boxes to say, “If you’re presented with a manifest file that has AVC and HEVC, go with HEVC to lower the bandwidth, overall.” The beauty is it’s a technical issue at this point and it’s a technical implementation issue, not a ‘can we make it work?’ Because we know that it works based around the HLS.
Mark Donnigan: 33:54 This is excellent. Tim, let’s wrap this up, as I knew it would be. It has just been an awesome conversation. Thank you for sharing all your years of collective experience to give some insight into what’s happening in the industry. Let’s look at 2019. I know we’ve been talking a little bit about … you’ve made references to ATSC 3.0. Some of our listeners will be going to CES. Maybe there’s some things that they should be looking at or keeping their eyes opened for. What can you tell us about 2019?
Tim Siglin: 34:35 Here’s what I think 2019 is bringing. We have moved in the cloud computing space and you all are part of this conversation at Beamr. We’ve moved from having cloud-based solutions that were not at parity with on-premise solutions to actually in 2018 reaching parity between what you could do in an on-premise solution versus the cloud. Now, I think in 2019, what we’re going to start seeing is a number of features in cloud-based services, whether it’s machine learning, which the popular nomenclature is AI, but I really like machine learning as a much better descriptor, whether it’s machine learning, whether it’s real-time transcoding of live content, whether it’s the ability to simultaneously spit out AVC and HEVC like we’ve been talking about here that the cloud-based solutions will move beyond parity with the on-premise solutions.
Tim Siglin: 35:35 There always will be needs for the on-premise parts from a security standpoint in sort of the industries, but I don’t think that will inhibit cloud-based in 2019. If people are going to CES, one of the things to look at there, for instance, is a big leap in power consumption savings for mobile devices. I’m not necessarily talking about smartphones, because the research I’ve done says the moment you turn GPS on, you lose 25% of battery. Tablets have the potential to make a resurgence in a number of areas for consumers and I think we’ll see some advances in battery (capacity).
Tim Siglin: 36:19 Part of that goes to HEVC, which as we know is a much harder codec to decode. I think the consumer companies are being forced into thinking about power consumption as HEVC becomes more mainstream. That’s something I think people should pay attention to as well. Then, finally, HDR and surround sound solutions, especially object placement like Dolby Atmos and some of these others, will become much more mainstream as a way to sell flat panels and surround sound systems.
Tim Siglin: 36:56 We sort of languished in that space. 4K prices have dropped dramatically in the last two years, but we’re not yet ready for 8K. But I think we’ll see a trend toward fixing some of the audio problems. In the streaming space, to fix those audio problems, we need to be able to encode and encapsulate into sort of the standard surround sound model. Those are three areas that I would suggest people pay attention.
Mark Donnigan: 37:25 Well, thank you for joining us, Tim. It’s really great to have you on. We’ll definitely do this again. We want to thank you, the listener, for supporting the Video Insiders. Until the next episode. Happy encoding!
Announcer: 37:39 Thank you for listening to the Video Insiders Podcast, a production of Beamr Imaging Limited. To begin using Beamr’s codecs today, go to Beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.
Anyone familiar with the streaming video industry knows that we love our acronyms. You would be hard-pressed to have a conversation about the online video industry without bringing one up…
In today’s episode, The Video Insiders focus on the future of three-character codecs: AVC, VP9, and VVC.
But before we can look at the future, we have to take a moment to revisit the past.
The year 2018 marks the 15-year anniversary of AVC and in this episode, we visit the process and lifecycle of standardization to adoption and what that means for the future of these codecs.
Mark Donnigan: 00:52 It is, it is episode three. So, today we have a really exciting discussion as we consider the future of codecs named with three characters.
Dror Gill: 01:03 Three character codecs, okay, let’s see.
Dror Gill: 01:10 Let’s see, that’s today’s trivia question. I can think of AVC, VP9, AV1, and VVC?
Mark Donnigan: 01:21 Well, you just named three that I was thinking about and we’re gonna discuss today! We’ve already covered AV1. Yeah, yeah, you answered correctly, but we haven’t really considered where AVC, VP9, and VVC fit into the codec stew. So when I think about AVC, I’m almost tempted to just skip it because isn’t this codec standard old news? I mean, c’mon. The entire video infrastructure of the internet is enabled by AVC, so what is there to discuss?
Dror Gill: 01:57 Yeah. You’re right. It’s like the default, but in fact, the interesting thing is that today, we’re (in) 2018 and this is the twenty year anniversary of AVC. I mean, ITU issued the call for proposals, their video coding expert group, issued the call for proposal for a project. At the time was called H26L, and their target was to double the coding efficiency, which effectively means halving the bit rate necessary for given level of fidelity. And that’s why it was called H26L, it was supposed to be low bit rate.
Mark Donnigan: 02:33 Ah! That’s an interesting trivia question.
Mark Donnigan: 02:36 I wonder how many of our listeners knew that? That’s kind of cool. H26L.
Dror Gill: 02:42 But they didn’t go alone. It was the first time they joined forces in 2001 with the ISO MPEG, that’s the same Motion Pictures Experts Group, you know we discussed in the first episode.
Dror Gill: 02:57 And they came together, they joined forced, and they created JVT, that was the Joint Video Team, and I think it’s a great example of collaboration. There are standards by dealing with video communication standards, and ISO MPEG, which is a standards body dealing with video entertainment standards. So, finally they understood that there’s no point in developing video standards for these two different types of applications, so they got all the experts together in the JVT and this group developed what was the best video compression standard at the time. It was launched May 30, 2003.
Dror Gill: 03:36 There was one drawback with this collaboration in that the video standard was known by two names. There was the ITU name which is H.264. And then there’s the ISO MPEG name which is AVC, so these created some confusion at the start. I think by now, most of our listeners know that H.264 and AVC are two of the same.
Mark Donnigan: 03:57 Yeah, definitely. So, AVC was developed 15 years ago and it’s still around today.
Dror Gill: 04:02 Yeah, yeah. I mean, that’s really impressive and it’s not only around, it’s the most popular video compression standard in the world today. I mean, AVC is used to deliver video over the internet, to computers, televisions, mobile devices, cable, satellite, broadcast, and even blu-ray disks. This just shows you how long it takes from standardization to adoption, right? 15 years until we get this mass market adoption market dominance of H.264, AVC as we have today.
Dror Gill: 04:31 And the reason it takes so long is that, we discussed it in our first episode, first you need to develop the standard. Then, you need to develop the chips that support the standard, then you need to develop devices that incorporate the chip. Even when initial implementation of the codec got released, they are still not as efficient as they can be, and it takes codec developers more time to refine it and improve the performance and the quality. You need to develop the tools, all of that takes time.
Mark Donnigan: 04:59 It does. Yeah, I have a background in consumer electronics and because of that I know for certainty that AVC is gonna be with us for a while and I’ll explain why. It’s really simple. Decoding of H.264 is fully supported in every chip set on the market. I mean literally every chip set. There is not a device that supports video which does not also support AVC today. It just doesn’t exist, you can’t find it anywhere.
Mark Donnigan: 05:26 And then when you look at in coding technologies for AVC, H.264, (they) have advanced to the point where you can really achieve state of the art for very low cost. There’s just too much market momentum where the encode and decode ecosystems are just massive. When you think about entertainment applications and consumer electronics, for a lot of us, that’s our primary market (that) we play in.
Mark Donnigan: 05:51 But, if you consider the surveillance and the industrial markets, which are absolutely massive, and all of these security cameras you see literally everywhere. Drone cameras, they all have AVC encoders in them. Bottom line, AVC isn’t going anywhere fast.
Dror Gill: 06:09 You’re right, I totally agree with that. It’s dominant, but it’s still here to stay. The problem is that, we talked about this, video delivery over the internet. The big problem is the bandwidth bottleneck. With so much video being delivered over the internet, and then the demand for quality is growing. People want higher resolution, they want HDR which is high dynamic range, they want higher frame rate. And all this means you need more and more bit rate to represent the video. The bit rate efficiency that is required today is beyond the standard in coding in AVC and that’s where you need external technologies such as content adaptive encoding perceptual optimization that will really help you push AVC to its limits.
Mark Donnigan: 06:54 Yeah. And Dror, I know you’re one of the inventors of a perceptual optimization technique based on a really unique quality measure, which I’ve heard some in the industry believe could even extend the life of AVC from a bit rate efficiency perspective. Tell us about what you developed and what you worked on.
Dror Gill: 07:13 Yeah, that’s right. I did have some part in this. We developed a quality measure and a whole application around it, and this is a solution that can reduce the bit rate of AVC by 30%, sometimes even 40%. It doesn’t get us exactly to where HEVC starts, 50% is pretty difficult and not for every content (type). But content distributors that recognize AVC will still be part of their codec mix for at least five years, I think what we’ve been able to do can really be helpful and a welcome relief to this bandwidth bottleneck issue.
Mark Donnigan: 07:52 It sounds like we’re in agreement that for at least the midterm horizon, the medium horizon, AVC is gonna stay with us.
Dror Gill: 08:01 Yeah, yeah. I definitely think so. For some applications and services and certain regions of the world where the device penetration of the latest, high end models is not as high as in other parts, AVC will be the primary codec for some time to come.
Dror Gill: 08:25 It’s interesting to me, essentially, it’s mostly a YouTube codec. It’s not a bad coded, it has some efficiency advantages over AVC, but outside of Google, you don’t see any large scale deployments. By the way, if you look at Wikipedia, you read about the section that says where is VP9 used, it says VP9 is used mostly by YouTube, some uses by Netflix, and it’s being used by Wikipedia.
Mark Donnigan: 08:50 VP9 is supported fairly well in devices. Though, it’s obviously hard to say exactly what the penetration is, I think there is support in hardware for decode for VP9. Certainly it’s ubiquitous on Android, and it’s in many of the UHD TV chip sets as well. So, it’s not always enabled, but again, from my background on the hardware side, I know that many of those SOCs, they do have a VP9 decoder built into them.
Mark Donnigan: 09:23 I guess the question in my mind is, it’s talked about. Certainly Google is a notable both developer and user, but why hasn’t it been adopted?
Dror Gill: 09:33 Well, I think there are several issues here. One of them is compression efficiency. VP9 brings maybe 20, 30% improvement in compression efficiency over AVC, but it’s not 50%. So, you’re not doubling your compression efficiency. If you want to replace the codec, that’s really a big deal. That’s really a huge investment. You need to invest in coding infrastructure, new players. You need to do compatibility testing. You need to make sure that your packaging and your DRM work correctly and all of that.
Dror Gill: 10:04 You really want to get a huge benefit to offset this investment. I think people are really looking for that 50% improvement, to double the efficiency, which is what you get with HEVC but not quite with VP9. I think the second point is that VP9, even though it’s an open source coder, it’s developed and the standard is maintained by Google. And some industry players are kind of afraid of the dominance of Google. Google has taken over the advertising market online.
Dror Gill: 10:34 You know, and search and mobile operating systems, except Apple, it’s all Android. So, those industry players might be thinking, I don’t want to depend on Google for my video compression format. I think this is especially true for traditional broadcasters. Cable companies, satellite companies, TV channels that broadcast over the air. These companies traditionally like to go with established, international standards. Compression technologies that are standardized, they have the seal of approval by ITU and ISO.
Dror Gill: 11:05 They’re typically following that traditional codec developer past. ISO MPEG too, now it’s AVC, starting with HEVC. What’s coming next?
Mark Donnigan: 11:16 Well, our next three letter codec is VVC. Tell us about VVC, Dror.
Dror Gill: 11:21 Yeah, yeah, VVC. I think this is another great example of collaboration between ITU and ISO. Again, they formed a joint video experts team. This time it’s called JVET.
Dror Gill: 12:10 So, JVET has launched a project to develop a new video coding standard. And you know, we had AVC that was advanced video coding. Then we had HEVC which is high efficiency video coding. So, they thought, what would be the next generation? It’s already advanced, it’s high efficiency. So, the next one, they called it VVC, which is versatile video code. The objective of VVC is obviously to provide a significant improvement in compression efficiency over the existing HEVC standard. Development already started. The JVET group is meeting every few in months in some exotic place in the world and this process will continue. They plan to complete it before the end of 2020. So, essentially in the next two years they are gonna complete the standard.
Dror Gill: 13:01 Today, already, even though VVC is in early development and they haven’t implemented all the tools, they already report a 30% better compression efficiency than HEVC. So, we have high hopes that we’ll be able to fight the video tsunami that is coming upon us with a much improved standard video coder which is VVC. I mean, its improved at least on the technical side and I understand that they also want to improve the process, right?
Mark Donnigan: 13:29 That’s right, that’s right. Well, technical capabilities are certainly important and we’re tracking of course VVC. 30% better efficiency this early in the game is promising. I wonder if the JVET will bring any learnings from the famous HEVC royalty debacles to VVC because I think what’s in everybody’s mind is, okay, great, this can be much more efficient, technically better. But if we have to go round and round on royalties again, it’s just gonna kill it. So, what do you think?
Dror Gill: 14:02 Yeah, that’s right. I think it’s absolutely true and many people in the industry have realized this, that you can’t just develop a video standard and then handle the patent and royalty issues later. Luckily some companies have come together and they formed an industry group called The Media Coding Industry Forum, or MC-IF. They held their first meeting a few weeks ago in Macau during empic meeting one through four. Their purpose statement, let me quote this from their website, and I’ll give you my interpretation of it. They say the media coding industry forum (MC-IF) is an open industry forum with a purpose of furthering the adoption of standards initially focusing on VVC, but establishing them as well accepted and widely used standards for the benefit of consumers and the industry.
Dror Gill: 14:47 My interpretation is that the group was formed in an effort for companies with interest in this next generation video codec to come together and attempt to influence the licensing policy of VVC and try to agree on a reasonable patent licensing policy in advance to prevent history from repeating itself. We don’t want that whole Hollywood story with the tragedy that took a few years until they reached the happy ending. So, what are you even talking about? This is very interesting. They’re talking about having a modular structure for the codec. These tools of the codecs, the features, can be plugged in and out, very easily.
Dror Gill: 15:23 So, if some company insists on reasonable licensing terms, this group can just decide not to support the feature and it will be very easily removed from the standard, or at least from the way that companies implement that standard.
Mark Donnigan: 15:37 That’s an interesting approach. I wonder how technically feasible it is. I think we’ll get into that in some other episodes.
Dror Gill: 15:46 Yeah. That may have some effect on performance.
Mark Donnigan: 15:49 Exactly. And again, are we back in the situation that the Alliance for Open Media is in with AV1. Where part of the issue of the slow performance is trying to work around patents. At the end of the day you end up with a solution that is hobbled technically.
Dror Gill: 16:10 Yeah. I hope it doesn’t go there.
Mark Donnigan: 16:13 Yeah, I hope we’re not there. I think you heard this too, hasn’t Apple joined the consortium recently?
Dror Gill: 16:21 Yeah, yeah, they did. They joined silently as they always do. Silently means that one day somebody discovers their logo… They don’t make any announcement or anything. You just see a logo on the website, and then oh, okay.
Mark Donnigan: 16:41 You know, maybe it’s good to kind of bring this discussion back to Earth and close out our three part series by giving the listeners some pointers. About how they should be thinking about the next codec that they adopt. I’ve been giving some thought as we’ve been doing these episodes. I think I’ll kick it off here Dror if you don’t mind, I’ll share some of my thoughts. You can jump in.
Mark Donnigan: 17:11 These are complex decisions of course. I completely agree, billing this as codec wars and codec battles, it’s not helpful at the end of the day. Maybe it makes for a catchy headline, but it’s not helpful. There’s real business decisions (to be made). There are technical decisions. I think a good place to start for somebody who’s listening and saying “okay great, I now have a better understanding of the lay of the land of HEVC, for AV1, I can understand VP9, I can understand AVC and what some of my options are to even further reduce bit rate. But now, what do I do?”
Mark Donnigan: 17:54 And I think a good place to start is to just look at your customers, and do they lean towards early adopters. Are you in a strong economic environment, which is to say quite frankly, do most of your customers carry around the latest devices? Like an iPhone X, or Galaxy 9. If largely your customers lean towards early adopter and they’re carrying around the latest devices, then you have an obligation to serve them with the highest quality and the best performance possible.
Dror Gill: 18:26 Right. If your customers can receive HEVC, and it’s half the bit rate, then why not deliver it to them better quality, or say when you see the end cost with this more efficient codec and everybody is happy.
Mark Donnigan: 18:37 Absolutely, and again, I think just using pure logic. If somebody could afford a more than $1000 device in their pocket, probably the TV hanging on the wall is a very new, UHD capable (one). They probably have a game console in the house. The point is that you can make a pretty strong argument and an assumption that you can go, what I like to think of as all in HEVC including even standard definition, just SDR content.
Mark Donnigan: 19:11 So, the industry has really lost sight in my mind of the benefits of HEVC as they apply across the board to all resolutions. All of the major consumer streaming services are delivering 4K using HEVC, but I’m still shocked at how many, it’s kind of like oh, we forget that the same advantages of bit rate efficiency that work at 4K apply at 480p. Obviously, the absolute numbers are smaller because the file sizes are smaller, etc.
Mark Donnigan: 19:41 But the point is, 30, 40, 50% savings applies at 4K as it does at 480p. I understand there’s different applications in use cases, right? But would you agree with that?
Dror Gill: 19:55 Yeah, yeah, I surely agree with that. I mean, for 4K, HEVC is really an enabler.
Dror Gill: 20:01 For HEVC, you would need like 30, 40 megabits of video. Nobody can stream that to the home, but change it to 10, 15, that’s reasonable, and you must use HEVC for 4k otherwise it won’t even fit the pipe. But for all other resolutions, you get the bang with the advantage or you can trade it off for a quality advantage and deliver higher quality to your users, or higher frame rate, or enable HDR. If all of these possibilities that you can do with HD and even SD content, give them a better experience using HEVC and being able to stream on devices that your users already have. So yeah, I agree. I think it’s an excellent analysis. Obviously if you’re up in an emerging market, or your consumers don’t have high end devices, then AVC is a good solution. If there are network constraints, and there are many places in the world that network conductivity isn’t that great. Or in rural areas where we have very large parts of the population spread out (in these cases) bandwidth is low and you will get into a bottleneck even with HD.
Dror Gill: 21:06 That’s where perceptual optimization can help you reduce the bit rate even for AVC and keep within the constraints that you have. When your consumers can upgrade their devices and when the cycle comes in a few years when every device has HEVC support, then obviously you upgrade your capability and support HEVC across the board.
Mark Donnigan: 21:30 Yeah, that’s a very important point Dror, is that this HEVC adoption curve in terms of silicon, on devices. It is in full motion. Just the planning life cycles. If you look at what goes into hardware, and especially on the silicon side, it doesn’t happen that way. Once these technologies are in the designs, once they are in the dies, once the codec is in silicon, it doesn’t get arbitrarily turned on and off like light switches.
Mark Donnigan: 22:04 How should somebody be looking at VP9, VVC, and AV1?
Dror Gill: 22:13 Well, VP9 is an easy one. Unless you’re Google, you’re very likely gonna skip over this codec. Not just that the VP9 isn’t the viable choice, it simply doesn’t go so far as HEVC in terms of bit rate efficiency and quality. Maybe two years back we would consider it as an option for reducing bit rate, but now with the HEVC support that you have, there’s no point in going to VP9. You might as well go to HEVC. If you talk about VVC, (the) standard is still a few years from being ratified so, we actually don’t have anything to talk about.
Dror Gill: 22:49 The important point is again to remember, even when VVC launches, it will still be another 2 to 3 years after ratifying the standard before you have even a very basic playback ecosystem in place. So, I would tell our listeners if you’re thinking, why should I adopt HEVC, because VVC is just around the corner, well, that corner is very far. It’s more like the corner of the Earth than the corner of the next block.
Dror Gill: 23:18 So, HEVC today, VVC will be the next step in a few years. And then there’s AV1. You know, we talked a lot about AV1. No doubt, AV1 has support from huge companies. I mean Google, Facebook, Intel, Netflix, Microsoft. And those engineers, they know what they’re doing. But now, it’s quite clear that compression efficiency is the same as HEVC. Meanwhile, after removing other royalty cost for content delivery, HEVC Advance removed it. The license situation is much more clear now. You add to this the fact that at the end of the day, two to three years, you’re gonna need five to ten times more compute power to encode AV1, reaching effectively the same result. Now Google, again. Google may be that they have unlimited compute resources, they will use it. They developed it.
Dror Gill: 24:13 The smaller content providers, all the other ones, the non Googles of the world and other broadcasters with growing support for HEVC that we expect in a few years. I think it’s obvious. They’re gonna support HEVC and then a few years later when VVC is ratified, when it’s supported in devices, they’re gonna move to VVC. Because this codec does have the required compression efficiency improvement over HEVC.
Mark Donnigan: 24:39 Yeah, that’s an excellent summary Dror. Thank you for breaking this all down for our listeners so succinctly. I’m sure this is really gonna provide massive value. I want to thank our amazing audience because without you, the Video Insiders Podcast would just be Dror and me taking up bits on a server somewhere.
Mark Donnigan: 25:01 As you can tell, video is really exciting to us and so we’re so happy that you’ve joined us to listen. And again, this has been a production of Beamr Imaging Limited. Please, subscribe on iTunes and if you would like to try out beamer codecs in your lab or your production environment, we are giving away up to $100 of HEVC and H264 in coding every month. That’s each and every month. Just go to https://beamer.com/free and get started immediately.
When it comes to comparing video codecs, it’s easy to get caught up in the “codec war” mentality. If analyzing and purchasing codecs was as easy as comparing fuel economy in cars, it would undoubtedly take a lot of friction out of codec comparison, but the reality is that it’s not that simple.
In Episode 02, The Video Insiders go head-to-head comparing two of the leading codecs in a three against one standoff over whether AV1 is more efficient than HEVC.
So, which is more efficient?
Listen in to this week’s episode, “Codec Efficiency Is in the Eye of the Measurer,” to find out.
Tune in here or click play on the video below to listen now.
https://youtu.be/ThkoOHuPEvo
Want to join the conversation? Reach out to TheVideoInsiders@beamr.com.
TRANSCRIPTION (lightly edited to improve readability only)
Mark Donnigan: 00:41 Hi everyone I am Mark Donnigan and I want to welcome you to episode two of the Video Insiders.
Mark Donnigan: 00:50 In every episode of the Video Insiders we bring the latest inside information about what’s happening in the video technology industry from encoding, to packaging, to delivery, and playback, and even the business behind the video business. Every aspect of the video industry is covered in detail on the Video Insiders podcast.
Dror Gill: 01:11 Oh yeah, we usually do cover everything from pixels, to blocks, to microblocks, to frames, to sequences. We go all the way up and down the video delivery chain and highlight the most important things you should know before you send any video bits over the wire.
Mark Donnigan: 01:28 In our first episode we talked about a very hot topic which asked, “Hasn’t this kind of been worn out?” The whole HEVC, AV1 discussion. But I think it was very interesting. I sure enjoyed the talk. What about you Dror?
Dror Gill: 01:47 Yeah, yeah, yeah. I sure did. It was great talking about the two leading codecs. I don’t want to say the word, codec war.
Mark Donnigan: 01:58 No, no, we don’t believe in codec wars.
Mark Donnigan: 02:00 Yeah, that’s true. Why is it so complicated to compare video codecs? Why can’t it be as simple as fuel economy of cars, this one gets 20 miles per gallon and that one gets 30 and then I make a decision based on that.
Dror Gill: 02:15 I wish it was that simple with video codecs. In video compression you have so many parameters to consider. You have the encoding tools, tools are grouped into what’s called profiles and levels, or as AV1 calls them “experiments.”
Dror Gill: 02:35 When you compare the codecs which profiles and levels do you use. What rate control method? Which specific parameters do you set for each codec? And each codec can have hundreds, and hundreds of parameters. Then there is the question of implementation. Which software implementation of the codec do you use? Some implementations are reference implementations that are used for research, and others are highly performance optimized commercial implementations. Which one do you select for the test? And then, which operating system, what hardware do you run on, and obviously what test content? Because encoding two people talking, or encoding an action scene for a movie, is completely different.
Dror Gill: 03:13 Finally, when you come to evaluate your video, what quality measure do you use? There’re various objective quality measures and some people use actual human viewers and they assesses subjective quality of the video. On that front also, there’re many possibilities that you need to choose from.
Mark Donnigan: 03:32 Yeah, so many questions and no wonder the answers are not so clear. I was quite surprised when I recently read three different technical articles published at IBC actually, effectively comparing AV1 versus HEVC and I can assume that each of the authors did their research independently. What was surprising was they came to the exact same conclusion, AV1 has the same compression efficiency as HEVC. This is surprising because some other studies and one in particular (I think we’ll talk about) out there says the contrary. So can you explain what this means exactly, Dror.
Dror Gill: 04:16 By saying that they have the same compression efficiency, this means that they can reach the same quality at the same bitrate or the other way round. You need the same bitrate to reach that same quality. If you need for example, two and a half megabits per second to encode an HD video file using HEVC at a certain quality, then with AV1 you would need roughly the same bitrate to reach that same quality and this means that AV1 and HEVC provide the same compression level. In other words, this means that AV1 does not have any technical advantage over HEVC because it has the same compression efficiency. Of course that’s if we put aside all the loyalty issues but we discussed that last time. Right?
Mark Donnigan: 04:56 That’s right. The guys who wrote the three papers that I’m referencing are really top experts in the field. It’s not seminar work done by a student, not to downplay those papers, but the point is these are professionals. One was written by the BBC in cooperation with the Multimedia and Vision Group at the Queen Mary University of London. I think nobody is going to say that the BBC doesn’t know a thing or two about video. The second was written by Ateme, and the third by Harmonic, leading vendors.
Mark Donnigan: 05:29 I actually pulled out a couple of phrases from each that I’d like to quote. First the BBC and Queen Mary University, here is a conclusion that they wrote, “The results obtained show in general a similar performance between AV1 and the reference HEVC both objectively and subjectively.” Which is interesting because they did take the time to both do the visual assessment as well as use a quality measure.
Mark Donnigan: 06:01 Ateme said, “Results demonstrate AV1 to have equivalent performance to HEVC in terms of both objective and subjective video quality test results.”
Mark Donnigan: 06:16 And then here is what Harmonic said, “The findings are that AV1 is not more advantageous today than HEVC on the compression side and much more complex to encode than HEVC.” What do you make of this?
Dror Gill: 06:32 I don’t know. It sounds pretty bad to me, even two of those papers also analyzed subjective quality so they used actual human viewers to check out the quality. But Mark what if I told you that researchers from the University of Klagenfurt in Austria together with Bitmovin published a paper which showed completely different results. What would you say about that?
Dror Gill: 06:58 Last month in Athens I was the ICIP conference that’s the IEEE International Conference on Image Compression and Image Processing. There was this paper presented by this University in Austria with Bitmovin and their conclusion was, let me quote, “When using weighted PSNR, AV1 performs consistently better for bit rate compared to AVC, HEVC, and VP9.” So they claim AV1 is better than three codecs but specifically it’s better than HEVC. And then they have a table in their article that compares AV1 to HEVC for six different video clips. The table shows that with AV1 you get up to 25% lower bitrate at the same quality than HEVC.
Dror Gill: 07:43 I was sitting there in Athens last month when they presented this and I was shocked.
Mark Donnigan: 07:50 What are the chances that three independent papers are wrong and only this paper got it right? And by the way, the point here is not three against one because presumably there’re some other papers. I’m guessing other research floating around that might side with Bitmovin. The point is that three companies who no one is going to say that any of them are not experts and not highly qualified to do a video assessment, came up with such a different result. Tell us what you think is going on here?
Dror Gill: 08:28 I was thinking the same thing. How can that be. During the presentation I asked one of the authors who presented the paper a few questions and it turned out that they made some very questionable decisions in all of that sea of possibility that I talked about before. Decisions related to coding tools, codec parameters, and quality measures.
Dror Gill: 08:51 First of all, in this paper they didn’t show any results of subjective viewing. Only the objective metrics. Now we all know that you should always your eyes, right?
Dror Gill: 09:04 Objective metrics, nice numbers, but obviously you need to view the video because that’s how the actual viewers are going to assess the (video) quality. The second thing is that they only used the single objective metric and this was PSNR. PSNR, it stands for peak signal-to-noise ratio and basically this measure is a weighted average of the difference in peaks between pixel values of the two images.
Dror Gill: 09:30 Now, we’re Video Insiders, but even if you’re not an insider you know that PSNR is not a very good quality measure because it does not correlate very well with human vision. This is the measure that they choose to look at but what was most surprising is that there is a flag in the HEVC open source encoder which they used that if chosen, the result is improved PNSR. What it does, it turns off some psycho-visual optimizations which make the video look better but reduce the PSNR, and that’s turned on by default. So you would expect that they’re measuring PSNR they would turn that flag on so you would get higher PSNR. Well, they didn’t. They didn’t turn the flag on!
Dror Gill: 10:17 Finally, even then AV1 is much slower than HEVC, and they also reported in this data that it was much, much slower than HEVC but still they did not use the slowest encoding standing of HEVC, which would provide the best quality. There’s always a trade off between performance and quality. The more tools you employ the better quality you can squeeze out of the video, of course that takes you more CPU cycles but they used for HEVC, the third slowest setting which means this is the third best quality you can get with that codec and not the very best quality. When you handicap an HEVC encoder in this way, it’s not surprising that you get such poor results.
Dror Gill: 11:02 I think based on all these points everybody can understand why the results of this comparison were quite different than all of the other comparison that were published a month earlier at IBC (by Ateme, BBC, Harmonic).
Mark Donnigan: 11:14 Another critical topic that we have to cover is performance. If you measure the CPU performance on encoding time of AV1, I believe that it’s pretty universally understood that you are going to find it currently is a hundred times slower than HEVC. Is that correct?
Dror Gill: 11:32 Yeah, that’s right. Typically, you measure the performance of an encoder and FPS which is frames per second. For HEVC it’s common to measure an FPM which is frames per minute.
Mark Donnigan: 11:42 Frames per minute, (more like) frames per hour, FPH.
Dror Gill: 11:45 A year and a half ago or a year ago when there were very initial implementation, it was really FPD or FPH, Frames per hour or per day and you really needed to have a lot of patience, but now after they’ve done some work it’s only a hundred times slower than HEVC.
Mark Donnigan: 12:02 Yeah, that’s pretty good. They’re getting there. But some people say that the open source implementation of AV1 I believe it’s AOM ENC.
Mark Donnigan: 12:16 ENC exactly has not been optimized for performance at all. One thing I like about speed is either your encoder produces X number of frames per second or per minute, or it doesn’t. It’s really simple. Here is my next question for you. Proponents of AV1 are saying, “well it’s true it’s slow but it hasn’t been optimized, the open source implementation,” which is to imply that there’s a lot of room (for improvement) and that we’re just getting started, “don’t worry we’ll close the gap.” But if you look at the code, and by the way I may be a marketing guy but my formal education is computer science.
Mark Donnigan: 13:03 You can see it already includes performance optimizations. I mean eptimizations like MMX, SSE, there’s AVX instructions, there’s CPU optimization, there’s multithreading. It seems like they’re already trying to make this thing go faster. So how are they going to close this a hundred X (time) gap?
Dror Gill: 13:22 I don’t think they can. I mean a hundred X, that’s a lot and you know even the AV1 guys they even admit that they won’t be able to close the gap. I talked to a few senior people who’re involved in the Alliance for Open Media and even they told me that they expect AV1 to five to 10 times more complex than HEVC at the end of the road. In two to three years after all optimization are done, it’s still going to be more complex than HEVC.
Dror Gill: 13:55 Now, if you ask me why it’s so complex I’ll tell you my opinion. Okay, this is my personal opinion. I think it’s because they invested a lot of effort in side stepping the patents (HEVC).
Dror Gill: 14:07 They need to get that compression efficiency which is the same as HEVC but they need to use algorithms that are not patented. They have methods that use much more CPU resources than the original patent algorithms to reach the same results. You can call it kind of brute force implementation of the same thing to avoid the patent issue. That’s my personal opinion, but the end result I think is clear, it’s going to be five to 10 times slower than HEVC. It has the same compression efficiency so I think it’s quite questionable. This whole notion of using AV1 to get better results.
Mark Donnigan: 14:45 Absolutely. If you can encode let’s say on a single computer with HEVC a full ABR stack, this is what people want to do. But here we’re talking speeds that are so slow let’s just try and do (encode) one stream. Literally what you’re saying is you’ll need five to 10 computers to do the same encode with AV1. I mean, that’s just not viable. It doesn’t make sense to me.
Dror Gill: 15:14 Yeah, why would you invest so much encoding into getting the same results. If you look at another aspect of this, let’s talk about hardware encode. Companies that have large data centers, companies that are encoding vast amount of video content are not looking into moving from the traditional software encoding and CPUs and GPUs, to dedicated hardware. We’re hearing talks about FPGAs even ASICs … by the way this is a very interesting trend in itself that we’ll probably cover in one of the next episodes. But in the context of AV1, imagine a chip that is five to 10 times larger than an HEVC chip and which is the same complexity efficiency. The question I ask again is why? Why would anybody design such a chip, and why would anybody use it when HEVC is available today? It’s much easier to encode, royalty issues have been practically solved so you know?
Mark Donnigan: 16:06 Yeah, it’s a big mystery for sure. One thing I can say is the Alliance for Open Media has done a great service to HEVC by pushing the patent holders to finalize their licensing terms … and ultimately make them much more rational shall we say?
Mark Donnigan: 16:25 Let me say that as we’re an HEVC vendor and speaking on behalf of others (in the industry), we’re forever thankful to the Alliance for Open Media.
Dror Gill: 16:36 Definitely, without the push from AOM and the development of AV1 we would be stuck with HEVC royalty issue until this day.
Mark Donnigan: 16:44 That was not a pretty situation a few years back, wow!
Dror Gill: 16:48 No, no, but as we said in the last episode we have a “happy ending” now. (reference to episode 1)
Dror Gill: 16:52 Billions of devices support HEVC and royalty issues are pretty much solved, so that’s great. I think we’ve covered HEVC and AV1 pretty thoroughly in two episodes but what about the other codecs? There’s VP9, you could call that the predecessor of AV1, and then there’s VVC, which is the successor of HEVC. It’s the next codec developed by MPEG. Okay, VP9 and VVC I guess we have a topic for our next episode, right?
Narrator: 17:23 Thank you for listening to the Video Insider podcast a production of Beamr limited. To begin using Beamr codecs today go to beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.
If you spend the majority of your time pondering hot topics in the streaming media space, you’re not alone.
Our team of image scientists and video encoding engineers are equally passionate about all things video related. Which means, not surprisingly, we spend our days discussing everything from video encoding, playback, workflow architecture, codecs, compression techniques, and more.
But as we looked around the industry for sources of information on these topics we noticed a dearth of information, which is why we thought to ourselves, “let’s do something about this,” and launched The Video Insiders podcast.
Hosted by our CTO Dror Gill and me, Mark Donnigan, the show takes on a refreshingly non-partisan tone as the purpose is to equip the industry with information that is free of bias, spin, and agenda.
Join us as we dive into today’s top of mind discussions: compression, codecs, encoding, transcoding, and video workflows. We cover it all, and we do hope that you find the information valuable as you contribute to the industry in your role, whether it be engineering, operations, or business.
Mark Donnigan: 00:15 Hi everyone, my name is Mark Donnigan and I am super excited to be opening this first episode of The Video Insiders. But before I say anything more I want to introduce you to my esteemed co-host, Dror Gill. Welcome Dror.
Dror Gill: 00:31 Thank you Mark. You know if you could do some e-motion estimation on me, you would also find out that I’m pretty excited to be opening this podcast and I think it’s gonna be awesome. Mark maybe we should tell everyone what they’re getting themselves into.
Mark Donnigan: 00:45 So, we are the Video Insiders and in every episode we will bring the latest inside information about what’s happening in the video technology industry from codecs both past, present and maybe even some not invented yet, you know. I don’t know, maybe we’ll invent one. What do you think Dror?
Mark Donnigan: 01:05 And obviously we’ll cover video in coding, trans coding and you know I’m a little bit more of a business guy. You’re a serious technologist and our listeners will get to know you but at the end of the day business drives this. So, we’ll cover business and I can’t think of too much that’s really off limits, right?
Dror Gill: 01:27 No, no, we’re gonna talk about everything, everything related to video. But first Mark I have to tell the listeners a small secret about you.
Dror Gill: 01:37 Those who know you best probably have been privileged to pound the keyboard.
Mark Donnigan: 01:43 Pound the keyboard, hmm, well you know when I, you know my new MacBook, you know it is pretty noisy. Man, what’s up with that keyboard, wow.
Dror Gill: 01:55 Not that kind of keyboard, I just happened to know that you’ve studied some classical piano performance and jazz in college, right?
Mark Donnigan: 02:03 Oo, this is true. Not too many people know that about me. That’s true, that is true. But it was a long time ago. But while we’re on the subject of music, you know, discussing beta release schedules kinda has a dual meaning for you, doesn’t it Dror?
Dror Gill: 02:21 Oh, yeah, yeah. You got me there. So, yeah, Beta is the name of my rock band. I’m a land singer of a rock band called Beta.
Dror Gill: 02:31 Yeah, we do some gigs, we’ve recorded a few tracks. So, when you discuss a beta schedule with me that could be just our tour schedule and not our milestones.
Mark Donnigan: 02:42 Well, there you have it. You know, look if we do our jobs this podcast just may rock our listeners video world, right?
Dror Gill: 02:50 Yeah, yeah, for sure. And if not, at least it will be the number one destination for their latest news and analysis about video compression, protection, monetization, applications, you know we’re gonna talk about all those buzzy words. Buzz words such as HDR, UHD, HEVC, AV1.
START OF DISCUSSION
HEVC triumph?
Mark Donnigan: 03:13 That’s pretty buzzy. That’s pretty buzzy. Well, hey HEVC and AV1, I’m glad you brought those codecs up, because some may say the topic is really been run into the ground. So, what do you think Dror?
Dror Gill: 03:26 I think it’s still a great topic, you know, because it has all the elements of a good story. It’s like a true hero’s journey. You know, you might say it has Hollywood glamour, has some tragedy, has its ups and downs and finally does have a happy ending.
Mark Donnigan: 03:42 Well, happy endings are good, so to kick this off for our listeners who don’t have a deep technical background. What is HEVC?
Dror Gill: 03:51 HEVC is the latest video standard. It was developed by the MPEG committee. MPEG is short for Motion Pictures Experts Group. Notice the motion pictures, yes they don’t call it video. We’re still in like the old days of when cinema just started. It’s not a video it’s just a bunch of moving pictures.
Dror Gill: 04:19 So, MPEG developed a lot of video standards. All the MPEG standards, there was MPEG 1, MPEG 2, MPEG 4 and then MPEG 4 AVC. Which some people know by the name H.264. And finally MPEG HEVC, which some people call H.265. But don’t do that Mark. Don’t call it H.265 because Leonardo might ban you.
Mark Donnigan: 04:43 Now, wait a second, how will Leonardo DiCaprio ban me?
Dror Gill: 04:46 Oh no, not Leonardo DiCaprio, Mark. You’re taking the Hollywood reference a bit too far. I’m talking about Leonardo Chiariglione. He’s been the founder and chairman of the MPEG committee for the past 30 years.
Dror Gill: 05:03 You know, many people call him the father of MPEG because you know he founded the committee and he’s been leading it. So, I heard a rumor that if he used the term H.265 instead of HEVC than he might ban you from the MPEG mail reflectors forever. So, you know, be careful.
Mark Donnigan: 05:19 Well, I mean pretty successful stuff that the MPEG committee has developed. MPEG 2, you know it’s used in all digital cable, satellite, terrestrial broadcast, DVD. MPEG 4, Blu-ray discs. Everything on the internet is streamed with MPEG 4.
Mark Donnigan: 05:53 Now, can you imagine the Leonardo that I’m thinking of and all the engineering geeks from MPEG standing there on the same stage as the other Leonardo from the MPEG committee. That’s really amazing. That’s pretty cool actually.
Dror Gill: 06:09 Yeah, yeah it is. You know, what’s even more amazing? Every generation of the standard they essentially double the compression efficiency. So, for example with HEVC you can get the same quality as H.264 but at half the bit rate. Now, I’m not saying you can do this all the time and not for every content and it’s not true for every resolution. I have to be very careful here cause you know, some people in the industry spread information that is not 100 percent accurate.
Dror Gill: 06:39 And sometimes it is possible that we will discuss such occasions on a podcast. But I’d like to be on the safe side so let’s say between 40 and 50 percent, depending on the resolution. That’s the improvement in compression efficiency would get with HEVC.
Mark Donnigan: 06:57 Sure, which is really great and pretty amazing. So, why isn’t everyone using HEVC instead of H.264? I mean, with video occupying, I mean the studies keep on, this number goes up seems like by the month by 70 percent the internet is video traffic. Netflix is like 15 percent of bandwidth consumed at peak viewing time. That’s staggering if you think about it. I would think everyone would just want to switch to HEVC and either enjoy the savings or be able to double their capacity. What’s going on there?
Dror Gill: 07:31 Yeah, yeah, that would have been really what you would expect with HEVC being much more efficient than H.264 but this is really where the tragedy part of our story comes in. There’s a lot of sophisticated algorithms which actually enable this great compression efficiency and many companies would jointly created the HEVC standard. They obviously have patents on all of these technologies that they developed. So, with H.264, the situation was simple. There were companies that developed algorithms, they had patents but there was a single patent pool that was organized by a company called MPEG LA. And you could just license all the patents very easily in this one stop shop. But with HEVC the problem was that the patent holders joined three different pool and some of them didn’t join any pools. So, you act of this unreasonable licensing terms which were published by one of the patent pools initially and their half of the industry was in complete chaos regarding HEVC licensing.
Mark Donnigan: 08:40 Wow, sounds pretty bad. Leonard Chiariglione wrote a pretty famous blog post called, ‘The Crisis, the Causes and it’s Solutions’, if I remember correctly.
Dror Gill: 08:55 Yes. It was very sad and he said that from his point of view the MPEG way has reached an end. They developed this great technology and then nobody uses them because of licensing issues. So, that was the peak of the tragedy but luckily for us and I think for the whole industry, finally, the patent holders came to their senses. And now everything is much more clear. I mean the two major patent’s pools which are MPEG LA and HEVC Advance, both of them are not charging any royalties for content.
Dror Gill: 09:34 And content providers around the industry and the third one Velos Media. They haven’t officially announced their licensing terms but everybody’s expecting them also to be reasonable and I’ve heard some things off the record that they’re also not going to charge for content. I mean, they don’t want to cut the branch they’re sitting on. If they’re unreasonable then nobody will license it and they get no money. So, I think basically where we are today you can say that HEVC patent world, these are mostly paid by the vendor of devices such as mobile phones and TVs. All those companies will put chips that support HEVC decoding and the devices and I understand the, Mark, that many of them are already supporting HEVC, right? A lot of devices out there.
Mark Donnigan: 10:18 You know earlier this year Beamr actually sponsored an industry survey which Tim Siglin a contributor to Streaming Media helped us with and also Streaming Media promoted it and helped put it together. The findings were really positive. This was completely a non partisan, fair and balanced survey. 66 percent of those that responded of which was a very high percentage of engineers by the way, more than 400. So, these are industry professionals – reported that HEVC was already in production. So, the reality is that though in some circles there still maybe a little bit of FUD. You know, that’s fear, uncertainty and doubt. The fact is is that more than half the industry has HEVC encoding pipelines running or they’re in the implementation stage. These things don’t happen overnight, so in some cases maybe they’re not encoding HEVC content yet but they will be. It’s a foregone conclusion.
Mark Donnigan: 11:33 77 percent so more than three out of four said HEVC was a very viable replacement for H.264. That’s obviously good news but if we’re making files that can’t be played then I suppose you could file this under the fake news category.
Dror Gill: 11:49 Yeah, if you encode the files somebody has to play them.
Mark Donnigan: 11:53 Exactly. And we’re gonna get to that when we talk about AV1, but when we look at the situation for device support it’s a little bit hard to get the exact numbers but industry seems to be kinda gathering around two billion devices is where most people, you know nod their head and say yes, we believe that number. That’s two billion today that support HEVC in hardware. So, if we think about, first of all it’s a massive number right? But you think about iPhones. Everything from the 6S forward supports HEVC and some markets iPhone penetration is way over 50 percent. And then when you factor in Samsung like the S8, S9, Note 8, Note 9, Galaxy Tab S3. I mean phones from LG, phones from Sony. You look at Macs, you look at PCs especially if they have the Intel Skylake or the Kaby Lake chips. Every 4K TV sold today, I mean I could go on and on.
Mark Donnigan: 12:54 Roku boxes, the new Apple TV, that is the Apple TV 4K. Chromecast even, Amazon Fire TV. So, all of those support HEVC. We heard from a very well placed industry insider that one MSO recently discovered of their users which number tens of millions, (that) more than 50 percent of their customers had at least one device in the home that supported HEVC. Which is very, encouraging.
Dror Gill: 13:32 Yeah, it’s very encouraging. It means that HEVC support is out there. I mean two billion devices, that’s a really large number. So, if you look at it, the situation with HEVC I would say it’s pretty clear. I mean, it’s half the bit rate of H.264 AVC. Royalty issues have mostly been solved, a few billion devices supported in hardware. So, there it is Mark. I think that’s the happy ending I was looking for. So, okay, that’s HEVC, where are we with AV1?
AV1 tragedy?
Mark Donnigan: 14:04 That’s a good question. You know, it’s unclear at the moment. Look, we are definitely fair and balanced here on the Video Insiders so we don’t present things from an agenda. First of all AV1’s open source, right? It is a supposedly a royalty free codec. It’s been developed by a extremely strong group of companies. Companies that if anybody can pull this off it’s Google, it’s Netflix, it’s Facebook, it’s Intel, it’s Microsoft, it’s Apple and there are others. These people are our friends and they’re good, they can build a codec. They could do this. They set out to develop the AV1 codec really in response to the royalty issue. You know, you might get different responses, I haven’t really gone around and asked privately any of these individual companies. But I’m guessing that off the record most would say yeah, if HEVC Advance had been rational, the Alliance for Open Media wouldn’t have been needed and AV1 wouldn’t exist and things would be more clear.
Dror Gill: 15:11 Right, and I understand they want to develop a royalty free codec but I think I heard you say “supposedly royalty free.” Cause I thought AV1 was royalty free, I mean look what it says here. I’m on the AOM website, I’m reading from the website, it says, “Collaborative effort to offer open royalty free and interoperable solutions for the next generation of media delivery.” So, it says royalty free, doesn’t it? I mean, can it be more clearer than that?
Mark Donnigan: 15:36 You can challenge me on that but it’s not that simple. Here’s the deal, the members of the Alliance for Open Media all agreed to contribute their own IP to the AV1 standard and not charge royalties for it, but it’s possible. Again, I’m saying “possible”, I’m not a lawyer, we’re not giving any legal advice so I wanna be careful, but it is possible that other parties who are not members of the Alliance for Open Media could have IP claims or at least could attempt to assert IP claims and to charge royalties on the codec. Now, alliance for open media, AOM, has set up a fund to indemnify licensees, or so I’ve heard. At the end of the day there’s just a lot of questions that don’t have answers. It’s one thing when it’s, you know, engineers sitting around at an industry conference or on a panel having a debate and discussion. It’s a whole different thing when a fortune 50 or a fortune 25 media company who could be facing hundreds of millions if not billions of dollars of fines or levees or whatever. You just can’t sort of sweep this stuff under the rug.
Mark Donnigan: 16:56 I think unfortunately that’s kinda what is happening a little bit in my opinion.
Dror Gill: 17:03 At the end of the day I think we need to encourage our listeners to do their own due diligence, you know, because in some industry conferences you can be led to believe the situation is very secure but really nobody is indemnifying you here. So, you really need to be careful. Anyway, on the device side Mark, how many devices support AV1 in hardware?
Mark Donnigan: 17:24 Yeah, so I’m glad you asked, and you know this is what my background is in, consumer electronics. I have a lot of experience there. We get into this situation where it’s easy to sit around and talk about creating files with these new exciting codecs but people kind of forget that VLC is not a viable solution to play back. I’ve been doing some research though and I actually was able to connect via email and in some cases have voice conversations with the SOC vendors, that is the system-on-the-chip vendors who are the ones at the end of the day that have to be convinced to integrate AV1 decode into their silicon.
Mark Donnigan: 18:25 But you know it’s these SOC vendors that at the end of the day is where I’m gonna get the real information. I heard what I anticipated to hear that obviously support today is zero. I heard varying degrees of responses such as we’re thinking about it. It’s getting onto the road map to a slightly more optimistic where some people said in 24 months we’ll have it, maybe sooner. But you should anticipate two years. And then on the other side I got the usual hey look, you know, when there’s a business case that is when I can’t sell a chip because it doesn’t have AV1, that’s when we’ll support it. You know the situation is really this, I am very confident again because the companies behind AV1 that somewhere in 2020 – It’s just not even possible to complete the engineering cycles, the fab cycles for the silicon any sooner than really the end of 2020. So, we’re really talking 24 months, we’re recording this podcast on the last day of October, October 31st, 2018. I anticipate that Christmas season 2020 there will be an (AV1) device.
Mark Donnigan: 19:44 Will it be a Roku box? Will it be a TV? Will it be a game console? Or even just some new product all together, I don’t know.
Mark Donnigan: 19:54 Maybe an Android phone, exactly. There will be an AV1 device and there’s no doubt that certain folks that are incented for AV1 to succeed will write articles saying that the world is about to flip. At the end of the day one device, a device does not build an ecosystem. It takes two billion devices that are in real consumer’s hands.
Mark Donnigan: 20:20 You know, that somebody doesn’t even have to download an app or install something, they just turn it on and it just works. And that we are years and years and years away.
Dror Gill: 20:33 Yeah, I think I agree with you. It will take time until the chips will come out and then tested and devices will come out. So, I’m very bullish about HEVC, I think the message for the industry now is very clear. HEVC is happening today and it’s here to stay.
Mark Donnigan: 21:06 Well let’s bring this to a close. It’s been an awesome discussion about how to compare HEVC and AV1.
Dror Gill: 21:13 Well, comparing codecs that’s a pretty big question and you never get the same answer. It really depends on who you ask so it’s a really interesting topic, but it seems like our time is up so how about we discuss that in our next episode?
Mark Donnigan: 21:29 I want to thank everyone for listening to The Video Insiders’ podcast. This is a production of Beamr Imaging Limited. Please subscribe on iTunes and we will be expanding to other platforms shortly. If you’d like to try out Beamr codecs in your lab or production environment we’re giving away up to 100 hours of HEVC and H.264 encoding every month. Go to https://beamr.com/free to get started immediately.
If you would like to join the discussion as a guest, send an email to thevideoinsiders@beamr.com with a suggested topic that you would like to share.
This year we presented at the NAB show – the world’s largest electronic media show covering the creation, management and delivery of content across all platforms – in two booths, following therecently announced acquisition of Vanguard Video.
Double demos
At the Beamr booth, we demonstrated our content-adaptive optimization solutions, which reduce the bitrate of H.264 and HEVC streams by 20-50%. Our booth visitors were especially impressed by the demo, which showed our ability to reduce the bitrate of a 4K HEVC file from 14.2 Mbps to only 6.8 Mbps, with no quality loss. We challenged the industry’s “Golden Eyes” to tell them apart, but not one visitor to our booth could find any difference…
We also demonstrated our compliance with MPEG-2 Transport Stream and Blu-ray, playing optimized streams on a Cable set-top box over QAM, and on a Blu-ray disc player.
In the Vanguard by Beamr booth, we showed our V.265 HEVC encoder, which supports 8/10/12-bit video with both Dolby Vision and HDR 10. We also let our visitors put PACE (Parallel Architecture for Cloud Encoding) through its paces, showing a significant performance boost for distributed encoding on multiple servers. Another demo which caught the eye of our visitors was the streaming of live 720p60 HEVC video, captured on an iPad, encoded in real-time by our V.265-ARM SDK in pure software, and played back on an Apple TV device.
Double excitement
Our week at NAB was packed with back-to-back meetings, as we introduced Beamr’s customers to Vanguard Video’s solutions, and vice versa. We were flattered and excited by the positive reactions of our customers and visitors at both booths to the acquisition, and we heard from everyone that they are looking forward to the huge potential of the combined product line. We are ready for this challenge, and have already started working on bringing the benefits of Beamr’s perceptual optimization into Vanguard Video’s encoders.
An interesting trend that we witnessed at the show was that content providers are holding back on deployment of 4K content offerings and broadcast channels, opting instead to put a stronger focus on HD content with HDR, and on VR content that was clearly the strongest emerging trend at the show. It seems that industry players have realized that the viewing experience enabled with HDR, and the storytelling techniques made possible with VR video, are more compelling to viewers than 4K content which “just” has more pixels.
Regardless of the type of video being used by the industry moving forward, it is clear that the amount of video delivered will continue to grow, and we are now even more excited to produce the world’s most optimized encoding solutions, enabling the highest video qualities at the lowest bitrates.
![2018, the Year HEVC Took Flight. [podcast]](https://blog.beamr.com/wp-content/uploads/2019/01/Beamr-Podcast-banner-D-800x544.jpg)
![The Future of 3 Character Codecs. [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-v3c-800x544.jpg)
![Codec Efficiency is in the Eye of the Measurer. [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-B-1-1-800x544.jpg)
![Will the Story End a Tragedy or Triumph? [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-v3a-1-800x544.jpg)
