In case you missed it, Apple just snuck a little surprise into the first few days of 2018 in the form of their name appearing on the Alliance for Open Media (AOM) website as a Founding Member. As an ardent HEVC supporter, some may be shocked at this move by Apple. Blog post detailing Apple’s HEVC announcement. Now let’s see what it means for HEVC.
As a codec engineering company, Beamr has invested heavily in our HEVC implementation and we are proud of the best in class customers who are using it around the world to distribute video that is higher quality and up to 50% smaller than the H.264 version.
This means we do have a “vested” interest in HEVC being successful. At the same time, whether VP9 or AV1, we are always tracking the development of new codec technology so that we are in the strongest position to bring our extensive codec development resources to bear on market leading solutions.
Though we have an interest in HEVC becoming successful, we have invested resources and continue to do so, in order to understand AV1 in the areas of market readiness and licensing preparedness including IP questions, playback support and more.
In this article I will share the Beamr perspective that Apple joining the AOM reinforces the possibility that AV1 will be the successor to HEVC. However, with 1 billion HEVC enabled end points in the market, HEVC has legs for many years before a sufficiently large AV1 ecosystem will be built.
This position is also in alignment with many of our customers who are serving hundreds of millions of end users and must make codec decisions based on streams they can reliably deliver today.
After all, speculating on what may be coming in the future is not a luxury most of the industry enjoys because if they bet wrong, it could impact tens of millions of users negatively. There is a huge difference between advanced technology development (e.g. what happens in the lab) and the realities of production (that which generates revenue).
Once the AOM locks down the AV1 spec, you can expect many shootouts and comparisons with HEVC to be published. But let’s take a look at how HEVC compares to AV1, given what we know now.
AV1 Readiness compared to HEVC.
HEVC was ratified in 2013 while the AV1 bitstream was set to be frozen in Q1 2017, yet even now the AV1 bitstream has not been completed. Developing software timelines, committing to them, and then meeting them, is far from an exact science. Thus the delay is not completely the fault of the AOM development community since it is endemic to the software development lifecycle. Innovation is difficult to schedule. The point is, AV1 will be ready when it is ready. Which means commercial plans that hinge on the delivery of AV1 before 2020 or 2021 could be at risk given the uncertainty of when the standard will be ratified.
AV1 Compression Efficiency compared to HEVC.
HEVC is recognized to be 40-50% more efficient than AVC (H.264), and AV1 is hoped to be up to 30% more efficient than HEVC (H.265).
However, while HEVC’s compression efficiency has already been reached by advanced encoder implementations such as Beamr 5, AV1’s 30% efficiency claim over HEVC has not been proven outside of an extremely limited (small) set of files.
In any case any improvement can be validated only after the spec is final and the tools included in AV1 are decided upon. At that point the race to realize these gains will start, balancing the computing resources needed and maturing the rate control algorithms. But just as HEVC did not reach its planned 50% efficiency in the first release, taking multiple years to achieve, the AOM developers will need to work very hard for the next 2 to 3 years before significant gains over HEVC will be seen.
AV1 Royalty and IP constraints compared to HEVC.
There are three HEVC Patent pools which license the technology used in HEVC implementations: MPEG-LA, HEVC Advance, and Velos Media. Both MPEG-LA and Velos Media do not charge license fees for content distribution (See the MPEG-LA HEVC License Summary and the Velos Media FAQ), and HEVC Advance does not charge a license fee for free content distribution, such as public broadcasts and ad-funded commercial broadcasts (see page 3 of the HEVC Royalty Rates document).
Even Technicolor, that licenses its HEVC patents outside of the 3 patent pools, has publicly declared that they will not charge license fees from content providers. In addition, royalty schedules are being (have been) amended down, and it seems the Patent pools are aware that a more friendly approach is needed.
AV1 cannot guarantee a royalty-free offer.
Yes, that’s correct, I said it! Now here’s why.
While AV1 claims to be royalty-free, many industry players have missed the fact that the Alliance for Open Media does not provide indemnification to companies who use AV1 against patent claim violations.
Since some of the algorithms used in AV1 bear a resemblance to corresponding H.264 and HEVC algorithms, there is some probability that the IP in AV1 could infringe on AVC and/or HEVC Patents. In fact, delays to the ratification of the AV1 standard might well be due to legal teams who are examining the final algorithms exactly for these cases.
To be fair, IP questions are hardly ever cut and dried, and there are many unknowns and “what-if’s” to be discussed. But the lack of clarity regarding the AV1 IP situation, and the fact that AOM is not offering indemnification for IP infringement, makes the “royalty free” claim at this point more of a wish than a solid fact.
If you are still not convinced that AV1 offering a royalty-free codec could be problematic, consider that for VP8 and VP9 Google needed to license the H.264 patents from MPEG-LA. If an infringement action is identified with AV1, and if the courts rule in the plaintiff’s favor, the legal exposure will be of the magnitude that headlines will be penned and stock prices hammered. Velos Media, one of the 3 HEVC patent pools, has already warned in its FAQ:
“As it relates to royalties, we know that VP9 incorporates patented technologies, including some of the patents being licensed by Velos Media for HEVC. And, while AV1 has not yet been publicly released, it may also incorporate patented technology from many parties.”
Try slipping that little disclaimer by a corporate IP attorney! When a licensing body directly references a new technology as being possibly infringing, it is worth paying attention to.
Let’s look at why the AOM members feel it’s so important to have a royalty-free platform.
- Royalties are a pain, that is, when you are the party that needs to pay. Not only do you need to factor this added cost into your business model but in some cases, the tracking and reporting burden by itself represents a real difficulty and may limit certain business models from being feasible.
- You can never know what patent holder’s next demand will be. The HEVC fragmented IP pools and shifting fee structures prove the point. The AOM is absolutely correct in wanting to address this with AV1.
- Confusion over who to pay. The lack of clarity overpayments is delaying adoption even further. It seems AOM members are seeking control over a critical component of their technology that is not royalty free.
While all these arguments make sense, AOM cannot guarantee its users that this is a royalty-free codec; all they can suggest is that AOM members will not ask for royalties. But the thousands of patent holders for block-based codecs that are not a part of AOM are still out there, and the generosity of AOM could come at their expense.
For an interesting analysis on the validity and value of the MPEG-LA and HEVC Advance HEVC patent pools, you will want to read this Unified Patents article as it provides a perspective on how the courts look at the patents that are contained in a pool. In short, it’s probably not a wise legal move to assume that the AOM has everyone adopting AV1 “covered.”
AV1 Encoding CPU Performance compared to HEVC.
HEVC encoding CPU performance is advancing at a rapid rate. At IBC 2017 Beamr demonstrated six simultaneous 10-bit 4Kp60 live channels being encoded on a single Intel Xeon Scalable Platinum 8180 dual-socket server.
This encoding speed is in contrast to AV1, which has not been optimized, but is currently running about 100 times slower than real-time on a single server. Aside from the fact that AV1 is not available for low latency live encoding workflows, the operational cost delta of running an AV1 encoding service versus HEVC is staggeringly higher for AV1.
Download the Intel solution guides detailing Beamr HEVC codec SDK performance and applications on Intel processors.
Beamr’s HEVC encoder has been under active development for more than five years, and our CPU performance has been consistently improved by way of algorithmic and code optimizations. AV1 developers will bring improvements to the encoding speed, but it will most certainly follow the same development trend of every codec before it, including HEVC.
In other words, it can only happen over a period of years, and only if a group of dedicated engineers focuses on it day and night. Optimizing a codec is not a hobby. But even in the end after it is fully optimized AV1 will be slower because of the added mathematical complexity needed for it to achieve higher efficiency.
Video distributors with capex, opex, or physical space constraints will find the bitrate efficiency gains of AV1 will come with a very high operational cost. This Jan Ozer article from Streaming Media provides further context on AV1 performance.
AV1 Playback and Decoding Performance compared to HEVC.
HEVC hardware decoder support exists today in more than 1 billion devices spanning the most popular computing and mobile operating systems in the market like iOS, macOS, Android, and Windows. And in addition, low power hardware implementations for HEVC exist on Intel and ARM-based chips, as well as hundreds of millions of SoC’s shipping in CE devices such as TV’s, media players and game consoles. HEVC is a de-facto standard in all UHD TV’s found in the market.
According to the Consumer Technology Association (CTA) between 2014 and thru 2017, somewhere in the neighborhood of 175 million UHD TV’s will have been sold. Source: CTA 2017 presentation, data from GfK
This is an impressive HEVC footprint, and it’s only going to grow stronger. In contrast, the only AV1 playback environment available today is an early alpha implementation of the Mozilla browser Nightly build.
As the AV1 spec is not yet locked by the AOM it’s no wonder that when we reached out to the best-known silicon vendors who are supplying the media processing and video decoding chips to the most popular media player vendors, game console makers and TV OEM’s, everyone reported without exception that they cannot begin planning AV1 support in the absence of a ratified spec.
AV1 support in silicon is a minimum of 24 months out (Q1 2020). But silicon is just that, silicon. Chips have to be designed and integrated into consumer products before the advanced capabilities that chip vendors are including can be available. But, this is a chicken and egg situation. Afterall, why would a CE company go through the extra engineering cycles and increase their BOM to support a codec that is yet to be adopted across the ecosystem?
Should Apple influence your codec adoption decision?
With Apple’s commitment to HEVC, it’s a mystery of what their intentions are in joining the AOM. But we do know this – Apple has a solid HEVC roadmap with a vertically integrated HEVC video encoding, distribution and playback technology stack across all devices.
This means that today, you can reach the Apple ecosystem with H.264 and HEVC.
For video distributors looking for smaller bandwidth footprints, opting out of HEVC means they will lose a whopping 54% of the North American mobile market assuming the selected codec isn’t supported by Apple. This is hardly a prospect that any executive or encoding head will agree to, which means the question of whether to support AV1 or HEVC could come down to compatibility.
Any video service unable to match the performance of HEVC on Apple, will have a difficult time competing with services delivering high quality 1080p HD video at bitrates well under 2 Mbps. HEVC is available today across the all too important Apple ecosystem, while AV1 is not supported (today).
HEVC is the codec for today. Is AV1 the codec for the future?
Remember how I said that Beamr is actively tracking all new video technology and codec developments and that this applies to AV1? Well, we do believe that AV1 could be a factor at some point in the future.
There are those who always wait for next years model. The trouble with this approach is that by delaying, you miss out on technology leaps that could have afforded a significant advantage to your company. This is especially true with the decision to move ahead or hold on adopting HEVC in anticipation that AV1 will be cheaper and provide an added efficiency benefit.
The reality is that HEVC is able to reach 40% to 50% efficiency gains over H.264 today. These aren’t theoretical numbers or only possible on a limited set of content. Beamr has customers distributing content around the world, and enjoying bitrate savings in this range. Even Apple in their WWDC2017 announcements of HEVC, used the numbers 40% and 50% savings when talking about their decision to adopt the HEVC standard.
Ask yourself, what is the opportunity cost incurred by continuing with H.264 for the next 24 to 36 months as you wait for a relatively small AV1 playback footprint to emerge?
This is why most of the industry operating a commercial service is opting to realize the benefits of HEVC today while keeping tabs on the development of AV1 (for the future).
Still not sure…
Consider that Amazon Prime and Netflix are both members of AOM. And both are active in AV1 development and testing, yet Amazon and Netflix are users of HEVC.
Why would they do this? It’s simple. HEVC serves them well by being compatible with more than 1 billion devices and enabling premium video experiences at bitrates that are 40 to 50% less than H.264. Now, one may think that HEVC is only being used for 4K content by these services, but we know that Amazon is using HEVC in emerging markets for lower resolutions.
But what about Google, they are Founding members of the AOM and ardent supporters of alternative codecs? It’s interesting to note that Google supports HEVC in Chromecast which is clearly required for content services (some who are in the AOM like Netflix and Amazon) to stream 4K HDR video. But will Google use HEVC for their own services on a wider basis in the future? We do not know.
One situation in the market that must be faced by Google and YouTube is what will happen if Apple deprecates H.264, and mandates all apps that stream video to Apple devices leverage HEVC. Remember Flash? A similar situation occurred with HLS, the mandatory streaming protocol for Apple devices.
Remember how I pointed out that Apple is vertically integrating with HEVC for video and HEIC (HEVC I-frame) for mobile image capture and display? I think now you can see how a codec selection decision by a vendor like Apple can move the entire industry.
Apple has chosen HEVC for production and any video distributor can encode in HEVC and transmit to a user with iOS 11 or macOS High Sierra and know that it will play perfectly.
HEVC is a robust standard that has broad support with extensive development from the largest encoding vendors in the industry. And HEVC is widely adopted on the device side with major services like Amazon, Apple, and Netflix using it now. HEVC is not going away.
Beamr’s view is that HEVC is the codec for today, and AV1 may possibly be the codec of tomorrow. But for sure, with HEVC, we can all enjoy more video and better quality as HEVC enables new applications, experiences, and innovations to be transmitted to users today.