Easy & Safe Codec Modernization with Beamr using Nvidia GPUs
Following a decade where AVC/H.264 was the clear ruler of the video encoding world, the last years have seen many video coding options battling to conquer the video arena. For some insights on the race between modern coding standards you can check out our corresponding blog post.
Today we want to share how easy it can be to upgrade your content to a new and improved codec in a fast, fully automatic process which guarantees the visual quality of the content will not be harmed. This makes the switchover to newer encoders a smooth, easy and low cost process which can help accelerate the adoption of new standards such as HEVC and AV1. When this transformation is done using a combination of Beamr’s technology with the Nvidia NVENC encoder, using their recently released APIs, it becomes a particularly cutting-edge solution, enjoying the benefits of the leading solution in hardware AV1 encoding.
The benefit of switching to more modern codecs lies of course in the higher compression efficiency that they offer. While the extent of improvement is very dependent on the actual content, bitrates and encoders used, HEVC is considered to offer gains of 30%-50% over AVC, meaning that for the same quality you can spend up to 50% fewer bits. For AV1 this increase is generally a bit higher.. As more and more on-device support is added for these newer codecs, the advantage of utilizing them to reduce both storage and bandwidth is clear.
Generally speaking, performing such codec modernization involves some non-trivial steps.
First, you need to get access to the modern encoder you want to use, and know enough about it in order to configure the encoder correctly for your needs. Then you can proceed to encoding using one of the following approaches.
The first approach is to perform bit-rate driven encoding. One possibility is to use conservative bitrates, in which case the potential reduction in size will not be achieved. Another possibility is to set target bitrates that reflect the expected savings, in which case there is a risk of losing quality. For example, In an experimental test of files which were converted from their AVC source to HEVC, we found that on average, a bitrate reduction of 50% could be obtained when using the Beamr CABR codec modernization approach. However, when the same files were all brute-force encoded to HEVC at 50% reduced bitrate, using the same encoder and configuration, the quality took a hit for some of the files.
This example shows the full AVC source frame on top, with the transcodes to HEVC below it. Note the distortion in the blind HEVC encode, shown on the left, compared to the true-to-source video transformed with CABR on the right.
The second approach is to perform the transcode using a quality driven encode, for instance using the constant QP (Quantization Parameter) or CRF (Constant Rate Factor) encoding modes with conservative values, which will in all likelihood preserve the quality. However, in this case you are likely to unnecessarily “blow up” some of your files to much higher bitrates. For example, for the UGC content shown below, transcoding to HEVC using a software encoder and CRF set to 21 almost doubled the file size.
Yet another approach is to use a trial and error encode process for each file or even each scene, manually verifying that a good target encoding setup was selected which minimizes the bitrate while preserving the quality. This is of course an expensive and cumbersome process, and entirely unscalable.
By using Beamr CABR this is all done for you under the hood, in a fully automatic process, which makes optimized choices for each and every frame in your video, selecting the lowest bitrate that will still perfectly preserve the source visual quality. When performed using the Nvidia NVENC SDK with interfaces to Beamr’s CABR technology, this transformation is significantly accelerated and becomes even more cost effective.
The codec modernization flow is demonstrated for AVC to HEVC conversion in the above high-level block diagram. As shown here, the CABR controller interacts with NVENC, Nvidia’s hardware video encoder, using the new APIs Nvidia has created for this purpose. At the heart of the CABR controller lies Beamr’s Quality Measure, BQM, a unique, patented, Emmy award winning perceptual video quality measure. BQM has now been adapted and ported to the Nvidia GPU platform, resulting in significant acceleration of the optimization process .
The Beamr optimization technology can be used not only for codec modernization, but also to reduce bitrate of an input video, or of a target encode, while guaranteeing the perceptual quality is preserved, thus creating encodes with the same perceptual quality at lower bitrates or file sizes. In any and every usage of the Beamr CABR solution, size or bitrate are reduced as much as possible while each frame of the optimized encode is guaranteed to be perceptually identical to the reference. The codec modernization use case is particularly exciting as it puts the ability to migrate to more efficient and sophisticated codecs, previously used primarily by video experts, into the hands of any user with video content.
For more information please contact us at info@beamr.com
2023 is a very exciting year for Beamr. In February Beamr became a public company on NASDAQ:BMR on the premise of making our video optimization technology globally available as a SaaS. This month we are already announcing a second milestone for 2023: Release of the Nvidia driver that enables running our technology on the Nvidia platform. This is a result of a 2 year joint project, where Beamr engineers worked alongside the amazing engineering team at Nvidia to ensure that the Beamr solution can be integrated with all Nvidia codecs – AVC, HEVC and AV1.
The new NVENC driver, just now made public, provides an API that allows external control over NVENC, enabling Nvidia partners such as Beamr to tightly integrate with the NVENC H/W encoders for AVC, HEVC and AV1. Beamr is excited to have been a design partner for development of this API and to be the first company that uses it, to accelerate and reduce costs of video optimization.
This milestone with Nvidia offers some important benefits. A significant cost reduction is achieved when performing Beamr video optimization using this platform. For example, for 4Kp60 encoded with advanced codecs, when using the Beamr video optimization on GPU the costs of video optimization can be cut by a factor of x10, compared to running on CPU.
Using the Beamr solution integrated on GPU means that the encoding can be performed using the built in H/W codecs, which offer very fast, high frame rate, encoding. This means the combined solution can support live and real time video encoding which is a new use case for the Beamr video optimization technology.
In addition, Nvidia recently announced their AV1 codec, considered to be the highest quality AV1 HW accelerated encoder. In this comparison Jarred Walton concluded that “”From an overall quality and performance perspective, Nvidia’s latest Ada Lovelace NVENC hardware comes out as the winner with AV1 as the codec of choice”. When using the new driver to combine the Beamr video optimization with this excellent AV1 implementation, a very competitive solution is obtained, with video encoding abilities exceeding other AV1 encoders on the market.
So, how does the new driver actually allow the integration of NVENC codecs with Beamr video optimization technology?
Above you can see a high level illustration of the system flow. The user video is ingested, and for each video frame the encoding parameters are controlled by the Beamr Quality Control block instructing NVENC on how to encode the frame, to reach the target quality while minimizing bit consumption. The New NVENC API layer is what enables the interactions between the Beamr Quality Control and the encoder to create the reduced bitrate, target optimized video. As part of the efforts towards the integrated solution, Beamr also ported its quality measurement IP to GPU and redesigned it to match the performance of NVENC, thus placing the entire solution on the GPU.
Beamr uses the new API to control the encoder and perform optimization which can reduce bitrate of an input video, or of a target encode, while guaranteeing the perceptual quality is preserved, thus creating encodes with the same perceptual quality at lower bitrates or file sizes.
The Beamr optimization may also be used for automatic, quality guaranteed codec modernization, where input content can be converted to a modern codec such as AV1, while guaranteeing each frame of the optimized encode is perceptually identical to the source video. This allows for faster migration to modern codecs, for example from AVC to HEVC or AVC to AV1, in an automated, always safe process – with no loss of quality.
In the below examples the risk of blind codec modernization is clearly visible, showcasing the advantage of using Beamr technology for this task. In these examples, we took AVC sources and encoded them to HEVC, to benefit from the increased compression efficiency offered by the more advanced coding standard. On the test set we used, Beamr reduced the source clips by 50% when converting to perceptually identical HEVC streams. We compare these encodes to the results obtained when performing ‘brute force’ compression to HEVC, using 50% lower bitrates. As is clear in these examples, using the blind conversion, shown on the left, can introduce disturbing artifacts compared to the source, shown in the middle. The Beamr encodes however, shown on the right, preserve the quality perfectly.
This driver release and the technology enablement it offers, while a significant milestone, is just the beginning. Beamr is now building a new SaaS that will allow a scalable, no code, implementation of its technology for reducing storage and networking costs. This service is planned to be publicly available in Q3 of 2023. In addition Beamr is looking for design partners that will get early access to its service and help us build the best experiences for our customers.
At the same time Beamr will continue to strengthen relationships with existing users by offering them low level API’s for enhanced controls and specific workflow adaptations.
For more information please contact us at info@beamr.com
There are several different video codecs available today for video streaming applications, and more will be released this year. This creates some confusion for video services who need to select their codec of choice for delivering content to their users at the best quality and lowest bitrate, also taking into account the encode compute requirements. For many years, the choice of video codecs was quite simple to make: Starting from MPEG-2 (H.262) when it took over digital TV in the late 90s, through MPEG-4 part 2 (H.263) dominating video conferencing early in the millennia and followed by MPEG4 part 10 or AVC (H.264) which has been enjoying significant market share for many years now in most video applications and markets including delivery, conferencing and surveillance. Simultaneously, Google’s natural choice for YouTube was their own video codec, VP9.
While HEVC, ratified in 2013, seemingly offered the next logical step, royalty issues put a major stick in its wheels. Add to this the concern over increased complexity, and delay in 4K adoption which was assumed to be the main use case for HEVC, and you get quite a grim picture. This situation triggered a strong desire in the industry to create an independent, royalty free, codec. Significantly reduced timelines in release of new video codec standards were thrown onto this fire and we find ourselves somewhat like Alice in Wonderland: signs leading us forward in various directions – but which do we follow?
Let’s begin by presenting our contenders for the “codec with significant market share in future video applications” competition:
We will not discuss LC-EVC (MPEG-5 Part 2), as it is a codec add-on rather than an alternative stand-alone video codec. If you want to learn more about it, https://lcevc.com/ is a good place to start.
If you are hoping that we will crown a single winner in this article – sorry to disappoint: It is becoming apparent that we are not headed towards a situation of one codec to rule them all. What we will do is provide information, highlight some features of each of the codecs, share some insights and opinions and hopefully help arm you for the ongoing codec wars.
Origin
The first point of comparison we will address is the origin, where each codec is coming from and what that implies. To date, most of the widely adopted video codecs have been standards created by the Joint Video Expert Team combing the efforts of the ITU-T Video Coding Expert Group (VCEG) and the ISO Moving Picture Experts Group (MPEG) to create joint standards. AVC and HEVC were born through this process, which involves clear procedures, from the CfP (Call for Proposals), through teams performing evaluation of the compression efficiency and performance requirements of each proposed tool, and up to creating a draft of the proposed standard. A few rounds of editing and fixes yields a final draft which is ratified to provide the final standard. This process is very well organized and has a long and proven track record of resulting in stable and successful video codecs. AVC, HEVC and VVC are all codecs created in this manner.
The EVC codec is an exception in that it is coming only from MPEG, without the cooperation of ITU-T. This may be related to the ITU VCEG traditionally not being in favor of addressing royalty issues as part of the standardization process, while for the EVC standard, as we will see, this was a point of concern.
Another source for video codecs is specific companies. A particularly successful example is the VP9 codec, developed by Google as a successor to VP8, that was created by On2 technologies (later acquired by Google). In addition, some companies have tried to push open source, royalty free, proprietary codecs, such as Daala by Mozilla or Dirac by BBC Research.
A third source of codecs is when a consortium or group of several companies that works independently, outside of official international standards bodies such as the ISO or ITU. AV1 is the perfect example of such a codec, where multiple companies have joined forces through the Alliance for Open Media (AOM), to create a royalty-free open-source video coding format, specifically designed for video transmissions over the Internet. AOM founding members include Google (who contributed their VP9 technology), Microsoft, Amazon, Apple, Netflix, FB, Mozilla and others, along with classic “MPEG supporters” such as Cisco & Samsung. The AV1 encoder was built from ‘experiments’, where each considered tool was added into the reference software along with a toggle to turn the experiment on or off, allowing flexibility during the decision process as to which tools will be used for each of the eventual profiles.
Timeline
An easy point of comparison between the codecs is the timeline. AVC was completed back in May 2003. HEVC was finalized almost 10 years later in April 2013. AV1 bitstream freeze was in March 2018, with validation in June of that year and Errata-1 published in January 2019. As of the 130th MPEG meeting in April 2020, VVC and EVC are both in Final Draft of International Standard (FDIS) stage, and are expected to be ratified this year.
Royalties
The next point of comparison is the painful issue of royalties. Unless you have been living under a rock you are probably aware that this is a pivotal issue in the codec wars. AVC royalty issues are well resolved and a known and inexpensive licensing model is in place, but for HEVC the situation is more complex. While HEVC Advance unifies many of the patent holders for HEVC, and is constantly bringing more on-board, MPEG LA still represents some others. Velos Media unify yet more IP holders and a few are still unaffiliated and not taking part in any of these pools. Despite the pools finally publishing reasonable licensing models over the last couple of years (over five years after HEVC finalization), the industry is for the most part taking a ‘once bitten, twice shy’ approach to HEVC royalties with some concern over the possibility of other entities coming out of the woodwork with yet further IP claims.
AV1 was a direct attempt to resolve this royalty mess, by creating a royalty-free solution, backed by industry giants, and even creating a legal defense fund to assist smaller companies that may be sued regarding the technology they contributed. Despite AOM never promising to indemnify against third party infringement, this seemed to many pretty air-tight. That is until in early March Sisvel announced a patent pool of 14 companies that hold over 1000 patents, which Sisvel claim are essential for the implementation of AV1. About a month later, AOM released a counter statement declaring AOM’s dedication to a royalty-free media ecosystem. Time, and presumably quite a few lawyers, will determine how this particular battle plays out.
VVC initially seemed to be heading down the same IP road as HEVC: According to MPEG regulations, anyone contributing IP to the standard must sign a Fair, Reasonable And Non-Discriminatory (FRAND) licensing commitment. But, as experience shows, that does not guarantee convergence to applicable patent pools. This time however the industry has taken action in the form of the Media Coding Industry Forum (MC-IF), an open industry forum established in 2018, with the purpose of furthering the adoption of MPEG standards, initially focusing on VVC. Their goal is to establish them as well-accepted and widely used standards for the benefit of consumers and industry. One of the MC-IF work groups is working on defining “sub-profiles”, which include either royalty free tools or tools for which MC-IF are able to serve as a registration authority for all relevant IP licensing. If this effort succeeds, we may yet see royalty free or royalty known sub-profiles for VVC.
EVC is tackling the royalty issue directly within the standardization process, performed primarily by Samsung, Huawei and Qualcomm, using a combination of two approaches. For EVC-Baseline, only tools which can be shown to be royalty-free are being incorporated. This generally means the technologies are 20+ years old and have the publications to prove it. While this may sound like a rather problematic constraint, once you factor in the facts that AVC technology is all 20+ years old, and a lot of non IP infringing know-how has accumulated over these years, one can conceive that this codec can still significantly exceed AVC compression efficiency. For EVC-Main a royalty-known approach has been adopted, where any entity contributing IP is committed to provide a reasonably priced licensing model within two years of the FDIS, meaning by April 2022.
Technical Features
Now that we have dealt with the elephant in the room, we will highlight some codec features and see how the different codecs compare in this regard. All these codecs use a hybrid block-based coding approach, meaning the encode is performed by splitting the frame into blocks, performing a prediction of the block pixels, obtaining a residual as the difference between the prediction and the actual values, applying a frequency transform to the residual obtaining coefficients which are then quantized, and finally entropy coding those coefficients along with additional data, such as Motion Vectors used for prediction, resulting in the bitstream. A somewhat simplified diagram of such an encoder is shown in FIG 1.
FIGURE 1: HYBRID BLOCK BASED ENCODER
The underlying theme of the codec improvements is very much a “more is better” approach. More block sizes and sub-partitioning options, more prediction possibilities, more sizes and types of frequency transforms and more additional tools such as sophisticated in-loop deblocking filters.
Partitioning
We will begin with a look at the block or partitioning schemes supported. The MBs of AVC are always 16×16, CTUs in HEVC and EVC-Baseline are up to 64×64, While for EVC-Main, AV1 and VCC block sizes of up to 128×128 are supported. As block sizes grow larger, they enable efficient encoding of smooth textures in higher and higher resolutions.
Regarding partitioning, while in AVC we had fixed-size Macro-Blocks, in HEVC the Quad-Tree was introduced allowing the Coding-Tree-Unit to be recursively partitioned into four additional sub-blocks. The same scheme is also supported in EVC-Baseline. VVC added Binary Tree (2-way) and Ternary Tree (3-way) splits to the Quad-Tree, thus increasing the partitioning flexibility, as illustrated in the example partitioning in FIG 2. EVC-Main also uses a combined QT, BT, TT approach and in addition has a Split Unit Coding Order feature, which allows it to perform the processing and predictions of the sub-blocks in Right-to-Left order as well as the usual Left-to-Right order. AV1 uses a slightly different partitioning approach which supports up to 10-way splits of each coding block.
Another evolving aspect of partitions is the flexibility in their shape. The ability to split the blocks asymmetrically and along diagonals, can help isolate localized changes and create efficient and accurate partitions. This has two important advantages: The need for fine granularity of sub-partitioning is avoided, and two objects separated by a diagonal edge can be correctly represented without introducing a “staircase” effect. The wedges partitioning introduced in AV1 and the geometric partitioning of VVC both support diagonal partitions between two prediction areas, thus enabling very accurate partitioning.
FIGURE 2: Partitioning example combining QT (blue), TT (green) and BT (red)
Prediction
A good quality prediction scheme which minimizes the residual energy is an important tool for increasing compression efficiency. All video codecs from AVC onwards employ both INTRA prediction, where the prediction is performed using pixels already encoded and reconstructed in the current frame, and INTER prediction, using pixels from previously encoded and reconstructed frames.
AVC supports 9 INTRA prediction modes, or directions in which the current block pixels can be predicted from the pixels adjacent to the block on the left, above and right-above. EVC-Baseline supports only 5 INTRA prediction modes, EVC- Main supports 33, HEVC defines 35 INTRA prediction modes, AV1 has 56 and VVC takes the cake with 65 angular predictions. While the “more is better” paradigm may improve compression efficiency, this directly impacts encoding complexity as it means the encoder has a more complex decision to make when choosing the optimal mode. AV1 and VVC add additional sophisticated options for INTRA prediction such as predicting Chroma from Luma in AV1, or the similar Cross-Component Linear Model prediction of VVC. Another interesting tool for Intra prediction is INTRA Block Copy (IBC) which allows copying of a full block from the already encoded and reconstructed part of the current frame, as the predictor for the current block. This mode is particularly beneficial for frames with complex synthetic texture, and is supported in AV1, EVC-Main and VVC. VVC also supports Multiple Reference Lines, where the number of pixels near the block used for INTRA prediction is extended.
The differences in INTER prediction are in the number of references used, Motion Vector (MV) resolution and associated sub-pel interpolation filters, supported motion partitioning and prediction modes. A thorough review of the various INTER prediction tools in each codec is well beyond the scope of this comparison, so we will just point out a few of the new features we are particularly fond of.
Overlapped Block Motion Compensation (OBMC), which was first introduced in Annex F of H.263 and in MPEG4 part 2 – but not included in any profile, is supported in AV1 and though considered for VVC, was not included in the final draft. This is an excellent tool for reducing those annoying discontinuities at prediction block borders when the block on either side uses a different MV.
FIGURE 3A: OBMC ILLUSTRATION. On the top is regular Motion Compensation which creates a discontinuity due to two adjacent blocks using different parts of reference frame for prediction, on the bottom OBMC with overlap between prediction blocks
FIGURE 3B: OBMC ILLUSTRATION. Zoom into OBMC for the border between middle and left shown blocks, showing the averaging of the two predictions at the crossover pixels.
One of the significant limitations of the block matching motion prediction approach, is its failure to represent motion that is not horizontal & vertical only, such as zoom or rotation. This is being addressed by support of warped motion compensation in AV1 and even more thoroughly with 6 Degrees-Of-Freedom (DOF) Affine Motion Compensation supported in VVC. EVC-main takes it a step further with 3 affine motion modes: merge, and both 4DOF and 6DOF Affine MC.
FIGURE 4: AFFINE MOTION PREDICTION Image credit: Cordula Heithausen – Coding of Higher Order Motion Parameters for Video Compression – ISBN-13: 978-3844057843
Another thing video codecs do is MV (Motion Vector) prediction based on previously found MV values. This reduces bits associated with MV transmission, beneficial at aggressive bitrates and/or when using high granularity motion partitions. It can also help to make the motion estimation process more efficient. While all five codecs define a process for calculating the MV Predictor (MVP), EVC-Main extends this with a history-based MVP, and VVC takes it further with improved spatial and temporal MV prediction.
Transforms
The frequency transforms applied to the residual data are another arena for the “more is better” approach. AVC uses 4×4 and 8×8 Discrete Cosine Transform (DCT), while EVC-Baseline adds more transform sizes ranging from 2×2 to 64×64. HEVC added the complementary Discrete Sine Transform (DST) and supports multi-size transforms ranging from 4×4 to 32×32. AV1, VVC and EVC-Main all use DCT and DST based transforms with a wide range of sizes including non-square transform kernels.
Filtering
In-loop filters have a crucial contribution to improving the perceptual quality of block-based codecs, by removing artifacts created in the separated processing and decisions applied to adjacent blocks. AVC uses a relatively simple in loop adaptive De-Blocking (DB) filter, which is also the case for EVC-Baseline which uses the filter from H.263 Annex J. HEVC adds an additional Sample Adaptive Offset (SAO) filter, designed to allow for better reconstruction of the original signal amplitudes by applying offsets stored in a lookup table in the bitstream, resulting in increased picture quality and reduction of banding and ringing artifacts. VVC uses similar DB and SAO filters, and adds an Adaptive Loop Filter (ALF) to minimize the error between the original and decoded samples. This is done by using Wiener-based adaptive filters, with suitable filter coefficients determined by the encoder and explicitly signaled to the decoder. EVC-main uses an ADvanced Deblocking Filter (ADDB) as well as ALF, and further introduces a Hadamard Transform Domain Filter (HTDF) performed on decoded samples right after block reconstruction using 4 neighboring samples. Wrapping up with AV1, a regular DB filter is used as well as a Constrained Directional Enhancement Filter (CDEF) which removes ringing and basis noise around sharp edges, and is the first usage of a directional filter for this purpose by a video codec. AV1 also uses a Loop Restoration filter, for which the filter coefficients are determined by the encoder and signaled to the decoder.
Entropy Coding
The entropy coding stage varies somewhat among the codecs, partially due to the fact that the Context Adaptive Binary Arithmetic Coding (CABAC) has associated royalties. AVC offers both Context Adaptive Variable Length Coding (CAVLC) and CABAC modes. HEVC and VVC both use CABAC, with VVC adding some improvements to increase efficiency such as better initializations without need for a LUT, and increased flexibility in Coefficient Group sizes. AV1 uses non-binary (multi symbol) arithmetic coding – this means that the entropy coding must be performed in two sequential steps, which limits parallelization. EVC-Baseline uses the Binary Arithmetic Coder described in JPEG Annex D combined with run-level symbols, while EVC-Main employs a bit-plane ADvanced Coefficient Coding (ADCC) approach.
To wrap up the feature highlights section, we’d like to note some features that are useful for specific scenarios. For example, EVC-main and VVC support Decoder side MV Refinement (DMVR), which is beneficial for distributed systems where some of the encoding complexity is offloaded to the decoder. AV1 and VVC both have tools well suited for screen content, such as support of Palette coding, with AV1 supporting also the Paeth prediction used in PNG images. Support of Film Grain Synthesis (FGS), first introduced in HEVC but not included in any profile, is mandatory in AV1 Professional profile, and is considered a valuable tool for high quality, low bitrate compression of grainy films.
Codec Comparison
Compression Efficiency
Probably the most interesting question is how do the codecs compare in actual video compression, or what is the Compression Efficiency (CE) of each codec: What bitrate is required to obtain a certain quality or inversely – what quality will be obtained at a given bitrate. While the question is quite simple and well defined, answering it is anything but. The first challenge is defining the testing points – what content, at what bitrates, in what modes. As a simple example, when screen content coding tools exist, the codec will show more of an advantage on that type of content. Different selections of content, rate control methodologies if used (which are outside the scope of the standards), GOP structures and other configuration parameters, have a significant impact on the obtained results.
Another obstacle on the way to a definitive answer stems from how to measure the quality. PSNR is sadly often still used in these comparisons, despite its poor correlation with perceptual quality. But even more sophisticated objective metrics, such as SSIM or VMAF, do not always accurately represent the perceptual quality of the video. On the other hand, subjective evaluation is costly, not always practical at scale, and results obtained in one test may not be repeated when tests are performed with other viewers or in other locations.
So, while you can find endless comparisons available, which might be slightly different and sometimes even entirely contradicting, we will take a more conservative approach, providing estimates based on a cross of multiple comparisons in the literature. There seems no doubt that among these codecs, AVC has the lowest compression efficiency, while VVC tops the charts. EVC-Baseline seemingly has a compression efficiency which is about 30% higher than AVC, not far from the 40% improvement attributed to HEVC. AV1 and EVC-Main are close, with the decision re which one is superior very dependent on who performed the comparisons. They are both approximately 5-10% behind VVC in their compression efficiency.
Computational Complexity
Now, a look at the performance or computational complexity of each of the candidates. Again, this comparison is rather naïve, as the performance is so heavily dependent on the implementation and testing conditions, rather than on the tools defined by the standard. The ability to parallelize the encoding tasks, the structure of the processor used for testing, the content type such as low or high motion or dark vs. bright are just a few examples of factors that can heavily impact the performance analysis. For example, taking the exact same preset of x264 and running it on the same content with low and high target bitrates, can cause a 4x difference in encode runtime. In another example, in the Beamr5 epic face off blog post, the Beamr HEVC encoder is on average 1.6x faster than x265 on the same content with similar quality, and the range of the encode FPS across files for each encoder is order of 1.5x. Having said all that, what we will try to do here is provide a very coarse, ball-park estimate as to the relative computational complexity of each of the reviewed codecs. AVC is definitely the lowest complexity of the bunch, with EVC-Baseline only very slightly more complex. HEVC has higher performance demands for both the encoder and decoder. VVC has managed to keep the decoder complexity almost on par with that of the HEVC decoder, but encoding complexity is significantly higher and probably the highest of all 5 reviewed codecs. AV1 is also known for its high complexity, with early versions having introduced the unit Frame Per Minute (FPM) for encoding performance, rather than the commonly used Frames Per Second (FPS). Though recent versions have gone a long way to making matters better, it is still safe to say that complexity is significantly higher than HEVC, and probably still higher than EVC-Main.
Summary
In the table below, we have summarized some of the comparison features which we outlined in this blog post.
The Bottom Line
So, what is the bottom line? Unfortunately, life is getting more complicated, and the case of one or two dominant codecs covering almost all the industry – will be no more. Only time will tell which will have the highest market share in 5 years’ time, but one easy assessment is that with AVC current market share estimated at around 70%, this one is not going to disappear anytime soon. AV1 is definitely gaining momentum, and with the giants backing we expect to see it used a fair bit in online streaming. Regarding the others, it is safe to assume that the improved compression efficiency offered by VVC and EVC-Main, and the attractive royalty situation of EVC-Baseline, along with growing number of devices that support HEVC in HW, mean that having to support a plurality of codecs in many video streaming applications is the new reality for all of us.
In late July, I reported on the “news” that Verizon was throttling video traffic for some users. As usual, the facts around this seemingly punitive act were not fully understood, which triggered this blog post.
At IBC last month (September 2017), I was interviewed by RapidTV where much of the conversation was around the Apple news of their support for HEVC across the device ecosystem running iOS 11 and High Sierra. As I was reviewing this interview, it seemed natural to publish it as a follow up to the original post.
There is no doubt that mobile operators are under pressure as a result of the network crushing video traffic they are being forced to deliver. But the good news is that for those operators who adopt HEVC, they are going to enjoy significant bitrate efficiencies, possibly as high as 50%. And for many services, though they will chose to take some savings, this means they’ll be able to upgrade their resolutions to full 1080p while simultaneously improving the video quality they are delivering.
I hope you find this video insightful. Our team has a very simple evaluation offer to discuss with all qualified video services and video distributors. Just send an email to sales@beamr.com and we’ll get in touch with the details.
Today’s very special, and oh so yummy cake celebration, was a recognition of the amazing milestone that we reached on May 31, 2017 as the result of Beamr acquiring Vanguard Video on April 1st, 2016. Our vision for buying Vanguard as a firmly entrenched leader in HEVC video encoding was to combine Beamr’s world class content-adaptive optimization technology with the world’s best HEVC encoder. The results as we demonstrated at NAB 2017, are nothing short of breathtaking.
Can you imagine second screen HD at 1.5Mbps and 4K UHD with HDR at just 10Mbps? With Beamr 5x, and now that WWDC2017 saw Apple enabling HEVC across their devices, the time is now to move to HEVC so your users can enjoy enhanced UX and improved video quality.
Keep an eye out for all our news, because we’ve only just begun. The technology that we have introduced to the video encoding industry has set a new standard for performance and savings, and what the future holds is nothing short of earth shattering.
And yes, precisely 23 seconds after this picture was taken, this cake was unrecognizable!
For streaming video service operators who have been searching for a more efficient video codec to deliver higher resolution and higher quality video, the announcement by Apple at the Worldwide Developers Conference 2017 (#WWDC2017) that they will be supporting HEVC across their Mac and iOS product families, is huge.
Driven by the breathtaking quality of the retina display that is now ubiquitous across Apple devices, and with more than 1 billion active iOS devices across the globe, consumer demand for quality has never been higher. Until today, video streaming service providers have been limited by the technology constraints of only having access to an H.264 codec across this massive device ecosystem.
Beamr identified the advantages of HEVC and made a strategic decision to invest heavily ahead of the market to develop Beamr 5. As the world’s best HEVC software encoder, Beamr 5 has been used since 2014 by the largest OTT streaming services, Hollywood studios, broadcasters, and MVPDs. Our HEVC software encoder SDK is two times faster than the nearest implementation and is up to 50% more efficient with a content-adaptive bitrate mode that extends the HEVC standard rate control options beyond CBR and VBR.
What does this announcement mean for the video industry?
#1. Higher quality at lower bitrates will become a basic requirement to compete successfully in the market.
Services that are competing for consumer attention or engagement, whether SVOD, TVOD, or AVOD, will be compared with competitors who are delivering content at up to 50% lower bitrates and with better video quality. Now, it is time to re-encode all HD libraries using HEVC. And with the 2x speed boost of Beamr 5, for the first time live and ultra-low latency HEVC software video encoding is available without expensive hardware.
#2. There is no time to wait to integrate HEVC encoding into your video workflow.
The industry has realized the benefits of HEVC as a robust and ultra high-quality next generation codec for some time, but with limited playback support outside the TV, questions about timing held back some deployments. Now, with iOS 11 and High Sierra public betas rolling out to developers in June and to users this fall, video distributors who have not yet commissioned an HEVC encoding workflow can feel confident to evaluate and deploy Beamr 5, the best HEVC software encoding implementation on the market.
#3. Apple is supporting playback of video that emulates real life and provides critical user engagement with HDR.
With the announcement of 500 nit displays, 10-bit graphics support and 7th generation Intel Kaby Lake with Iris Pro Xeon processors in the new iMac line, Apple is showing us where they are going on the decode and display side. They’ve selected HEVC as the codec that will carry their video services such as iTunes into the future along with third party services who take advantage of the built-in HEVC support.
The benefits of HEVC include higher order entertainment experiences, such as HDR high dynamic range which only HEVC is capable of managing the quality and bitrate requirements. Though not every home has an HDR capable TV, with Apple pushing their device performance ever higher, consumers will grow to expect HDR even on their Mac (and perhaps iOS devices). Services who do not adopt HDR encoding capabilities will find their position in the market difficult to maintain.
At NAB 2017 Beamr announced and demonstrated with Intel Beamr 5 supporting real-time live HEVC 4K HDR encoding on a single Xeon E3 processor with Iris Pro GPU.
It’s go-time for HEVC!
Beamr would like to be the first to congratulate Apple on taking a leadership role in the video ecosystem with your support of HEVC in iOS11 and High Sierra. Apple devices are a top viewing platform for all consumer services which is why Monday, June 5th, 2017 will be marked as the date that HEVC became the winner in the next generation video codec war.
For more information on Beamr and our class leading HEVC software encoding solutions, check out beamr.com
Every video encoding professional faces the dilemma of how best to detect artifacts and measure video quality. If you have the luxury of dealing with high bitrate files then this becomes less of an “issue” since for many videos throwing enough bits at the problem means an acceptably high video quality is nearly guaranteed. However, for those living in the real world where 3 Mbps is the average bitrate that they must target, then compressing at scale requires metrics (algorithms) to help measure and analyze the visual artifacts in a file after encoding. This process is becoming even more sophisticated as some tools enable a quality measure to feed back into the encoding decision matrix, but more commonly quality measures are used as a part of the QC step. For this post we are going to focus on the application of quality measures used as part of the encoding process.
There are two common quality measures, PSNR and SSIM that we will discuss, but as you will see there is a third one and that is the Beamr quality measure that the bulk of this article will focus on.
PSNR, the Original Objective Quality Measure
PSNR, peak signal-to-noise ratio represents the ratio between the highest power of an original signal and the power level of the distortion. PSNR is one of the original engineering metrics that is used to measure the quality of image and video codecs. When comparing or measuring the quantitative quality of two files such as an original and a compressed version, PSNR attempts to approximate the difference between the compressed and the original. A significant shortcoming is that PSNR may indicate that the reconstruction is of suitably high quality when in some cases it is not. For this reason a user must be careful to not hold the results in high regard.
What is SSIM?
SSIM or the structured similarity index is a technique to predict the perceived quality of digital images and videos. The initial version was developed at the University of Texas at Austin while the full SSIM routine was developed jointly at New York University’s Laboratory for Computational Vision. SSIM is a perceptual model based algorithm that takes into account image degradation as a perceived shift in structural information, while including crucial perceptual detail, such as luminance and contrast masking. The difference compared with other techniques like PSNR is that this approach attempts to estimate absolute errors.
The basis of SSIM is the assumption that pixels have strong inter-dependencies and these dependencies contain needed information about the structure of the object in the scene, GOP or adjacent frames. Put simply, structured similarity is used for computing the similarity of two images. SSIM is a full reference metric where the computation and measurement of image quality is based on an uncompressed image as a reference. SSIM was developed as a step up over traditional methods such as PSNR (peak signal-to-noise ratio) which has proven to be uncorrelated with human vision. Yet, unfortunately SSIM itself is not perfect and can be easily fooled as shown by the following graphic which illustrates that though the original and compressed are closely correlated visually, PSNR and SSIM scored them as being not similar. Meanwhile, Beamr and MOS (mean opinion score), show them as being closely correlated.
Beamr Quality Measure
The Beamr quality measure is based on a proprietary, low complexity, reliable, perceptually aligned quality measure. The existence of this measure enables controlling a video encoder, to obtain an output clip with (near) maximal compression of the video input, while still maintaining the input video resolution, format and visual quality (PQ). This is performed by controlling the compression level of each frame, or GOP, in the video sequence, in such a way that is as deeply compressed as it can be, while still resulting in a perceptually identical output.
The Beamr quality measure is also a full-reference measure, i.e. it indicates a quality of a recompressed image or video frame when compared to a reference or original image or video frame, which is in accordance with the challenges our technology aims to tackle such as reducing bitrates to the maximum extent possible without imposing any quality degradation from the original. (as perceived by the human visual system). The Beamr quality measure calculation consists of two parts: A pre-process of the input video frames in order to obtain various score configuration parameters, and an actual score calculation done per candidate recompressed frame. Following is a system diagram of how the Beamr quality measure would interact with an encoder.
Application of the Beamr Quality Measure in an Encoder
The Beamr quality measure when integrated with an encoder enables the bitrate of video files to be reduced by up to an additional 50% over the current state of the art standard compliant block based encoders, without compromising image quality or changing the artistic intent. If you view a source video and a Beamr-optimized video side by side, they will look exactly the same to the human eye.
A question we get asked frequently is “How do you perform the “magic” of removing bits with no visual impact?”
Well, believe it or not there is no magic here, just solid technology that has been actively in development since 2009, and is now covered by 26 granted patents and over 30 additional patent applications.
When we first approached the task of reducing video bitrates based on the needs of the content and not a rudimentary bitrate control mechanism, we asked ourselves a simple starting question, “Given that the video file has already been compressed, how many additional bits can the encoder remove before the typical viewer would notice?”
There is a simple manual method of answering this question, just take a typical viewer, show them the source video and the processed video side by side, and then start turning down the bitrate knob on the processed video, by gradually increasing the compression. And at some point, the user will say “Stop! Now I can see the videos are no longer the same!”
At that point, turn the compression knob slightly backwards, and there you have it – a video clip that has an acceptably lower bitrate than the source, and just at the point before the average user can notice the visual differences.
Of course I recognize what you are likely thinking, “Yes, this solution clearly works, but it doesn’t scale!” and you are correct. Unfortunately many academic solutions suffer from this problem. They make for good hand built demos in carefully controlled environments with hand picked content, but put them out in the “wild” and they fall down almost immediately. And I won’t even go into the issues of varying perception among viewers of different ages, or across multiple viewing conditions.
Another problem with such a solution is that different parts of the videos, such as different scenes and frames, require different bitrates. So the question is, how do you continually adjust the bitrate throughout the video clip, all the time confirming with your test viewer that the quality is still acceptable? Clearly this is not feasible.
Automation to the Rescue
Today, it seems the entire world is being infected with artificial intelligence which in many cases is not much more than automation that is smart and able to adapt to its environment. So we too looked for a way to automate this image analysis process. That is take a source video, and discover a way to reduce the “non-visible” bits in a fully automatic manner, with no human intervention involved. A suitable solution would enable the bitrate to vary continuously throughout the video clip based on the needs of the content at that moment.
What is CABR?
You’ve heard of VBR or variable bitrate, Beamr has coined the term CABR or content-adaptive bitrate to summarize the process just described where the encoder is adjusted at the frame level based on quality requirements, rather than relying only on a bit budget to make decisions of where bits are applied and the number needed. But we understood that in order to accomplish the vision of CABR, we would need to be able to simulate perception of a human viewer.
We needed an algorithm that would answer the question, “Given two videos, can a human viewer tell them apart?” This algorithm is called a Perceptual Quality Measure and it is the very essence of what sets Beamr so far apart from every other encoding solution in the market today.
A quality measure is a mathematical formula, which tries to quantify the differences between two video frames. To implement our video optimization technology, we could have used one of the well-known quality measures, such as PSNR (Peak Signal to Noise Ratio) or SSIM (Structural SIMilarity). But as already discussed, the problem with these existing quality measures is that they are simply not reliable enough as they do not correlate highly enough with human vision.
There are other sophisticated quality measures which correlate highly enough with human viewer opinions to be useful, but since they require extensive CPU power they cannot be utilized in an encoding optimization process, which requires computing the quality measures several times for each input frame.
Advantages of the Beamr Quality Measure
With the constraints of objective quality measures we had no choice but to develop our own quality measure, and we developed it with a very focused goal: To identify and quantify the specific artifacts created by block-based compression methods.
All of the current image and video compression standards, including JPEG, MPEG-1, MPEG-2, H.264 (AVC) and H.265 (HEVC) are built upon block based principles.
They divide an image into blocks, attempt to predict the block from previously encoded pixels, and then transform the block into the frequency domain, and quantize it.
All of these steps create specific artifacts, which the Beamr quality measure is trained to detect and measure. So instead of looking for general deformations, such as out of focus images, missing pixels etc. which is what general quality measures do, in contrast, we look for artifacts that were created by the video encoder.
This means that our quality measure is tightly focused and extremely efficient, and as a result, the CPU requirements of our quality measure are much lower than quality measures that try to model the Human Visual System (HVS).
Beamr Quality Measure and the Human Visual System
After years of developing our quality measure, we put it to the test, under the strict requirements of ITU BT-500, which is an international standard for testing image quality. We were happy to find that the correlation of our quality measure with subjective (human) results was extremely high.
When the testing was complete, we felt certain this revolutionary quality measure was ready for the task of accurately comparing two images for similarity, from a human point of view.
But compression artifacts are only part of the secret. When a human looks at an image or video, the eye and the brain are drawn to particular places in the scene, for example, places where there is movement, and in fact we are especially “tuned” to capture details in faces.
Since our attention is focused on these areas, artifacts are more disturbing than the same artifacts in other areas of the image, such as background regions or out-of-focus areas. For this reason the Beamr quality measure takes this into account, and it ensures that when we measure quality proper attention is given to the areas that require it.
Furthermore, the Beamr quality measure takes into account temporal artifacts, introduced by the encoder, because it is not sufficient to ensure that each frame is not degraded, it is also necessary to preserve the quality and feel of the video’s temporal flow.
The Magic of Beamr
With the acquisition last year of Vanguard Video many industry observers have gone public with the idea that the combination of our highly innovative quality measure tightly integrated with the world’s best encoder, could lead to a real shake up of the ecosystem.
We encourage you to see for yourself what is possible when the world’s most advanced perceptual quality measure becomes the rate-control mechanism for the industry’s best quality software encoder. Check out Beamr Optimizer.
2016 has been a significant year for video compression as 4K, HDR, VR and 360 video picked up steam, paving the road for an EXPLOSION of HEVC adoption in 2017. With HEVC’s ability to reduce bitrate and file sizes up to 50% over H.264, it is no surprise that HEVC has transitioned to be the essential enabler of high-quality and reliable streaming video powering all the new and exciting entertainment experiences being launched.
Couple this with the latest announcement from HEVC Advance removing royalty uncertainties that plagued the market in 2016 and we have a perfect marriage of technology and capability with HEVC.
In this post we’ll discuss 2016 from the lenses of Beamr’s own product and company news, combined with notable trends that will shape 2017 in the advanced video encoding space.
>> The Market Speaks: Setting the Groundwork for an Explosion of HEVC
The State of 4K
With 4K content creation growing and the average selling price of UHD 4K TVs dropping (and being adopted faster than HDTVs), 4K is here and the critical mass of demand will follow closely. We recently did a little investigative research on the state of 4K and four of the most significant trends pushing its adoption by consumers:
The upgrade in picture quality is significant and will drive an increase in value to the consumer – and, most importantly, additional revenue opportunities for services as consumers are preconditioned to pay more for a premium experience. It only takes a few minutes viewing time to see that 4K offers premium video quality and enhances the entertainment experience.
Competitive forces are operating at scale – Service Providers and OTT distributors will drive the adoption of 4K. MSO are upping their game and in 2017 you will see several deliver highly formidable services to take on pure play OTT distributors. Who’s going to win, who’s going to lose? We think it’s going to be a win-win as services are able to increase ARPUs and reduce churn, while consumers will be able to actually experience the full quality and resolution that their new TV can deliver.
Commercially available 4K UHD services will be scaling rapidly – SNL Kagan forecasts the number of global UHD Linear channels at 237 globally by 2020, which is great news for consumers. The UltraHD Forum recently published a list of UHD services that are “live” today numbering 18 VOD and 37 Live services with 8 in the US and 47 outside the US. Clearly, content will not be the weak link in UHD 4K market acceptance for much longer.
Geographic deployments — 4K is more widely deployed in Asia Pacific and Western Europe than in the U.S. today. But we see this as a massive opportunity since many people are traveling abroad and thus will be exposed to the incredible quality. They will then return home to question their service provider, why they had to travel outside the country to see 4K. Which means as soon as the planned services in the U.S. are launched, they will likely attract customer more quickly than we’ve seen in the past.
HDR adds WOW factor to 4K
High Dynamic Range (HDR) improves video quality by going beyond more pixels to increase the amount of data delivered by each pixel. HDR video is capable of capturing a larger range of brightness and luminosity to produce an image closer to what can be seen in real life. Show anyone HDR content encoded in 4K resolution, and it’s no surprise that content providers and TV manufacturers are quickly jumping on board to deliver content with HDR. Yes, it’s “that good.” There is no disputing that HDR delivers the “wow” factor that the market and consumers are looking for. But what’s even more promising is the industry’s overwhelmingly positive reaction to it. Read more here.
Beamr has been working with Dolby to enable Dolby Vision HDR support for several years now, even jointly presenting a white paper at SMPTE in 2014. The V.265 codec is optimized for Dolby Vision and HDR10 and takes into account all requirements for both standards including full support for VUI signaling, SEI messaging, SMPTE ST 2084:2014 and ITU-R BT.2020. For more information visit http://beamr.com/vanguard-by-beamr-content-adaptive-hevc-codec-sdk
Beamr is honored to have customers who are best in class and span OTT delivery, Broadcast, Service Providers and other entertainment video applications. From what we see and hear, studios are uber excited about HDR, cable companies are prepping for HDR delivery, Satellite distributors are building the capability to distribute HDR, and of course OTT services like Netflix, FandangoNow (formerly M-GO), VUDU, and Amazon are already distributing content using either Dolby Vision or HDR10 (or both). If your current video encoding workflow cannot fully support or adequately encode content with HDR, it’s time to update. Our V.265 video encoder SDK is a perfect place to start.
VR & 360 Video at Streamable Bitrates
360-degree video made a lot of noise in 2016. YouTube, Facebook and Twitter added support for 360-degree videos, including live streaming in 360 degrees, to their platforms. 360-degree video content and computer-generated VR content is being delivered to web browsers, mobile devices, and a range of Virtual Reality headsets. The Oculus Rift, HTC Vive, Gear VR and Daydream View have all shipped this year, creating a new market for immersive content experiences.
But, there is an inherent problem with delivering VR and 360 video on today’s platforms. In order to enable HD video viewing in your “viewport” (the part of the 360-degree space that you actually look at), the resolution of the full 360 video delivered to you should be 4K or more. On the other hand, the devices on the market today which are used to view this content, including desktops, mobile devices and VR headsets only support H.264 video decoding. So delivering the high-resolution video content requires very high bitrates – twice as much as using the more modern HEVC standard.
The current solution to this issue is lowered video quality in order to fit the H.264 video stream into a reasonable bandwidth. This creates an experience for users which is not the best possible, a factor that can discourage them from consuming this newly-available VR and 360 video content. But there’s one thing we know for sure – next generation compression including HEVC and content adaptive encoding – and perceptual optimization – will be a critical part of the final solution. Read more about VR and 360 here.
Patent Pool HEVC Advance Announces “Royalty Free” HEVC software
As 4K, HDR, VR and 360 video gathers steam, Beamr has seen the adoption rate moving faster than expected, but with the unanswered questions around royalties, and concerns of who would shoulder the cost burden, distributors have been tentative. The latest move by HEVC Advance to offer a royalty free option is meant to encourage and accelerate the adoption (implementation) of HEVC, by removing royalty uncertainties. Internet streaming distributors and software application providers can be at ease knowing they can offer applications with HEVC software decoders without incurring onerous royalties or licensing fees. This is important as streaming app content consumption continues to increase, with more and more companies investing in its future. By initiating a software-only royalty solution, HEVC Advance expects this move to push the rest of the market i.e. device manufacturers and browser providers to implement HEVC capability in their hardware and offer their customers the best and most efficient video experience possible.
>> 2017 Predictions
Mobile Video Services will Drive the Need for Content-adaptive Optimization
Given the trend toward better quality and higher resolution (4K), it’s more important than ever for video content distributors to pursue more efficient methods of encoding their video so they can adapt to the rapidly changing market, and this is where content-adaptive optimization provides a massive benefit.
The boundaries between OTT services and traditional MSO (cable and satellite) are being blurred now that all major MSOs include TVE (TV Everywhere streaming services with both VOD and Linear channels) in their subscription packages (some even break these services out separately as is the case with SlingTV). And in October, AT&T CEO Randall Stephenson vowed that DirecTV Now would disrupt the pay-TV business with revolutionary pricing for an Internet-streaming service at a mere $35 per month for a package with more than 100 channels.
And get this – AT&T wireless is adopting the practice of “zero rating” for their customers, that is, they will not count the OTT service streaming video usage toward the subscriber’s monthly data plan. This represents a great value for customers, but there is no doubt that it puts pricing pressure on the operational side of all zero rated services.
2017 is the year that consumers will finally be able to enjoy linear as well as VOD content anywhere they wish even outside the home.
Beamr’s Contribution to MSOs, Service Providers, and OTT Distributors is More Critical Than Ever
When reaching to consumers across multiple platforms, with different constraints and delivery cost models, Beamr’s content adaptive optimizer perfects the encoding process to the most efficient quality and bitrate combination.
Whether you pay by the bit delivered to a traditional CDN provider, or operate your own infrastructure, the benefits of delivering less traffic are realized with improved UX such as faster stream start times and reduced re-buffering events, in addition to the cost savings. One popular streaming service reported to us that after implementing our content-adaptive optimization solution their rebuffering events as measured on the player were reduced by up to 50%, while their stream start times improved 20%.
Recently popularized by Netflix and Google, content-adaptive encoding is the idea that not all videos are created equal in terms of their encoding requirements. Content-adaptive optimization complements the encoding process by driving the encoder to the lowest bitrate possible based on the needs of the content, and not a fixed target bitrate (as seen in traditional encoding processes and products).
A content-adaptive solution can optimize more efficiently by analyzing already-encoded video on a frame-by-frame and scene-by-scene level, detecting areas of the video that can be further compressed without losing perceptual quality (e.g. slow motion scenes, smooth surfaces). Provided the perceptual quality calculation is performed at the frame level with an optimizer that contains a closed loop perceptual quality measure, the output can be guaranteed to be the highest quality at the lowest bitrate possible. Click the following link to learn how Beamr’s patented content adaptive optimization technology achieves exactly this result.
Encoding and Optimization Working Together to Build the Future
Since the content-adaptive optimization process is applied to files that have already been encoded, by combining an industry leading H.264 and HEVC encoder with the best optimization solution (Beamr Video), the market will be sure to benefit by receiving the highest quality video at the lowest possible bitrate and file size. As a result, this will allow content providers to improve the end-user experience with high quality video, while meeting the growing network constraints due to increased mobile consumption and general Internet congestion.
Beamr made a bold step towards delivering on this stated market requirement by disrupting the video encoding space when in April 2016 we acquired Vanguard Video – a premier video encoding and technology company. This move will benefit the industry starting in 2017 when we introduce a new class of video encoder that we call a Content Adaptive Encoder.
As content adaptive encoding techniques are being adopted by major streaming services and video platforms like YouTube and Netflix, the market is gearing up for more advanced rate control and optimization methods, something that fits our perceptual quality measure technology perfectly. This fact when combined with Beamr having the best in class HEVC software encoder in the industry, will yield exciting benefits for the market. Read the Beamr Encoder Superguide that details the most popular methods for performing content adaptive encoding and how you can integrate them into your video workflow.
One Year from Now…
In one year from now when you read our post summarizing 2017 and heralding 2018, what you will likely hear is that 2017 was the year that advanced codecs like HEVC combined with efficient perceptually based quality measures, such as Beamr’s, provide an additional 20% or greater bitrate reduction.
The ripple effect of this technology leap will be that services struggling to compete today on quality or bitrate, may fall so far behind that they lose their ability to grow the market. We know of many multi-service operator platforms who are gearing up to increase the quality of their video beyond the current best of class for OTT services. That is correct, they’ve watched the consumer response to new entrants in the market offering superior video quality, and they are not sitting still. In fact, many are planning to leapfrog the competition with their aggressive adoption of content adaptive perceptual quality driven solutions.
If any one service assumes they have the leadership position based on bitrate or quality, 2017 may prove to be a reshuffling of the deck.
For Beamr, the industry can expect to see an expansion of our software encoder line with the integration of our perceptual quality measure which has been developed over the last 7 years, and is covered by more than 50 patents granted and pending. We are proud of the fact that this solution has been shipping for more than 3 years in our stand-alone video and photo optimizer solutions.
It’s going to be an exciting year for Beamr and the industry and we welcome you to join us. If you are intrigued and would like to learn more about our products or are interested in evaluating any of our solutions, check us out at beamr.com.
With video consumption rising and consumer preferences shifting to 4K UHD this is contributing to an even faster adoption rate than what we saw with the move to HD TV. Consumer demand for a seamless (buffer-free) video experience is a new expectation, and with the latest announcement from HEVC Advance removing royalty uncertainties in the market it’s time to start thinking about building and deploying an HEVC workflow, starting with a robust HEVC encoder.
As you may know, Beamr’s V.265 was the first commercially deployed HEVC codec SDK and it is in use today by the world’s largest OTT streaming service. Even still, we receive questions regarding V.265 in comparison to x265 and in this post we’d like to address a few of them.
In future posts, we will discuss the differences in two distinct categories, performance (speed) and quality, but in this post we’ll focus on feature-related differences between V.265 and x265.
Beginning with our instruction set, specifically support for X86/x64 SMP Architecture, V.265 is able to improve encoding performance by leveraging a resource efficient architecture that is used by most multiprocessors today. Enabling this type of support allows each processor to execute different programs while working on discrete data sets to afford the capability of sharing common resources (memory, I/O device interrupt system and so on) that are connected using a system bus or a crossbar. The result is a notable increase in overall encoding speed with V.265 over x265. For any application where speed is important, V.265 will generally pull ahead as the winner.
Another area V.265 shines compared to x265 is with its advanced preprocessing algorithm support that provides resizing and de-interlacing. As many of you know, working with interlaced video can lead to poor video quality so to try and minimize the various visual defects V.265 uses a variety of techniques like line doubling where our smart algorithms are able to detect and fill in an empty row by averaging the line above and the line below. The advantages of having a resizing feature is recognizable, largely saving time and resources, and out of the box V.265 allows you to easily convert video from one resolution to another (i.e. 4K to HD). One note, we are aware that x265 supports these features via FFMPEG. However in the case that a user is not able to use FFMPEG, the fact that V.265 supports them directly is a benefit.
V.265 boasts an unmatched pre-analysis library with fading detection and complexity analysis capabilities not supported in x265. An application for the V.265 library is video segmentation that is problematic with many encoders because of the different ways two consecutive shots may be linked. In V.265, the fading detection method detects the type of gradual transition, fade type etc. which is needed to detect hard to recognize soft cuts. V.265’s complexity analysis is able to discriminate temporal and spatial complexity in video sequences with patented multi-step motion estimation methods that are more advanced than standard “textbook” motion estimation algorithms. The information gained from doing a video complexity analysis is used during the encoding process to improve encoding quality especially during transitions between scenes.
One of the most significant features V.265 offers compared to x265 is multistreaming (ABR) support. V.265 can produce multiple GOP-aligned video output streams that are extremely important when encoding for adaptive streaming. It is critical that all bitrates have IDRs aligned to enable seamless stream switching, which V.265 provides.
Additionally, with V.265 users can produce multiple GOP-aligned HEVC streams from a single input. This is extremely important for use cases when a user has one chance to synchronize video of different resolutions and bitrates. Multistreaming helps to provide encoded data to HLS or DASH packagers in an optimal way and it provides performance savings – especially when the service must output multiple streams of the same resolution, but at varying bitrates.
Another significant feature V.265 has over x265 is its content adaptive speed settings that makes codec configuration more convenient such as real-time compared to VOD workflows. Currently we offer presets ranging from ultra fast for extremely low latency live broadcast streams to the highest quality VOD.
To combat packet losses and produce the most robust stream possible, V.265 supports slicing by slice compressed size which produces encoded slices of limited sized (typically the size of a network packet) for use in an error prone network. This is an important feature for anyone distributing content on networks with highly variable QoS.
Continuing on to parallel processing features, V.265 offers support for tiles that divides the frame into a grid of rectangular regions that can be independently decoded and encoded. Enabling this feature increases encoding performance.
V.265 is regarded as one of the most robust codecs in the market because of its ability to suit both demanding real-time and offline file based workflows. To deliver the industry leading quality that makes V.265 so powerful, it offers motion estimation features like patented high performance search algorithms and motion vectors over a picture boundary to provide additional quality improvements over x265.
For encoding by frame-type, V.265 offers Bi- and uni-directional non-reference P-frames which is useful where low-delay encoding is needed to improve temporal scalability
As for encoding tools, V.265 offers a unique set of tools over x265:
Joint bi-directional Motion Vector Search which is an internal motion estimation encoding technique that provides a better bi-direction motion vector search.
Sub-LCU QP modulation that allows the user to change QP from block to block inside LCU as a way to control in-frame bits/quality more precisely.
Support for up to 4 temporal layers of multiple resolutions in the same bitstream to help with changing network conditions.
Region of Interest (ROI) control which allows for encoding of a specific ROI with a particular encoding parameter (qp) to add flexibility and improve encoding quality.
Another major advantage over x265 is the proprietary rate control implementation offered with V.265. This ensures target bitrates are always maintained.
The more supplemental enhancement information (SEI) messages a codec supports the more video usability information (VUI) metadata that may be delivered to the decoder in an encoded bitstream. For this reason, Beamr found it necessary to include in V.265 support for Recovery point, Field indication, Decoded Picture Hash, User data unregistered, and User data as specified by ITU-T T.35.
V.265’s ability to change encoding parameters on the fly is another extremely important feature that sets it apart from x265. With the ability to change encoder resolution, bitrate, and other key elements of the encoding profile, video distributors can achieve a significant advantage by creating recipes appropriate to each piece of content without needing to interrupt their workflows or processing cycles to reset and restart an encoder.
We trust this feature comparison was useful. In the event that you require more information or would like to evaluate the V.265, feel free to reach out to us at http://beamr.com/info-request and someone will get in touch to discuss your application and interest.
November 22nd 2016 may be shown by history as the day that wholesale adoption of HEVC as the preferred next generation codec began. For companies like Beamr who are innovating on next-generation video encoding technologies such as HEVC, the news HEVC Advance announced on to drop royalties (license fees) on certain applications of their patents is huge.
In their press release, HEVC Advance, the patent pool for key HEVC technologies stated that they will not seek a license fee or royalties on software applications that utilize the HEVC compression standard for encoding and decoding. This carve out only applies to software which is able to be run on commodity servers, but we think the restriction fits beautifully with where the industry is headed.
Did you catch that? NO HEVC ROYALTIES FOR SOFTWARE ENCODERS AND DECODERS!
Specifically, the policy will protect “application layer software downloaded to mobile devices or personal computers after the initial sales of the device, where the HEVC encoding or decoding is fully executed in software on a general purpose CPU” from royalty and licensing fees.
Requirements of Eligible Software
For those trying to wrap their heads around eligibility, the new policy outlines three requirements which the software products performing HEVC decoding or encoding must meet:
Application layer software, or codec libraries used by application layer software, enabling software-only encoding or decoding of HEVC.
Software downloaded after the initial sale of a related product (mobile device or desktop personal computer). In the case of software which otherwise would fit the exclusion but is being shipped with a product, then the manufacturer of the product would need to pay a royalty.
Software must not be specifically excluded.
Examples of exempted software applications where an HEVC decode royalty will likely not be due includes web browsers, personal video conferencing software and video players provided by various internet streaming distributors or software application providers.
As stated previously, driven by the rise of virtual private and public cloud encoding workflows, provided an HEVC encoder meets the eligibility requirements, for many companies it appears that there will not be an added cost to utilize HEVC in place of H.264.
A Much Needed Push for HEVC Adoption
As 4k, HDR, VR and 360 video are gathering steam, Beamr has seen the adoption rate moving faster than expected, but with the unanswered questions around royalties, and concerns of the cost burden, even the largest distributors have been tentative. This move by HEVC Advance is meant to encourage and accelerate the adoption (implementation) of HEVC, by removing uncertainties in the market.
Internet streaming distributors and software application providers can be at ease knowing they can offer applications with HEVC software decoders without incurring onerous royalties or licensing fees. This is important as streaming app content consumption continues to increase, with more and more companies investing in its future.
By initiating a software-only royalty solution, HEVC Advance expects this move to push the rest of the market i.e. device manufacturers and browser providers to implement HEVC capability in their hardware and offer their customers the best and most efficient video experience possible.
What this Means for a Video Distributor
Beamr is the leader in H.265/HEVC encoding. With 60 engineers around the world working at the codec level to produce the highest performing HEVC codec SDK in the market, Beamr V.265 delivers exceptional quality with much better scalability than any other software codec.
Industry benchmarks are showing that H.265/HEVC provides on average a 30% bitrate efficiency for the same quality and resolution over H.264. Which given the bandwidth pressure all networks are under to upgrade quality while minimizing the bits used, there is only one video encoding technology available at scale to meet the needs of the market, and that is HEVC.
The classic chicken and egg problem no longer exists with HEVC.
The challenge every new technology faces as it is introduced into the market is the classic problem of needing to attract implementers and users. In the case of a video encoding technology, without an appropriately scaled video playback ecosystem, no matter the benefits, it cannot be deployed without a sufficiently large number of players in the market.
But the good news is that over the last few years, and as consumers have propelled the TV upgrade cycle forward, many have opted to purchase UHD 4k TVs.
Most of the 2015-2016 models of major brand TVs have built-in HEVC decoders and this trend will continue in 2017 and beyond. Netflix, Amazon, VUDU, and FandangoNow (M-GO) are shipping their players on most models of UHD TVs that are capable of decoding and playing back H.265/HEVC content from these services. These distributors were all able to utilize the native HEVC decoder in the TV, easing the complexity of launching a 4k app.
For those who wonder if there is a sufficiently large ecosystem of HEVC playback in the market, just look at the 90 million TVs that are in homes today globally (approximately 40 million are in the US). And consider that in 2017 the number of 4k HEVC capable TV’s will nearly double to 167 million according to Cisco, as illustrated below.
The industry has spoken regarding the superior quality and performance of Beamr’s own HEVC encoder, and we will be providing benchmarks and documentation in future blog posts. Meanwhile our team of architects and implementation specialists who work with the largest service providers, SVOD consumer streaming services, and broadcasters in the world are ready to discuss your migration plans from H.264 to HEVC.
Just fill out our short Info Request form and the appropriate person will get in touch.
