TL;DR 

Post COVID-19, the work from home trend will continue, and this will extend the pressure on the Internet from video traffic. Even with the EU Commissioner’s call for video services to reduce their traffic by 25%, as Internet traffic patterns shift from corporate networks to mobile, fixed wireless, and broadband networks, the need to reduce video bandwidths will continue beyond COVID-19. Consumers will still demand the highest quality, and those streaming services meeting their expectations while delivering video in as small a footprint as possible will dominate the market. Now is the time for the streaming video industry to play an active role in adopting more efficient codecs and content-adaptive bitrate technology so that streaming video services can ensure a great user experience without disrupting the Internet. 

The Internet is a shared resource to preserve.

For the video streaming industry, Thursday, March 19th, marked the day of reckoning for runaway bitrates and seemingly never-ending network capacity. On March 19th, Thierry Breton, the European Commissioner for the Internal Market tweeted, “let’s #SwitchToStandard definition when HD is not necessary.” The result is that most of the best known US video services, including Facebook and Instagram, agreed to a 25% reduction in bandwidth used for video delivered in Europe, UK, and Israel. With other countries rumored to follow suit. 

We can blame COVID-19 for the strain as a result of closed schools and businesses, leading to increased use of video conferencing, streaming video services, and cloud gaming. Verizon reported that total web traffic is up 22% between March 12th and March 19th, while week-over-week usage patterns for streaming video services increased by 12%. However, it’s easily predictable that these numbers are trending even higher as the quarantine and shelter in place orders expanded, as evidenced by Cloudflare reporting Internet traffic is 40% higher than pre COVID-19 levels.

The purpose of this article is to provide a framework for how video streaming services may want to think about the Internet post COVID-19 where video streaming services and video-centric applications will need to consider their utilization of the Internet as a shared and not an unlimited resource.

Content-Adaptive Encoding is no longer a nice to have for streaming services. 

There are multiple technical and technology options available for reducing video bitrate. The fastest to implement, however, is to drop the resolution of the video. By manipulating the video playlist (called a manifest) that organizes the various resolutions and bit rates that enable the video player to adapt to the speed of the network, a video service can achieve immediate savings by merely serving a lighter weight version of the video. Standard-definition (SD) instead of high-definition (HD). This approach is what most of the complying services have taken, but, it is not a sustainable answer since dropping resolution impacts video experience negatively.

A more advanced technique known as Content-Adaptive Encoding works by guiding the encoder to adapt the bitrate allocation to the needs of the video content. 

Reducing resolution is not what consumers want, and this will make content-adaptive encoding essential for many video encoding workflows. Because content-adaptive encoding solutions require integration, for some services, it was relegated to the “nice to have” list. But now, with the sweeping changes to video consumption that is driving network saturation, those services that must compete with high visual quality, are shifting the priority to “must-have.” 

Effective tools and methods to be a good citizen of the Internet.

If we are going to be a good citizen of the Internet, we should understand what tools and methods are available to preserve this precious shared resource while delivering a suitable UX and visual quality. 

Engineering for video encoding is about tradeoffs. The three primary levers are 1) bitrate, 2) resolution, 3) encoder performance. These levers are interconnected and dependent. For example, it’s not possible to achieve high bitrate efficiency at higher resolutions without affecting encoder performance (increasing CPU cycles).  

From a video quality perspective, lever one and lever two are the levers available to most video encoding engineers. While from an operational point of view, the third lever is what most impacts bitrate and quality.

The tools that we can use to reduce bandwidth include the use of advanced video codecs such as HEVC. HEVC (H.265) provides up to 50% reduction in bitrate at the same quality level as H.264, the current dominant codec used around the world. The other tool available is advanced technology, such as content-adaptive encoding, implemented inside the encoder. 

Beamr’s Content-Adaptive Bitrate (CABR) rate-control is an example of advanced technology that brings an additional 20-40% reduction in bitrate. Using HEVC and CABR, a 4K HDR video file can be as small as 10Mbps, an added savings of as much 6Mbps without CABR. With the promise of a 50% bitrate reduction using HEVC, and over 2 billion devices supporting HEVC decoding in hardware, it’s the obvious thing to do for a video service concerned about the sustainability of the Internet.

If a technical integration of a new codec is not possible, the three most popular methods for reducing bitrate are Per-Category, Per-Title, and Per-Frame Encoding optimization.

Per-Category Encoding optimization.

The Per-Category Encoding approach is least practical for premium movies, and TV shows since the range of encoding complexity within a category can vary significantly. Animated videos are typically easier to compress than video captured from a camera sensor, given the wide range of complexity. Animation techniques are highly diverse, from hand-drawn to 2D to 3D, and that makes it challenging to create an encoding ladder that works across animated content equally. 

Per-Category Encoding is the easiest of all the methods to implement, but also produces the lowest real bitrate reduction because of the variability of scenes. For example, a sports broadcast may include talking head in-studio shots along with fast action gameplay and slow-motion recaps, each requiring different bitrate values to preserve the quality level. 

Per-Title Encoding optimization.

Per-Title Encoding received a big boost when Netflix published a blog post explaining their encoding schema that creates a custom encoding ladder for each video file. The system performs a series of test encodes at different CRF levels and resolutions that are analyzed using the Video Multimethod Assessment Fusion (VMAF) quality metric. Netflix uses the scores to identify the best quality resolution at each applicable data rate. 

Though many video services have adopted their variation, Per-Title Encoding, or some variation of it can now be found in many video encoding workflows. It’s a great way to rethink fixed ABR recipes that are the primary source of wasted bandwidth, or poor video quality. Per-Title Encoding only works when you have a smaller library as it requires extensive computing resources to run the hundreds of fractional encodes needed for each title. 

Per-Title Encoding helps to reduce bitrate but is limited in its ability since the rudimentary VBR rate-control bounds the encoder QP setting with no additional intelligence. 

Per-Frame Encoding optimization.

The weakness of a category or title based optimization method is that this approach cannot adapt to the specific needs of the video at the frame level. Only by steering the encoder decisions frame by frame is it possible to achieve the ultimate result of producing high quality with the least number of bits required. 

Beamr’s CABR technology is the primary feature of the Beamr 4x and Beamr 5x encoding engines. CABR operates at the frame level to deliver the smallest possible size for each video frame while ensuring the highest overall quality of each frame within the video sequence. This approach avoids transient quality issues in other optimization techniques. The Beamr Quality Measure Analyzer has a higher correlation with subjective results than existing quality measures such as PSNR, and SSIM. CABR is protected by the majority of Beamr’s 48 granted patents. 

To learn more about Beamr’s Content-Adaptive Bitrate technology, you can hear Tamar Shoham, Head of Algorithms at Beamr, explain CABR here.

We must all play our part in preserving the integrity of the Internet.

Just as environmental sustainability is an essential initiative for companies who want to be good citizens of the world, in the COVID-19 world that we are living in, video sustainability is now an equally vital initiative. And this is likely to be unchanged in the future as the work from home and virtual meeting trends continue post COVID-19. Now is the time for the streaming video industry to play an active role in adopting more efficient codecs and content-adaptive bitrate technology so that we can ensure a great user experience without disrupting the Internet. 

Game of Thrones entered its eighth and final season on April 14th, 2019. Though Game of Thrones has been a cultural phenomenon since the beginning of its airing, the attention and eyeballs on these final episodes have been higher than ever. Right now, every aspect of season eight is under a microscope and up for discussion, from the battle of Winterfell to theories on the Azor Ahai prophecy, super fans are taking to the internet and social media to debate and swap theories. Yet, even if you’ve installed the Chrome extension GameOfSpoilers to block Game of Thrones news from popular social networks, you probably did not miss all the fans who flocked to social media to report their dissatisfaction with the poor quality of Season 8 Episode 3, “The Longest Night.”

CHECK OUT THIS PODCAST EPISODE

If nothing else that episode of #GameofThrones is proof positive that @Xfinity needs to start broadcasting in 4K pronto because the night scenes in that kind of looked like ass in their HBO broadcast. If I’m going to pay this much for cable I want better picture quality.

— Michael S (@TheMovieVampire) April 29, 2019

So can we talk about the bootleg quality of the war scenes or nah? #GameOfThrones

— Marquis Johns (@weaksauceradio) April 29, 2019

Though not all viewers experienced degraded visual quality for “The Longest Night”, a sufficiently high number did report a poor viewer experience, which triggered TechCrunch to write an article titled, “Why did last night’s ‘Game of Thrones’ look so bad?”

And TechCrunch wasn’t alone, The Verge also wrote an entire piece on how to setup your TV for a rewatch of “The Longest Night”, something that seems hardly possible. After all how is it possible that fans could need to rewatch an episode, not because the plot was so twisted or complicated that they needed a second pass at deciphering it, but because they couldn’t see what was happening on the screen? And in fact, Game of Thrones super fans were not shy in taking to Twitter with their quality assessments.

Why does this look so bad?

@HBO @GameOfThrones the picture quality tonight is absolutely terrible. All that time spent filming this beautiful episode and it looks like it’s running off a 1989 VHS tape…

— Chris Bohn (@CBohn83) April 29, 2019

Before you throw a Valyrian steel dagger at your TV, let’s take a close look at what happened to create this poor video quality by diving into the operational structure of the underlying video codecs that are used by all commercial streaming services.

The video compression schemes used in video streaming, including AVC which is also used by most PayTV cable and satellite distributors, utilize hybrid block-based video codecs. These codecs use block-based encoding methods which mean each video frame or picture is partitioned into blocks during the compression process, and they also apply motion estimation between frames. Though the effective compression that is made possible by these techniques is impressive, hybrid block-based compression schemes are inherently prone to creating blockiness and banding artifacts, which can be particularly evident in dark scenes.

Blockiness is a video artifact where areas of a video image appear to be comprised of small squares, rather than proper detail and smooth edges as the viewer would expect to see. The blockiness artifact happens when not enough detail is preserved in each of the coding blocks, resulting in inconsistencies between adjacent blocks and making each block appear separate from its neighbors. The video quality will suffer from blockiness when too much detail is lost within each block.

There are two main causes of blockiness. The first is when there is a mismatch between the content complexity and the target bitrate. It can be present either in highly complex content which is encoded at typical bitrates, or in standard content which is compressed to overly aggressive bitrates. Content providers can avoid this by using content adaptive solutions which match the encoder bitrate to the content complexity. The second cause of blockiness is from poor quality decisions made by the encoder, such as discarding information which is crucial for visual quality.

As noted by TechCrunch, for the specific Game of Thrones episode “The Longest Night”, the images are very dark and have a limited range of colors and brightness levels, basically between grey and dark grey. Encoding this limited range of grey shades, which filled up most of the screen, resulted in “banding” artifacts which is where there are visible transitions as a result of the video being represented by just a few shades of grey, which look like “stripes” instead of smooth gradients. Video suffers from banding when the color or brightness palette being used has too few values to accurately describe the shades present in part or all of the video frame.

The prevalent assumption even among some video engineers is that increasing bitrate is the cure-all to video quality problems. But as we’ll see in this case, it’s likely that even if the bitrate had been doubled, the systemic artifacts would still be present. Thus, the solution is not likely external to the video encoding process, but rather can only be addressed at the codec implementation level.

That is, the video encoding engine must be improved to prevent situations like this in the future. That, or HBO and other premium content owners could instruct their filmmakers to avoid dark scenes – We’ll stick with Option #1!

In this case, the video quality issues were not caused by the video encoding bitrate being too low. In fact, the bitrate used was more than sufficient to represent the limited range of colors and brightness. The issue was in the decisions made by the specific video encoder used by HBO. These are decisions regarding how to allocate the available bitrate, or how the bits should be used for different elements of the compressed video stream.

Without getting too deep into video compression technology, it is sufficient to say that a compressed video stream consists basically of two types of data.

The first type is prediction data which enables the creation of a prediction block from previously decoded pixels (in either the same or a reference frame). This prediction block acts as a rough estimate of the block and is complemented by the residual or error block, encoded in the stream. This is essentially a block that fills in the difference between the predicted block and the actual source video block.

The second type of data that is key to how a block-based video encoder works can be found in the rate-distortion algorithm which optimizes the selection of the prediction modes which the prediction data represents. This determines the level of detail to preserve in the residual block. The decisions are made in an attempt to minimize distortion, or maximize quality, for a specific bit allocation which is derived from the target bit-rate.

When a scene is very dark and consists of small variations of pixel colors, the numerical estimates of distortion may be skewed. Components of the video encoder including motion estimation and the rate-distortion algorithm should adapt to optimize the allocations and decisions for this particular use case.

For example, the motion estimation might classify the differences in pixel values as noise instead of actual motion, thus providing inferior prediction information. In another example, if the distortion measures are not tuned correctly, the residual may be considered noise rather than true pixel information and may be discarded or aggressively quantized.

Another common encoder technique that is affected and often “fooled” by very dark scenes is early terminations. Many encoders use this technique to “guess” what would be the best encoding decision they should make, instead of making an exhaustive search of all possibilities, and computing their cost. This technique improves the performance of the video encoder, but in the case of dark scenes with small variations, it can cause the encoder to make the wrong decision.

Some encoding engineers use a technique called “capped CRF” for encoding video at a constant quality instead of a pre-defined bitrate. This is a simple form of “content-adaptive” or “content-aware” encoding, which produces different bitrates for each video clip or scene based on its content. In some implementations, when this technique is used for dark scenes, it can also be “fooled” by the limited range of color and brightness values and may perform very aggressive quantization thus removing too much information form the residual blocks, resulting in these blockiness and banding artifacts.

In summary, we can conclude that dark scenes can lead to various encoding issues if the encoder is not “well prepared” for this type of content, and it seems that this is what happened with this Game of Thrones episode.

Better luck quality, next time.

Hey HBO, why is #GameofThrones in potato quality?

— Rabbit Raccoon (@PollyQPublic) April 29, 2019

In order to ensure good quality video across different content types, the encoder must be able to correctly adapt to each and every frame being encoded. In Beamr encoders we tackle this with a combination of tools and algorithms to provide the best video quality possible.

Beamr encoders use unique, patented and patent-pending approaches, to calculate psycho-visual distortions to be used by the rate control module when deciding on prediction modes and on the bit allocations for different components of the compressed video data. This means that the actual visual impact of different decisions is taken into account, resulting in the improved visual quality of the content across a variety of content types.

Beamr encoders offer a wide variety of encoding speeds for different use cases, ranging from lightning fast to enable a full 4K ABR stack to be generated on a single server for live or VOD. When airing premium content of the caliber of Game of Thrones, one should opt for the maximum quality by using slower encoder speeds.

In these speeds, the encoder is wary of invoking early termination methods and thus does not overlook the data that may be hiding in the small deviations of the dark pixel values. We invest a huge effort to discover the optimal combinations for all the internal controls of the algorithm such as the most optimum lambda values for the different rate-distortion cost calculations, optimal values for the deblocking filter (and SAO in the case of Beamr 5 for HEVC) , and many other details – none of which are overlooked.

Rather than use a CRF like based approach for constant quality encoding, Beamr employs a sophisticated technique for content-adaptive encoding called CABR. The content-adaptive bitrate mode of encoding operates in a closed loop and examines the quality of each frame using a patented perceptual quality measure. Our perceptual quality measure is also specifically tuned to adapt to the “darkness” of the frame and each region within the frame, which makes it highly effective even when processing very dark scenes such as the “The Longest Night”, or fade-in and fade-out sequences.

Looking to the Future

For content providers, viewer expectations and demands for quality will continue to rise each year.  A decade ago, you could slide by not delivering a consistent experience across devices. Today, not only is video degradation noticed by your viewers, it can have a massive impact on your audience and churn if you’re not delivering an experience inline with their quality expectations.  

To see what high quality at the lowest bitrate should look like, try Beamr Transcoder for free or contact our team by sending an email to sales@beamr.com to learn about our comparison tool Beamr View.