TL;DR: Beamr CABR operating with the Intel Media SDK hardware encoder powered by Intel GPUs is the perfect video encoding engine for cloud gaming services like Google Stadia. The Intel GPU hardware encoder reaches real-time performance with a power envelope that is 90% less than a CPU based software solution. When combined with Beamr CABR (Content-Adaptive Bitrate) technology, the required bandwidth for cloud gaming is reduced by as much as 49% while delivering higher quality 65% of the time. Using the Intel hardware encoder combined with Beamr CABR enables players to enjoy a gaming experience that is competitive to a console and able to be streamed by cloud gaming platforms. Get more information about how CABR works.
The era of cloud gaming.
With the launch of Google Stadia, we have entered a new era in the games industry called cloud gaming. Just as streaming video services opened media and entertainment content to a broader audience by freeing it from the fixed frameworks of terrestrial (over-the-air), cable, and satellite distribution, so to will cloud gaming open gameplay to a larger audience. Besides extending gameplay to virtually anywhere the user has a network-connected device, the ability for a player to access an extensive library of games without needing to use a specific piece of hardware will push 25.9 million players to cloud gaming platforms by 2023, according to the media research group Kagan.
In addition to opening up gameplay to an “anywhere/anytime” experience. A major user experience benefit of cloud gaming is that players will not necessarily need to purchase a game, but in many cases will be free to access a vast library of their choosing instantaneously. Cloud gaming services promise the quality of a console or PC experience, but without the need to own expensive hardware and the configuration and software installation work that comes with that.
The one constraint that could cause cloud gaming to never catch up with the console experience.
With the wholesale transition of video entertainment content from traditional broadcast and physical media to streaming distribution, it is not hard to project the same pattern will occur for games. Except now, unlike the early days of video streaming where a 3Mbps home Internet connection was “high speed,” and the number of devices able to decode and reliably play back H.264 video was limited, even the lowest cost smartphone can stream video with acceptable quality.
Yet, there is a fundamental constraint that must be overcome for cloud gaming to reach its full market potential, and that is the bandwidth required to deliver a competitive video experience at 1080p60 or 4kp60 resolution. To better understand the bandwidth squeeze that is unique to cloud gaming, let’s examine the data and signal flow.
In FIGURE 1 we see the cloud gaming architecture moves compute-intensive operations, like the graphics rendering engine, to the cloud.
FIGURE 1
Shifting the compute-intensive function to the cloud eliminates device technical capability from being a bottleneck. However, as a result of the video rendering and encoding function not being local to the user, it means the video stream needs to be delivered over the network, with latency in the tens of milliseconds. And, at a framerate that is double the entertainment video frame rate of 24, 25, or 30 frames per second. Additionally, video game resolutions need to be HD with 4K preferable. Also, HDR is an increasingly important capability for many AAA game titles.
None of these requirements are impossible to meet, except as a result of needing fast encoding speed, the encoder must be operated in a mode that makes it difficult to produce high-quality and with small stream size. Because of the added time needed for the encoder to create B frames, and without the benefit of a look-ahead buffer, producing high quality with low bitrate is not possible. Hence why cloud gaming services require a significantly higher bitrate than what is possible with traditional video on demand streaming video services.
Beamr has been innovating in the area of performance, allowing us to encode H.264 and HEVC in software with breathtaking speed, even when running our most advanced Content-Adaptive Bitrate (CABR) rate-control. For video applications where a single encoder can serve hundreds of thousands or even millions of users, the compute requirement to do this in software, given the tremendous benefits of lower bitrate and higher quality, makes it easy to justify. But, in an application like cloud gaming, where the video encoder is matched 1:1 to every user, the computing cost to do this in software makes it uneconomical. The answer is to use a hardware encoder controlled by software, and running a content-adaptive optimization process which can deliver the additional bitrate savings needed.
FIGURE 2 illustrates the required Google Stadia bitrates.
FIGURE 2
The answer is to leverage hardware and software.
The Intel Media SDK and GPU engines occupy a well-established position in the market, with many video services relying on its included HEVC hardware encoder for real-time encoding. However, using the VBR rate-control only, there is a limit to the quality available when bitrate efficiency is essential. The advantage of Beamr’s next-generation rate-control technology, CABR (Content-Adaptive Bitrate), combined with Intel GPUs, is the secret to delivering bitrate efficiency and quality, in real-time, with 90% less power than software alone.
In verified testing, Beamr has shown that the Intel Media SDK hardware encoder controlled by CABR will produce the same perceptual quality as VBR encodes, with a confidence level greater than 95%. Using CABR gives a meaningful impact on user experience. 65% of the time, the player will perceive better quality at the same bandwidth, even while the gaming platform experiences up to a 49% reduction in the bandwidth required to provide the same quality level.
Watch Beamr Founder Sharon Carmel present Beamr CABR integrated with Intel Gen 11 hardware encoder at Intel Experience Day October 29, 2019 in Moscow.
Proof of performance.
As an image science company, Beamr is committed to proof of performance with all claims. For this reason, the industry recognizes that all technology, products, and solutions which carry the Beamr name, represent the pinnacle of quality. For this reason, it was insufficient to integrate CABR with the Intel Media SDK without being able to prove that the original quality of the stream is always preserved and that the user experience is improved. Testing comprised corresponding 10-second segments extracted from clips created with the Intel hardware encoder using VBR, and clips encoded using the Intel hardware encoder but with the integrated Beamr CABR rate-control.
The only way to test perceptual quality is with subjective techniques. We used a process similar to forced-choice double stimulus (FCDS), and closely approximating the ITU BT.500 method. Using the Beamr Auto-VISTA framework, we recruited anonymous viewers from Amazon Mechanical Turk where each viewer was shown corresponding segment pairs and asked to select which video had lower quality. The VBR and CABR encoded files were placed at random on the left and right sides. Validation pairs were used to verify the user’s capabilities with visible artifacts inserted, and only test results for users who correctly answered all four validation pairs were incorporated into the analysis. The viewers had up to five attempts to view the pairs before making a decision. Each viewer watched 20 segment pairs consisting of sixteen actual CABR, and VBR encodes, and four validation pairs.
Games used for testing were: CSGO, Fallout, and GTA5. To reflect realistic bitrates, we only tested the middle four bitrates out of the six bitrates provided. This was because the bitrate for the top layer was very high, and the bottom layer quality was very low. The four bitrates tested were spaced one JND (just noticeable difference) apart. Each target test pair was viewed 13 to 21 times by valid users, with a total of 800 target pair viewings, or about 17 viewings per pair on average. The total number of valid test sessions were 50, completed by more than 40 unique viewers.
Peeling back the data, you will notice that the per-pair statistical distribution is quite symmetrical above and below 50%. With the sampling base, this phenomenon is no surprise; human perception varies. The overall results had 800 views of 48 pairs, which make the statistical certainty higher, indicating that CABR is not compromising perceptual quality.
FIGURE 4 shows CABR encodes had the same perceptual quality as VBR and with a confidence level of more than 95%.
FIGURE 4
Better quality, lower bitrate.
Beamr CABR encoded streams offer higher quality when compared subjectively to a VBR equivalent encode, while offering a bitrate savings of up to 49%. Benefits of CABR for cloud gaming or any live streaming service, are quantified by better quality, greater bandwidth savings, and a reduction in storage cost. For the files that we tested, the aggregated metrics were as follows:
65% of the time, users will experience better quality for a given bandwidth.
40% bandwidth savings on average across all three titles (GTA5 had a savings of 49%).
30% overall storage savings.
FIGURE 5, 6, and 7 illustrate for the three video samples used that for a given User Bandwidth, CABR provides higher quality. You will interpret the chart by observing that where VBR is blue, CABR is BLACK (higher quality), and where VBR is turquoise, CABR is BLUE.
FIGURE 5
FIGURE 6FIGURE 7
Conclusion.
Beamr CABR controlling the Intel Media SDK hardware encoder is the perfect video encoding engine for cloud gaming services like Google Stadia. The Beamr CABR rate-control and optimization process works with all Intel codecs, including AVC, HEVC, VP9, and AV1. All bitstreams produced by the Intel + Beamr CABR solution are fully standard-compliant and work with every player in the field today. Beamr CABR is proven and protected by 46 International patents, meaning there is no other solution that can reduce bitrate by as much as 49% while working in real-time using a closed-loop perceptually aligned quality measure to guarantee the original quality.
The single most important technical hurdle for anyone building or operating a cloud gaming service or platform is the bandwidth consumption required to deliver a player experience on par with the console. Now, with Intel + Beamr CABR, the ideal solution is here; one that can reach the performance and density needed for cloud gaming at scale, so that more players can enjoy a premium gaming experience. Streaming video upended the media and entertainment business, with the rise of Netflix, Hulu, Amazon Prime Video, Disney+, Apple TV Plus, and dozens of other tier-one streaming services. In the same way, cloud gaming will create new service platforms, gaming experiences, and business models.
To experience the power of Beamr CABR controlling the Intel hardware encoder, send an email to info@beamr.com.
It has been two years since we published a comparison of the two leading HEVC software encoder SDKs; Beamr 5, and x265. In this article you will learn how Beamr’s AVC and HEVC software codec SDKs have widened the computing performance gap further over x264 and x265 for live broadcast quality streaming.
Why Performance Matters
With the performance of our AVC (Beamr 4) and HEVC (Beamr 5) software encoders improving several orders of magnitude over the 2017 results, it felt like the right time to refresh our benchmarks, this time with real-time operational data.
It’s no secret that x264 and x265 have benefited greatly, as open-source projects, from having thousands of developers working on the code. This is what makes x264 and x265 a very high bar to beat. Yet even with so many bright and talented engineers donating tens of thousands of development hours to the code base, the architectural constraints of how these encoders were built limit the performance on multicore processors as you will see in the data below.
Creative solutions have been developed which enable live encoding workflows to be built using open-source. But it’s no secret that they come with inherent flaws that include being overly compute intensive and encumbered with quality issues as a result of not being able to encode a full ABR stack, or even a single 4K profile, on a single machine.
The reason this matters is because the resolutions that video services deliver continues to increase. And as a result of exploding consumer demand for video, data consumption is consuming network bandwidth to the point that Cisco reports in 2022, 82% of Internet traffic will be video.
Cisco says in their Visual Networking Index that by 2022 SD resolution video will comprise just 23.4% of Internet video traffic, compared to the 60.2% of Internet video traffic that SD video consumed in 2017. What use to represent the middle-quality tier, 480p (SD), has now become the lowest rung of the ABR ladder for many video distributors.
1080p (HD) will makeup 56.2% of Internet video traffic by 2022, an increase from 36.1% in 2017. And if you thought the resolution expansion was going to end with HD, Cisco is claiming in 2022, 4K (UHD) will comprise 20.3% of all Internet-delivered video.
Live video services are projected to expand 15x between 2017 and 2022, meaning within the next three years, 17.1% of all Internet video traffic will be comprised of live streams.
These trends demonstrate the industry’s need to prepare for this shift to higher resolution video and real-time delivery with software encoding solutions that can meet the requirement for live broadcast quality 4K.
Blazing Software Performance on Multicore Processors
The Beamr 5 software encoder utilizes an advanced thread management architecture. This represents a key aspect of how we can achieve such fantastic speed over x265 at the same quality level.
x265 works by creating software threads and adding them to a thread pool where each task must wait its turn. In contrast, Beamr 5 tracks all the serial dependencies involved with the video coding tasks it must perform, and it creates small micro-tasks which are efficiently distributed across all of the CPU cores in the system. This allows the Beamr codec to utilize each available core at almost 100% capacity.
All tasks added to the Beamr codec thread pool may be executed immediately so that no hardware thread is wasted on tasks where the data is not yet available. Interestingly, under certain conditions, x265 can appear to have higher CPU utilization. But, this utilization includes software threads which are not doing any useful work. This means they are “active” but not processing data that is required for the encoding process.
Adding to Beamr encoders thread efficiency, we have implemented patented algorithms for more effective and efficient video encoding, including a fast motion estimation process and a heuristic early-termination algorithm which enables the encoder to reach a targeted quality using fewer compute resources (cycles). Furthermore, Beamr encoders utilize the latest AVX-512 SIMD instruction set for squeezing even more performance out of advanced CPUs.
The end result of the numerous optimizations found in the Beamr 4 (AVC) and Beamr 5 (HEVC) software encoders is that they are able to operate nearly twice as fast as x264 and x265 at the same quality, and with similar settings and tools utilization.
Video streaming services can benefit from this performance advantage in many ways, such as higher density (more channels per server) which reduces operational costs. To illustrate what this performance advantage can do for you- consider that at the top-end, Beamr 5 is able to encode 4K, 10-bit video at 60 FPS in real-time using just 9 Intel Xeon Scalable cores where x265 is unable to achieve this level of performance with any number of computing cores (at least on a single machine). And, as a result of being twice as efficient, Beamr 4 and Beamr 5 can deliver higher quality at the same computing envelope as x264 and x265.
The Test Results
For our test to be as real-world as possible, we devised two methodologies. In the first, we measured the compute performance of an HEVC ABR stack operating both Beamr 5 and x265 at live speed. And for the second test, our team measured the number of simultaneous live streams at 1080p, comparing Beamr 4 with x264, and Beamr 5 with x265; and for 4K comparing Beamr 5 with x265. All tests were run on a single machine.
Live HEVC ABR Stack: Number of ABR Profiles (Channels)
This test was designed to find the maximum number of full ABR channels which can be encoded live by Beamr 5 and x265 on an AWS EC2 c5.24xlarge instance.
Each AVC channel comprises 4 layers of 8-bit 60 FPS video starting from 1080p, and the HEVC channel comprises either 4 layers of 10-bit 60 FPS video (starting from 1080p), or 5 layers of 10-bit 60 FPS video (starting from 4K).
Live HEVC ABR Stack Test – CONFIGURATION
Platform:
AWS EC2 c5.24xlarge instance
Intel Xeon Scaleable Cascade Lake @ 3.6 GHz
48 cores, 96 threads
Presets:
Beamr 5: INSANELY_FAST
x265: ultrafast
Content: Netflix 10-bit 4Kp60 sample clips (DinnerScene and PierSeaside)
Encoded Frame Rate (all layers): 60 FPS
Encoded Bit Depth (all layers): 10-bit
Encoded Resolutions and Bitrates:
4Kp60@18000 Kbps (only in 4K ABR stack)
1080p60@3750 Kbps
720p60@2500 Kbps
576p60@1250 Kbps
360p@625 Kbps
Live HEVC ABR Stack Test – RESULTS
NOTES:
(1) When encoding 2 full ABR stacks with Beamr 5, 25% of the CPU is unused and available for other tasks.
(2) x265 cannot encode even a single 4K ABR stack channel at 60 FPS. The maximum FPS for the 4K layer of a single 4K ABR stack channel using x265 is 35 FPS.
Live AVC & HEVC Single-Resolution: Number of Channels (1080p & 4K)
In this test, we are trying to discover the maximum number of single-resolution 4K and HD channels that can be encoded live by Beamr 4 and Beamr 5 as compared with x264 and x265, on a c5.24xlarge instance. As with the Live ABR Channels test, the quality between the two encoders as measured by PSNR, SSIM and VMAF was always found to be equal, and in some cases better with Beamr 4 and Beamr 5 (see the “Quality Results” section below).
Live AVC Beamr 4 vs. x264 Channels Test – CONFIGURATION
Platform:
AWS EC2 c5.24xlarge instance
Intel Xeon Scaleable Cascade Lake @ 3.6 GHz
48 cores, 96 threads
Speeds / Presets:
Beamr 4: speed 3
x264: preset medium
Content: Netflix 10-bit 4Kp60 sample clips (DinnerScene and PierSeaside)
Encoded Frame Rate (all channels): 60 FPS
Encoded Bit Depth (all channels): 8-bit
Channel Resolutions and Bitrates:
1080p60@5000 Kbps
Live AVC Beamr 4 vs. x264 Channels Test – RESULTS
Live HEVC Beamr 5 vs. x265 Channels Test – CONFIGURATION
Platform:
AWS EC2 c5.24xlarge instance
Intel Xeon Scaleable Cascade Lake @ 3.6 GHz
48 cores, 96 threads
Speeds / Presets:
Beamr 5: INSANELY_FAST
x265: ultrafast
Content: Netflix 10-bit 4Kp60 sample clips (DinnerScene and PierSeaside)
Encoded Frame Rate (all channels): 60 FPS
Encoded Bit Depth (all channels): 10-bit
Channel Resolutions and Bitrates:
4K@18000 Kbps
1080p60@3750 Kbps
Live HEVC Beamr 5 vs. x265 Channels Test – RESULTS
NOTES:
(1) x265 was unable to reach 60 FPS for a single 4K channel, achieving just 35 FPS at comparable quality.
Quality Comparisons (PSNR, SSIM, VMAF)
Beamr 5 vs. x265
NOTES:
As previously referenced, x265 was unable to reach 4Kp60 and thus PSNR, SSIM, and VMAF scores could not be calculated, hence the ‘N/A’ designation in the 3840×2160 cells.
Video engineers are universally focused on the video encoding pillars of computing efficiency (performance), bitrate efficiency, and quality. Even as technology has enabled each of these pillars to advance with new tool sets, it’s well known that there is still a tradeoff between each that is required.
On one hand, bitrate efficiency requires tools that sap performance, and on the other hand, to reach a performance (speed) target, tools which could positively affect quality cannot be used without harming the performance characteristics of the encoding pipeline. As a result, many video encoding practitioners have adapted to the reality of these tradeoffs and simply accept them for what they are. Now, there is a solution…
The impact of adopting Beamr 4 for AVC and Beamr 5 for HEVC transcends a TCO calculation. With Beamr’s high-performance software encoders, services can achieve bitrate efficiency and performance, all without sacrificing video quality.
The use of Beamr 4 and Beamr 5 opens up improved UX with an increase in resolution or frame-rate which means it is now possible for everyone to stream higher quality video. As the competitive landscape for video delivery services continues to evolve, never has the need been greater for an AVC and HEVC codec implementation that can deliver the best of all three pillars: performance, bitrate efficiency, and quality. With the performance data presented above, it should be clear that Beamr 4 and Beamr 5 continue to be the codec implementations to beat.
Going Inside Beamr’s Frame-Level Content-Adaptive Rate Control for Video Coding
When it comes to video, the tradeoff between quality and bitrate is an ongoing dance. Content producers want to maximize quality for viewers, while storage and delivery costs drive the need to reduce bitrate as much as possible. Content-adaptive encoding addresses this challenge, by striving to reach the “optimal” bitrate for each unique piece of content, be it a full clip or a single scene. Our CABR technology takes it a step further by adapting the encoding at the frame level. CABR is a closed-loop content-adaptive rate control mechanism enabling video encoders to lower the bitrate of their encode, while simultaneously preserving the perceptual quality of the higher bitrate encode. As a low-complexity solution, CABR also works for live or real-time encoding.
All Eyes are on Video
According to Grand View Research, the global video streaming market is expected to grow at a CAGR of 19.6% from 2019 to 2025. This shift, fueled by the increasing popularity of direct-to-consumer streaming services such as Netflix, Amazon and Hulu, the growth of video on social media networks and user-generated video platforms such as Facebook and YouTube, and other applications like online education & video surveillance, has all eyes on video workflows. Therefore, efficient video encoding, in terms of encoding and delivery costs, and meeting the viewer’s rising quality expectations, are at the forefront of video service provider’s minds. Beamr’s CABR solution can reduce bitrates without compromising quality while keeping a low computational overhead to enhance video services.
Comparing Content-Adaptive Encoding Solutions
Instead of using fixed encoding parameters, content-adaptive encoding configures the video encoder according to the content of the video clip to reach the optimal tradeoff between bitrate and quality. Various content-adaptive encoding techniques have been used in the past to provide a better user experience with reduced delivery costs. Some of them have been entirely manual, where encoding parameters are hand-tuned for each content category and sometimes, like in the case of high-volume Blu-ray titles, at the scene level. Manual content-adaptive techniques are restricted in the sense that they can’t be scaled, and they don’t provide granularity lower than the scene level.
Other techniques, such as those used by YouTube and Netflix, use “brute force” encoding of each title by applying a wide range of encoding parameters, and then by employing rate-distortion models or machine learning techniques, try to select the best parameters for each title or scene. This approach requires a lot of CPU resources since many full encodes are performed on each title, at different resolutions and bitrates. Such techniques are suitable for diverse content libraries that are limited in size, such as premium content including TV series and movies. These methods do not apply well to vast repositories of videos such as user-generated content, and are not applicable to live encoding.
Beamr’s CABR solution is different from the techniques described above in that it works in a closed-loop and adapts the encoding per frame. The video encoder first encodes a frame using a configuration based on its regular rate control mechanism, resulting in an initial encode. Then, Beamr’s CABR rate control instructs the encoder to encode the same frame again with various values of encoding parameters, creating candidate encodes. Using a patented perceptual quality measure, each candidate encode is compared with the initial encode, and then the best candidate is selected and placed in the output stream. The best candidate is the one that has the lowest bitrate but still has the same perceptual quality as the initial encode.
Taking Advantage of Beamr’s CABR Rate Control
In order for Beamr’s CABR technology to encode video to the minimal bitrate and still retain the perceptual quality of a higher bitrate encode, it compresses each video frame to the maximum extent that provides the same visual quality when the video is viewed in motion. Figure 1 shows a block diagram of an encoding solution which incorporates CABR technology.
Figure 1 – A block diagram of the CABR encoding solution
An integrated CABR encoding solution consists of a video encoder and the CABR rate control engine. The CABR engine is comprised of the CABR control module responsible for managing the optimization process and a module which evaluates video quality.
As seen in Figure 2, the CABR encoding process consists of multiple steps. Some of these steps are performed once for each encoding session, some are performed once for each frame, and some are performed for each iteration of candidate frame encoding. When starting a content-adaptive encoding session, the CABR engine and the encoder are initialized. At this stage, we set system-level parameters such as the maximum number of iterations per frame. Then, for each frame, the encoder rate control module selects the frame types by applying its internal logic.
Figure 2. A block diagram of a video encoder incorporating Content Adaptive Bit-Rate encoding.
The encoder provides the CABR engine with each original input frame for pre-analysis within the quality measure calculator. The encoder performs an initial encode of the frame, using its own logic for bit allocation, motion estimation, mode selections, Quantization Parameters (QPs), etc. After encoding the frame, the encoder provides the CABR engine with the reconstructed frame corresponding to this initially encoded frame, along with some side information – such as the frame size in bits and the QP selected for each MacroBlock or Coding Tree Unit (CTU).
In each iteration, the CABR control module first decides if the frame should be re-encoded at all. This is done, for example, according to the frame type, the bit consumption of the frame, the quality of previous frames or iterations, and according to the maximum number of iterations set for the frame. In some cases, the CABR control module may decide not to re-encode a frame at all – in that case, the initial encoded frame becomes the output frame, and the encoder continues to the next frame. When the CABR control module decides to re-encode, the CABR engine provides the encoder with modified encoding parameters, for example, a proposed average QP for the frame, or the difference from the QP used for the initial encode. Note that the QP or delta QP values are an average value, and QP modulation for each encoding block can still be performed by the encoder. In more sophisticated implementations a QP map of value per encoding block may be provided, as well as additional encoder configuration parameters.
The encoder performs a re-encode of the frame with the modified parameters. Note that this re-encode is not a full encode, since it can utilize many encoding decisions from the initial encode. In fact, the encoder may perform only re-quantization of the frame, reusing all previous motion vectors and mode decisions. Then, the encoder provides the CABR engine with the reconstructed re-encoded frame, which becomes one of the candidate frames. The quality measure module then calculates the quality of the candidate re-encoded frame relative to the initially encoded frame, and this quality score, along with the bit consumption reported by the encoder is provided to the CABR control module, which again determines if the frame should be re-encoded. When that is the case, the CABR control module sets the encoding parameters for the next iteration, and the above process is repeated. If the control module decides that the search for the optimal frame parameters is complete, it indicates which frame, among all previously encoded versions of this frame, should be used in the output video stream. Note that the encoder rate control module receives its feedback from the initial encode of the current frame, and in this way the initial encode of the next frames (which determines the target quality of the bitstream) is not affected.
The CABR engine can operate in either a serial iterative approach or a parallel approach. In the serial approach, the results from previous iterations can be used to select the QP value for the next iteration. In the parallel approach, all candidate QP values are provided simultaneously and encodes are done in parallel – which reduces latency.
Integrating the CABR Engine with Software & Hardware Encoders
Beamr has integrated the CABR engine into its AVC software encoder, Beamr 4, and into its HEVC software encoder, Beamr 5. However, the CABR engine can be integrated with any software or hardware video encoder, supporting any block-based video standard such as MPEG-2, AVC, HEVC, EVC, VVC, VP9, and AV1.
To integrate the CABR engine with a video encoder, the encoder should support several requirements. First and foremost, the encoder should be able to re-encode an input frame (that has already been encoded) with several different encoding parameters (such as QP values), and save the “state” of each of these encodes, including the initial encode. The reason for saving the state is that when the CABR control module selects one of the candidate frame encodes (or the initial encode) as the one to use in the output stream, the encoder’s state should correspond to the state it was right after encoding that candidate frame. Encoders that support multi-threaded operation and hardware encoders typically have this capability, since each frame encode is performed by a stateless unit.
Second, the encoder should support an interface to provide the reconstructed frame and the per block QP and bit consumption information for the encoded frame. To improve compute performance, we also recommend that the encoder supports a partial re-encode mode, where information related to motion estimation, partitioning and mode decisions found in the initial encode can be re-used for re-encoding without being computed again, and only the quantization and entropy coding stages are repeated for each candidate encode. This results in a minimal encoding efficiency drop for the optimized encoding result, with significant speed-up compared to full re-encode. As described above, we recommend that the encoder will use the initial encoded data (QPs, compressed size, etc.) for its Rate Control state update. However, the selected frame and accompanying data must be used for reference frames and other reference data, such as temporal MV predictors, as it is the only data available in the bitstream for decoding.
When integrating with hardware encoders that support parallel encoding with no increase in latency, we recommend using the parallel search approach where multiple QP values per frame are evaluated simultaneously. If the hardware encoder can perform parallel partial encodes (for example, re-quantization and entropy coding only), while all parallel encodes use the analysis stage of the initial encode, such as motion estimation and mode decisions, better CPU performance will be achieved.
Sample Results
Below, we provide two sample results of the CABR engine, when integrated with Beamr 5, Beamr’s HEVC software encoder, each illustrating different aspects of CABR.
For the first example, we encoded various 4K 24 FPS source clips to a target bitrate of 10 Mbps. Sample frames from each of the clips can be seen in Figure 3. The clips vary in their content complexity: “Crowd Run” has very high complexity since it has great detail and very significant motion of the runners. “StEM” has medium complexity, with some video compression challenges such as different lighting conditions and reasonably high film grain. Finally, a promotional clip of JPEGmini by Beamr has low complexity due to relatively low motion and simple scenes.
Figure 3. Sample frames from the test clips. top: crowd-run, bottom left: StEM bottom right: JPEGmini.
We encoded 500 frames from each clip to a target bitrate of 10 Mbps, using the VBR mode of the Beamr 5 HEVC encoder, which performs regular encoding, and using the CABR mode, which creates a lower bit-rate, perceptually identical stream. For the high complexity clip “Crowd Run,” where providing excellent quality at such an aggressive bitrate is very challenging, CABR reduced the bitrate by only 3%. For the intermediate complexity clip “StEM,” bitrate savings were higher and reached 17%. For the lowest complexity clip “JPEGmini,” CABR reduced the bitrate by a staggering 45%, while still obtaining excellent quality which matches the quality of the 10 Mbps VBR encode. This extensive range of bitrate reduction percentage demonstrates the fully automatic content-adaptive nature of CABR-enhanced encoder, which reaches a different final bitrate, according to the content complexity.
The second example uses a 500 frame 1080p 24 FPS clip from the well-known “Tears Of Steel” movie by the Blender open movie project. The same clip was encoded using the VBR and CABR modes of the Beamr 5 HEVC software encoder, with three target bitrates: 1.5, 3 and 5 Mbps. Savings, in this case, were 13% for the lowest bitrate resulting in a 1.4 Mbps encode, 44% for the intermediate bitrate resulting in an encode of 1.8 Mbps, and 62% for the highest bitrate, resulting in a 2 Mbps encode. Figures 4 and 5 show sample frames from the encoded clips with VBR encoding on the left vs. CABR encoding on the right. The top two images are from encodes to a bitrate of 5 Mbps, while the bottom two were taken from the 1.5 Mbps encodes. As can be seen here, both 5 Mbps target encodes preserve the details, such as the texture of the bottom lip or the two hairs on the forehead above the right eye, while in the lower bitrate encodes these details are somewhat blurred. This is the reason that when starting from different target bitrates, CABR does not converge to the same bitrate. We also see, however, that the more generous the initial encoding, generally the more savings can be obtained. This example shows that CABR adapts not only to the content complexity, but also to the quality of the target encode, and preserves perceptual quality in motion while offering significant savings.
Figure 4. A sample from the “Tears of Steel” 1080p 24 FPS encode to 5 Mbps (top) and 1.5 Mbps (bottom), encoded in VBR mode (left) and CABR mode (right)
Figure 5. Closer view of the face in Figure 4, showing detail of lips and forehead from the encode to 5 Mbps (top) and 1.5 Mbps (bottom), encoded in VBR mode (left) and CABR mode (right).
By now, most of us have seen the data and know that online video consumption is soaring at a rate that is historically unrivaled. It’s no surprise that in the crux of the streaming era, so many companies are looking to innovate and figure out how to make their workflows or customers workflows better, less expensive, and faster.
In Episode 4 of The Video Insiders, we caught up with streaming veteran Tim Siglin to discuss HEVC implementation trends that counter previous assumptions, notable 2018 streaming events, and what’s coming in 2019. Tune in to hear The Video Insiders cover top-of-mind topics:
Want to join the conversation? Reach out to TheVideoInsiders@beamr.com
TRANSCRIPTION (lightly edited to improve readability only)
Mark Donnigan: 00:00 On today’s episode, the Video Insiders sit down with an industry luminary who shares results of a codec implementation study, while discussing notable streaming events that took place in 2018 and what’s on the horizon for 2019. Stay tuned. You don’t want to miss receiving the inside scoop on all this and more.
Announcer: 00:22 The Video Insiders is the show that makes sense of all that is happening in the world of online video, as seen through the eyes of a second generation Kodak nerd and a marketing guy who knows what I frames and macroblocks are. Here are your hosts, Mark Donnigan and Dror Gill.
Mark Donnigan: 00:40 Welcome, everyone. I am Mark Donnigan, and I want to say how honored Dror and I are to have you with us. Before I introduce this very special guest and episode, I want to give a shout of thanks for all of the support that we’re receiving. It’s really been amazing.
Mark Donnigan: 00:59 In the first 48 hours, we received 180 downloads. It’s pretty amazing.
Dror Gill: 01:06 Yeah. Yeah, it is. The industry is not that large, and I think it’s really an amazing number that they’re already listening to the show from the start before the word of mouth starts coming out, and people spread the news and things like that. We really appreciate it. So, if it’s you that is listening, thank you very much.
Mark Donnigan: 01:29 We really do aim for this to be an agenda-free zone. I guess we can put it that way. Obviously, this show is sponsored by Beamr, and we have a certain point of view on things, but the point is, we observed there wasn’t a good place to find out what’s going on in the industry and sort of get unbiased, or maybe it’s better to say unfiltered, information. That’s what we aim to do in every episode.
Mark Donnigan: 01:57 In this one, we’re going to do just that. We have someone who you can definitely trust to know what’s really happening in the streaming video space, and I know he has some juicy insights to share with us. So, without further ado, let’s bring on Tim Siglin.
Tim Siglin: 02:15 Hey, guys. Thank you for having me today and I will definitely try to be either as unfiltered or unbiased as possible.
Mark Donnigan: 02:21 Why don’t you give us a highlight reel, so to speak, of what you’ve done in the industry and, even more specifically, what are you working on today?
Tim Siglin: 02:31 Sure. I have been in streaming now for a little over 20 years. In fact, when Eric Schumacher-Rasmussen came on as the editor at StreamingMedia.com, he said, “You seemed to be one of the few people who were there in the early days.” It’s true. I actually had the honor of writing the 10-year anniversary of Streaming Media articles for the magazine, and then did the 20-year last year.
Tim Siglin: 02:57 My background was Motion Picture production and then I got into video conferencing. As part of video conferencing, we were trying to figure out how to include hundreds of people in a video conference, but not need necessarily have them have two-way feedback. That’s where streaming sort of caught my eye, because, ultimately, for video conferencing we maybe needed 10 subject matter experts who would talk back and forth, and together a hundred, then it went to thousands, and now hundreds of thousands. You can listen in and use something like chat or polling to provide feedback.
Tim Siglin: 03:31 For me, the industry went from the early revolutionary days of “Hey, let’s change everything. Let’s get rid of TV. Let’s do broadcast across IP.” That was the mantra in the early days. Now, of course, where we are is sort of, I would say, two-thirds of the way there, and we can talk a little bit about that later. The reality is that the old mediums are actually morphing to allow themselves to do heap, which is good, to compete with over the top.
Tim Siglin: 04:01 Ultimately, what I think we’ll find, especially when we get to pure IP broadcast with ATSC 3.0 and some other things for over-the-air, is that we will have more mediums to consume on rather than fewer. I remember the early format ways and of course we’re going to talk some in this episode about some of the newer codec like HEVC. Ultimately, it seems like the industry goes through the cycles of player wars, format wars, browser wars, operating system wars, and we hit brief periods of stability which we’ve done with AVC or H.264 over the last probably eight years.
Tim Siglin: 04:46 Then somebody wants to stir the pot, figure out how to either do it better, less expensively, faster. We go back into a cycle of trying to decide what the next big thing will be. In terms of what I’m working on now, because I’ve been in the industry for almost 21 years. Last year, I helped start a not-for-profit called Help Me Stream, which focuses on working with NGOs in emerging economies, trying to help them actually get into the streaming game to get their critical messages out.
Tim Siglin: 05:18 That might be emerging economies like African economies, South America, and just the idea that we in the first world have streaming down cold, but there are a lot of messages that need to get out in emerging economies and emerging markets that they don’t necessarily have the expertise to do. My work is to tie experts here with need there and figure out which technologies and services would be the most appropriate and most cost effective.
Tim Siglin: 05:48 The other thing I’m working on here, just briefly, is we’re getting ready for the Streaming Media Sourcebook, the 2019 sourcebook. I’m having to step back for the next 15 days, take a really wide look at the industry and figure out what the state of affairs are.
Dror Gill: 06:06 That’s wonderful. I think because this is exactly the right point, is one you end and the other one begins, kind of to summarize where we’ve been in 2018, what is the state of the industry and the fact that you’re doing that for the sourcebook, I think, ties in very nicely with our desire to hear from you an overview of what were the major milestones or advancements that were made in the streaming industry in 2018, and then looking into next year.
Dror Gill: 06:39 Obviously, the move to IP, getting stronger and stronger, now the third phase after analog and digital, now we have broadcast over IP. It’s interesting what you said about broadcasters not giving up the first with the pure OTT content providers. They have a huge business. They need to keep their subscribers and lower their churn and keep people from cutting the cord, so to speak.
Dror Gill: 07:04 The telcos and the cable companies still need to provide the infrastructure for Internet on top of which the over-the-top providers and their content, but they still need to have more offering and television and VLD content in order to keep their subscribers. It’s very interesting to hear how they’re doing it and how they are upgrading themselves to the era of IP.
Tim Siglin: 07:30 I think, Dror, you hit a really major point, which is we, the heavy lift … I just finished an article in ATSC 3.0 where I talk about using 2019 to prepare for 2020 when that will go live in the U.S. The heavy lift was the analog to digital conversion. The slightly easier lift is the conversion from digital to IP, but it still requires significant infrastructure upgrade and even transmission equipment to be able to do it correctly for the over-the-year broadcasters and cable.
Dror Gill: 08:07 That’s right. I think on the other hand, there is one big advantage to broadcast, even broadcast over-the-air. That is the ability to actually broadcast, the ability to reach millions, tens of millions, hundreds of millions of people over a single channel that everybody is receiving. Whereas, because of historic reasons and legacy reasons in IP, we are limited, still, when you broadcast to the end user to doing everything over unicast. When you do this, it creates a tremendous load on your network. You need to manage your CDNs.
Dror Gill: 08:46 I think we’ve witnessed in 2018 on one hand very large events being streamed to our record audience. But, on the other hand, some of them really failed in terms of user experience. It wasn’t what they expected because of the high volume of users, and because more and more people have discovered the ability to stream things over IP to their televisions and mobile devices. Can you share with us some of the experience that you have, some of the things that you’re hearing about in terms of these big events where they had failures and what were the reasons for those failures?
Tim Siglin: 09:30 I want to reiterate the point you made on the OTA broadcast. It’s almost as if you have read the advanced copy of my article, which I know you haven’t because it’s only gone to the editor.
Dror Gill: 09:42 I don’t have any inside information. I have to say, even though we are the Video Insiders.
Mark Donnigan: 09:47 We are the Video Insiders. That’s right.
Mark Donnigan: 09:49 But no inside information here.
Dror Gill: 09:51 No inside information. I did not steal that copy.
Tim Siglin: 09:55 What I point out in that article, Dror, I think which will come out in January shortly after CES is basically this. We have done a good job in the streaming industry, the OTT space of pushing the traditional mediums to upgrade themselves. One of the things as you say with OTA, that ability to do essentially a multicast from a tower wirelessly is a really, really good thing, because to get us to scale, and I think about things like the World Cup, the Olympics and even the presidential funeral that’s happened here in December, there are large-scale events that we in the OTT space just can’t handle, if you’re talking about having to build the capacity.
Tim Siglin: 10:39 The irony is, one good ATSC transmission tower could hit as many people as we could handle essentially globally with the unicast (OTT) model. If you look at things like that and then you look at things like EMBMS in the mobile world, where there is that attempt to do essentially a multicast, and it goes to points like the World Cup. I think one of the horror stories in the World Cup was in Australia. There was a mobile provider named Optus who won the rights to actually do half of the World Cup preliminary games. In the first several days, they were so overwhelmed by the number of users who wanted to watch and were watching, as you say, in a unicast model that they ended up having to go back to the company they had bid against who had the other half of the preliminaries and ask them to carry those on traditional television.
Tim Siglin: 11:41 The CEO admitted that it was such a spectacular failure that it damaged the brand of the mobile provider. Instead of the name Optus being used, everybody was referring to it as “Floptus.” You don’t want your brand being known as the butt of jokes for an event that only happens once every four years that you have a number of devotees in your market. And heaven forbid, it had been the World Cup for cricket, there would have been riots in the street in Sydney and Melbourne. Thank goodness it was Australia with soccer as opposed to Australia with cricket.
Tim Siglin: 12:18 It brings home the point that we talk about scale, but it’s really hard to get to scale in a unicast environment. The other event, this one happened, I believe, in late 2017, was the Mayweather fight that was a large pay-per-view event that was streamed. It turned out the problem there wasn’t as much the streams as it was the authentication servers were overwhelmed in the first five minutes of the fight. So, with authentication gone, it took down the ability to actually watch the stream.
Tim Siglin: 12:53 For us, it’s not just about the video portion of it, it’s actually about the total ecosystem and who you’re delivering to, whether you’re going to force caps into place because you know you can’t go beyond a certain capacity, or whether you’re going to have to partner up with traditional media like cable service providers or over-the-air broadcasters.
Mark Donnigan: 13:14 It’s a really good point, Tim. In the World Cup, the coverage that I saw, it was more of, I’d almost say or use the phrase, dashed expectations. Consumers, they were able to watch it. In most cases, I think it played smoothly. In other words, the video was there, but HDR signaling didn’t work or didn’t work right. Then it looked odd on some televisions or …
Tim Siglin: 13:43 20 frames a second instead of 60 frames a second.
Mark Donnigan: 13:48 Exactly. What’s interesting to me is that, what I see is, the consumer, they’re not of course walking around thinking as we are, like frame rate and color space and resolution. They are getting increasingly sensitive to where they can look at video now and say, “That’s good video,” or “That doesn’t look right to me.” I know we were talking before we started recording about this latest Tom Cruise public service announcement, which is just super fascinating, because it …
Tim Siglin: 14:24 To hear him say motion interpolation.
Mark Donnigan: 14:26 Yeah. Maybe we should tell the audience, for those, since it literally just came out I think today, even. But you want to tell the audience what Tom Cruise is saying?
Tim Siglin: 14:38 Essentially, Tom Cruise was on the set of Top Gun, as they’re shooting Top Gun. Another gentleman did a brief PSA for about a minute asking people to turn off motion interpolation on their televisions, which motion interpolation essentially takes a 24-frame per second and converts it to 30 frames per second by adding phantom frames in the middle. Because Mission Impossible: Fallout is just being released for streaming, Cruise was concerned and obviously others were concerned that some of the scenes would not look nearly as good with motion interpolation turned on.
Tim Siglin: 15:17 I think, Mark, we ought to go to a PSA model, asking for very particular things like, “How do you turn HDR on? How do you …” Those types of things, because those get attention in a way that you and I or a video engineer can’t get that attention.
Dror Gill: 15:33 How do you know if what you’re getting is actually 4K or interpolate HD, for example?
Tim Siglin: 15:38 Especially in our part of the industry, because we will call something OTT 4K streaming. That may mean that it fits in a 4K frame, but it doesn’t necessarily mean it’s that number of pixels being delivered.
Dror Gill: 15:52 It can also mean that the top layer in your adaptive bit rate stream is 4K, but then if you don’t have enough bandwidth, you’re actually getting the HD layer or even lower.
Dror Gill: 16:02 Even though it is a 4K broadcast and it is 4K content. Sometimes, you can be disappointed by that fact as well.
Mark Donnigan: 16:11 I have to give a very, very funny story directly related, and this happened probably, I don’t know, maybe, at least 18 months ago, maybe two years ago. I’m sitting on an airplane next to this guy. It’s the usual five-minute, get acquainted before we both turn on our computers. Anyway, when someone asks, “What do you do?” I generally just say, “I work for a video software company,” because how do you explain digital encoding? Most people just sort of stop at that, and don’t really ask more.
Mark Donnigan: 16:44 But this guy is like, “Oh, really?” He said, “So, I just bought a 4K TV and I love it.” He was raving about his new Samsung TV. Of course, he figured I’m a video guy. I would appreciate that. I said, “Hey.” “So, you must subscribe to Netflix.” “Yes. Yes, of course,” he says. I said, “What do you think of the Netflix quality? It looks great, doesn’t it?”
Mark Donnigan: 17:10 He sort of hem and hawed. He’s like, “Well, it really … I mean, yeah. Yeah, it looks great, but it’s not quite … I’m just not sure.” Then, I said, “I’m going to ask you two questions. First of all, are you subscribed to the 4K plan?” He was. Then I said, “How fast is your Internet at home.” He’s like, “I just have the minimum. I don’t know. I think it’s the 20 megabit package,” or whatever it was. I don’t remember the numbers.
Mark Donnigan: 17:38 I said, “There’s this thing.” And I gave him like a 30-second primer on adaptive bit rate, and I said, “It is possible, I have no idea of your situation, that you might be watching the HD version.” Anyway, he’s like, “Hah, that’s interesting.” I connect with the guy on LinkedIn. Three days later, I get this message. He says, “I just upgraded my Internet. I now have 4K on my TV. It looks awesome.”
Mark Donnigan: 18:04 On one hand, the whole situation was not surprising and, yet, how many thousands, tens of thousands, maybe millions of people are in the exact same boat? They’ve got this beautiful TV. It could be because they’re running some low-end router in the house. It could be they truly have a low end bandwidth package. There could be a lot of reasons why they’re not getting the bandwidth. They’re so excited about their 4K TV. They’re paying Netflix to get the top layer, the best quality, and they’re not even seeing it. It’s such a pity.
Tim Siglin: 18:37 I had a TSA agent asked me that same question, Mark, when I came through customs. I’m like, “Sure. I’ll stand here and answer that question for you.” The router was actually what I suggested that he upgrade, because he said his router was like this (old unit).
Mark Donnigan: 18:53 In a lot of homes, it’s a router that’s 15 years old and it just isn’t (up to the task).
Tim Siglin: 18:58 But it brings out the point that even as we’re talking about newer codecs and better quality, even if we get a lower sweet spot in terms of 4K content (streaming bandwidth), or as we found in the survey that we worked on together, that using HEVC for 1080p or 720p, if the routers, if the software in the chain is not updated, the delivery quality will suffer in a way that people who have a tuned television and seen the consistent quality aren’t certain what to do to fix when they use an over-the-top service.
Tim Siglin: 19:34 I think this is a key for 2019. As we prepare for ATSC 3.0 on over-the-air broadcast where people will be able to see pristine 4K, it will actually force those of us in the OTT space to up our game to make sure that we’re figuring out how to deliver across these multiple steps in a process that we don’t break.
Dror Gill: 19:54 You really see ATSC 3.0 as a game-changer in 2019?
Tim Siglin: 19:59 What I see it as is the response from the broadcast industry to, A) say that they’re still relevant, which I think is a good political move. And, B) it provides the scale you were talking about, Dror. See, I think what it does is it at least puts us in the OTT space on notice that there will be in certain first world countries a really decent quality delivery free of charge with commercials over the air.
Tim Siglin: 20:31 It takes me back to the early days of video compression when, if you had a good class-one engineer and an analog NTSC transmission system, they could give you really good quality if your TV was tuned correctly. It only meant having to tune your TV. It didn’t mean having to tune your router or having to tune your cable modem, having to tune your settings on your TV. I think that’s where the game-changer may be, is that those tuner cards, which will send HDR signaling and things like that with the actual transmission, are going to make it much easier for the consumer to consume quality in a free scenario. I think that part of it is a potential game-changer.
Mark Donnigan: 21:19 That’s interesting. Tim, we worked together earlier this year on a survey, an industry survey that I think it would be really, really interesting to listeners to talk about. Shall we pivot into that? Maybe you can share some of the findings there.
Tim Siglin: 21:38 Why don’t you take the lead on why Beamr wanted to do that? Then I’ll follow up with some of the points that we got out of it.
Mark Donnigan: 21:46 Obviously, we are a codec developer. It’s important for us to always be addressing the market the way that the market wants to be addressed, meaning that we’re developing technologies and solutions and standards that’s going to be adopted. Clearly, there has been, especially if we rewind a year ago or even 18 months ago, AV1 was just recently launched. There were still questions about VP9.
Mark Donnigan: 22:19 Obviously, H264 AVC is the standard, used everywhere. We felt, “Let’s go out to the industry. Let’s really find out what the attitudes are, what the thinking is, what’s going on ‘behind closed doors’ and find out what are people doing.” Are they building workflows for these new advanced codecs? How are they going to build those workflows? That was the impetus, if you will, for it.
Mark Donnigan: 22:49 We are very happy, Tim, to work with you on that and of course Streaming Media assisted us with promoting it. That was the reason we did it. I know there were some findings that were pretty predictable, shall we say, no surprises, but there were some things that I think were maybe a little more surprising. So, maybe if you like to share some of those.
Tim Siglin: 23:12 Yeah. I’ll hit the highlights on that. Let me say too that one of the things that I really like about this particular survey, there was another survey that had gone on right around that time that essentially was, “Are you going to adopt HEVC?” What we took the approach on with this survey was to say, “Okay. Those of you who’ve already adopted HEVC, what are the lessons that we can learn from that?”
Tim Siglin: 23:36 We didn’t exclude those who were looking at AV1 or some of the other codes, even VP9, but we wanted to know those people who used HEVC. Were they using it in pilot projects? Were they thinking about it? Were they using it in actual production? What we found in the survey is that AVC, or H.264, was still clearly dominant in the industry, but that the ramp-up to HEVC was moving along much faster than at least I … I believed. Mark, I told you when we started the survey question creation, which was about a year ago and then launched it in early 2018, I expected we wouldn’t see a whole lot of people using HEVC in production.
Tim Siglin: 24:23 I was pleasantly surprised to say that I was wrong. In fact, I think you mentioned in our recent Streaming Media West interview that there was a statistic you gave about the number of households that could consume HEVC. Was it north of 50%?
Mark Donnigan: 24:40 Yeah, it’s more than 50%. What’s interesting about that number is that that actually came from a very large MSO. Of course, they have a very good understanding of what devices are on their network. They found that there was at least one device in at least 50% of their homes that could receive and decode, playback, HEVC. That’s about as real world as you can get.
Tim Siglin: 25:06 What was fascinating to me too in this study was, we asked open-ended questions, which is what I’ve done in the research projects for the last 25 years both the video conferencing and streaming. One of the questions we asked was, “Do you see HEVC as only a 4K solution or do you see it as an option for lower resolutions?” It turned out overwhelmingly, people said, “We not only see it for 4K. We see it for high-frame rate (HFR) 1080p, standard frame rate 1080p, with some HDR.”
Tim Siglin: 25:40 Not a majority, but a large number of respondents said they would even see it as a benefit at 720p. What that tells me is, because we had a large number of engineers, video engineers, and we also have people in business development who answer these questions, what it tells me is that companies know as we scale because of the unicast problem that Dror pointed out in the beginning that scaling with a codec that consumes more bandwidth is a good way to lose money, kind of like the joke that the way a rich man can lose money really fast is to invest in an airline.
Tim Siglin: 26:19 If indeed you get scale with AVC, you could find yourself with a really large bill. That look at HEVC is being not just for 4K, HDR, or high frame rate in the future, but also for 1080p with some HDR and high frame rate. It tells me that the codec itself or the promise of the codec itself was actually really good. What was even more fascinating to me was the number of companies that had AVC pipelines that were actually looking to integrate HEVC into those same production pipe.
Tim Siglin: 26:55 It was much easier from a process standpoint to integrate HEVC into an AVC pipeline, so in other words, H265 into H264 pipeline than it was to go out of house and look at something like AV1 or VP9, because the work that was done on HEVC builds on the benefits that were already in place in AVC. Of course, you got Apple who has HLS, HTTP Live Streaming, and a huge ecosystem in terms of iPhones and iPads, laptops and desktops supporting HEVC not just as a standard for video delivery, but also with the HEIC or HEIF image format, now having all of their devices shoot images using HEVC instead of JPEG. That in and of itself drives forward adoption of HEVC. I think you told me since that survey came out, probably now seven months ago, you all have continued to see the model of all-in HEVC adoption.
Dror Gill: 28:03 This is what we promote all the time. It’s kind of a movement. Are you all in HEVC or are you doing it just for 4K, just where you have to do it? We really believe in all-in HEVC. Actually, this week, I had an interesting discussion with one of our customers who is using our optimization product for VOD content, to reduce bit-rate of H.264 (streams). He said, “I want to have a product. I want to have a solution for reducing bit-rates on our live channels.”
Dror Gill: 28:32 So, I asked them, “Okay. Why don’t you just switch your codec to HEVC?” He said, “No, I can’t do that.” I said, “Why not?” He said, “You know compatibility and things like that.” I asked, “Okay. What are you using? What are you delivering to?” He said, “We have our own set-top boxes (STB), IP set-top boxes which we give out to our customers. Well, these are pretty new.” So, they support HEVC. I’m okay there. “Then we have an Apple TV app.” “Okay, Apple TV has a 4K version. So, it supports HEVC. All of the latest Apple TV devices have HEVC. That’s fine.” “Then we have smartphone apps, smart TV apps for Android TV and for the LG platform.”
Dror Gill: 29:15 Obviously, TV’s support 4K. So, I’m okay there. With delivering to mobile devices, all the high-end devices already support HEVC. He was making this estimate that around 50 to 60% of his viewers are using devices that are HEVC capable. Suddenly, he’s thinking, “Yeah, I can do that. I can go all in HEVC. I will continue, of course, to support H.264 for all of the devices that don’t support HEVC. But if I can save 50% of the bandwidth to 50 to 60% of my customers, that’s a very big savings.”
Mark Donnigan: 29:48 What’s interesting about this conversation, Dror, is first of all I’m pretty certain that the operator you’re talking with is different than the operator that I shared, found the exact same thing. This is a consistent theme, is that pretty much in developed parts of the world, it really is true that 50% or more of the users can today receive HEVC. This number is only growing. It’s not like it’s static It is just growing. Next year, I don’t know if that number will be 60% or 70%, but it’s going to be even bigger.
Mark Donnigan: 30:27 What’s fascinating is that, again, we’ve said earlier, that the consumer is getting just more aware of quality, and they’re getting more aware of when they’re being underserved. For operators who are serving to lowest common denominator, which is to say, AVC works across all my devices, and it’s true. AVC works on all the high-end devices equally well, but you’re under-serving a large and growing number of your users.
Mark Donnigan: 31:01 If your competitors are doing the same, then I guess you could say … well, “Who are they going to switch to?” But there are some fast-moving leaders in the space who are either planning or they’re shortly going to be offering better quality. They’re going to be extending HEVC into lower bit rates or lower resolutions, that is, and therefore lower bit rates, and the consumers are going to begin to see like, “Well, wait a second. This service over here that my friend has or we have another subscription in the household, how come the video looks better?” They just begin to migrate there. I think it’s really important when we have these sorts of conversations to connect to this idea that don’t underserve your consumer in an effort to be something to everybody.
Tim Siglin: 31:57 I would add two other quick things to that, Mark. One is, we’ve always had this conversation in the industry about the three-legged stool of speed, quality and bandwidth in terms of the encoding.
Tim Siglin: 32:09 Two of those are part of the consumer equation, which is quality and bandwidth. Then, oftentimes, we’ve had to make the decision between quality and bandwidth. If the argument is ostensibly that HEVC as it stands right now, had a couple years of optimization, can get us to about, let’s say, 40%. Let’s not even say 50%. For equivalent quality, it can get us to 40% bandwidth reduction. Why wouldn’t you switch over to something like that?
Tim Siglin: 32:39 Then the second part, and I have to put a plugin for what Eric Schumacher-Rasmussen and the Streaming Media team did at Streaming Media West by having Roger Pantos come and speak, Roger Pantos being of course the inventor of HLS, and I’m not a huge fan of HLS, just because of the latency issues. But he pointed out in his presentation, his tutorial around HLS that you can put two different codecs in a manifest file. There is absolutely no reason that an OTT provider could not provide both HEVC and AVC within the same manifest file and then allow the consumer device to choose.
Tim Siglin: 33:22 When Dror mentioned the company who has the OTT boxes that they give away, they could easily set a flag in those boxes to say, “If you’re presented with a manifest file that has AVC and HEVC, go with HEVC to lower the bandwidth, overall.” The beauty is it’s a technical issue at this point and it’s a technical implementation issue, not a ‘can we make it work?’ Because we know that it works based around the HLS.
Mark Donnigan: 33:54 This is excellent. Tim, let’s wrap this up, as I knew it would be. It has just been an awesome conversation. Thank you for sharing all your years of collective experience to give some insight into what’s happening in the industry. Let’s look at 2019. I know we’ve been talking a little bit about … you’ve made references to ATSC 3.0. Some of our listeners will be going to CES. Maybe there’s some things that they should be looking at or keeping their eyes opened for. What can you tell us about 2019?
Tim Siglin: 34:35 Here’s what I think 2019 is bringing. We have moved in the cloud computing space and you all are part of this conversation at Beamr. We’ve moved from having cloud-based solutions that were not at parity with on-premise solutions to actually in 2018 reaching parity between what you could do in an on-premise solution versus the cloud. Now, I think in 2019, what we’re going to start seeing is a number of features in cloud-based services, whether it’s machine learning, which the popular nomenclature is AI, but I really like machine learning as a much better descriptor, whether it’s machine learning, whether it’s real-time transcoding of live content, whether it’s the ability to simultaneously spit out AVC and HEVC like we’ve been talking about here that the cloud-based solutions will move beyond parity with the on-premise solutions.
Tim Siglin: 35:35 There always will be needs for the on-premise parts from a security standpoint in sort of the industries, but I don’t think that will inhibit cloud-based in 2019. If people are going to CES, one of the things to look at there, for instance, is a big leap in power consumption savings for mobile devices. I’m not necessarily talking about smartphones, because the research I’ve done says the moment you turn GPS on, you lose 25% of battery. Tablets have the potential to make a resurgence in a number of areas for consumers and I think we’ll see some advances in battery (capacity).
Tim Siglin: 36:19 Part of that goes to HEVC, which as we know is a much harder codec to decode. I think the consumer companies are being forced into thinking about power consumption as HEVC becomes more mainstream. That’s something I think people should pay attention to as well. Then, finally, HDR and surround sound solutions, especially object placement like Dolby Atmos and some of these others, will become much more mainstream as a way to sell flat panels and surround sound systems.
Tim Siglin: 36:56 We sort of languished in that space. 4K prices have dropped dramatically in the last two years, but we’re not yet ready for 8K. But I think we’ll see a trend toward fixing some of the audio problems. In the streaming space, to fix those audio problems, we need to be able to encode and encapsulate into sort of the standard surround sound model. Those are three areas that I would suggest people pay attention.
Mark Donnigan: 37:25 Well, thank you for joining us, Tim. It’s really great to have you on. We’ll definitely do this again. We want to thank you, the listener, for supporting the Video Insiders. Until the next episode. Happy encoding!
Announcer: 37:39 Thank you for listening to the Video Insiders Podcast, a production of Beamr Imaging Limited. To begin using Beamr’s codecs today, go to Beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.
Anyone familiar with the streaming video industry knows that we love our acronyms. You would be hard-pressed to have a conversation about the online video industry without bringing one up…
In today’s episode, The Video Insiders focus on the future of three-character codecs: AVC, VP9, and VVC.
But before we can look at the future, we have to take a moment to revisit the past.
The year 2018 marks the 15-year anniversary of AVC and in this episode, we visit the process and lifecycle of standardization to adoption and what that means for the future of these codecs.
Mark Donnigan: 00:52 It is, it is episode three. So, today we have a really exciting discussion as we consider the future of codecs named with three characters.
Dror Gill: 01:03 Three character codecs, okay, let’s see.
Dror Gill: 01:10 Let’s see, that’s today’s trivia question. I can think of AVC, VP9, AV1, and VVC?
Mark Donnigan: 01:21 Well, you just named three that I was thinking about and we’re gonna discuss today! We’ve already covered AV1. Yeah, yeah, you answered correctly, but we haven’t really considered where AVC, VP9, and VVC fit into the codec stew. So when I think about AVC, I’m almost tempted to just skip it because isn’t this codec standard old news? I mean, c’mon. The entire video infrastructure of the internet is enabled by AVC, so what is there to discuss?
Dror Gill: 01:57 Yeah. You’re right. It’s like the default, but in fact, the interesting thing is that today, we’re (in) 2018 and this is the twenty year anniversary of AVC. I mean, ITU issued the call for proposals, their video coding expert group, issued the call for proposal for a project. At the time was called H26L, and their target was to double the coding efficiency, which effectively means halving the bit rate necessary for given level of fidelity. And that’s why it was called H26L, it was supposed to be low bit rate.
Mark Donnigan: 02:33 Ah! That’s an interesting trivia question.
Mark Donnigan: 02:36 I wonder how many of our listeners knew that? That’s kind of cool. H26L.
Dror Gill: 02:42 But they didn’t go alone. It was the first time they joined forces in 2001 with the ISO MPEG, that’s the same Motion Pictures Experts Group, you know we discussed in the first episode.
Dror Gill: 02:57 And they came together, they joined forced, and they created JVT, that was the Joint Video Team, and I think it’s a great example of collaboration. There are standards by dealing with video communication standards, and ISO MPEG, which is a standards body dealing with video entertainment standards. So, finally they understood that there’s no point in developing video standards for these two different types of applications, so they got all the experts together in the JVT and this group developed what was the best video compression standard at the time. It was launched May 30, 2003.
Dror Gill: 03:36 There was one drawback with this collaboration in that the video standard was known by two names. There was the ITU name which is H.264. And then there’s the ISO MPEG name which is AVC, so these created some confusion at the start. I think by now, most of our listeners know that H.264 and AVC are two of the same.
Mark Donnigan: 03:57 Yeah, definitely. So, AVC was developed 15 years ago and it’s still around today.
Dror Gill: 04:02 Yeah, yeah. I mean, that’s really impressive and it’s not only around, it’s the most popular video compression standard in the world today. I mean, AVC is used to deliver video over the internet, to computers, televisions, mobile devices, cable, satellite, broadcast, and even blu-ray disks. This just shows you how long it takes from standardization to adoption, right? 15 years until we get this mass market adoption market dominance of H.264, AVC as we have today.
Dror Gill: 04:31 And the reason it takes so long is that, we discussed it in our first episode, first you need to develop the standard. Then, you need to develop the chips that support the standard, then you need to develop devices that incorporate the chip. Even when initial implementation of the codec got released, they are still not as efficient as they can be, and it takes codec developers more time to refine it and improve the performance and the quality. You need to develop the tools, all of that takes time.
Mark Donnigan: 04:59 It does. Yeah, I have a background in consumer electronics and because of that I know for certainty that AVC is gonna be with us for a while and I’ll explain why. It’s really simple. Decoding of H.264 is fully supported in every chip set on the market. I mean literally every chip set. There is not a device that supports video which does not also support AVC today. It just doesn’t exist, you can’t find it anywhere.
Mark Donnigan: 05:26 And then when you look at in coding technologies for AVC, H.264, (they) have advanced to the point where you can really achieve state of the art for very low cost. There’s just too much market momentum where the encode and decode ecosystems are just massive. When you think about entertainment applications and consumer electronics, for a lot of us, that’s our primary market (that) we play in.
Mark Donnigan: 05:51 But, if you consider the surveillance and the industrial markets, which are absolutely massive, and all of these security cameras you see literally everywhere. Drone cameras, they all have AVC encoders in them. Bottom line, AVC isn’t going anywhere fast.
Dror Gill: 06:09 You’re right, I totally agree with that. It’s dominant, but it’s still here to stay. The problem is that, we talked about this, video delivery over the internet. The big problem is the bandwidth bottleneck. With so much video being delivered over the internet, and then the demand for quality is growing. People want higher resolution, they want HDR which is high dynamic range, they want higher frame rate. And all this means you need more and more bit rate to represent the video. The bit rate efficiency that is required today is beyond the standard in coding in AVC and that’s where you need external technologies such as content adaptive encoding perceptual optimization that will really help you push AVC to its limits.
Mark Donnigan: 06:54 Yeah. And Dror, I know you’re one of the inventors of a perceptual optimization technique based on a really unique quality measure, which I’ve heard some in the industry believe could even extend the life of AVC from a bit rate efficiency perspective. Tell us about what you developed and what you worked on.
Dror Gill: 07:13 Yeah, that’s right. I did have some part in this. We developed a quality measure and a whole application around it, and this is a solution that can reduce the bit rate of AVC by 30%, sometimes even 40%. It doesn’t get us exactly to where HEVC starts, 50% is pretty difficult and not for every content (type). But content distributors that recognize AVC will still be part of their codec mix for at least five years, I think what we’ve been able to do can really be helpful and a welcome relief to this bandwidth bottleneck issue.
Mark Donnigan: 07:52 It sounds like we’re in agreement that for at least the midterm horizon, the medium horizon, AVC is gonna stay with us.
Dror Gill: 08:01 Yeah, yeah. I definitely think so. For some applications and services and certain regions of the world where the device penetration of the latest, high end models is not as high as in other parts, AVC will be the primary codec for some time to come.
Dror Gill: 08:25 It’s interesting to me, essentially, it’s mostly a YouTube codec. It’s not a bad coded, it has some efficiency advantages over AVC, but outside of Google, you don’t see any large scale deployments. By the way, if you look at Wikipedia, you read about the section that says where is VP9 used, it says VP9 is used mostly by YouTube, some uses by Netflix, and it’s being used by Wikipedia.
Mark Donnigan: 08:50 VP9 is supported fairly well in devices. Though, it’s obviously hard to say exactly what the penetration is, I think there is support in hardware for decode for VP9. Certainly it’s ubiquitous on Android, and it’s in many of the UHD TV chip sets as well. So, it’s not always enabled, but again, from my background on the hardware side, I know that many of those SOCs, they do have a VP9 decoder built into them.
Mark Donnigan: 09:23 I guess the question in my mind is, it’s talked about. Certainly Google is a notable both developer and user, but why hasn’t it been adopted?
Dror Gill: 09:33 Well, I think there are several issues here. One of them is compression efficiency. VP9 brings maybe 20, 30% improvement in compression efficiency over AVC, but it’s not 50%. So, you’re not doubling your compression efficiency. If you want to replace the codec, that’s really a big deal. That’s really a huge investment. You need to invest in coding infrastructure, new players. You need to do compatibility testing. You need to make sure that your packaging and your DRM work correctly and all of that.
Dror Gill: 10:04 You really want to get a huge benefit to offset this investment. I think people are really looking for that 50% improvement, to double the efficiency, which is what you get with HEVC but not quite with VP9. I think the second point is that VP9, even though it’s an open source coder, it’s developed and the standard is maintained by Google. And some industry players are kind of afraid of the dominance of Google. Google has taken over the advertising market online.
Dror Gill: 10:34 You know, and search and mobile operating systems, except Apple, it’s all Android. So, those industry players might be thinking, I don’t want to depend on Google for my video compression format. I think this is especially true for traditional broadcasters. Cable companies, satellite companies, TV channels that broadcast over the air. These companies traditionally like to go with established, international standards. Compression technologies that are standardized, they have the seal of approval by ITU and ISO.
Dror Gill: 11:05 They’re typically following that traditional codec developer past. ISO MPEG too, now it’s AVC, starting with HEVC. What’s coming next?
Mark Donnigan: 11:16 Well, our next three letter codec is VVC. Tell us about VVC, Dror.
Dror Gill: 11:21 Yeah, yeah, VVC. I think this is another great example of collaboration between ITU and ISO. Again, they formed a joint video experts team. This time it’s called JVET.
Dror Gill: 12:10 So, JVET has launched a project to develop a new video coding standard. And you know, we had AVC that was advanced video coding. Then we had HEVC which is high efficiency video coding. So, they thought, what would be the next generation? It’s already advanced, it’s high efficiency. So, the next one, they called it VVC, which is versatile video code. The objective of VVC is obviously to provide a significant improvement in compression efficiency over the existing HEVC standard. Development already started. The JVET group is meeting every few in months in some exotic place in the world and this process will continue. They plan to complete it before the end of 2020. So, essentially in the next two years they are gonna complete the standard.
Dror Gill: 13:01 Today, already, even though VVC is in early development and they haven’t implemented all the tools, they already report a 30% better compression efficiency than HEVC. So, we have high hopes that we’ll be able to fight the video tsunami that is coming upon us with a much improved standard video coder which is VVC. I mean, its improved at least on the technical side and I understand that they also want to improve the process, right?
Mark Donnigan: 13:29 That’s right, that’s right. Well, technical capabilities are certainly important and we’re tracking of course VVC. 30% better efficiency this early in the game is promising. I wonder if the JVET will bring any learnings from the famous HEVC royalty debacles to VVC because I think what’s in everybody’s mind is, okay, great, this can be much more efficient, technically better. But if we have to go round and round on royalties again, it’s just gonna kill it. So, what do you think?
Dror Gill: 14:02 Yeah, that’s right. I think it’s absolutely true and many people in the industry have realized this, that you can’t just develop a video standard and then handle the patent and royalty issues later. Luckily some companies have come together and they formed an industry group called The Media Coding Industry Forum, or MC-IF. They held their first meeting a few weeks ago in Macau during empic meeting one through four. Their purpose statement, let me quote this from their website, and I’ll give you my interpretation of it. They say the media coding industry forum (MC-IF) is an open industry forum with a purpose of furthering the adoption of standards initially focusing on VVC, but establishing them as well accepted and widely used standards for the benefit of consumers and the industry.
Dror Gill: 14:47 My interpretation is that the group was formed in an effort for companies with interest in this next generation video codec to come together and attempt to influence the licensing policy of VVC and try to agree on a reasonable patent licensing policy in advance to prevent history from repeating itself. We don’t want that whole Hollywood story with the tragedy that took a few years until they reached the happy ending. So, what are you even talking about? This is very interesting. They’re talking about having a modular structure for the codec. These tools of the codecs, the features, can be plugged in and out, very easily.
Dror Gill: 15:23 So, if some company insists on reasonable licensing terms, this group can just decide not to support the feature and it will be very easily removed from the standard, or at least from the way that companies implement that standard.
Mark Donnigan: 15:37 That’s an interesting approach. I wonder how technically feasible it is. I think we’ll get into that in some other episodes.
Dror Gill: 15:46 Yeah. That may have some effect on performance.
Mark Donnigan: 15:49 Exactly. And again, are we back in the situation that the Alliance for Open Media is in with AV1. Where part of the issue of the slow performance is trying to work around patents. At the end of the day you end up with a solution that is hobbled technically.
Dror Gill: 16:10 Yeah. I hope it doesn’t go there.
Mark Donnigan: 16:13 Yeah, I hope we’re not there. I think you heard this too, hasn’t Apple joined the consortium recently?
Dror Gill: 16:21 Yeah, yeah, they did. They joined silently as they always do. Silently means that one day somebody discovers their logo… They don’t make any announcement or anything. You just see a logo on the website, and then oh, okay.
Mark Donnigan: 16:41 You know, maybe it’s good to kind of bring this discussion back to Earth and close out our three part series by giving the listeners some pointers. About how they should be thinking about the next codec that they adopt. I’ve been giving some thought as we’ve been doing these episodes. I think I’ll kick it off here Dror if you don’t mind, I’ll share some of my thoughts. You can jump in.
Mark Donnigan: 17:11 These are complex decisions of course. I completely agree, billing this as codec wars and codec battles, it’s not helpful at the end of the day. Maybe it makes for a catchy headline, but it’s not helpful. There’s real business decisions (to be made). There are technical decisions. I think a good place to start for somebody who’s listening and saying “okay great, I now have a better understanding of the lay of the land of HEVC, for AV1, I can understand VP9, I can understand AVC and what some of my options are to even further reduce bit rate. But now, what do I do?”
Mark Donnigan: 17:54 And I think a good place to start is to just look at your customers, and do they lean towards early adopters. Are you in a strong economic environment, which is to say quite frankly, do most of your customers carry around the latest devices? Like an iPhone X, or Galaxy 9. If largely your customers lean towards early adopter and they’re carrying around the latest devices, then you have an obligation to serve them with the highest quality and the best performance possible.
Dror Gill: 18:26 Right. If your customers can receive HEVC, and it’s half the bit rate, then why not deliver it to them better quality, or say when you see the end cost with this more efficient codec and everybody is happy.
Mark Donnigan: 18:37 Absolutely, and again, I think just using pure logic. If somebody could afford a more than $1000 device in their pocket, probably the TV hanging on the wall is a very new, UHD capable (one). They probably have a game console in the house. The point is that you can make a pretty strong argument and an assumption that you can go, what I like to think of as all in HEVC including even standard definition, just SDR content.
Mark Donnigan: 19:11 So, the industry has really lost sight in my mind of the benefits of HEVC as they apply across the board to all resolutions. All of the major consumer streaming services are delivering 4K using HEVC, but I’m still shocked at how many, it’s kind of like oh, we forget that the same advantages of bit rate efficiency that work at 4K apply at 480p. Obviously, the absolute numbers are smaller because the file sizes are smaller, etc.
Mark Donnigan: 19:41 But the point is, 30, 40, 50% savings applies at 4K as it does at 480p. I understand there’s different applications in use cases, right? But would you agree with that?
Dror Gill: 19:55 Yeah, yeah, I surely agree with that. I mean, for 4K, HEVC is really an enabler.
Dror Gill: 20:01 For HEVC, you would need like 30, 40 megabits of video. Nobody can stream that to the home, but change it to 10, 15, that’s reasonable, and you must use HEVC for 4k otherwise it won’t even fit the pipe. But for all other resolutions, you get the bang with the advantage or you can trade it off for a quality advantage and deliver higher quality to your users, or higher frame rate, or enable HDR. If all of these possibilities that you can do with HD and even SD content, give them a better experience using HEVC and being able to stream on devices that your users already have. So yeah, I agree. I think it’s an excellent analysis. Obviously if you’re up in an emerging market, or your consumers don’t have high end devices, then AVC is a good solution. If there are network constraints, and there are many places in the world that network conductivity isn’t that great. Or in rural areas where we have very large parts of the population spread out (in these cases) bandwidth is low and you will get into a bottleneck even with HD.
Dror Gill: 21:06 That’s where perceptual optimization can help you reduce the bit rate even for AVC and keep within the constraints that you have. When your consumers can upgrade their devices and when the cycle comes in a few years when every device has HEVC support, then obviously you upgrade your capability and support HEVC across the board.
Mark Donnigan: 21:30 Yeah, that’s a very important point Dror, is that this HEVC adoption curve in terms of silicon, on devices. It is in full motion. Just the planning life cycles. If you look at what goes into hardware, and especially on the silicon side, it doesn’t happen that way. Once these technologies are in the designs, once they are in the dies, once the codec is in silicon, it doesn’t get arbitrarily turned on and off like light switches.
Mark Donnigan: 22:04 How should somebody be looking at VP9, VVC, and AV1?
Dror Gill: 22:13 Well, VP9 is an easy one. Unless you’re Google, you’re very likely gonna skip over this codec. Not just that the VP9 isn’t the viable choice, it simply doesn’t go so far as HEVC in terms of bit rate efficiency and quality. Maybe two years back we would consider it as an option for reducing bit rate, but now with the HEVC support that you have, there’s no point in going to VP9. You might as well go to HEVC. If you talk about VVC, (the) standard is still a few years from being ratified so, we actually don’t have anything to talk about.
Dror Gill: 22:49 The important point is again to remember, even when VVC launches, it will still be another 2 to 3 years after ratifying the standard before you have even a very basic playback ecosystem in place. So, I would tell our listeners if you’re thinking, why should I adopt HEVC, because VVC is just around the corner, well, that corner is very far. It’s more like the corner of the Earth than the corner of the next block.
Dror Gill: 23:18 So, HEVC today, VVC will be the next step in a few years. And then there’s AV1. You know, we talked a lot about AV1. No doubt, AV1 has support from huge companies. I mean Google, Facebook, Intel, Netflix, Microsoft. And those engineers, they know what they’re doing. But now, it’s quite clear that compression efficiency is the same as HEVC. Meanwhile, after removing other royalty cost for content delivery, HEVC Advance removed it. The license situation is much more clear now. You add to this the fact that at the end of the day, two to three years, you’re gonna need five to ten times more compute power to encode AV1, reaching effectively the same result. Now Google, again. Google may be that they have unlimited compute resources, they will use it. They developed it.
Dror Gill: 24:13 The smaller content providers, all the other ones, the non Googles of the world and other broadcasters with growing support for HEVC that we expect in a few years. I think it’s obvious. They’re gonna support HEVC and then a few years later when VVC is ratified, when it’s supported in devices, they’re gonna move to VVC. Because this codec does have the required compression efficiency improvement over HEVC.
Mark Donnigan: 24:39 Yeah, that’s an excellent summary Dror. Thank you for breaking this all down for our listeners so succinctly. I’m sure this is really gonna provide massive value. I want to thank our amazing audience because without you, the Video Insiders Podcast would just be Dror and me taking up bits on a server somewhere.
Mark Donnigan: 25:01 As you can tell, video is really exciting to us and so we’re so happy that you’ve joined us to listen. And again, this has been a production of Beamr Imaging Limited. Please, subscribe on iTunes and if you would like to try out beamer codecs in your lab or your production environment, we are giving away up to $100 of HEVC and H264 in coding every month. That’s each and every month. Just go to https://beamer.com/free and get started immediately.
When it comes to comparing video codecs, it’s easy to get caught up in the “codec war” mentality. If analyzing and purchasing codecs was as easy as comparing fuel economy in cars, it would undoubtedly take a lot of friction out of codec comparison, but the reality is that it’s not that simple.
In Episode 02, The Video Insiders go head-to-head comparing two of the leading codecs in a three against one standoff over whether AV1 is more efficient than HEVC.
So, which is more efficient?
Listen in to this week’s episode, “Codec Efficiency Is in the Eye of the Measurer,” to find out.
Tune in here or click play on the video below to listen now.
https://youtu.be/ThkoOHuPEvo
Want to join the conversation? Reach out to TheVideoInsiders@beamr.com.
TRANSCRIPTION (lightly edited to improve readability only)
Mark Donnigan: 00:41 Hi everyone I am Mark Donnigan and I want to welcome you to episode two of the Video Insiders.
Mark Donnigan: 00:50 In every episode of the Video Insiders we bring the latest inside information about what’s happening in the video technology industry from encoding, to packaging, to delivery, and playback, and even the business behind the video business. Every aspect of the video industry is covered in detail on the Video Insiders podcast.
Dror Gill: 01:11 Oh yeah, we usually do cover everything from pixels, to blocks, to microblocks, to frames, to sequences. We go all the way up and down the video delivery chain and highlight the most important things you should know before you send any video bits over the wire.
Mark Donnigan: 01:28 In our first episode we talked about a very hot topic which asked, “Hasn’t this kind of been worn out?” The whole HEVC, AV1 discussion. But I think it was very interesting. I sure enjoyed the talk. What about you Dror?
Dror Gill: 01:47 Yeah, yeah, yeah. I sure did. It was great talking about the two leading codecs. I don’t want to say the word, codec war.
Mark Donnigan: 01:58 No, no, we don’t believe in codec wars.
Mark Donnigan: 02:00 Yeah, that’s true. Why is it so complicated to compare video codecs? Why can’t it be as simple as fuel economy of cars, this one gets 20 miles per gallon and that one gets 30 and then I make a decision based on that.
Dror Gill: 02:15 I wish it was that simple with video codecs. In video compression you have so many parameters to consider. You have the encoding tools, tools are grouped into what’s called profiles and levels, or as AV1 calls them “experiments.”
Dror Gill: 02:35 When you compare the codecs which profiles and levels do you use. What rate control method? Which specific parameters do you set for each codec? And each codec can have hundreds, and hundreds of parameters. Then there is the question of implementation. Which software implementation of the codec do you use? Some implementations are reference implementations that are used for research, and others are highly performance optimized commercial implementations. Which one do you select for the test? And then, which operating system, what hardware do you run on, and obviously what test content? Because encoding two people talking, or encoding an action scene for a movie, is completely different.
Dror Gill: 03:13 Finally, when you come to evaluate your video, what quality measure do you use? There’re various objective quality measures and some people use actual human viewers and they assesses subjective quality of the video. On that front also, there’re many possibilities that you need to choose from.
Mark Donnigan: 03:32 Yeah, so many questions and no wonder the answers are not so clear. I was quite surprised when I recently read three different technical articles published at IBC actually, effectively comparing AV1 versus HEVC and I can assume that each of the authors did their research independently. What was surprising was they came to the exact same conclusion, AV1 has the same compression efficiency as HEVC. This is surprising because some other studies and one in particular (I think we’ll talk about) out there says the contrary. So can you explain what this means exactly, Dror.
Dror Gill: 04:16 By saying that they have the same compression efficiency, this means that they can reach the same quality at the same bitrate or the other way round. You need the same bitrate to reach that same quality. If you need for example, two and a half megabits per second to encode an HD video file using HEVC at a certain quality, then with AV1 you would need roughly the same bitrate to reach that same quality and this means that AV1 and HEVC provide the same compression level. In other words, this means that AV1 does not have any technical advantage over HEVC because it has the same compression efficiency. Of course that’s if we put aside all the loyalty issues but we discussed that last time. Right?
Mark Donnigan: 04:56 That’s right. The guys who wrote the three papers that I’m referencing are really top experts in the field. It’s not seminar work done by a student, not to downplay those papers, but the point is these are professionals. One was written by the BBC in cooperation with the Multimedia and Vision Group at the Queen Mary University of London. I think nobody is going to say that the BBC doesn’t know a thing or two about video. The second was written by Ateme, and the third by Harmonic, leading vendors.
Mark Donnigan: 05:29 I actually pulled out a couple of phrases from each that I’d like to quote. First the BBC and Queen Mary University, here is a conclusion that they wrote, “The results obtained show in general a similar performance between AV1 and the reference HEVC both objectively and subjectively.” Which is interesting because they did take the time to both do the visual assessment as well as use a quality measure.
Mark Donnigan: 06:01 Ateme said, “Results demonstrate AV1 to have equivalent performance to HEVC in terms of both objective and subjective video quality test results.”
Mark Donnigan: 06:16 And then here is what Harmonic said, “The findings are that AV1 is not more advantageous today than HEVC on the compression side and much more complex to encode than HEVC.” What do you make of this?
Dror Gill: 06:32 I don’t know. It sounds pretty bad to me, even two of those papers also analyzed subjective quality so they used actual human viewers to check out the quality. But Mark what if I told you that researchers from the University of Klagenfurt in Austria together with Bitmovin published a paper which showed completely different results. What would you say about that?
Dror Gill: 06:58 Last month in Athens I was the ICIP conference that’s the IEEE International Conference on Image Compression and Image Processing. There was this paper presented by this University in Austria with Bitmovin and their conclusion was, let me quote, “When using weighted PSNR, AV1 performs consistently better for bit rate compared to AVC, HEVC, and VP9.” So they claim AV1 is better than three codecs but specifically it’s better than HEVC. And then they have a table in their article that compares AV1 to HEVC for six different video clips. The table shows that with AV1 you get up to 25% lower bitrate at the same quality than HEVC.
Dror Gill: 07:43 I was sitting there in Athens last month when they presented this and I was shocked.
Mark Donnigan: 07:50 What are the chances that three independent papers are wrong and only this paper got it right? And by the way, the point here is not three against one because presumably there’re some other papers. I’m guessing other research floating around that might side with Bitmovin. The point is that three companies who no one is going to say that any of them are not experts and not highly qualified to do a video assessment, came up with such a different result. Tell us what you think is going on here?
Dror Gill: 08:28 I was thinking the same thing. How can that be. During the presentation I asked one of the authors who presented the paper a few questions and it turned out that they made some very questionable decisions in all of that sea of possibility that I talked about before. Decisions related to coding tools, codec parameters, and quality measures.
Dror Gill: 08:51 First of all, in this paper they didn’t show any results of subjective viewing. Only the objective metrics. Now we all know that you should always your eyes, right?
Dror Gill: 09:04 Objective metrics, nice numbers, but obviously you need to view the video because that’s how the actual viewers are going to assess the (video) quality. The second thing is that they only used the single objective metric and this was PSNR. PSNR, it stands for peak signal-to-noise ratio and basically this measure is a weighted average of the difference in peaks between pixel values of the two images.
Dror Gill: 09:30 Now, we’re Video Insiders, but even if you’re not an insider you know that PSNR is not a very good quality measure because it does not correlate very well with human vision. This is the measure that they choose to look at but what was most surprising is that there is a flag in the HEVC open source encoder which they used that if chosen, the result is improved PNSR. What it does, it turns off some psycho-visual optimizations which make the video look better but reduce the PSNR, and that’s turned on by default. So you would expect that they’re measuring PSNR they would turn that flag on so you would get higher PSNR. Well, they didn’t. They didn’t turn the flag on!
Dror Gill: 10:17 Finally, even then AV1 is much slower than HEVC, and they also reported in this data that it was much, much slower than HEVC but still they did not use the slowest encoding standing of HEVC, which would provide the best quality. There’s always a trade off between performance and quality. The more tools you employ the better quality you can squeeze out of the video, of course that takes you more CPU cycles but they used for HEVC, the third slowest setting which means this is the third best quality you can get with that codec and not the very best quality. When you handicap an HEVC encoder in this way, it’s not surprising that you get such poor results.
Dror Gill: 11:02 I think based on all these points everybody can understand why the results of this comparison were quite different than all of the other comparison that were published a month earlier at IBC (by Ateme, BBC, Harmonic).
Mark Donnigan: 11:14 Another critical topic that we have to cover is performance. If you measure the CPU performance on encoding time of AV1, I believe that it’s pretty universally understood that you are going to find it currently is a hundred times slower than HEVC. Is that correct?
Dror Gill: 11:32 Yeah, that’s right. Typically, you measure the performance of an encoder and FPS which is frames per second. For HEVC it’s common to measure an FPM which is frames per minute.
Mark Donnigan: 11:42 Frames per minute, (more like) frames per hour, FPH.
Dror Gill: 11:45 A year and a half ago or a year ago when there were very initial implementation, it was really FPD or FPH, Frames per hour or per day and you really needed to have a lot of patience, but now after they’ve done some work it’s only a hundred times slower than HEVC.
Mark Donnigan: 12:02 Yeah, that’s pretty good. They’re getting there. But some people say that the open source implementation of AV1 I believe it’s AOM ENC.
Mark Donnigan: 12:16 ENC exactly has not been optimized for performance at all. One thing I like about speed is either your encoder produces X number of frames per second or per minute, or it doesn’t. It’s really simple. Here is my next question for you. Proponents of AV1 are saying, “well it’s true it’s slow but it hasn’t been optimized, the open source implementation,” which is to imply that there’s a lot of room (for improvement) and that we’re just getting started, “don’t worry we’ll close the gap.” But if you look at the code, and by the way I may be a marketing guy but my formal education is computer science.
Mark Donnigan: 13:03 You can see it already includes performance optimizations. I mean eptimizations like MMX, SSE, there’s AVX instructions, there’s CPU optimization, there’s multithreading. It seems like they’re already trying to make this thing go faster. So how are they going to close this a hundred X (time) gap?
Dror Gill: 13:22 I don’t think they can. I mean a hundred X, that’s a lot and you know even the AV1 guys they even admit that they won’t be able to close the gap. I talked to a few senior people who’re involved in the Alliance for Open Media and even they told me that they expect AV1 to five to 10 times more complex than HEVC at the end of the road. In two to three years after all optimization are done, it’s still going to be more complex than HEVC.
Dror Gill: 13:55 Now, if you ask me why it’s so complex I’ll tell you my opinion. Okay, this is my personal opinion. I think it’s because they invested a lot of effort in side stepping the patents (HEVC).
Dror Gill: 14:07 They need to get that compression efficiency which is the same as HEVC but they need to use algorithms that are not patented. They have methods that use much more CPU resources than the original patent algorithms to reach the same results. You can call it kind of brute force implementation of the same thing to avoid the patent issue. That’s my personal opinion, but the end result I think is clear, it’s going to be five to 10 times slower than HEVC. It has the same compression efficiency so I think it’s quite questionable. This whole notion of using AV1 to get better results.
Mark Donnigan: 14:45 Absolutely. If you can encode let’s say on a single computer with HEVC a full ABR stack, this is what people want to do. But here we’re talking speeds that are so slow let’s just try and do (encode) one stream. Literally what you’re saying is you’ll need five to 10 computers to do the same encode with AV1. I mean, that’s just not viable. It doesn’t make sense to me.
Dror Gill: 15:14 Yeah, why would you invest so much encoding into getting the same results. If you look at another aspect of this, let’s talk about hardware encode. Companies that have large data centers, companies that are encoding vast amount of video content are not looking into moving from the traditional software encoding and CPUs and GPUs, to dedicated hardware. We’re hearing talks about FPGAs even ASICs … by the way this is a very interesting trend in itself that we’ll probably cover in one of the next episodes. But in the context of AV1, imagine a chip that is five to 10 times larger than an HEVC chip and which is the same complexity efficiency. The question I ask again is why? Why would anybody design such a chip, and why would anybody use it when HEVC is available today? It’s much easier to encode, royalty issues have been practically solved so you know?
Mark Donnigan: 16:06 Yeah, it’s a big mystery for sure. One thing I can say is the Alliance for Open Media has done a great service to HEVC by pushing the patent holders to finalize their licensing terms … and ultimately make them much more rational shall we say?
Mark Donnigan: 16:25 Let me say that as we’re an HEVC vendor and speaking on behalf of others (in the industry), we’re forever thankful to the Alliance for Open Media.
Dror Gill: 16:36 Definitely, without the push from AOM and the development of AV1 we would be stuck with HEVC royalty issue until this day.
Mark Donnigan: 16:44 That was not a pretty situation a few years back, wow!
Dror Gill: 16:48 No, no, but as we said in the last episode we have a “happy ending” now. (reference to episode 1)
Dror Gill: 16:52 Billions of devices support HEVC and royalty issues are pretty much solved, so that’s great. I think we’ve covered HEVC and AV1 pretty thoroughly in two episodes but what about the other codecs? There’s VP9, you could call that the predecessor of AV1, and then there’s VVC, which is the successor of HEVC. It’s the next codec developed by MPEG. Okay, VP9 and VVC I guess we have a topic for our next episode, right?
Narrator: 17:23 Thank you for listening to the Video Insider podcast a production of Beamr limited. To begin using Beamr codecs today go to beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.
If you spend the majority of your time pondering hot topics in the streaming media space, you’re not alone.
Our team of image scientists and video encoding engineers are equally passionate about all things video related. Which means, not surprisingly, we spend our days discussing everything from video encoding, playback, workflow architecture, codecs, compression techniques, and more.
But as we looked around the industry for sources of information on these topics we noticed a dearth of information, which is why we thought to ourselves, “let’s do something about this,” and launched The Video Insiders podcast.
Hosted by our CTO Dror Gill and me, Mark Donnigan, the show takes on a refreshingly non-partisan tone as the purpose is to equip the industry with information that is free of bias, spin, and agenda.
Join us as we dive into today’s top of mind discussions: compression, codecs, encoding, transcoding, and video workflows. We cover it all, and we do hope that you find the information valuable as you contribute to the industry in your role, whether it be engineering, operations, or business.
Mark Donnigan: 00:15 Hi everyone, my name is Mark Donnigan and I am super excited to be opening this first episode of The Video Insiders. But before I say anything more I want to introduce you to my esteemed co-host, Dror Gill. Welcome Dror.
Dror Gill: 00:31 Thank you Mark. You know if you could do some e-motion estimation on me, you would also find out that I’m pretty excited to be opening this podcast and I think it’s gonna be awesome. Mark maybe we should tell everyone what they’re getting themselves into.
Mark Donnigan: 00:45 So, we are the Video Insiders and in every episode we will bring the latest inside information about what’s happening in the video technology industry from codecs both past, present and maybe even some not invented yet, you know. I don’t know, maybe we’ll invent one. What do you think Dror?
Mark Donnigan: 01:05 And obviously we’ll cover video in coding, trans coding and you know I’m a little bit more of a business guy. You’re a serious technologist and our listeners will get to know you but at the end of the day business drives this. So, we’ll cover business and I can’t think of too much that’s really off limits, right?
Dror Gill: 01:27 No, no, we’re gonna talk about everything, everything related to video. But first Mark I have to tell the listeners a small secret about you.
Dror Gill: 01:37 Those who know you best probably have been privileged to pound the keyboard.
Mark Donnigan: 01:43 Pound the keyboard, hmm, well you know when I, you know my new MacBook, you know it is pretty noisy. Man, what’s up with that keyboard, wow.
Dror Gill: 01:55 Not that kind of keyboard, I just happened to know that you’ve studied some classical piano performance and jazz in college, right?
Mark Donnigan: 02:03 Oo, this is true. Not too many people know that about me. That’s true, that is true. But it was a long time ago. But while we’re on the subject of music, you know, discussing beta release schedules kinda has a dual meaning for you, doesn’t it Dror?
Dror Gill: 02:21 Oh, yeah, yeah. You got me there. So, yeah, Beta is the name of my rock band. I’m a land singer of a rock band called Beta.
Dror Gill: 02:31 Yeah, we do some gigs, we’ve recorded a few tracks. So, when you discuss a beta schedule with me that could be just our tour schedule and not our milestones.
Mark Donnigan: 02:42 Well, there you have it. You know, look if we do our jobs this podcast just may rock our listeners video world, right?
Dror Gill: 02:50 Yeah, yeah, for sure. And if not, at least it will be the number one destination for their latest news and analysis about video compression, protection, monetization, applications, you know we’re gonna talk about all those buzzy words. Buzz words such as HDR, UHD, HEVC, AV1.
START OF DISCUSSION
HEVC triumph?
Mark Donnigan: 03:13 That’s pretty buzzy. That’s pretty buzzy. Well, hey HEVC and AV1, I’m glad you brought those codecs up, because some may say the topic is really been run into the ground. So, what do you think Dror?
Dror Gill: 03:26 I think it’s still a great topic, you know, because it has all the elements of a good story. It’s like a true hero’s journey. You know, you might say it has Hollywood glamour, has some tragedy, has its ups and downs and finally does have a happy ending.
Mark Donnigan: 03:42 Well, happy endings are good, so to kick this off for our listeners who don’t have a deep technical background. What is HEVC?
Dror Gill: 03:51 HEVC is the latest video standard. It was developed by the MPEG committee. MPEG is short for Motion Pictures Experts Group. Notice the motion pictures, yes they don’t call it video. We’re still in like the old days of when cinema just started. It’s not a video it’s just a bunch of moving pictures.
Dror Gill: 04:19 So, MPEG developed a lot of video standards. All the MPEG standards, there was MPEG 1, MPEG 2, MPEG 4 and then MPEG 4 AVC. Which some people know by the name H.264. And finally MPEG HEVC, which some people call H.265. But don’t do that Mark. Don’t call it H.265 because Leonardo might ban you.
Mark Donnigan: 04:43 Now, wait a second, how will Leonardo DiCaprio ban me?
Dror Gill: 04:46 Oh no, not Leonardo DiCaprio, Mark. You’re taking the Hollywood reference a bit too far. I’m talking about Leonardo Chiariglione. He’s been the founder and chairman of the MPEG committee for the past 30 years.
Dror Gill: 05:03 You know, many people call him the father of MPEG because you know he founded the committee and he’s been leading it. So, I heard a rumor that if he used the term H.265 instead of HEVC than he might ban you from the MPEG mail reflectors forever. So, you know, be careful.
Mark Donnigan: 05:19 Well, I mean pretty successful stuff that the MPEG committee has developed. MPEG 2, you know it’s used in all digital cable, satellite, terrestrial broadcast, DVD. MPEG 4, Blu-ray discs. Everything on the internet is streamed with MPEG 4.
Mark Donnigan: 05:53 Now, can you imagine the Leonardo that I’m thinking of and all the engineering geeks from MPEG standing there on the same stage as the other Leonardo from the MPEG committee. That’s really amazing. That’s pretty cool actually.
Dror Gill: 06:09 Yeah, yeah it is. You know, what’s even more amazing? Every generation of the standard they essentially double the compression efficiency. So, for example with HEVC you can get the same quality as H.264 but at half the bit rate. Now, I’m not saying you can do this all the time and not for every content and it’s not true for every resolution. I have to be very careful here cause you know, some people in the industry spread information that is not 100 percent accurate.
Dror Gill: 06:39 And sometimes it is possible that we will discuss such occasions on a podcast. But I’d like to be on the safe side so let’s say between 40 and 50 percent, depending on the resolution. That’s the improvement in compression efficiency would get with HEVC.
Mark Donnigan: 06:57 Sure, which is really great and pretty amazing. So, why isn’t everyone using HEVC instead of H.264? I mean, with video occupying, I mean the studies keep on, this number goes up seems like by the month by 70 percent the internet is video traffic. Netflix is like 15 percent of bandwidth consumed at peak viewing time. That’s staggering if you think about it. I would think everyone would just want to switch to HEVC and either enjoy the savings or be able to double their capacity. What’s going on there?
Dror Gill: 07:31 Yeah, yeah, that would have been really what you would expect with HEVC being much more efficient than H.264 but this is really where the tragedy part of our story comes in. There’s a lot of sophisticated algorithms which actually enable this great compression efficiency and many companies would jointly created the HEVC standard. They obviously have patents on all of these technologies that they developed. So, with H.264, the situation was simple. There were companies that developed algorithms, they had patents but there was a single patent pool that was organized by a company called MPEG LA. And you could just license all the patents very easily in this one stop shop. But with HEVC the problem was that the patent holders joined three different pool and some of them didn’t join any pools. So, you act of this unreasonable licensing terms which were published by one of the patent pools initially and their half of the industry was in complete chaos regarding HEVC licensing.
Mark Donnigan: 08:40 Wow, sounds pretty bad. Leonard Chiariglione wrote a pretty famous blog post called, ‘The Crisis, the Causes and it’s Solutions’, if I remember correctly.
Dror Gill: 08:55 Yes. It was very sad and he said that from his point of view the MPEG way has reached an end. They developed this great technology and then nobody uses them because of licensing issues. So, that was the peak of the tragedy but luckily for us and I think for the whole industry, finally, the patent holders came to their senses. And now everything is much more clear. I mean the two major patent’s pools which are MPEG LA and HEVC Advance, both of them are not charging any royalties for content.
Dror Gill: 09:34 And content providers around the industry and the third one Velos Media. They haven’t officially announced their licensing terms but everybody’s expecting them also to be reasonable and I’ve heard some things off the record that they’re also not going to charge for content. I mean, they don’t want to cut the branch they’re sitting on. If they’re unreasonable then nobody will license it and they get no money. So, I think basically where we are today you can say that HEVC patent world, these are mostly paid by the vendor of devices such as mobile phones and TVs. All those companies will put chips that support HEVC decoding and the devices and I understand the, Mark, that many of them are already supporting HEVC, right? A lot of devices out there.
Mark Donnigan: 10:18 You know earlier this year Beamr actually sponsored an industry survey which Tim Siglin a contributor to Streaming Media helped us with and also Streaming Media promoted it and helped put it together. The findings were really positive. This was completely a non partisan, fair and balanced survey. 66 percent of those that responded of which was a very high percentage of engineers by the way, more than 400. So, these are industry professionals – reported that HEVC was already in production. So, the reality is that though in some circles there still maybe a little bit of FUD. You know, that’s fear, uncertainty and doubt. The fact is is that more than half the industry has HEVC encoding pipelines running or they’re in the implementation stage. These things don’t happen overnight, so in some cases maybe they’re not encoding HEVC content yet but they will be. It’s a foregone conclusion.
Mark Donnigan: 11:33 77 percent so more than three out of four said HEVC was a very viable replacement for H.264. That’s obviously good news but if we’re making files that can’t be played then I suppose you could file this under the fake news category.
Dror Gill: 11:49 Yeah, if you encode the files somebody has to play them.
Mark Donnigan: 11:53 Exactly. And we’re gonna get to that when we talk about AV1, but when we look at the situation for device support it’s a little bit hard to get the exact numbers but industry seems to be kinda gathering around two billion devices is where most people, you know nod their head and say yes, we believe that number. That’s two billion today that support HEVC in hardware. So, if we think about, first of all it’s a massive number right? But you think about iPhones. Everything from the 6S forward supports HEVC and some markets iPhone penetration is way over 50 percent. And then when you factor in Samsung like the S8, S9, Note 8, Note 9, Galaxy Tab S3. I mean phones from LG, phones from Sony. You look at Macs, you look at PCs especially if they have the Intel Skylake or the Kaby Lake chips. Every 4K TV sold today, I mean I could go on and on.
Mark Donnigan: 12:54 Roku boxes, the new Apple TV, that is the Apple TV 4K. Chromecast even, Amazon Fire TV. So, all of those support HEVC. We heard from a very well placed industry insider that one MSO recently discovered of their users which number tens of millions, (that) more than 50 percent of their customers had at least one device in the home that supported HEVC. Which is very, encouraging.
Dror Gill: 13:32 Yeah, it’s very encouraging. It means that HEVC support is out there. I mean two billion devices, that’s a really large number. So, if you look at it, the situation with HEVC I would say it’s pretty clear. I mean, it’s half the bit rate of H.264 AVC. Royalty issues have mostly been solved, a few billion devices supported in hardware. So, there it is Mark. I think that’s the happy ending I was looking for. So, okay, that’s HEVC, where are we with AV1?
AV1 tragedy?
Mark Donnigan: 14:04 That’s a good question. You know, it’s unclear at the moment. Look, we are definitely fair and balanced here on the Video Insiders so we don’t present things from an agenda. First of all AV1’s open source, right? It is a supposedly a royalty free codec. It’s been developed by a extremely strong group of companies. Companies that if anybody can pull this off it’s Google, it’s Netflix, it’s Facebook, it’s Intel, it’s Microsoft, it’s Apple and there are others. These people are our friends and they’re good, they can build a codec. They could do this. They set out to develop the AV1 codec really in response to the royalty issue. You know, you might get different responses, I haven’t really gone around and asked privately any of these individual companies. But I’m guessing that off the record most would say yeah, if HEVC Advance had been rational, the Alliance for Open Media wouldn’t have been needed and AV1 wouldn’t exist and things would be more clear.
Dror Gill: 15:11 Right, and I understand they want to develop a royalty free codec but I think I heard you say “supposedly royalty free.” Cause I thought AV1 was royalty free, I mean look what it says here. I’m on the AOM website, I’m reading from the website, it says, “Collaborative effort to offer open royalty free and interoperable solutions for the next generation of media delivery.” So, it says royalty free, doesn’t it? I mean, can it be more clearer than that?
Mark Donnigan: 15:36 You can challenge me on that but it’s not that simple. Here’s the deal, the members of the Alliance for Open Media all agreed to contribute their own IP to the AV1 standard and not charge royalties for it, but it’s possible. Again, I’m saying “possible”, I’m not a lawyer, we’re not giving any legal advice so I wanna be careful, but it is possible that other parties who are not members of the Alliance for Open Media could have IP claims or at least could attempt to assert IP claims and to charge royalties on the codec. Now, alliance for open media, AOM, has set up a fund to indemnify licensees, or so I’ve heard. At the end of the day there’s just a lot of questions that don’t have answers. It’s one thing when it’s, you know, engineers sitting around at an industry conference or on a panel having a debate and discussion. It’s a whole different thing when a fortune 50 or a fortune 25 media company who could be facing hundreds of millions if not billions of dollars of fines or levees or whatever. You just can’t sort of sweep this stuff under the rug.
Mark Donnigan: 16:56 I think unfortunately that’s kinda what is happening a little bit in my opinion.
Dror Gill: 17:03 At the end of the day I think we need to encourage our listeners to do their own due diligence, you know, because in some industry conferences you can be led to believe the situation is very secure but really nobody is indemnifying you here. So, you really need to be careful. Anyway, on the device side Mark, how many devices support AV1 in hardware?
Mark Donnigan: 17:24 Yeah, so I’m glad you asked, and you know this is what my background is in, consumer electronics. I have a lot of experience there. We get into this situation where it’s easy to sit around and talk about creating files with these new exciting codecs but people kind of forget that VLC is not a viable solution to play back. I’ve been doing some research though and I actually was able to connect via email and in some cases have voice conversations with the SOC vendors, that is the system-on-the-chip vendors who are the ones at the end of the day that have to be convinced to integrate AV1 decode into their silicon.
Mark Donnigan: 18:25 But you know it’s these SOC vendors that at the end of the day is where I’m gonna get the real information. I heard what I anticipated to hear that obviously support today is zero. I heard varying degrees of responses such as we’re thinking about it. It’s getting onto the road map to a slightly more optimistic where some people said in 24 months we’ll have it, maybe sooner. But you should anticipate two years. And then on the other side I got the usual hey look, you know, when there’s a business case that is when I can’t sell a chip because it doesn’t have AV1, that’s when we’ll support it. You know the situation is really this, I am very confident again because the companies behind AV1 that somewhere in 2020 – It’s just not even possible to complete the engineering cycles, the fab cycles for the silicon any sooner than really the end of 2020. So, we’re really talking 24 months, we’re recording this podcast on the last day of October, October 31st, 2018. I anticipate that Christmas season 2020 there will be an (AV1) device.
Mark Donnigan: 19:44 Will it be a Roku box? Will it be a TV? Will it be a game console? Or even just some new product all together, I don’t know.
Mark Donnigan: 19:54 Maybe an Android phone, exactly. There will be an AV1 device and there’s no doubt that certain folks that are incented for AV1 to succeed will write articles saying that the world is about to flip. At the end of the day one device, a device does not build an ecosystem. It takes two billion devices that are in real consumer’s hands.
Mark Donnigan: 20:20 You know, that somebody doesn’t even have to download an app or install something, they just turn it on and it just works. And that we are years and years and years away.
Dror Gill: 20:33 Yeah, I think I agree with you. It will take time until the chips will come out and then tested and devices will come out. So, I’m very bullish about HEVC, I think the message for the industry now is very clear. HEVC is happening today and it’s here to stay.
Mark Donnigan: 21:06 Well let’s bring this to a close. It’s been an awesome discussion about how to compare HEVC and AV1.
Dror Gill: 21:13 Well, comparing codecs that’s a pretty big question and you never get the same answer. It really depends on who you ask so it’s a really interesting topic, but it seems like our time is up so how about we discuss that in our next episode?
Mark Donnigan: 21:29 I want to thank everyone for listening to The Video Insiders’ podcast. This is a production of Beamr Imaging Limited. Please subscribe on iTunes and we will be expanding to other platforms shortly. If you’d like to try out Beamr codecs in your lab or production environment we’re giving away up to 100 hours of HEVC and H.264 encoding every month. Go to https://beamr.com/free to get started immediately.
If you would like to join the discussion as a guest, send an email to thevideoinsiders@beamr.com with a suggested topic that you would like to share.
The news of Beamr building a transcoder is spreading fast. Many are applauding the move and have expressed interest in adopting – but some may be wondering why a pioneering codec technology company feels the need to build a transcoder? Well, here’s the story.
Beamr was founded on a simple but technically complex idea that there are opportunities to deliver massive value to the video streaming and distribution ecosystem by guaranteeing the smallest video file and stream size possible without compromising video quality.
With the introduction of Beamr Video (now Beamr Optimizer), we proved that we could deliver an additional bitrate savings of up to 50% at the same quality. An incredible feat acknowledged by well-known streaming services and tier-one studios in Hollywood who adopted the technology for use in workflows including Blu-ray disc authoring which represents the highest pinnacle of direct-to-consumer quality.
By incorporating our perceptually driven optimization process (content-adaptive) into the codec, we enable MSO’s and streaming video delivery services to meet their objective of not delivering a single bit more than needed to represent the absolute quality they desire to transmit. This led us to acquire Vanguard Video in April 2016, a company providing codec engineering and software development services for many of the best-known video encoder OEM’s and streaming services. Now with our best-in-class codec, combined with our highly patented and well established perceptual optimization technology, we have the most advanced HEVC video codec in the market: Beamr 5x.
As a video codec does not function standalone outside of a lab environment, all Beamr Codec SDKs work inside the framework of FFmpeg through our plug-in interface. Through the development of our FFmpeg plug-in and as we surveyed the needs of our Telecom and MSO customers, it became clear that FFmpeg could not meet their requirements for industry-standard integration points such as SCTE-35 ad insertion.
What modern video services need in a transcoder.
To deliver uncompromising performance and system integration that meets our customer’s expectations, we built a transcoder that can enable new video workflow architectures with a value-added and performance-oriented engineering philosophy. The Beamr Transcoder is designed to work seamlessly in virtualized video encoding workflows and provides a full transcoding solution which includes Beamr codec SDKs: Beamr 4, Beamr 5, and Beamr 5x.
The first thing we recognized after talking with customers about what they were looking for in a transcoder is that the days of the monolithic black box “workflow solution” that includes content management, transcoding, packaging, DRM, etc. were over.
Customers were unanimous in reporting that they are tired of being held hostage by vendors.
Everyone said that they need a flexible media processing framework to fulfill the requirements of their video encoding service today and in the future.
We took this feedback as inspiration and adopted a critical design philosophy. The transcoder must satisfy the functional framework requirements to work in today’s containerized virtual computing and cloud environments while connecting to complementary solutions from any vendor.
Why FFmpeg isn’t enough.
A major performance advantage with Beamr codec SDKs is our multi-bitrate encoding capability. With multi-bitrate encoding, the SDK can share analysis information from the highest quality tier to the lowest bitrate tiers of the same picture size (resolution). This feature offers significant CPU and performance efficiencies, which allows the quality level to be dialed up for a given job, in a specified turnaround (processing) time. For anyone generating ABR profile stacks, Beamr’s multi-bitrate functionality is a game changer.
Today’s transcoder is a video processing framework that supports video codecs and the associated processing required for audio, metadata, etc. The performance advantages of Beamr’s H.264 and HEVC codec SDKs which ship inside the Beamr Transcoder start with much better parallelism (tiled encoding). Additionally, Beamr codec SDKs enable highly efficient CPU utilization on a many-core server and include better overall thread pool management.
Over the years as OTT workflows developed, many video distributors found FFmpeg as a media processing framework a good solution, but in time and as technical requirements and compatibility needs increased it has become less viable. FFmpeg is flexible, with many libraries and filters supporting container files and audio/video codecs. It also works with any video transformation during the transcoding process (decode – transform – encode): scaling function, frame rate conversion, color space conversion, bit depth conversion, filtering (noise reduction), etc.
Because the libraries and filters were developed separately, to make everything work in every combination needed, each filter must be run sequentially, passing the resulting uncompressed video frame to the next filter after it is ready. When the first processing step is complete, and the pixels or data needed for the next step are in the memory cache of the processor, it is more efficient to perform the following processing step immediately, as a tightly synchronized operation, all without saving the intermediate results to system memory, or disk. FFmpeg was developed asynchronously by hundreds and thousands of contributors, meaning this operational structure was not possible to build outside of a closely coordinated product roadmap, which by definition is lacking in open source projects.
Beamr Transcoder was built using a single thread pool that runs all processing tasks easily while outperforming FFmpeg as a media processing framework that runs multiple libraries with their own separate thread pool’s.
How we approached development of the Beamr Transcoder.
The Beamr Transcoder is lightweight, flexible, and not loaded down with legacy bloat. It may not meet a long laundry list of supported technologies, but, as service providers and OTT streaming services are shedding their legacy requirements in favor of lightweight, agile solutions that run 100% in software spanning virtualized environments, this is an advantage to the product. Designed for today’s video workflows, the Beamr Transcoder can scale across on-prem, public, private, or hybrid cloud infrastructures.
The heart of this highly efficient and flexible transcoder is an engine that was developed in native C++ to run on Linux while being highly scalable across public clouds such as AWS using Docker containers.
Beamr Transcoder VOD 1.0 features overview.
Beamr Transcoder is finely tuned and automatically allocates the optimal number of threads to the video encoding function, reserving the minimum needed for decoding, pipeline management, and network function control. Following is a summary of the critical capabilities of the first version of Beamr Transcoder VOD. (Beamr Transcoder Live will ship later in 2018)
– Supports high resolution, high frame rate (HFR) up to 8Kp120
– SCTE-35 pass-through
– Encoding Boundary Points (EBP) support
– EIA 608/708 closed captioning
– Logo insertion
– Support for high dynamic range (HDR): HDR-10/HLG
Where we are going.
With Beamr Transcoder, Beamr’s HEVC & H.264 codec SDKs are now full video encoding solutions that can power your service to deliver the highest quality at the lowest bitrate possible. Like all Beamr solutions, the Beamr Transcoder has a fully committed road-map with the industries most reliable video technology engineering team executing it.
By leveraging Beamr’s video transcoder and the included encoding solutions, video encoding operations will operate as much as three times faster creating files that are up to 50% smaller.
To experience the performance of Beamr Transcoder which comes with an innovative business model that includes a free and premium version along with our highest rated HEVC and H.264 codec SDKs, send us an email to transcoder@beamr.com.
Visit beamr.com/transcoder to download the Beamr Transcoder VOD product information sheet or CLICK HERE for the product page.
In case you missed it, Apple just snuck a little surprise into the first few days of 2018 in the form of their name appearing on the Alliance for Open Media (AOM) website as a Founding Member. As an ardent HEVC supporter, some may be shocked at this move by Apple. Blog post detailing Apple’s HEVC announcement. Now let’s see what it means for HEVC.
As a codec engineering company, Beamr has invested heavily in our HEVC implementation and we are proud of the best in class customers who are using it around the world to distribute video that is higher quality and up to 50% smaller than the H.264 version.
This means we do have a “vested” interest in HEVC being successful. At the same time, whether VP9 or AV1, we are always tracking the development of new codec technology so that we are in the strongest position to bring our extensive codec development resources to bear on market leading solutions.
Though we have an interest in HEVC becoming successful, we have invested resources and continue to do so, in order to understand AV1 in the areas of market readiness and licensing preparedness including IP questions, playback support and more.
In this article I will share the Beamr perspective that Apple joining the AOM reinforces the possibility that AV1 will be the successor to HEVC. However, with 1 billion HEVC enabled end points in the market, HEVC has legs for many years before a sufficiently large AV1 ecosystem will be built.
This position is also in alignment with many of our customers who are serving hundreds of millions of end users and must make codec decisions based on streams they can reliably deliver today.
After all, speculating on what may be coming in the future is not a luxury most of the industry enjoys because if they bet wrong, it could impact tens of millions of users negatively. There is a huge difference between advanced technology development (e.g. what happens in the lab) and the realities of production (that which generates revenue).
Once the AOM locks down the AV1 spec, you can expect many shootouts and comparisons with HEVC to be published. But let’s take a look at how HEVC compares to AV1, given what we know now.
AV1 Readiness compared to HEVC.
HEVC was ratified in 2013 while the AV1 bitstream was set to be frozen in Q1 2017, yet even now the AV1 bitstream has not been completed. Developing software timelines, committing to them, and then meeting them, is far from an exact science. Thus the delay is not completely the fault of the AOM development community since it is endemic to the software development lifecycle. Innovation is difficult to schedule. The point is, AV1 will be ready when it is ready. Which means commercial plans that hinge on the delivery of AV1 before 2020 or 2021 could be at risk given the uncertainty of when the standard will be ratified.
AV1 Compression Efficiency compared to HEVC.
HEVC is recognized to be 40-50% more efficient than AVC (H.264), and AV1 is hoped to be up to 30% more efficient than HEVC (H.265).
However, while HEVC’s compression efficiency has already been reached by advanced encoder implementations such as Beamr 5, AV1’s 30% efficiency claim over HEVC has not been proven outside of an extremely limited (small) set of files.
In any case any improvement can be validated only after the spec is final and the tools included in AV1 are decided upon. At that point the race to realize these gains will start, balancing the computing resources needed and maturing the rate control algorithms. But just as HEVC did not reach its planned 50% efficiency in the first release, taking multiple years to achieve, the AOM developers will need to work very hard for the next 2 to 3 years before significant gains over HEVC will be seen.
AV1 Royalty and IP constraints compared to HEVC.
There are three HEVC Patent pools which license the technology used in HEVC implementations: MPEG-LA, HEVC Advance, and Velos Media. Both MPEG-LA and Velos Media do not charge license fees for content distribution (See the MPEG-LA HEVC License Summary and the Velos Media FAQ), and HEVC Advance does not charge a license fee for free content distribution, such as public broadcasts and ad-funded commercial broadcasts (see page 3 of the HEVC Royalty Rates document).
Even Technicolor, that licenses its HEVC patents outside of the 3 patent pools, has publicly declared that they will not charge license fees from content providers. In addition, royalty schedules are being (have been) amended down, and it seems the Patent pools are aware that a more friendly approach is needed.
AV1 cannot guarantee a royalty-free offer.
Yes, that’s correct, I said it! Now here’s why.
While AV1 claims to be royalty-free, many industry players have missed the fact that the Alliance for Open Media does not provide indemnification to companies who use AV1 against patent claim violations.
Since some of the algorithms used in AV1 bear a resemblance to corresponding H.264 and HEVC algorithms, there is some probability that the IP in AV1 could infringe on AVC and/or HEVC Patents. In fact, delays to the ratification of the AV1 standard might well be due to legal teams who are examining the final algorithms exactly for these cases.
To be fair, IP questions are hardly ever cut and dried, and there are many unknowns and “what-if’s” to be discussed. But the lack of clarity regarding the AV1 IP situation, and the fact that AOM is not offering indemnification for IP infringement, makes the “royalty free” claim at this point more of a wish than a solid fact.
If you are still not convinced that AV1 offering a royalty-free codec could be problematic, consider that for VP8 and VP9 Google needed to license the H.264 patents from MPEG-LA. If an infringement action is identified with AV1, and if the courts rule in the plaintiff’s favor, the legal exposure will be of the magnitude that headlines will be penned and stock prices hammered. Velos Media, one of the 3 HEVC patent pools, has already warned in its FAQ:
“As it relates to royalties, we know that VP9 incorporates patented technologies, including some of the patents being licensed by Velos Media for HEVC. And, while AV1 has not yet been publicly released, it may also incorporate patented technology from many parties.”
Try slipping that little disclaimer by a corporate IP attorney! When a licensing body directly references a new technology as being possibly infringing, it is worth paying attention to.
Let’s look at why the AOM members feel it’s so important to have a royalty-free platform.
Royalties are a pain, that is, when you are the party that needs to pay. Not only do you need to factor this added cost into your business model but in some cases, the tracking and reporting burden by itself represents a real difficulty and may limit certain business models from being feasible.
You can never know what patent holder’s next demand will be. The HEVC fragmented IP pools and shifting fee structures prove the point. The AOM is absolutely correct in wanting to address this with AV1.
Confusion over who to pay. The lack of clarity overpayments is delaying adoption even further. It seems AOM members are seeking control over a critical component of their technology that is not royalty free.
While all these arguments make sense, AOM cannot guarantee its users that this is a royalty-free codec; all they can suggest is that AOM members will not ask for royalties. But the thousands of patent holders for block-based codecs that are not a part of AOM are still out there, and the generosity of AOM could come at their expense.
For an interesting analysis on the validity and value of the MPEG-LA and HEVC Advance HEVC patent pools, you will want to read this Unified Patents article as it provides a perspective on how the courts look at the patents that are contained in a pool. In short, it’s probably not a wise legal move to assume that the AOM has everyone adopting AV1 “covered.”
AV1 Encoding CPU Performance compared to HEVC.
HEVC encoding CPU performance is advancing at a rapid rate. At IBC 2017 Beamr demonstrated six simultaneous 10-bit 4Kp60 live channels being encoded on a single Intel Xeon Scalable Platinum 8180 dual-socket server.
This encoding speed is in contrast to AV1, which has not been optimized, but is currently running about 100 times slower than real-time on a single server. Aside from the fact that AV1 is not available for low latency live encoding workflows, the operational cost delta of running an AV1 encoding service versus HEVC is staggeringly higher for AV1.
Download the Intel solution guides detailing Beamr HEVC codec SDK performance and applications on Intel processors.
Beamr’s HEVC encoder has been under active development for more than five years, and our CPU performance has been consistently improved by way of algorithmic and code optimizations. AV1 developers will bring improvements to the encoding speed, but it will most certainly follow the same development trend of every codec before it, including HEVC.
In other words, it can only happen over a period of years, and only if a group of dedicated engineers focuses on it day and night. Optimizing a codec is not a hobby. But even in the end after it is fully optimized AV1 will be slower because of the added mathematical complexity needed for it to achieve higher efficiency.
Video distributors with capex, opex, or physical space constraints will find the bitrate efficiency gains of AV1 will come with a very high operational cost. This Jan Ozer article from Streaming Media provides further context on AV1 performance.
AV1 Playback and Decoding Performance compared to HEVC.
HEVC hardware decoder support exists today in more than 1 billion devices spanning the most popular computing and mobile operating systems in the market like iOS, macOS, Android, and Windows. And in addition, low power hardware implementations for HEVC exist on Intel and ARM-based chips, as well as hundreds of millions of SoC’s shipping in CE devices such as TV’s, media players and game consoles. HEVC is a de-facto standard in all UHD TV’s found in the market.
According to the Consumer Technology Association (CTA) between 2014 and thru 2017, somewhere in the neighborhood of 175 million UHD TV’s will have been sold. Source: CTA 2017 presentation, data from GfK
This is an impressive HEVC footprint, and it’s only going to grow stronger. In contrast, the only AV1 playback environment available today is an early alpha implementation of the Mozilla browser Nightly build.
As the AV1 spec is not yet locked by the AOM it’s no wonder that when we reached out to the best-known silicon vendors who are supplying the media processing and video decoding chips to the most popular media player vendors, game console makers and TV OEM’s, everyone reported without exception that they cannot begin planning AV1 support in the absence of a ratified spec.
AV1 support in silicon is a minimum of 24 months out (Q1 2020). But silicon is just that, silicon. Chips have to be designed and integrated into consumer products before the advanced capabilities that chip vendors are including can be available. But, this is a chicken and egg situation. Afterall, why would a CE company go through the extra engineering cycles and increase their BOM to support a codec that is yet to be adopted across the ecosystem?
Should Apple influence your codec adoption decision?
With Apple’s commitment to HEVC, it’s a mystery of what their intentions are in joining the AOM. But we do know this – Apple has a solid HEVC roadmap with a vertically integrated HEVC video encoding, distribution and playback technology stack across all devices.
This means that today, you can reach the Apple ecosystem with H.264 and HEVC.
For video distributors looking for smaller bandwidth footprints, opting out of HEVC means they will lose a whopping 54% of the North American mobile market assuming the selected codec isn’t supported by Apple. This is hardly a prospect that any executive or encoding head will agree to, which means the question of whether to support AV1 or HEVC could come down to compatibility.
Any video service unable to match the performance of HEVC on Apple, will have a difficult time competing with services delivering high quality 1080p HD video at bitrates well under 2 Mbps. HEVC is available today across the all too important Apple ecosystem, while AV1 is not supported (today).
HEVC is the codec for today. Is AV1 the codec for the future?
Remember how I said that Beamr is actively tracking all new video technology and codec developments and that this applies to AV1? Well, we do believe that AV1 could be a factor at some point in the future.
There are those who always wait for next years model. The trouble with this approach is that by delaying, you miss out on technology leaps that could have afforded a significant advantage to your company. This is especially true with the decision to move ahead or hold on adopting HEVC in anticipation that AV1 will be cheaper and provide an added efficiency benefit.
The reality is that HEVC is able to reach 40% to 50% efficiency gains over H.264 today. These aren’t theoretical numbers or only possible on a limited set of content. Beamr has customers distributing content around the world, and enjoying bitrate savings in this range. Even Apple in their WWDC2017 announcements of HEVC, used the numbers 40% and 50% savings when talking about their decision to adopt the HEVC standard.
Ask yourself, what is the opportunity cost incurred by continuing with H.264 for the next 24 to 36 months as you wait for a relatively small AV1 playback footprint to emerge?
This is why most of the industry operating a commercial service is opting to realize the benefits of HEVC today while keeping tabs on the development of AV1 (for the future).
Still not sure…
Consider that Amazon Prime and Netflix are both members of AOM. And both are active in AV1 development and testing, yet Amazon and Netflix are users of HEVC.
Why would they do this? It’s simple. HEVC serves them well by being compatible with more than 1 billion devices and enabling premium video experiences at bitrates that are 40 to 50% less than H.264. Now, one may think that HEVC is only being used for 4K content by these services, but we know that Amazon is using HEVC in emerging markets for lower resolutions.
But what about Google, they are Founding members of the AOM and ardent supporters of alternative codecs? It’s interesting to note that Google supports HEVC in Chromecast which is clearly required for content services (some who are in the AOM like Netflix and Amazon) to stream 4K HDR video. But will Google use HEVC for their own services on a wider basis in the future? We do not know.
One situation in the market that must be faced by Google and YouTube is what will happen if Apple deprecates H.264, and mandates all apps that stream video to Apple devices leverage HEVC. Remember Flash? A similar situation occurred with HLS, the mandatory streaming protocol for Apple devices.
Remember how I pointed out that Apple is vertically integrating with HEVC for video and HEIC (HEVC I-frame) for mobile image capture and display? I think now you can see how a codec selection decision by a vendor like Apple can move the entire industry.
Apple has chosen HEVC for production and any video distributor can encode in HEVC and transmit to a user with iOS 11 or macOS High Sierra and know that it will play perfectly.
HEVC is a robust standard that has broad support with extensive development from the largest encoding vendors in the industry. And HEVC is widely adopted on the device side with major services like Amazon, Apple, and Netflix using it now. HEVC is not going away.
Beamr’s view is that HEVC is the codec for today, and AV1 may possibly be the codec of tomorrow. But for sure, with HEVC, we can all enjoy more video and better quality as HEVC enables new applications, experiences, and innovations to be transmitted to users today.
It’s no secret that Intel-based hardware is ubiquitous in many video encoding data centers, which explains the high level of interest in the new Intel® Xeon® Scalable Platinum processor family. This article examines the live HEVC 4Kp60 10-bit encoding performance from the perspective of speed with Beamr 5 running on the scalable Platinum 8180 where the results are nothing short of amazing.
Intel is known for pushing the state of art with ever faster and more capable processors, which enable software encoding vendors like Beamr to achieve performance benchmarks some thought impossible. Since Intel’s announcement of their new processor series, Beamr has been excited to see what is possible with the version 5 Xeon® processor.
The result?
Mind-blowing.
Video services needing to encode live 4Kp60 10-bit HDR video can achieve 6 simultaneous streams on a dual-socket Intel® Xeon® Scalable Platinum 8180 processor using Beamr 5 v4.1. This performance represents a 6x speed advantage over x265 and establishes an entirely new benchmark for HEVC software encoders.
CLICK HERE for a speed comparison of the Beamr 5 HEVC software encoder and x265 running on Amazon EC2.
Executive Introduction
As over-the-top (OTT) and IP overtake traditional linear delivery systems, the ability to encode video on software, in real time, is now a requirement.
According to Cisco, consumer Video-on-Demand (VoD) traffic will almostdouble by 2021. This trend is driven largely by new video services and entertainment formats such as Cloud DVR, UHD (4K), High Dynamic Range (HDR) and 360 degree AR/VR immersive video. These advanced video formats carry with them much greater encoding complexity which places high operational demands on the computing environment. This means more efficient software requires less server time which translates to fewer machines and lower capex and opex.
With content and operational costs rising, and end user pricing under pressure, it is essential for operators to invest in video encoding technology that can provide advanced services in the most efficient way possible. As NFV and virtualized container based architectures advance in the data center, encoding density and speed is becoming a critical vector within the encoder selection process. Also, the ability to operate across a wide range of general purpose platforms is essential. Many GPU bound solutions are inextricably linked to a single processor or limited series. Beamr’s HEVC encoder scales across the entire Intel® Xeon® family.
Apple is playing a pivotal role in enabling the HEVC ecosystem starting with iOS 11 and High Sierra. It is estimated that up to 1 billion devices in the Apple ecosystem can now play HEVC without any special update or third-party apps needed. With HEVC files now supported from big screens to mobile devices, video services can transition their entire library with all resolutions to HEVC, and benefit from the reduced bitrate and improved quality that HEVC is able to deliver. This represents hundreds of thousands if not millions of hours of content needing to be encoded by each video distributor.
With Beamr 5 running on an Intel® Xeon® Scalable processor, video encoding engineers can perform up to 6 times more live video encoding, which leads to a reduction in:
The new Beamr 5 HEVC software encoder exploits multiple features of the Intel® Xeon® Scalable platform, making it possible to deliver the performance of hardware with the scale and flexibility that only software allows. Beamr 5 is able to utilize the entire range of Intel® Xeon® processors from the entry-level Intel® Xeon® Processor E3 v5 family to the best in class Intel® Xeon® Platinum Scalable 8180 processor. For real-time live video encoding operations that require resolutions up to 4K and frame rates as high as 60 FPS, higher performance means less computing resources required.
Solutions like Beamr 5 running on Intel® Xeon® Scalable Platinum processors contribute to decidedly lower operational and capital investment costs.
In addition to the live workflow benefits, offline VoD applications can also benefit from greater performance as the shorter turnaround time to complete VoD encodes and transcodes means the content will be available for distribution more quickly.
Beamr 5 opens up compelling options for MSOs, CDNs, and broadcasters to build video workflows that exceed their service expansion plans and technical goals while also enabling service operators to deploy media processing workflows across public clouds and on-premise data centers.
With Beamr 5 running on Intel® Xeon® processors, new video encoding workflows can be imagined as edge encoding services running Intel® Xeon® E3 processor-based points of presence (PoPs) for JIT transcoding are now possible. The high performance of Beamr 5 directly enables workflows to be split and re-deployed without a need to redesign workflows.
Beamr’s Next-Generation Video Encoding Technology on Intel
At the foundation of Beamr’s HEVC software encoder is technology that stems from more than a dozen years of codec development by our codec engineering team. Though we’ve developed our solution and technical advantages entirely in-house, working closely with Intel gives us a significant technical and business advantage.
Of the many points related to how we achieved our massive speed advantage, the two we will highlight are motion estimation advantage and micro-level parallelization.
Incoming frames are extensively analyzed by Beamr 5. It is this step which determines the complexity of the scene as rough motion vectors are calculated, and estimates for the bit demand of the encoded frame are made. These estimates guide the second stage of the encoder and allow those activities to focus on visually meaningful aspects of the frame. By partitioning the encoding process, unproductive calculations can be avoided, thus improving the speed of the encoder and the quality it produces.
Second, Beamr 5 features micro-level parallelization which is the result of the codec engineering team leveraging software engineering lessons learned from earlier generations of Beamr real-time software encoders. This experience led the team to design a micro-level parallelization approach that stages portions of the encoding tasks in a controlled manner, staggering their execution so each micro-task begins when the data is available (and still in the cache). This results in wasteful power and CPU cycles spent writing and fetching data being eliminated. Careful design of these micro-tasks assures that they are executed efficiently across the whole frame and in an even manner so that all cores are kept uniformly busy, and none are left waiting for their next task.
Uniquely, the Beamr encoder does not rely on the operating system to manage these critical execution threads but instead is under full control of the pooling and process allocation between the available cores and threads. Beamr’s advanced encoder controls the execution priority based on the availability of pipelined data.
Test Methodology
As in part 1 of ourx265 vs. Beamr 5 performance test, we encourage you to experience our speed claims first hand. For qualified video distributors, services and platforms, we are happy to offer a FREE evaluation agreement. To take advantage of this, please contact sales@beamr.com.
The purpose of this test was to measure the speed of Beamr 5 for live encoding applications running on Intel® Xeon® Scalable Platinum processors. In our first comparison of codec performance based on CPU, we decided to run a few comparisons with x265. For this comparison, x265 was benchmarked in its fastest speed setting ‘ultrafast’ – while for Beamr 5 we operated the encoder at its highest speed setting “15” with the performance boost modifier ‘TURBO1’ which activated our latest algorithmic improvements that are available in version 4.1. (All files were 4Kp60.)
For this second test, we wanted to dig deeper using Beamr 5 version 4.1 running on the same 2S Intel® Xeon® Platinum 8180 processor-based machine that we tested with in September, to see what gains were possible. What we found was nothing short of stunning.
In Graphic 1, HTOP shows Beamr 5 loaded 108 threads (from 112 available) at an impressive 90% utilization rate. This demonstrates the high degree of effectiveness with our Intel specific optimization.
Graphic 1: HTOP Intel 8180 dual socket 108 thread utilization at 90% across each thread.
More speed and performance benchmarks from Intel: CLICK HERE.
Conclusion
The drive to increase density with software-based video encoding and transcoding infrastructure is key to securing a competitive advantage for multi-service operators, OTT video distributors, and content distribution networks. At the same time, video architects must enable encoding and delivery of advanced entertainment content, by embracing new technologies, capabilities, and codecs such as HEVC, HDR, and 4K.
With a Beamr + Intel® Xeon® optimized video encoding solution, density – efficiency – quality – and flexibility of video encoding operations for on-premises, cloud, and hybrid data centers can be realized. Beamr 5 running on Intel® Xeon® Scalable processors offers TCO benefits and provides a meaningful improvement to the video processing capabilities of any video distribution solution.
If you missed part 1 of this post, be sure to check it out since additional technical details about Beamr 5 and its operational and performance advantages against x265 were specifically discussed. Find the x265 vs. Beamr 5 Speed Test here.
![2018, the Year HEVC Took Flight. [podcast]](https://blog.beamr.com/wp-content/uploads/2019/01/Beamr-Podcast-banner-D-800x544.jpg)
![The Future of 3 Character Codecs. [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-v3c-800x544.jpg)
![Codec Efficiency is in the Eye of the Measurer. [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-B-1-1-800x544.jpg)
![Will the Story End a Tragedy or Triumph? [podcast]](https://blog.beamr.com/wp-content/uploads/2018/12/Beamr-Podcast-banner-v3a-1-800x544.jpg)
