2021年9月27日
By 每Hultin iSIZE的业务发展总监
聚光灯

基于差分的深度感知预处理

每Hultin discusses meeting the challenge of a rising volume of content and the growing carbon footprint of online video 流媒体.

Consumers today have higher expectations for the quality of the content they consume than ever before. 他们也有了比以往更多的选择. 结果是, delivering more high-quality content as cost-effectively as possible is very much top of mind for content providers.

在应对网络媒体消费激增的问题上，媒体部门面临着一系列挑战, 这给全球网络基础设施带来了前所未有的压力. 互联网基础设施的巨大负载不仅造成了内容传输的瓶颈, but also affects how content can be distributed efficiently to larger numbers of viewers and contributes to its environmental footprint.

思科预测，超过一半的全球IP视频流量(56%.8%的人会选择高清，四分之一(22%)的人会选择高清.3%) will be Ultra HD by 2022; this demand for high-resolution video means an inevitable trade-off between bandwidth and the end-user experience. 高分辨率视频通常也需要极高的比特率, 这可能导致启动缓慢, 视频缓冲, 内容分发网络(CDN)和存储成本高.

继续努力平衡效率和能力, 对视频感知优化的兴趣——换句话说, the processing of digital video streams to deliver the uncompromising quality that users expect at the minimum bandwidth—is rising. 传统上, 数字视频的世界依赖于压缩, 哪个是处理器密集型进程, 要解决这些问题. 提供更高质量的内容, 同时降低了带宽要求, the industry has worked to increase the efficiency and sophistication of the codecs it uses—but this brings much higher levels of complexity.

我们现在所处的阶段是，视频编码复杂性的增长速度超过了摩尔定律. 即使有更多的GPU和CPU容量来编码视频内容, the sheer volume of content being produced and watched means we will very quickly outstrip the compute cycles available. We are also facing a situation where the carbon footprint of the internet is estimated to be greater than that of the aviation industry.

作为一家公司，我们认为对视频流媒体的颠覆性创新是迫切需要的. 我们需要新的预处理和后处理, 编码, 以及设备感知和跨编解码器兼容的交付工具. 这是我们满足日益增长的在线视频需求的唯一途径, 减少加工, 能源, 存储要求. 红杉资本(SEQUOIA)， 96万美元&iSIZE, BBC R&D, 和伦敦玛丽女王大学(QMUL), 作为一个公司，我们正在努力实现这些目标吗.

红杉资本专注于带来创新技术, 包括人工智能, to improve the way video content is distributed and responds to the pressing need for video 流媒体 to become more sustainable. The SEQUOIA project is looking at perceptual optimization of video streams as a way of making significant reductions in bandwidth required for equal quality. 这是iSIZE工作的核心，我们在这一领域积累了广泛的专业知识.

新影像时代的新途径

iSIZE takes a unique approach to solving what is an increasingly urgent challenge of finding trade-offs between the various metrics, 在比特率和感知之间，同时管理处理和编码的复杂性. 而不是依赖于更复杂的编解码器和更大的GPU/CPU容量, we have developed a preprocessing solution that reduces the 编码 bitrate while operating before the encoder, 并且不需要任何关于编码器规范的信息.

Our innovative BitSave solution leverages patent-pending artificial intelligence (AI) features and machine learning, 结合感知质量指标的最新进展. It enhances details of the areas of each frame that affect the perceptual quality score of the content after typical block-based prediction and quantization and attenuates details that are not important. By reducing the bits required for elements of the image that perceptual metrics tell us are not important to human viewers, BitSave确保感知质量与编码比特率达到最佳平衡.

BitSave is a server-side preprocessing enhancement that is cross-codec applicable and optimizes legacy encoders like AVC/H.264以及HEVC/H.265、AV1和VVC/H.266，而不需要知道每个编码器的编码细节. 最重要的是, BitSave不改变编码, 包装, transport or decoding mechanisms—unlike solutions such as LCEVC—making it fully-compatible to any 编码, 流媒体, 没有任何修改的播放设备.

Most preprocessing solutions use some variant of a sharpening or contrast adjustment technique to deliver perceptual optimization, e.g.， HEVC或AV1编码器中的tune-vmaf选项. What sets BitSave apart is that it maintains the perceptual characteristics of the source without sharpening or changing contrast/brightness/color properties, and it eliminates the need for in-the-loop integration used by many other 编码 or perceptual optimization tools. BitSave is a single-pass preprocessing solution that needs no metadata or integration with the subsequent 编码 engine(s) and delivers significant gains in quality.

把我们的技术放在编码器之前, 我们确保它不依赖于特定的编解码器, 它优化了低层次的指标，如SSIM(结构相似指数指标), as well as for higher-level (and more perceptually oriented) metrics like Netflix’s VMAF and Apple’s AVQT metric or AI-based perceptual quality metrics like LPIPS. 对于这些指标，BitSave被证明提供了平均比特率节省. 同样的编码器和配方往往超过20%. 我们还以不违反编码标准的方式设计了我们的解决方案, 允许在现有的分销链和现有的客户端设备中使用它.

为了更大的灵活性, iSIZE软件开发工具包(SDK)允许BitSave作为Linux二进制文件进行试验, Docker容器服务, 或者作为CPU或GPU与内部编码器集成的Linux SDK. SDK的CPU运行时可与低复杂度的编码(例如.g., AVC x264中预置), a result obtained in part via our recent partnership with Intel in order to optimize our framework for inference on Intel CPUs. 此外, its runtime on mainstream NVIDIA hardware like T4 GPUs can be as fast as 3ms/frame for 1080p resolution.

Widereaching好处

那么，使用BitSave的好处是什么? 简而言之，它在两个关键领域提供了显著的节省. 首先，它降低了标准编解码器提供一定质量水平所需的比特率. 除了, 如果比特率节省不是唯一的目标, BitSave’s modest runtime means it can also be used to make the actual 编码 much faster—up to 500%—or even faster in cases like VP9, AV1, 或VVC编码.

整体, BitSave改进了多个最先进的质量指标, 并跨越多种视频编码标准. We believe we can go further since our approach offers compounded gains to any encoder-specific perceptual quality optimization: a real, 可衡量的, 显著节省比特率而不影响视觉质量.

This innovative technology elegantly answers one of the growing challenges faced by the industry: sustainable distribution of Ultra High Definition content, 同时限制视频对互联网流量的影响，降低分发成本. We believe that our solution will make an impact at every stage in the media distribution chain, delivering benefits for the whole sector by proactively reducing 能源 consumption at all stages within the media value chain.

iSIZE目前正与客户合作，在游戏中推广这项技术, 社交媒体, 以及娱乐视频流媒体行业. 在接下来的几个月, we are looking forward to making some important announcements on the commercial benefits offered by our framework.

http://www.isize.co/

这篇文章是赞助内容