The Vulkan® Working Group at Khronos® has developed a set of video format decode and encode extensions, collectively referred to as "Vulkan Video." Today, with the release of Vulkan 1.3.277, the Working Group is proud to announce the new Decode AV1 video extension. Similar to the Vulkan project in its goals, AV1 is a royalty-free open standard for video compression developed by the Alliance for Open Media (AOM) delivering industry-leading performance and quality. The result of broad industry collaboration and support, the Decode AV1 extension is a major milestone that builds on the foundation of Vulkan Video to bring cross-platform portable and performant AV1 decode to engines and applications everywhere. 

The figure below shows the released Vulkan Video extensions and some of the extensions under development.

Figure 1. Vulkan Video extensions

Details about the new Decode AV1 functionality can be found in the extension proposal document and specification.

Vulkan drivers supporting both Decode AV1 and the recently released Encode H.264/H.265 extensions are already available, including:

• NVIDIA: Windows and Linux beta drivers support Decode AV1 today, with an imminent  production driver that will add Decode AV1 to already shipping H.264/H.265 Encode support.
• AMD: Windows beta driver for Encode H.264/H.265 and Decode AV1.
• Intel: Windows driver coming soon.

There is strong open-source community adoption of Vulkan Video, including decode and encode support in the GStreamer and FFmpeg multimedia frameworks, as well as the RADV/ANVs open-source Vulkan driver for Intel and AMD GPUs.

NVIDIA’s open source Vulkan Video sample is also being updated to illustrate the use of the Encode H.264/H.265 and the Decode AV1 extensions.

The open-source community has been deeply involved in the development of the Vulkan Video Decode AV1 extension, starting with the initial Mesa extension proposal from Dave Airlie and Lynne Iribarren, and including key contributions from Igalia for conformance test development, and detailed feedback from Collabora on the extension design. The experienced support and guidance from RasterGrid for specification and validation layer development was also essential. The Vulkan Video Decode AV1 extension is a true testament to the strong industry collaboration fostered by Khronos, and the Vulkan Working Group is grateful for all its members’ hard work and contributions to reach this important milestone for the Vulkan Video ecosystem!

Vulkan Video Support in the Vulkan SDK

The Vulkan SDK 1.3.275.0 now integrates all the components necessary for developers to easily use the Vulkan Video Encode H.264/H.265 and maintenance1 extensions released in Dec 2023, and the video core and decode H.264/H.265 extensions released in Dec 2022. An upcoming SDK release will add support for the Decode AV1 extension.

The Vulkan validation layers, API headers, and API registry now include support for:

• VK_KHR_video_queue: common APIs for all video coding operations
• VK_KHR_video_decode_queue: common APIs for all video decode operations
• VK_KHR_video_decode_h264: H.264 decode-specific capabilities and parameters
• VK_KHR_video_decode_h265: H.265 decode-specific capabilities and parameters
• VK_KHR_video_encode_queue: common APIs for all video encode operations
• VK_KHR_video_encode_h264: H.264 encode-specific capabilities and parameters
• VK_KHR_video_encode_h265: H.265 encode-specific capabilities and parameters
• VK_KHR_video_maintenance1: some API flexibility improvements and simplifications for video decode and encode operations

The new Vulkan SDK also provides Vulkan Video codec-specific headers for encode H.264 and encode H.265 operations. The full list of codec-specific headers included is:

• vulkan_video_codec_h264std.h: defines structures and types shared by H.264 decode and encode operations
• vulkan_video_codec_h264std_decode.h: defines structures used only by H.264 decode operations
• vulkan_video_codec_h264std_encode.h: defines structures used only by H.264 encode operations
• vulkan_video_codec_h265std.h: defines structures and types shared by H.265 decode and encode operations
• vulkan_video_codec_h265std_decode.h: defines structures used only by H.265 decode operations
• vulkan_video_codec_h265std_encode.h: defines structures used only by H.265 encode operations
• vulkan_video_codecs_common.h: defines a versioning macro used by other standard headers for version maintenance

Call for Action, Feedback & Support!

The Khronos Vulkan Video subgroup welcomes all developer feedback and is carefully monitoring the Vulkan issue trackers on GitHub for Decode AV1 and Encode H.264/H.265. We also look forward to hearing about any additional features important for your use cases that should be added to Vulkan Video!

In addition, all are invited to attend the various presentations about Vulkan Video and other topics at the Vulkanised 2024 event in Sunnyvale, CA., on February 5-7. Videos for the presentations will be posted online after the event.

Industry Support for Vulkan Video

“AMD is proud to continue collaborating with industry leaders to complete the standardization of video acceleration support in Vulkan®. The finalization of encode H.264/H.265 and decode AV1 extensions marks a significant milestone, and we’re even more excited about future codec and feature extensions,” said Andrej Zdravkovic, senior vice president and chief software officer at AMD. “We look forward to supporting end-users with an upcoming release of an AMD Software: Adrenaline Edition™ driver supporting optimized Vulkan Video for AMD RDNA™ architecture-based products and solutions.” 

"Furthering support for AV1, a state-of-the-art, open video compression format, aligns with Collabora's mission to accelerate the adoption of open technologies," said Nicolas Dufresne, principal software engineer at Collabora. "We are delighted to have had the opportunity to leverage our deep experience enabling embedded hardware codecs on Linux to help the Vulkan Working Group bring this new extension to life."

“We have been eagerly awaiting the ratification of the Vulkan Video encode extensions as they will pave the way to the definition of the Vulkan Safety Critical variant. Congratulations to all that have made this a reality. We are extremely encouraged to be part of the growth and evolution of the Vulkan ecosystem, and we as a collective have come a long way. Furthermore, we at CoreAVI are extremely excited to further empower safety critical systems with the addition of Vulkan Video,” said Dan Joncas, chief sales and marketing officer, CoreAVI.

“Over the course of 2023, Igalia has been working in collaboration with the vendors, and exceptional individuals from the wider community, on the delivery of the latest Vulkan Video extensions. As always at Igalia, we pay particular attention to the wider FOSS adoption of our work. We believe strongly in the approach taken by these new extensions for adding more flexibility and portability in video processing across the industry. To aid in the FOSS adoption of these new extensions, we have developed the GStreamer framework support for Vulkan Video, worked on the development and maintenance of the Vulkan Mesa drivers for Intel and AMD, and assisted in ensuring the Vulkan conformance testshave a good standard of coverage to ensure the implementations and specification are in unison. We look forward to continuing this work going into 2024 and beyond,” said Samuel Iglesias, director of GPU driver development, Igalia.

“We work closely with the Vulkan Working Group to drive more open standards that will benefit users at large. The ratification of the Vulkan Video extension for AV1 decode acceleration will set the stage for next-generation visual computing experiences,” said Vivian Lien, vice president Client Graphics and AI group at Intel. “By delivering dedicated AV1 hardware capabilities in our products, we aim to empower developers to build immersive applications across platforms and in the near future all Intel® Arc™ Graphics products will fully support hardware AV1 video decoding through the new Vulkan Video extension.”

“AV1 represents the final product of years of work to develop a royalty-free video codec. The open-source community was amongst the first to embrace it, by creating dav1d, the fastest software decoder implementation, and now, by offering support for multi-platform hardware decoding via the new extension,” said Lynne Iribarren, Khronos member and FFmpeg developer. 

“AV1 support is one of the main reasons I wanted to get involved in Vulkan video, having a royalty-free video codec available across vendors and platforms will go a long way to making AV1 a baseline for future Linux desktop use cases. I welcome Khronos ratification of the work started as a vendor extension in the Mesa project,” said Dave Airlie, Mesa developer and Linux Kernel maintainer.

“NVIDIA is proud to continue to drive innovation with the Vulkan Working Group. By being one of the first to support Vulkan-based H.264, H.265 encoding and AV1 decoding, we are expanding video API support for Vulkan developers. We look forward to feedback from our developers and partners to advance our GPUs’ video capabilities, including Vulkan Video, and to enable applications to take advantage of the latest video technologies,” said Bob Pette, vice president of enterprise platforms, NVIDIA.

"RasterGrid is delighted to continue the journey with the Vulkan Working Group in enabling efficient access to hardware-accelerated video coding through Vulkan Video," said Daniel Rakos, CEO, RasterGrid. "We are happy to see our work on the extension specifications and Validation Layer support finally landing in the hands of the developer community, and looking forward to helping the Vulkan Video TSG expand the support for royalty-free video codecs in the API."