A deep dive into how the Vortek Vulkan driver works

This series explores the inner workings of Vortek, motivated by its seemingly magical ability to enable directx gaming on Mali devices.

In the second part of this series, we will go through some of the workarounds that Vortek implements in order to get directx games working on, for e.g., Mali devices.

  1. Part 1: https://leegao.github.io/winlator-internals/2025/06/01/Vortek1.html
  2. Part 2 (this one): https://leegao.github.io/winlator-internals/2025/06/02/Vortek2.html

Disclaimers

This analysis is done with the Winlator 10.0 Final Hotfix APK and its Vortek libraries (libvulkan_vortek.so and libvortekrenderer.so) from https://github.com/brunodev85/winlator/releases/tag/v10.0.0

Note that everything here is inferred from binary reverse engineering of complex native libraries, as a result, the deep dive here is by no means exhaustive. It’s only meant to illustrate the design/architecture and some specific implementation details for certain workarounds found within Vortek.

The decompiled Java code is taken unmodified from JADX, the C code is reinterpreted from Ghidra by me + an LLM assistant into a more human-readable form. Reverse engineering artifacts are in https://github.com/leegao/vortek-deep-dive

Part 2: Driver-Specific Workarounds

As we saw in Part 1 of this series, a core goal of Vortek is to enable support for system drivers to be used within Winlator. However, just proxying vulkan commands isn’t enough, many system drivers tend to have poor Vulkan support. As a result, running a directx game directly on these drivers will often lead to glitches or crashes.

Vortek implements a (growing) set of driver-specific workarounds to help enable directx gaming, these include:

  1. Add support for drivers that lack WSI extensions
  2. Add support for drivers that lack placed memory extensions used by 32bit emulation (for x86-32 games) via emulation
  3. Add support for drivers that lack BCn compressed texture formats used by DX games via emulation + JIT decompression of these compressed textures
  4. Add support for drivers that lack gl_ClipDistance capability on Mali devices by removing all SPIR-V instructions associated with it. (This may however cause graphical glitches as proper clipping is no longer guaranteed)
  5. Add support for drivers that lack scaled texture formats on some mobile GPUs by emulating them on the GPU via SPIR-V instruction patching.

(BCn) Compressed Texture Support

Some Vk drivers (such as Mali’s) do not support certain texture compression formats that are crucial for dxvk to work. In particular, several compressed texture formats known as the BCn (Block Compression n) formats are unsupported on Mali.

Vortek deals with this by emulating texture compression via just-in-time (CPU-bound) decompression of these BCn compressed textures. In particular, it will intercept all image creation commands, check if the texture format is one of the unsupported BCn formats, and if so, perform (CPU-bound) decoding of the texture buffer data into a standard uncompressed format supported by the GPU. It will then stub out the original parameters (using compressed formats and potential flags exclusive to compressed formats) with the uncompressed variant.

More specifically:

  1. Intercept vkCreateImage (and variants), Vortek will check to see if the format of the image createInfo is compressed or not
    • If not, Vortek calls the real vkCreateImage directly with the parameters from the client (since the format is supported on the GPU)
  2. If the format is a compressed image, it’ll:
    • Record the original (compressed) options (the createInfo)
    • Replace the format parameter within the createInfo option with ~VK_FORMAT_R8G8B8A8_UNORM~ VK_FORMAT_B8G8R8A8_UNORM (as well as all pNext entries of VK_STRUCTURE_TYPE_IMAGE_FORMAT_LIST_CREATE_INFO_KHR)
    • Unset any flags that are for compressed images only - (VK_IMAGE_CREATE_BLOCK_TEXEL_VIEW_COMPATIBLE_BIT_KHR)
    • Call the real vkCreateImage with the modified (the uncompressed) createInfo and return that image handle to the client
  3. When the client wants to actually fill the image buffer (e.g. vkCmdCopyBufferToImage or variants) with a (compressed) “source” buffer, it’ll
    • Allocate a second destination buffer (with the uncompressed image data) for this image that will actually be rendered
    • Enqueue a request to decompress the source buffer (with the compressed image data) into the destination buffer (uncompressed).
    • Call the real vkCmdCopyBufferToImage with the (yet unfilled) destination buffer (uncompressed).
      • Note that while the destination buffer (uncompressed) doesn’t contain any data yet, Vortek will guarantee that this buffer is filled before the actual command is executed
  4. When the client calls vkQueueSubmit (and variants) to execute all of the current Vk command buffer, Vortek will need to ensure that all of the destination buffers of the images with compressed source buffers must be filled with the uncompressed data. It does this with a call to the TextureDecoder::decodeAll function (see below).
  5. Finally, when Vulkan actually uses the image, it will have already been filled with the complete uncompressed image data (decoded from the original compressed source buffer from the client).

Note: the target format is VK_FORMAT_B8G8R8A8_UNORM as Bruno himself reveals in https://github.com/brunodev85/winlator/issues/852#issuecomment-2962526789

TextureDecoder::decodeAll - Decompressing BCn Compressed Textures

A crucial part of Vortek is the need to decode/decompress BCn compressed texture format into plain old VK_FORMAT_B8G8R8A8_UNORM format (basically 8bit R, G, B, and Alpha encoding, AKA a classic RGB-with-alpha format you see so frequently on the internet). While these decode operations are batched and deferred, they still need to happen. This falls to the TextureDecoder::decodeAll function.

Here’s a sketch of what it does for each decode operation in the current batch:

  • Map the source (compressed texture) buffer into the current process’s memory map (allows the process to modify this buffer which is shared with the GPU)
  • Map the destination (final uncompressed texture) buffer as well, this will eventually be used to render the current texture
  • Based on the compression format of the source buffer, it will process each of the 4x4 blocks of the image (the unit at which BCn compression works on) and perform the proper decoding step depending on the format
    • Includes some fancy NEON acceleration in some paths to help vectorize this, but this is still very much bound to the CPU for every compressed texture submitted to the GPU

Shader Patching

Vortek also does some complex shader (SPIR-V) modifications for certain drivers (e.g. Mali). To do this, the general pattern is:

  1. [During Initialization/ShaderInspector::create]
    Check to see if a crucial capability/feature needed by the shaders for this game may be missing (e.g. shaderClipDistance)
    • Varies by the type of modification/patch requested
  2. [During Shader Module Building/vkCreateShaderModule]
    Intercept vertex shader modules and push them onto a queue to add necessary patches/modifications
    • This is done by inspecting if the OpEntryPoint (op=0x0F) instruction has execution model of Vertex (0x00) within the SPIR-V bytecode sent by the client, if so:
      • Defer the actual vkCreateShaderModule until later (e.g. vkCreateGraphicsPipelines)
      • Otherwise, call the real vkCreateShaderModule function if no shader modifications are needed
  3. [During Pipeline Building/vkCreateGraphicsPipelines]
    Ensure all unbuilt shader modules have their patches applied and then built into shader modules before creating the pipeline.
    • This is done through a dedicated modification function (ShaderInspector::inspectShaderStages) which patches the SPIR-V instructions and then finally builds the shader module (by calling the real vkCreateShaderModule that was deferred earlier with the modified shader code)
    • Generally requires some auxiliary data such as the VkPipelineShaderStageCreateInfo* pStages and the VkPipelineVertexInputStateCreateInfo* pVertexInputState)
    • This function acts as a chokepoint before shader modules are actually used, hence the design to defer shader modifications to this point before they are actually built. See Setting up render pipeline - Vulkan Guide for e.g.

gl_ClipDistance Removal (Mali)

What

Traditionally, GPUs will automatically clip (removing objects) outside of the current viewing volume. However, the developer can also specify custom clipping planes with the ClipDistance capability.

The Problem

Some Vk drivers (like Mali’s) do not support the ClipDistance shader capability.

When you try to create a graphics pipeline (vkCreateGraphicsPipelines) with a shader stage that writes to gl_ClipDistance, the pipeline creation will fail if the feature isn’t enabled. As a result, this will cause many pipelines to fail to be created on Mali GPUs.

How Vortek Deals With This

Vortek deals with this problem by simply removing the relevant ClipDistance SPIR-V (shader-opcode) instructions when compiling shader modules (also just-in-time). In particular, Vortek will:

  1. [ShaderInspector::create] Check to see if shaderClipDistance is a supported device feature by checking for VkPhysicalDeviceFeatures->shaderClipDistance (+0x94)
  2. [ShaderInspector_inspectShaderStages] Remove all SPIR-V instructions related to ClipDistance, specifically:

So in effect, removing the ClipDistance decoration and the capability will disable all user-defined clipping. It might cause some visual glitches/artifacts as some objects that should be clipped may show up, or certain effects that rely on custom clipping may render incorrectly. However, the pipeline will now be created successfully albeit with some potential graphical glitches.

SPIR-V Instruction Removal

The design for how to remove instructions is pretty simple:

  1. Record the index of the instruction to remove
  2. Sort all of the removals in descending order to allow stable modifications of the codebuffer without changing the previous indices
  3. Later, in the removal loop, Vortek will allocate a new buffer and:
    • memcpy the prefix
    • Ignore the current instruction
    • memcpy the suffice
    • Invariant: all of the remaining indices of instructions to remove occur prior to this modification and are unaffected
    • Replace the old codebuffer with the new codebuffer

Emulating Scaled Texture Format Support (Mali/Qcom)

What

Texture data can be stored in a variety of ways. As we’ve seen above, they can be block-encoded using the BCn format (which is not a supported format on Mali). Three of the most common formats are:

  • R8G8B8A8_UINT / R8G8B8A8_SINT - represents a block of colors+alpha (RGB+A) with each component represented and presented to the GPU as an 8-bit (unsigned/signed) integer
  • R8G8B8A8_UNORM / R8G8B8A8_SNORM - represents the RGB+A with each component represented as an 8-bit int and presented to the GPU as a float with range from 0.0f-to-1.0f or -1.0f-to-1.0f
  • R8G8B8A8_USCALED / R8G8B8A8_SSCALED - represents the RGB+A with each component represented as an 8-bit int and presented to the GPU as a float with range from 0.0f-to-255.0f or -127.0f-to-128.0f
Feature _UINT _UNORM _USCALED
Memory Storage 8-bit unsigned int (0-255) 8-bit unsigned int (0-255) 8-bit unsigned int (0-255)
Shader Receives uint float float
Shader Value Range 0 to 255 0.0f to 1.0f 0.0f to 255.0f
Conversion (Read) None int_val / 255.0f (float)int_val AKA OpConvertUToF
SPIR-V Sampled Type OpTypeInt OpTypeFloat OpTypeFloat

The Problem

VertexBuffer support (the ability to use buffers with these formats in vertex shaders) for _[US]SCALED formats seem to be very spotty on mobile GPUs. However, dxvk/games will sometimes attempt to use these scaled texture formats within vertex shaders, causing errors/glitches on GPUs without this support.

How Vortek Deals With This

Vortek does something very cool here. It’ll actually emulate _USCALED and _SSCALED vertex buffers directly within the shader. It does this by performing the conversion from _[US]INT to _[US]SCALED directly.

For example, the code layout(location = 0) in vec4 in_Position to create a new vec4 layout called position will generate SPIR-V code like

  • OpDecorate %in_Position Location 0 ; tagged as VK_FORMAT_R8G8B8A8_SSCALED in the pVertexInputState in the pipeline
  • %float = OpTypeFloat 32
  • %vec4_float = OpTypeVector %float 4
  • %ptr_vec4_float = OpTypePointer Input %vec4_float
  • %in_Position = OpVariable %ptr_vec4_float Input
  • %loaded = OpLoad %vec4_float %in_Position

Vortek will modify this into:

  • OpDecorate %in_Position Location 0 ; same
  • %float = OpTypeFloat 32 ; same
  • %vec4_float = OpTypeVector %float 4 ; same
  • %ptr_vec4_float = OpTypePointer Input %vec4_float ; same
  • %in_Position = OpVariable %ptr_vec4_float Input ; removed
    ; Add vec4 int32 as well as change in_Position to int32
  • %int32 = OpTypeInt 32 signed ; added
  • %vec4_int32 = OpTypeVector %int32 4 ; added
  • %ptr_vec4_int32 = OpTypePointer Input %vec4_int32 ; added
  • %in_Position = OpVariable %ptr_vec4_int32 Input ; changed
    ; Load the vec4_int32, then convert it into the expected type
  • %loaded = OpLoad %vec4_float %in_Position ; removed
  • %temp_loaded = OpLoad %vec4_int32 %in_Position ; added
  • %loaded = OpConvertSToF %vec4_float %temp_loaded ; changed

In essence, Vortek will add new instructions to define the proper type for the unscaled format (int32/uint32, vec4_[u]int32, … Then, it will load the underlying vec4_int32 into a temporary object and convert it into the proper type with the OpConvertSToF or UToF.

In particular, Vortek will:

  1. [ShaderInspector::create] Check to see if VertexBuffer is a supported format feature of scaled formats by checking for formatProps.linearTilingFeatures & VK_FORMAT_FEATURE_VERTEX_BUFFER_BIT for the VK_FORMAT_R8G8B8A8_SSCALED format.
  2. [ShaderInspector_inspectShaderStages]
    • Map all X_[US]SCALED formats into X_[U]INT instead within VkPipelineVertexInputStateCreateInfo.pVertexAttributeDescriptions and save their locations
    • Look for the OpVariables decorated with the location of a remapped scaled format from the pVertexAttributeDescriptions
    • Generate the SPIR-V instructions to construct new int32, vec4_int32 and ptr_vec4_int32 types needed by this approach
      • Optimization: if an int32 type has already been generated, reuse that instead of creating new ones (though the vec_int32 and ptr_vec4_int32 will be recreated)
    • Modify those OpVariables to use a vec4_int32 type
    • Modify OpLoads into the OpLoad + OpConvert[US]ToF combination instead
Identifying OpVariables to map

This is done by consulting the VkPipelineVertexInputStateCreateInfo*, which stores an array (pVertexAttributeDescriptions) that contains a list of (location: format) bindings. Vortek will then check each entry for their texture format and remap all scaled variants to their integer variants instead:

Original Format Original VkFormat New Format New VkFormat
0xb VK_FORMAT_R8_USCALED 0xd VK_FORMAT_R8_UINT
0xc VK_FORMAT_R8_SSCALED 0xe VK_FORMAT_R8_SINT
0x12 VK_FORMAT_R8G8_USCALED 0x14 VK_FORMAT_R8G8_UINT
0x13 VK_FORMAT_R8G8_SSCALED 0x15 VK_FORMAT_R8G8_SINT
0x27 VK_FORMAT_R8G8B8_USCALED 0x29 VK_FORMAT_R8G8B8_UINT
0x28 VK_FORMAT_R8G8B8_SSCALED 0x2a VK_FORMAT_R8G8B8_SINT
0x48 VK_FORMAT_R16G16_USCALED 0x4a VK_FORMAT_R16G16_UINT
0x49 VK_FORMAT_R16G16_SSCALED 0x4b VK_FORMAT_R16G16_SINT
0x4f VK_FORMAT_R16G16B16_USCALED 0x51 VK_FORMAT_R16G16B16_UINT
0x50 VK_FORMAT_R16G16B16_SSCALED 0x52 VK_FORMAT_R16G16B16_SINT
0x5d VK_FORMAT_R16G16B16A16_USCALED 0x5f VK_FORMAT_R16G16B16A16_UINT
0x5e VK_FORMAT_R16G16B16A16_SSCALED 0x60 VK_FORMAT_R16G16B16A16_SINT

Additionally, Vortek will save the locations for each of the modified attributes. During SPIR-V patching, Vortek will look for OpDecorate %target Location %LOCATION instructions where %LOCATION is in one of the modified attributes to find the correct %targets to modify.

vecN_int32 Generation

Vortek will additionally inspect the target (sint/uint format) to identify some additional parameters such as whether or not the vector should be signed (for sint), or how many vector components to use (e.g. VK_FORMAT_R8 has just a single R component, while R8G8B8A8 has 4 RGB+A components)

switch (format) {
    case VK_FORMAT_R8G8B8A8_UINT:    // 0xd
    case VK_FORMAT_R64G64B64_UINT:   // 0x4a
        isSigned = 0;
        componentSize = 1;
        break;
    case VK_FORMAT_R8G8B8A8_SINT:    // 0xe  
    case VK_FORMAT_R64G64B64_SINT:   // 0x4b
        isSigned = 1;
        componentSize = 1;
        break;
    case VK_FORMAT_R16G16_UINT:      // 0x14
    case VK_FORMAT_R32G32B32A32_UINT: // 0x51
        isSigned = 0;
        componentSize = 2;
        break;
    case VK_FORMAT_R16G16_SINT:      // 0x15
    case VK_FORMAT_R32G32B32A32_SINT: // 0x52
        isSigned = 1;
        componentSize = 2;
        break;
    case VK_FORMAT_R32G32B32_UINT:   // 0x29
    case VK_FORMAT_R64G64B64A64_UINT: // 0x5f
        isSigned = 0;
        componentSize = 4;
        break;
    case VK_FORMAT_R32G32B32_SINT:   // 0x2a
    case VK_FORMAT_R64G64B64A64_SINT: // 0x60
        isSigned = 1;
        componentSize = 4;
        break;
}

It will then generate the actual replacement vec4 definition using the proper isSigned and componentSize:

%int32 = OpTypeInt 32 $isSigned ; added  
%vecN_int32 = OpTypeVector %int32 $componentSize ; added  
%ptr_vecN_int32 = OpTypePointer Input %vecN_int32 ; added
OpLoad handling

The original OpLoad instruction

%loaded = OpLoad %vec4_float %in_Position

expects that the Vk variable (e.g. %in_Position) must be a ptr_vec4_float type as well in order to work nicely with the rest of the shader. Unfortunately, we’ve just converted the variable (e.g. the %in_Position) into an integer vector, so what do we do?

Turns out that there’s a handy instruction OpConvertS/UToF that effectively converts any integer vectors into float vectors in a way that’s directly compliant with the S/USCALED format. As a result, we merely need to introduce a temporary variable and rewrite the OpLoad into

%temp_loaded = OpLoad %vec4_int32 %in_Position ; added  
%loaded = OpConvertSToF %vec4_float %temp_loaded ; changed

Below are several cases where Vortek modifies extensions/features of the underlying driver. Generally speaking, these modifications occur on the server handler side.

vkCreateDevice

One of the places where this happens is within the handling of vkCreateDevice, let’s take a look at it more closely

vt_call_vkCreateDevice

The client-side is a standard vt_call_ dispatch function. It serializes (the very complicated arguments of) the VkDeviceCreateInfo onto the serverRing buffer, and then it reads a single VkDevice object out of the clientRing buffer. In particular, it serializes the following:

  • nullable(VkObject) physicalDevice (‘0’ or ‘1’;physicalDevicePtr)
  • VkDeviceCreateInfo (‘0’ or sizeof(VkDeviceCreateInfo);payload)
    • sType
    • list(pNext) (‘0’ or len(pNext);pNext…)
      • Note: each pNext is serialized differently based on pNext->sType
    • struct(queueCreateInfo)
    • list(layerNames)
    • list(extensionNames)
    • list(enabledFeatures)
int vt_call_vkCreateDevice(uint64_t physicalDevice, 
                            VkDeviceCreateInfo *createInfo, 
                            uint64_t allocator, 
                            uint64_t *device) {
    pthread_mutex_lock(&vt_call_mutex);
    // Get VkObject from handle
    VortekVkObject *physDevObj = VkObject_fromHandle(physicalDevice);
    // Calculate buffer size needed
    ...
    // Allocate serialization buffer
    uint8_t *buffer = vt_alloc(bufferSize);
    // Serialize physical device handle (as a nullable VortekVkObject - [0] or [1][handle_ptr])
    ...
    // Serialize VkDeviceCreateInfo
    if (createInfo == NULL) {
        *(uint32_t*)(buffer + headerSize) = 0;
    } else {
        *(int*)(buffer + headerSize) = vt_sizeof_VkDeviceCreateInfo(createInfo);
        // Serialize createInfo fields
        *(uint32_t*)(buffer + offset) = createInfo->sType;
        // Complicated serialization format for pNext
        uint32_t *pNext = (uint32_t*)createInfo->pNext;
        if (pNext != NULL) {
	        // For each pNext in the chain
	        *(uint32_t*)(buffer + pNextOffset + offset) = pNext->sType;
			...
			
			// Giant switch statement for structure types and different serialization logic based on the extension
			switch (structType) {
				// Example case
				case 0x33: { // VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2
					*(uint32_t*)(buffer + nextStructOffset + offset) = 0xc0;
					pNextOffset += 200;
					*(uint32_t*)(buffer + structOffset + offset) = pNext[0];
					// Copy all 48 feature fields
					for (int i = 1; i < 49; i++) {
						*(uint32_t*)(buffer + structOffset + offset + i*4) = pNext[i+3];
					}
					break;
				}
				...
			}
        } else {
	        *(int32_t*)(buffer + offset + 4) = -1;
        }
        
        // Serialize queue create infos
        ... 
		// Serialize layer names
		...
		// Serialize extension names
		...
		// Serialize enabled features
		...
    }
    
    // Mark end of buffer
    buffer[offset] = 0;
    
    // Send serialized data to server
    uint64_t serverRingBuffer = serverRing;
    requestHeader = {0x6d, bufferSize}; // Cmd and size
    
    RingBuffer_write(serverRing, &requestHeader, 8);
    RingBuffer_write(serverRingBuffer, buffer, bufferSize);
    
    // Read response from client
    RingBuffer_read(clientRing, &responseHeader, 8);
    int resultCode = responseHeader.vk_result;
    int responseSize = responseHeader.size;
    
    RingBuffer_read(clientRingBuffer, &vkDevice, responseSize);
	VkObject_create(3, vkDevice);
	...
	pthread_mutex_unlock(&vt_call_mutex);
	return resultCode;
}

vt_handle_vkCreateDevice

void vt_handle_vkCreateDevice(long param_1) {
    ...
    // Parse all of the arguments
    void *physical_device_id
    void *pNext_chain;
    VkDeviceCreateInfo create_info = {0};
    VkDeviceQueueCreateInfo *queue_infos;
    char **layer_names;
    char **extension_names;
    void *enabled_features;
    ...
    
    // Get VkPhysicalDevice object
    uint64_t physical_device = VkObject_fromId(physical_device_id);
    
    // Disable unsupported features
    disableUnsupportedFeatures(physical_device, &create_info);
    
    // Define required extensions to inject
    const char *required_extensions[] = {
        "VK_KHR_swapchain",
        "VK_KHR_external_memory", 
        "VK_KHR_external_memory_fd",
        "VK_KHR_get_memory_requirements2",
        "VK_KHR_dedicated_allocation",
        "VK_KHR_sampler_ycbcr_conversion",
        "VK_EXT_queue_family_foreign",
        "VK_ANDROID_external_memory_android_hardware_buffer",
        "VK_KHR_external_semaphore",
        "VK_KHR_external_semaphore_fd", 
        "VK_KHR_external_fence",
        "VK_KHR_external_fence_fd"
    };
    
    const char *swapchain_ext[] = {"VK_KHR_swapchain"};
    
    // Inject required extensions
    injectExtensions(&create_info.ppEnabledExtensionNames, 
                     &create_info.enabledExtensionCount,
                     required_extensions, 11,
                     swapchain_ext, 1);
    
    // Call actual vkCreateDevice
    int result = (*_DAT_vkCreateDevice)(physical_device, &create_info, NULL, &vkDevice);
    
    // Initialize Vulkan device wrapper if successful
    if (result == 0) {
        initVulkanDevice(param_1, physical_device);
    }
    
    ...
    
    if (RingBuffer_write(ring_buffer, response_header, 8) & 1) {
        RingBuffer_write(ring_buffer, vkDevice, 8);
    }
}
disableUnsupportedFeatures

This function will look for the VkPhysicalDeviceFeatures2 object within the CreateInfo (either as a direct field or within its pNext chain). Once found, it will:

  1. vkGetPhysicalDeviceFeatures(physical_device, &supported_features) to get the set of actual supported device features on this driver
  2. Disable any user-requested features that are not supported

While this might cause potential glitches or bugs, it prevents Vulkan from failing immediate on attempting to create the device due to missing features.

injectExtensions
void injectExtensions(
	const char* initialExtensions, initialExtensionsCount,
	const char* addExtensions, addExtensionsCount,
	const char* rmExtensions, rmExtensionsCount
)

The injectExtensions function will take an initial set of extensions (create_info.ppEnabledExtensionNames) and:

  1. Remove any extensions from the rmExtensions list first ("VK_KHR_swapchain")
  2. Add in all extensions from the addExtensions list, including any that may have been removed in the same call

In this case, we’re just adding in the whole list of required extensions (required_extensions, including VK_KHR_swapchain) into the requested create_info.ppEnabledExtensionNames:

  • “VK_KHR_swapchain” - needed for windows/surface presentation
  • “VK_KHR_external_memory”
  • “VK_KHR_external_memory_fd” - fd based memory sharing
  • “VK_KHR_get_memory_requirements2”
  • “VK_KHR_dedicated_allocation”
  • “VK_KHR_sampler_ycbcr_conversion” - Hardware-accelerated YUV color format support
  • “VK_EXT_queue_family_foreign”
  • “VK_ANDROID_external_memory_android_hardware_buffer” - enables integration with AHBs as Vulkan buffers/memory
  • “VK_KHR_external_semaphore”
  • “VK_KHR_external_semaphore_fd”
  • “VK_KHR_external_fence”
  • “VK_KHR_external_fence_fd”

Note that Vortek assumes that the underlying Vulkan driver already natively supports these device extensions and do not need to emulate them.

vkCreateInstance

void vt_handle_vkCreateInstance(long param_1)
{
    // Deserialize the VkInstanceCreateInfo struct
    // sType
    // flags
    // struct(pApplicationInfo)
    // list(ppEnabledLayerNames)
    // list(ppEnabledExtensionNames)
    VkInstanceCreateInfo localCreateInfo = {0};
    VkApplicationInfo* appInfo = NULL;
    char** layerNames = NULL;
    char** extensionNames = NULL;
    ...
    VkInstance instance = VK_NULL_HANDLE;
    
    // Define extensions to be added and removed
    char* extensionsToAdd[] = {
        "VK_KHR_external_memory_capabilities",
        "VK_KHR_external_semaphore_capabilities", 
        "VK_KHR_external_fence_capabilities",
        "VK_KHR_get_physical_device_properties2"
    };
    
    char* extensionsToRemove[] = {
        "VK_KHR_surface",
        "VK_KHR_xlib_surface"
    };
    
    // Inject extensions: remove unwanted ones, add required ones
    injectExtensions(&extensionNames, &extensionCount, 
                    extensionsToAdd, 4,
                    extensionsToRemove, 2);
    
    // Update the create info with modified extensions
    localCreateInfo.enabledExtensionCount = extensionCount;
    localCreateInfo.ppEnabledExtensionNames = (const char* const*)extensionNames;
    
    // Call the original vkCreateInstance function
    VkResult result = vkCreateInstance(&localCreateInfo, NULL, &instance);
    
    if (result == VK_SUCCESS) {
        // Initialize Vulkan instance tracking
        initVulkanInstance(param_1, instance, appInfo);
    }
    
    // Write the instance back to the client
    if (RingBuffer_write(clientRing, &response, sizeof(response)) & 1) {
        RingBuffer_write(clientRing, instance, sizeof(VkInstance));
    }
}

Here, Vortek will add the following extensions to the underlying Vulkan call (assuming native support from the driver):

  • “VK_KHR_external_memory_capabilities”,
  • “VK_KHR_external_semaphore_capabilities”,
  • “VK_KHR_external_fence_capabilities”,
  • “VK_KHR_get_physical_device_properties2” - Get extended device properties and features, needed by other parts of Vortek

Curiously, Vortek also removes two extensions:

  • “VK_KHR_surface” - essential for connecting vulkan to a windowing system (like x11) and to display graphics on-screen
  • “VK_KHR_xlib_surface” - allows creation of surfaces for X11 windows using the Xlib library

This is because Vortek will add in its own WSI implementation directly with the Lorie renderer.

vkEnumerateInstanceExtensionProperties

void vt_handle_vkEnumerateInstanceExtensionProperties(long context_param) {
    ...
    // Call original Vulkan function to get base extensions
    uint original_extension_count;
    VkResult result = original_vkEnumerateInstanceExtensionProperties(
        NULL, 
        &original_extension_count, 
        NULL
    );
    
    // Allocate buffer for all extensions (original + injected)
    uint total_count = original_extension_count;
    long extension_list_buffer = vt_alloc(total_count * 0x104); // VkExtensionProperties size
    
    // Get the actual extension list
    result = original_vkEnumerateInstanceExtensionProperties(
        NULL, 
        &original_extension_count, 
        extension_list_buffer
    );
    
    // Define extensions to inject
    char* extensions_to_add[] = {
        "VK_KHR_surface",
        "VK_KHR_android_surface"
    };
    
    char* extensions_to_remove[] = {
        "VK_KHR_xlib_surface"
    };
    
    // Inject additional extensions (remove first, then add)
    injectExtensions2(
        &extension_list_buffer,     // Extension list buffer
        &original_extension_count,  // Current count
        &extensions_to_add,         // Extensions to add
        2,                          // Count of extensions to add
        &extensions_to_remove,      // Extensions to remove  
        1                           // Count of extensions to remove
    );
    
    // Write all of extension_list_buffer to the response
    ...
    // Write response to ring buffer
    struct {
        VkResult vk_result;
        int size;
    } response_header = { result, response_size };
    if (RingBuffer_write(clientRing, &response_header, 8) & 1) {
        RingBuffer_write(clientRing, response_buffer, response_size);
    }
}

This function is called to report all available instance extensions to the client. In particular, it will always report the following 2 extensions to be available:

  • VK_KHR_surface
  • VK_KHR_android_surface

While it will remove the VK_KHR_xlib_surface extension

This note covers a few ways in which Vortek is able to work around driver incompatibilities with directx / dxvk games and enabling better support for them on, for example, Mali devices.

In the next part of this series, we will look into other implementation details of Vortek outside of the standard vt_call_ and vt_handle_ patterns seen in part 1.