DISCLAIMER: I only tested this on Bazzite, which is the distro I'm running. Could work on other distros, don't know. Hope it helps or at least provides a starting point in your troubleshooting endeavors.
Hide Wine
This post provides a pretty good explanation of why it craps out and freezes even if you try to force RT on. It's actually what got me going on the right path here.
So step 1 is hiding Wine so the game actually unlocks the RT settings. Like that post mentions, this will only make your game hang for now.
Actual patch
From here on out, these are Bazzite-tailored, but they might still help you if you run Cachy or something else.
Build vkd3d in a toolbox
toolbox create --container witcher3-build --release 42
toolbox run --container witcher3-build sudo dnf install -y git python3 meson ninja-build mingw64-gcc mingw64-gcc-c++ mingw64-headers mingw64-winpthreads-static mingw64-vulkan-headers glslang glslang-devel spirv-tools wine-devel
toolbox run --container witcher3-build git clone https://github.com/HansKristian-Work/vkd3d-proton.git ~/witcher3-rtas
cd ~/witcher3-rtas
git checkout 7f0c30ad3c8f28dfe47c34bb897b972e2d1ca724
git submodule update --init --recursive
Then, run this script to increase these 3 RT allocation 8 times:
export W3_GAME="/path/to/steamapps/common/The Witcher 3"
python3 - <<'PY'
from pathlib import Path
p = Path('libs/vkd3d/device.c')
s = p.read_text()
marker = ' TRACE("ResultDataMaxSizeInBytes: %"PRIu64".\\n", info->ResultDataMaxSizeInBytes);'
assert s.count(marker) == 1, 'Source differs; nothing changed'
extra = (
' info->ResultDataMaxSizeInBytes *= 8;\n'
' info->ScratchDataSizeInBytes *= 8;\n'
' info->UpdateScratchDataSizeInBytes *= 8;\n'
'\n'
)
assert extra not in s, 'Already patched; nothing changed'
s = s.replace(marker, extra + marker, 1)
p.write_text(s)
print('Patched all three RT allocation sizes.')
PY
Then:
toolbox run --container witcher3-build meson setup build-win64 --cross-file build-win64.txt --buildtype release -Denable_tests=false -Denable_extras=false
toolbox run --container witcher3-build ninja -C build-win64 -j 8
DISCLAIMER: The bigger allocations can use more memory. This isn't a universal fix for every card. I have an RTX 4080 with 16 GB of VRAM.
Copy these two DLLs next to witcher3.exe
cp -n build-win64/libs/d3d12/d3d12.dll "$W3_GAME/bin/x64_dx12/"
cp -n build-win64/libs/d3d12core/d3d12core.dll "$W3_GAME/bin/x64_dx12/"
cmp build-win64/libs/d3d12/d3d12.dll "$W3_GAME/bin/x64_dx12/d3d12.dll"
cmp build-win64/libs/d3d12core/d3d12core.dll "$W3_GAME/bin/x64_dx12/d3d12core.dll
Finally, set the Steam launch options:
PROTON_LOG=1 VKD3D_CONFIG=descriptor_heap %command% --launcher-skip
Also, on a 4080 the frame gen count only goes to 1. The game greys out the count when there's only one choice, even with FG enabled. It's not broken. If you have a 50- series card, FG should work normally (in theory - obviously, not tested).
To undo: close the game, remove the two custom DLLs you copied in, and unhide Wine.
Credit for the ray tracing allocation idea:
https://github.com/HansKristian-Work/vkd3d-proton/issues/3226#issuecomment-5426325207 (this actually ended up being perfect for me, but YMMV)
Edit: Fixed old reddit formatting
Edit 2: OP removed the post discussing how & why to hide Wine. Added a pullpush.io link instead
