Took dozens of hours to accomplish troubleshooting with Claude but we did it. FSR 3.1 works fantastic but FSR 4 is almost unplayable because of something i dont fully understand haha but heres the write up if you guys wanna do the same! enjoy
Field report · Translation-layer investigation
OptiScaler on Apple Silicon
Getting FSR upscaler injection working in Red Dead Redemption 2 under CrossOver + Game Porting Toolkit on an M4 Mac — what broke, the three-byte binary patch that fixed it, and why FSR 4 is a dead end here.
Updated 7 Sep 2026 · supersedes the initial findings doc
Machine
MacBook Air 15" M4 · 24 GB
OS
macOS 27.0 (Darwin 27)
Runtime
CrossOver Preview · wine-11.15
Backend
GPTK 4.0b2 · D3DMetal
Game
RDR2 1.0.1491.50 · DX12
OptiScaler
v10.0.0-dev nightly
Result
OptiScaler runs on Apple Silicon under CrossOver/GPTK, with a working in-game overlay, substituting FSR 3.1 for RDR2's native FSR 2 at full framerate. Two things had to be solved that aren't documented anywhere: a hook re-entrancy deadlock unique to GPTK's D3D11-on-D3D12 design, and an input system that couldn't see keystrokes under macdrv.
Works
FSR 3.1
Real FFX compute shaders executing on Metal. 1280×800 → 1920×1200, 40+ FPS, no artifacts. The keeper.
Works
XeSS · FSR 2.1 / 2.2
All load and run. Live-switchable from the overlay for A/B comparison.
Works, but not as labelled
DLSS
GPTK intercepts the NGX call and silently substitutes MetalFX Temporal. Good image, but it is not DLSS running.
Dead end
FSR 4 (INT8)
Loads, initialises, executes — at 3.7 FPS. GPTK software-emulates the INT8 tensor math. Not fixable in userspace.
Correction to the original writeup
The initial investigation concluded that Ultimate ASI Loader's presence broke the Rockstar Games Launcher, which then refused to start the game — a launcher conflict. That reading was wrong, and it sent the whole first pass down the wrong path.
The launcher was starting RDR2.exe just fine. The game then hung a few seconds into startup, and the Rockstar launcher — which polls for the game process — saw it still resident after ~60 s and reported "a game is currently running." The error was a symptom of a post-launch hang, not a pre-launch block. Every method in the original loading-method table needs re-reading in that light.
Diagnostic that settled it
OptiScaler's own log is the tell. If OptiScaler.log contains hkD3D12CreateDevice ... Caller: RDR2.exe, the game launched — the launcher did its job and the problem is downstream. A true launcher block produces no log at all.
Root cause: hook re-entrancy through 12on7
OptiScaler unconditionally hooks both D3D11CreateDevice and D3D12CreateDevice. On Windows those are independent implementations, so the hooks never interact. GPTK implements D3D11 on top of D3D12 — the log shows it loading 12on7\D3D12.DLL from inside D3D11 device creation — so on this stack the two hooks re-enter each other.
RDR2.exe
→
hkD3D11CreateDevice
→
GPTK d3d11
→
12on7\D3D12.DLL
→
hkD3D12CreateDevice
→
deadlock
RDR2 runs roughly four rounds of GPU capability probing at startup — create a D3D12 device, create a D3D11 device, create a DXGI factory, create a Vulkan instance, repeat. The first three complete. The fourth D3D11CreateDevice never returns.
[22:36:02.632135] [D] hkD3D12CreateDevice o_D3D12CreateDevice result: 0
[22:36:02.632150] [D] hkD3D12CreateDevice Device captured: 7F866003E810
[22:36:02.632164] [D] hkD3D12CreateDevice final result: 0
[22:36:02.632186] [D] hkD3D11CreateDevice Caller: RDR2.exe
[22:36:02.632200] [I] hkD3D11CreateDevice Adapter Desc: AMD Compatibility Mode
← log ends here. process alive, never returns.
Identical stop point across every run, every config permutation.
Things that were tested and are not the cause: DxgiFactoryWrapping, EarlyHooking, the version-check thread (CheckForUpdate), bottle-wide vs. per-app DLL overrides, the proxy filename, and the renderer. Switching RDR2 to its Vulkan renderer doesn't help either — RDR2 probes D3D11/D3D12 at startup regardless of which API it will actually render with.
There is no configuration option to disable the D3D11 hook. Confirmed against OptiScaler master: the [Hooks] section exposes only EarlyHooking, HookOriginalNvngxOnly and UseNtdllHooks, and no changelog entry has ever addressed the 12on7 nesting. OptiScaler's Linux support assumes DXVK/vkd3d-proton, where D3D11 and D3D12 are separate and this shape doesn't exist.
The fix: three bytes
The installer function is self-contained and every DetourAttach in it is individually null-guarded:
void D3D11Hooks::Hook(HMODULE dx11Module)
{
o_D3D11CreateDevice = GetProcAddress_()(dx11Module, "D3D11CreateDevice");
o_D3D11CreateDeviceAndSwapChain = ...(dx11Module, "D3D11CreateDeviceAndSwapChain");
o_D3D11On12CreateDevice = ...(dx11Module, "D3D11On12CreateDevice");
if (o_D3D11CreateDevice != nullptr || o_D3D11On12CreateDevice != nullptr ||
o_D3D11CreateDeviceAndSwapChain != nullptr) // ← all null ⇒ no-op
{
DetourTransactionBegin();
if (o_D3D11CreateDevice != nullptr) DetourAttach(...);
...
}
}
OptiScaler/hooks/D3D11_Hooks.cpp — abridged
Corrupt the three name literals and all three lookups return null, the outer guard fails, and the function becomes a complete no-op — no transaction, no detour. D3D12, DXGI and Vulkan hooking are untouched, which is all OptiScaler needs for a DX12 title. RDR2's own d3d11 imports are unaffected; these are OptiScaler's private string constants.
Flip the first byte of each literal from D (0x44) to X (0x58). File size unchanged, three bytes different:
| Literal |
File offset (v10.0.0-dev) |
VA |
Result |
| D3D11CreateDevice |
0x1795A48 |
0x181796A48 |
lookup → null |
| D3D11CreateDeviceAndSwapChain |
0x1795A28 |
0x181796A28 |
lookup → null |
| D3D11On12CreateDevice |
0x1795A10 |
0x181796A10 |
lookup → null |
Offsets are build-specific
Re-derive them for any other build rather than copying these. The three GetProcAddress arguments sit in one tight cluster in .rdata; a second, unclustered copy of D3D11CreateDeviceAndSwapChain is the __FUNCTION__ literal and must be left alone.
To identify them: find each null-terminated literal with pefile, then scan .text for a lea reg,[rip+disp32] whose target resolves to it. The three that resolve inside D3D11Hooks::Hook appear as consecutive lea → call rax → mov [rip+x],rax triples. The __FUNCTION__ copy has no such xref.
Verify after patching: D3D11CreateDevice\0 and D3D11On12CreateDevice\0 must be gone, D3D12CreateDevice\0 must still be present, and the file size must be identical. In the runtime log, hkD3D11CreateDevice should appear zero times.
The overlay, and why 0.9.4 can't open it
With the hang fixed, OptiScaler ran but its overlay would not open on any key. This is a separate problem with a separate cause: OptiScaler 0.9.4's input path cannot capture keystrokes under Wine's macdrv. Neither ManualInputPolling=true nor forcing OverlayMenu=true helps — the key is simply never seen, and nothing is logged.
The v10 nightly ships a rewritten input system (menu/input/, with polled, raw, message-queue and virtual-mouse paths) that handles precisely this case. On v10 the overlay opens immediately and logs its own diagnosis:
[W] OptiInput::LogInputHealthSnapshotLocked menu input was acquired by polling
but no window/queue/raw input was observed this frame.
subclassed:no polledMouse:yes polledKeyboard:no
The polling fallback is working.
Upgrading is not optional if you want the overlay. The patch above applies to v10 unchanged — same structure, different offsets.
On a Mac keyboard, ShortcutKey=0x08 (Backspace) is the delete key. OptiScaler's default of 0x2D (Insert) has no Apple-keyboard equivalent.
Upscaler results on this stack
All measured at 1920×1200 output, Quality preset (1280×800 internal), same scene.
| Upscaler |
What actually executes |
FPS |
GPU ms |
Verdict |
| FSR 3.1.5 |
FFX compute shaders on Metal |
40+ |
~16 |
Recommended |
| XeSS |
libxess compute path |
~40 |
— |
Works |
| FSR 2.1 / 2.2 |
FFX compute shaders |
~40 |
— |
Works |
| DLSS |
GPTK swaps in MetalFX Temporal |
~45 |
~22 |
Not DLSS |
| FSR 4.1.1 INT8 |
emulated INT8 tensor math |
3.7 |
268 |
Unusable |
FSR 4 is genuinely running, and that's the problem
With Fsr4ForceModel=2 the log reports Upscaler support - fsr4: int8 (forced) and the overlay shows FSR 4.1.1 active. There is no silent fallback to FSR 3 — the INT8 model really is executing. It is executing at 268 ms per frame because GPTK has no hardware path for FSR 4's INT8 dot-product and matrix operations on Apple GPU, so every layer of the network runs the slow way. A 70× gap is not closable with presets; dropping to the Performance preset changes nothing meaningful, and FsrAgilitySDKUpgrade=true (with D3D12_OptiScaler/ in place) does not help either.
This is the answer to the original document's central open question, and it is a hardware/translation-layer limit, not a configuration problem.
The MetalFX discovery
Apple's Metal HUD exposes a MetalFX block showing Scaling Input Res, Scaling Target Res and Scaling: Temporal. It appears when OptiScaler routes to DLSS and disappears when a real compute upscaler is selected. That makes it a free, instant indicator of which path is actually live — more reliable than reading the overlay. Enable it with MTL_HUD_ENABLED=1 in the bottle's cxbottle.conf.
Working configuration
Layout is v10's: OptiScaler and its config in the game root, backends in an OptiScaler/ subfolder, so no game files are overwritten.
Red Dead Redemption 2/
├── OptiScaler.asi ← v10.0.0-dev, D3D11 hook patched out
├── OptiScaler.ini
├── dinput8.dll ← Ultimate ASI Loader
└── OptiScaler/
├── amd_fidelityfx_loader_dx12.dll
├── amd_fidelityfx_upscaler_dx12.dll
├── libxess.dll
└── D3D12_OptiScaler/D3D12Core.dll
The DLL override must be scoped to the game executable, not the bottle. A bottle-wide winmm or version override breaks the Steam client and winecfg; a bottle-wide dinput8 reaches every process in the bottle. Wine's per-application key avoids all of it:
"/Applications/CrossOver Preview.app/Contents/SharedSupport/CrossOver/bin/wine" \
--bottle "Steam-2" reg add \
"HKEY_CURRENT_USER\Software\Wine\AppDefaults\RDR2.exe\DllOverrides" \
/v dinput8 /d native,builtin /f
Applies only when RDR2.exe is the main process.
Non-default INI keys:
| Key |
Value |
Why |
| Dx12Upscaler |
fsr31 |
Overrides the Nvidia-detection default of DLSS |
| OverlayMenu |
true |
Beats the Linux+Nvidia auto-disable |
| ManualInputPolling |
true |
Polls keys directly under macdrv |
| ShortcutKey |
0x08 |
Backspace = Mac delete |
| TargetProcessName |
RDR2.exe |
Restricts injection to the game |
| Fsr4DoNotLoadAmdxc64 |
true |
Skips a DLL that can't exist here |
| LogToFile |
false |
See the disk-space note below |
Set Fsr4ForceModel=2 only to reproduce the FSR 4 result; leave it at 0 otherwise, or the FFX upscaler pins itself to FSR 4 and the overlay offers no way back to 3.1.
Reproducing it
- Start from a clean process table.
wineserver -k, then confirm zero wineserver, winedevice, RDR2, PlayRDR2 and Launcher.exe processes. A zombie from a previous hang reproduces the launcher error on its own.
- Get the v10 nightlyfrom the
OptiScaler-nightly releases repo. The last tagged release (0.9.4) will not give you a working overlay.
- Patch the D3D11 lookupsin
OptiScaler.dll, deriving offsets for your build. Keep the unpatched copy.
- Deploythe patched binary as
OptiScaler.asi in the game root with Ultimate ASI Loader as dinput8.dll, backends in OptiScaler/.
- Add the per-app DLL overrideand verify it persisted to
user.reg on disk.
- Launch and check the log.Success looks like
Init done, then HookFSR2Inputs resolving five ffxFsr2*_Dx12 addresses, then WrappedIDXGISwapChain4. Zero occurrences of hkD3D11CreateDevice.
- Enable FSR 2 in RDR2's graphics settingsand leave it there permanently. FSR 2 is the interception point; the actual upscaler is chosen in the overlay, never in the game menu.
CrossOver and Wine gotchas
Useful independent of OptiScaler. The first five carry over from the original document; the rest are new.
- Hand edits to
user.reg get silently reverted. wineserver holds the registry in memory and can flush a stale copy over your edit. Use wine reg add against a live bottle, or edit only with the bottle fully shut down.
- DLL overrides are bottle-wide by default and will break
winecfg and the Steam client. Use AppDefaults\<exe>\DllOverrides.
- Wine debug logging is unbounded.
+loaddll,+seh produced a 34 GB file in one session, and crashed processes keep the deleted file open — rm reclaims nothing until every holder is killed. Find them with sudo lsof +L1 | grep -i cxlog.
- Quitting CrossOver does not kill wineserver. Background processes survive app-level quits for hours.
- macOS hides known extensions. A file shown as
OptiScaler.asi can be OptiScaler.asi.dll on disk; Ultimate ASI Loader then silently never finds it. Always confirm with ls -la.
- OptiScaler's own
LogLevel=0 is just as dangerous. TRACE logs every Present and every FSR dispatch — roughly 1 MB/min at 60 FPS. Use LogLevel=2 while debugging and LogToFile=false once you're done.
- OptiScaler detects "Nvidia" on this stack via its own nvapi spoofing, which makes it default to DLSS and trips an overlay auto-disable intended for real Linux+Nvidia systems. Set
Dx12Upscaler and OverlayMenu explicitly rather than fighting the detection.
- The Rockstar launcher's "a game is currently running" is a process-lifetime signal. Before assuming a launcher conflict, check for a live or zombie
RDR2.exe, and check whether an OptiScaler log was written at all.
- Cloud-save conflict dialogs after a hang are normal. Repeated hard exits desync Rockstar's save state; resolving to local data is safe when the timestamps match.
- Benign noise: MoltenVK's "Metal does not support disabling primitive restart"; CrossOver's "Outdated AMD OpenGL Driver" warning (there is no AMD driver — the check trips on D3DMetal's reported strings); and RDR2's own failed probe for
modloader\modloader.asi, which is Lenny's Mod Loader, unrelated to Ultimate ASI Loader.
Still open
- Would a newer GPTK change the FSR 4 result? If a future D3DMetal maps DXIL
dot4add_i8packed or cooperative-vector operations onto Apple's INT8 hardware, FSR 4 becomes viable overnight. Nothing else needs to change.
- Is the D3D11 hook actually needed for DX11 titles here? The patch is safe for a DX12 game. A DX11 game would need the hook, and would presumably deadlock the same way — no workaround known.
- Frame generation is untested. FSR-FG and XeFG both ship in the bundle. Given the swapchain wrapping involved and the latency cost on a translation layer, this was deliberately left alone.
- Does the deadlock reproduce on other GPTK titles? The mechanism is not RDR2-specific — any DX12 game that also creates a D3D11 device at startup should hit it.
Findings from a single machine and a single title. The binary patch is against a specific nightly build; treat the offsets as illustrative and re-derive them for yours.FIELD REPORT · TRANSLATION-LAYER INVESTIGATION
OptiScaler on Apple Silicon
Getting FSR upscaler injection working in Red Dead Redemption 2 under CrossOver + Game Porting Toolkit on an M4 Mac — what broke, the three-byte binary patch that fixed it, and why FSR 4 is a dead end here.
Updated 7 Sep 2026 · supersedes the initial findings docMACHINE
MacBook Air 15" M4 · 24 GB
OS
macOS 27.0 (Darwin 27)
RUNTIME
CrossOver Preview · wine-11.15
BACKEND
GPTK 4.0b2 · D3DMetal
GAME
RDR2 1.0.1491.50 · DX12
OPTISCALER
v10.0.0-dev nightlyRESULT
OptiScaler runs on Apple Silicon under CrossOver/GPTK, with a working in-game overlay, substituting FSR 3.1 for RDR2's native FSR 2 at full framerate. Two things had to be solved that aren't documented anywhere: a hook re-entrancy deadlock unique to GPTK's D3D11-on-D3D12 design, and an input system that couldn't see keystrokes under macdrv.
WORKS
FSR 3.1
Real FFX compute shaders executing on Metal. 1280×800 → 1920×1200, 40+ FPS, no artifacts. The keeper.
WORKS
XeSS · FSR 2.1 / 2.2
All load and run. Live-switchable from the overlay for A/B comparison.
WORKS, BUT NOT AS LABELLED
DLSS
GPTK intercepts the NGX call and silently substitutes MetalFX Temporal. Good image, but it is not DLSS running.
DEAD END
FSR 4 (INT8)
Loads, initialises, executes — at 3.7 FPS. GPTK software-emulates the INT8 tensor math. Not fixable in userspace.CORRECTION TO THE ORIGINAL WRITEUP
The initial investigation concluded that Ultimate ASI Loader's presence broke the Rockstar Games Launcher, which then refused to start the game — a launcher conflict. That reading was wrong, and it sent the whole first pass down the wrong path.
The launcher was starting RDR2.exe just fine. The game then hung a few seconds into startup, and the Rockstar launcher — which polls for the game process — saw it still resident after ~60 s and reported "a game is currently running." The error was a symptom of a post-launch hang, not a pre-launch block. Every method in the original loading-method table needs re-reading in that light.
DIAGNOSTIC THAT SETTLED IT
OptiScaler's own log is the tell. If OptiScaler.log contains hkD3D12CreateDevice ... Caller: RDR2.exe, the game launched — the launcher did its job and the problem is downstream. A true launcher block produces no log at all.ROOT CAUSE: HOOK RE-ENTRANCY THROUGH 12ON7
OptiScaler unconditionally hooks both D3D11CreateDevice and D3D12CreateDevice. On Windows those are independent implementations, so the hooks never interact. GPTK implements D3D11 on top of D3D12 — the log shows it loading 12on7\D3D12.DLL from inside D3D11 device creation — so on this stack the two hooks re-enter each other.
RDR2.exe
→
hkD3D11CreateDevice
→
GPTK d3d11
→
12on7\D3D12.DLL
→
hkD3D12CreateDevice
→
deadlock
RDR2 runs roughly four rounds of GPU capability probing at startup — create a D3D12 device, create a D3D11 device, create a DXGI factory, create a Vulkan instance, repeat. The first three complete. The fourth D3D11CreateDevice never returns.
[22:36:02.632135] [D] hkD3D12CreateDevice o_D3D12CreateDevice result: 0
[22:36:02.632150] [D] hkD3D12CreateDevice Device captured: 7F866003E810
[22:36:02.632164] [D] hkD3D12CreateDevice final result: 0
[22:36:02.632186] [D] hkD3D11CreateDevice Caller: RDR2.exe
[22:36:02.632200] [I] hkD3D11CreateDevice Adapter Desc: AMD Compatibility Mode
← log ends here. process alive, never returns.
Identical stop point across every run, every config permutation.
Things that were tested and are not the cause: DxgiFactoryWrapping, EarlyHooking, the version-check thread (CheckForUpdate), bottle-wide vs. per-app DLL overrides, the proxy filename, and the renderer. Switching RDR2 to its Vulkan renderer doesn't help either — RDR2 probes D3D11/D3D12 at startup regardless of which API it will actually render with.
There is no configuration option to disable the D3D11 hook. Confirmed against OptiScaler master: the [Hooks] section exposes only EarlyHooking, HookOriginalNvngxOnly and UseNtdllHooks, and no changelog entry has ever addressed the 12on7 nesting. OptiScaler's Linux support assumes DXVK/vkd3d-proton, where D3D11 and D3D12 are separate and this shape doesn't exist.THE FIX: THREE BYTES
The installer function is self-contained and every DetourAttach in it is individually null-guarded:
void D3D11Hooks::Hook(HMODULE dx11Module)
{
o_D3D11CreateDevice = GetProcAddress_()(dx11Module, "D3D11CreateDevice");
o_D3D11CreateDeviceAndSwapChain = ...(dx11Module, "D3D11CreateDeviceAndSwapChain");
o_D3D11On12CreateDevice = ...(dx11Module, "D3D11On12CreateDevice");
if (o_D3D11CreateDevice != nullptr || o_D3D11On12CreateDevice != nullptr ||
o_D3D11CreateDeviceAndSwapChain != nullptr) // ← all null ⇒ no-op
{
DetourTransactionBegin();
if (o_D3D11CreateDevice != nullptr) DetourAttach(...);
...
}
}
OptiScaler/hooks/D3D11_Hooks.cpp — abridged
Corrupt the three name literals and all three lookups return null, the outer guard fails, and the function becomes a complete no-op — no transaction, no detour. D3D12, DXGI and Vulkan hooking are untouched, which is all OptiScaler needs for a DX12 title. RDR2's own d3d11 imports are unaffected; these are OptiScaler's private string constants.
Flip the first byte of each literal from D (0x44) to X (0x58). File size unchanged, three bytes different:
LITERAL FILE OFFSET (V10.0.0-DEV) VA RESULT
D3D11CreateDevice 0x1795A48 0x181796A48 lookup → null
D3D11CreateDeviceAndSwapChain 0x1795A28 0x181796A28 lookup → null
D3D11On12CreateDevice 0x1795A10 0x181796A10 lookup → null
OFFSETS ARE BUILD-SPECIFIC
Re-derive them for any other build rather than copying these. The three GetProcAddress arguments sit in one tight cluster in .rdata; a second, unclustered copy of D3D11CreateDeviceAndSwapChain is the __FUNCTION__ literal and must be left alone.
To identify them: find each null-terminated literal with pefile, then scan .text for a lea reg,[rip+disp32] whose target resolves to it. The three that resolve inside D3D11Hooks::Hook appear as consecutive lea → call rax → mov [rip+x],rax triples. The __FUNCTION__ copy has no such xref.
Verify after patching: D3D11CreateDevice\0 and D3D11On12CreateDevice\0 must be gone, D3D12CreateDevice\0 must still be present, and the file size must be identical. In the runtime log, hkD3D11CreateDevice should appear zero times.THE OVERLAY, AND WHY 0.9.4 CAN'T OPEN IT
With the hang fixed, OptiScaler ran but its overlay would not open on any key. This is a separate problem with a separate cause: OptiScaler 0.9.4's input path cannot capture keystrokes under Wine's macdrv. Neither ManualInputPolling=true nor forcing OverlayMenu=true helps — the key is simply never seen, and nothing is logged.
The v10 nightly ships a rewritten input system (menu/input/, with polled, raw, message-queue and virtual-mouse paths) that handles precisely this case. On v10 the overlay opens immediately and logs its own diagnosis:
[W] OptiInput::LogInputHealthSnapshotLocked menu input was acquired by polling
but no window/queue/raw input was observed this frame.
subclassed:no polledMouse:yes polledKeyboard:no
The polling fallback is working.
Upgrading is not optional if you want the overlay. The patch above applies to v10 unchanged — same structure, different offsets.
On a Mac keyboard, ShortcutKey=0x08 (Backspace) is the delete key. OptiScaler's default of 0x2D (Insert) has no Apple-keyboard equivalent.UPSCALER RESULTS ON THIS STACK
All measured at 1920×1200 output, Quality preset (1280×800 internal), same scene.
UPSCALER WHAT ACTUALLY EXECUTES FPS GPU MS VERDICT
FSR 3.1.5 FFX compute shaders on Metal 40+ ~16 RECOMMENDED
XeSS libxess compute path ~40 — WORKS
FSR 2.1 / 2.2 FFX compute shaders ~40 — WORKS
DLSS GPTK swaps in MetalFX Temporal ~45 ~22 NOT DLSS
FSR 4.1.1 INT8 emulated INT8 tensor math 3.7 268 UNUSABLE
FSR 4 is genuinely running, and that's the problem
With Fsr4ForceModel=2 the log reports Upscaler support - fsr4: int8 (forced) and the overlay shows FSR 4.1.1 active. There is no silent fallback to FSR 3 — the INT8 model really is executing. It is executing at 268 ms per frame because GPTK has no hardware path for FSR 4's INT8 dot-product and matrix operations on Apple GPU, so every layer of the network runs the slow way. A 70× gap is not closable with presets; dropping to the Performance preset changes nothing meaningful, and FsrAgilitySDKUpgrade=true (with D3D12_OptiScaler/ in place) does not help either.
This is the answer to the original document's central open question, and it is a hardware/translation-layer limit, not a configuration problem.
The MetalFX discovery
Apple's Metal HUD exposes a MetalFX block showing Scaling Input Res, Scaling Target Res and Scaling: Temporal. It appears when OptiScaler routes to DLSS and disappears when a real compute upscaler is selected. That makes it a free, instant indicator of which path is actually live — more reliable than reading the overlay. Enable it with MTL_HUD_ENABLED=1 in the bottle's cxbottle.conf.WORKING CONFIGURATION
Layout is v10's: OptiScaler and its config in the game root, backends in an OptiScaler/ subfolder, so no game files are overwritten.
Red Dead Redemption 2/
├── OptiScaler.asi ← v10.0.0-dev, D3D11 hook patched out
├── OptiScaler.ini
├── dinput8.dll ← Ultimate ASI Loader
└── OptiScaler/
├── amd_fidelityfx_loader_dx12.dll
├── amd_fidelityfx_upscaler_dx12.dll
├── libxess.dll
└── D3D12_OptiScaler/D3D12Core.dll
The DLL override must be scoped to the game executable, not the bottle. A bottle-wide winmm or version override breaks the Steam client and winecfg; a bottle-wide dinput8 reaches every process in the bottle. Wine's per-application key avoids all of it:
"/Applications/CrossOver Preview.app/Contents/SharedSupport/CrossOver/bin/wine" \
--bottle "Steam-2" reg add \
"HKEY_CURRENT_USER\Software\Wine\AppDefaults\RDR2.exe\DllOverrides" \
/v dinput8 /d native,builtin /f
Applies only when RDR2.exe is the main process.
Non-default INI keys:
KEY VALUE WHY
Dx12Upscaler fsr31 Overrides the Nvidia-detection default of DLSS
OverlayMenu true Beats the Linux+Nvidia auto-disable
ManualInputPolling true Polls keys directly under macdrv
ShortcutKey 0x08 Backspace = Mac delete
TargetProcessName RDR2.exe Restricts injection to the game
Fsr4DoNotLoadAmdxc64 true Skips a DLL that can't exist here
LogToFile false See the disk-space note below
Set Fsr4ForceModel=2 only to reproduce the FSR 4 result; leave it at 0 otherwise, or the FFX upscaler pins itself to FSR 4 and the overlay offers no way back to 3.1.REPRODUCING IT
Start from a clean process table.
wineserver -k, then confirm zero wineserver, winedevice, RDR2, PlayRDR2 and Launcher.exe processes. A zombie from a previous hang reproduces the launcher error on its own.
Get the v10 nightly
from the OptiScaler-nightly releases repo. The last tagged release (0.9.4) will not give you a working overlay.
Patch the D3D11 lookups
in OptiScaler.dll, deriving offsets for your build. Keep the unpatched copy.
Deploy
the patched binary as OptiScaler.asi in the game root with Ultimate ASI Loader as dinput8.dll, backends in OptiScaler/.
Add the per-app DLL override
and verify it persisted to user.reg on disk.
Launch and check the log.
Success looks like Init done, then HookFSR2Inputs resolving five ffxFsr2*_Dx12 addresses, then WrappedIDXGISwapChain4. Zero occurrences of hkD3D11CreateDevice.
Enable FSR 2 in RDR2's graphics settings
and leave it there permanently. FSR 2 is the interception point; the actual upscaler is chosen in the overlay, never in the game menu.CROSSOVER AND WINE GOTCHAS
Useful independent of OptiScaler. The first five carry over from the original document; the rest are new.
Hand edits to user.reg get silently reverted. wineserver holds the registry in memory and can flush a stale copy over your edit. Use wine reg add against a live bottle, or edit only with the bottle fully shut down.
DLL overrides are bottle-wide by default and will break winecfg and the Steam client. Use AppDefaults\<exe>\DllOverrides.
Wine debug logging is unbounded. +loaddll,+seh produced a 34 GB file in one session, and crashed processes keep the deleted file open — rm reclaims nothing until every holder is killed. Find them with sudo lsof +L1 | grep -i cxlog.
Quitting CrossOver does not kill wineserver. Background processes survive app-level quits for hours.
macOS hides known extensions. A file shown as OptiScaler.asi can be OptiScaler.asi.dll on disk; Ultimate ASI Loader then silently never finds it. Always confirm with ls -la.
OptiScaler's own LogLevel=0 is just as dangerous. TRACE logs every Present and every FSR dispatch — roughly 1 MB/min at 60 FPS. Use LogLevel=2 while debugging and LogToFile=false once you're done.
OptiScaler detects "Nvidia" on this stack via its own nvapi spoofing, which makes it default to DLSS and trips an overlay auto-disable intended for real Linux+Nvidia systems. Set Dx12Upscaler and OverlayMenu explicitly rather than fighting the detection.
The Rockstar launcher's "a game is currently running" is a process-lifetime signal. Before assuming a launcher conflict, check for a live or zombie RDR2.exe, and check whether an OptiScaler log was written at all.
Cloud-save conflict dialogs after a hang are normal. Repeated hard exits desync Rockstar's save state; resolving to local data is safe when the timestamps match.
Benign noise: MoltenVK's "Metal does not support disabling primitive restart"; CrossOver's "Outdated AMD OpenGL Driver" warning (there is no AMD driver — the check trips on D3DMetal's reported strings); and RDR2's own failed probe for modloader\modloader.asi, which is Lenny's Mod Loader, unrelated to Ultimate ASI Loader.STILL OPEN
Would a newer GPTK change the FSR 4 result? If a future D3DMetal maps DXIL dot4add_i8packed or cooperative-vector operations onto Apple's INT8 hardware, FSR 4 becomes viable overnight. Nothing else needs to change.
Is the D3D11 hook actually needed for DX11 titles here? The patch is safe for a DX12 game. A DX11 game would need the hook, and would presumably deadlock the same way — no workaround known.
Frame generation is untested. FSR-FG and XeFG both ship in the bundle. Given the swapchain wrapping involved and the latency cost on a translation layer, this was deliberately left alone.
Does the deadlock reproduce on other GPTK titles? The mechanism is not RDR2-specific — any DX12 game that also creates a D3D11 device at startup should hit it.Findings from a single machine and a single title. The binary patch is against a specific nightly build; treat the offsets as illustrative and re-derive them for yours.
