Diablo 4 - Peripherals losing connection when connected to a USB-switch

This is the only game where I have this issue. My peripherals(mouse, keyboard, headset(wireless)) are losing the connection for 1-2 seconds every 1-5 minuts while in game. It seems to be because of the USB-switch.

Is it because of a power draw issue?
Because when I unplugged 2 devices it got better.

Anyone else experiencing this issue?

Note: other games work fine while my peripherals are connected to the USB-switch.