I had the same and managed to fix it for me.
I had the same issues since yesterday, right after updating my graphics driver (I hadn’t updated in like 3 months). I had no issues at all before then, but right after updating the driver I had exactly these issues like OP: freezes when opening inv or world map, random 5 to 10 sec freezes and crashes having to force close the game etc.
I then went to a bit earlier driver, but still had the same problems. So I just went back to the earliest driver I could get from Nvidia (I have a RTX 2070 Super), and it completely fixed all the issues. I haven’t had a single freeze, crash or anything at all since then. The driver version I went back to is from 13 April 2023, 31.0.15.3168, though it probably doesn’t need to be that far back.
Maybe this helps some of you too.