Some of our servers ignore or deprioritise ping requests as a protection against DDoS attacks, so it’s not uncommon to see either poor or no results, and those aren’t indicative of a problem.
The fact that it works properly via a VPN suggests there is some throttling happening somewhere along the route, as VPNs typically use different routing to what your ISP defaults to, and is something you would need to reach out to them about.