Sanctum of Domination Lag Update

For the past several weeks, we’ve received and acted on reports of lag in the Sanctum of Domination raid, particularly during the Sylvanas encounter. We’ve been working hard to understand and resolve the issues, but we overlooked telling you that. We apologize for that oversight, and we will strive to do better in the future.

The most common complaint we received was that there was a burst of lag when Sylvanas is first engaged. Our server performance group dug into those reports, and we found the cause – there are a higher-than-usual number of calculations being done at the start of the fight to facilitate the complex mechanics of the encounter. We’re looking into ways we can potentially improve this, but we have to be careful to not change the design or mechanics of the encounter. While this is something that we think we can improve over time, there will still be a few seconds of server lag on pull for now.

Outside of that moment, our main challenge in diagnosing these lag reports came from the fact that lag was reported in almost every phase of the fight, and we were unable to reproduce that lag in our test cases. In addition to examining our own internal testing, when we utilized our performance analysis tools on actual live raids that reported they were experiencing lag, the server always appeared to be functioning smoothly. This guided our focus elsewhere.

What we eventually discovered is that the Sylvanas fight in particular is very messaging-heavy, the mechanics involved require sending many frequent updates to the player’s game clients, which in itself isn’t a problem. However, we discovered that an unrelated change to how we organize and distribute server processes across our server hardware, combined with accumulated changes over time to how our hardware server infrastructure is organized, resulted in our server processes being more concentrated across our hardware.

We believe all of these factors combined to make the messaging overload happen more commonly and cause lag.

With our most recent weekly maintenance, we’ve deployed several changes to how we distribute server processes across our infrastructure, which should alleviate this potential source of lag. Please let us know about your experiences in Sanctum of Domination going forward, especially if you can capture video of any new misbehavior.

Thank you very much for your valuable feedback!

2 Likes

Thank you for the update, but there is also extreme lag across the Maw / Korthia when the world boss is engaged, which can render the game practically unplayable at times - I had this occur today after maintenance so this aspect does not seem any different at least, assuming it is impacted by a similar issue. Maybe it is different. The last time I remember performance this bad was that little thing called the WoD launch. I’m sure you guys remember that too.

I am also very dubious given the general quality of content for considerable time now, that this was simply a case of “we tested before shipping and it worked fine”. For several years now the content has gone live with numerous bugs, some of which completely break things, and the cutbacks to QA have been very noticeable. Ultimately this is not acceptable, and another thing that drives players away from the game, along with cutting support staff to save a few pennies.

But assuming it was tested adequately internally, why was this not caught when it seems such an easy repo on production servers? This would seem to suggest something is seriously wrong with the testing methodology and it does not come close to emulating the production environment.

Content should not be pushed to live as a badly performing buggy mess. Invest in your QA staff. Your engineering teams need to figure out exactly why this was not caught in testing, and seriously improve the methodology to reflect real world usage and ensure this level of performance never occurs on live again.

Not sure if it’s helpful but I recorded my wing 4 LFR yesterday (after reset / updates I assume) and what I often have is slight stutter of other players on the bridges (sometimes also last phase). But a few weeks ago it was a lot worse and I kinda got used to it.

Edit: It’s in every start of the 2nd phase, first bridge and sometimes on later bridges or phase 3

When we first enter phase 2: 5:40
On the bridge 7:14 (mage with frost barrier)
Second try 2nd phase: 13:51 on the first bridge (tauren tank later mage)
3rd try second phase: 27:15
4th try 2nd phase: 37:41
and 3rd phase: 43:18 (hunter on the left side)

Will also copy these timestamps in a comment on youtube so you can click on it.

In an NHC run the same day later I didn’t have it though,

while you at it fix the stutters when opening the map and inside torghast when the power is ready! the stutters in revendreth and bastion ,periodic stutters since patch 9.1.

im going to unsub this month if this stutters in dx 12 are not fixed.

reported this many times since patch 9.1 broke the game with stutters and nothing was done in months.