What's up with the servers?


  • Moderator

    Due to COVID-19, there have been a lot more people playing Paladins over the past couple months and there's also been a drastic increase in internet usage with people stuck at home and/or working from home. There has also been a steep rise in the number of players encountering matches where their ping looks fine, but they're getting high packet loss which feels similar to having 1000+ ping, so I wanted to give a detailed explanation of the status for each region.

    Europe
    When we started investigating reports back in March, we found overloaded internet nodes around our London and Amsterdam data centers, but we noticed no such issue around some temporary servers we stood up in Frankfurt. As such, we decided to try moving all of our server capacity from London and Amsterdam to Frankfurt. As soon as we did this, we found overloaded internet nodes around the Frankfurt data center. This lead us to the conclusion that we need to spread our servers across multiple data centers to at least reduce the chances of encountering overloaded internet nodes. We've continued to receive reports of laggy matches on every server across all data centers, but analyzing all of the matches being played on these servers, there are a lot more matches on each server that are not encountering lag than ones with lag. It would appear that there's intermittently too much internet traffic being routed through certain internet nodes leading to players encountering packet loss along the route to our data centers.

    NOTE: EU experienced a 24 hour period of things being much worse on May 14th to 15th, details can be found here: https://forums.hirezstudios.com/paladins/topic/3605/to-our-champions-regarding-servers

    Southeast Asia
    Over the past few years, numerous SEA players have intermittently experienced laggy matches which have been linked to various ISPs and/or overloaded internet nodes. Over the past couple months, more players have been consistently experiencing high packet loss during matches and our investigations have concluded there are more overloaded internet nodes than previously seen in that region. SEA seems to have a larger number of internet nodes spread throughout the region that are intermittently getting overloaded when compared to what we're seeing in EU.

    South America (aka Brazil)
    This region is in a very similar state as SEA. This region has a history of intermittent laggy matches and it's been exacerbated by the increased internet traffic. Investigations have drawn us to similar conclusions of internet nodes spread throughout the region being intermittently overloaded.

    North America
    Other than a bad data center and a few bad servers, NA hasn't experienced the same issues with overloaded internet nodes causing intermittent packet loss. I'm sure some players have experienced some issues here and there, but generally packet loss issues are not plaguing this region. To be crystal clear, we are using similar data centers and servers in NA that we use in other regions. This is not a situation where we're concentrating our efforts on this region. In fact, we've spent several times as many hours trying to investigate and resolve issues in EU and other regions than we have for NA.

    NOTE: NA experienced the same issue as EU during a 24 hour period where things were much worse on May 14th to 15th, details can be found here: https://forums.hirezstudios.com/paladins/topic/3605/to-our-champions-regarding-servers

    We know that our players are suffering and I've read plenty of comments saying that we simply don't care. This couldn't be further from the truth, but I honestly don't know what else we could possibly do to resolve this issue for you all. We don't control the internet nodes between your house/ISP and our data centers. We also can't change the route that your packets take from your house/ISP to our data center. If there's a bad node in the middle dropping packets and not pushing them along the route, it's simply not within our control to fix. I know that this isn't the answer that you came here looking for and I wish that I had a better answer, but know that we've done everything that we can think of to combat the increased internet congestion that COVID-19 has caused.

    by @Xienen


Log in to reply