Server Lag Update

  • Moderator

    Based on the responses to my post from last week, we have thrown out the results of our previous investigation, made a few server configuration changes last week, and have continued to explore different angles of analysis.

    So, what changes did we make last week? We've brought even more data centers online and continued to increase the number of servers available for players, which seems to have improved things, but has not entirely removed the packet loss issues. We've also tested out some super expensive servers that have fewer but much faster processor cores, but that didn't seem to fix it either. While we didn't expect that to be the issue, we figured it was worth a try, despite it being an expensive test.

    In order to continue our analysis, we need players to give us any information you have available. In my last post, players started pointing out that they experience lag during off-peak hours in the EU where previously we had players reporting that this issue only happened during peak hours. We also had players pointing out that they don't experience lag issues in training matches, which players had previously stated only to have others disagree. We've also noticed that players reporting this issue all seemed to be playing on PC (or in a party with PC players). Please feel free to share anything that you believe may be relevant and please include which region you're playing in!

    If you just want to complain about the servers still sucking, that's helpful too as long as you include the region! A little extra detail would go a long way, though! Do you place on PC or console? Is it every single match or just most of them? Do you also experience lag in training matches? Has it gotten better or worse in the past week?

    I've managed to get some super smart people to start looking into this issue with me, but even they are scratching their heads trying to understand what's going wrong. Do know that we are trying to figure this out, because it's a critically important problem to solve, but whatever is happening is definitely not easy to track down and fix.

    by @Xienen

  • @Lukash369

    Are the new servers using VMware vitulization connected to a isci storage solutuon?

    Delayed Ack is an issue that causes high latency during low IO periods and performance improves with high IO.

    Since all this occured after the new server tech i would assume that there is a configuration issue some where, my be several things nic bindings, flow control, or a compatibility issue with nic drivers and the version of Esxi.

    Kinda shooting in the dark with no info on your hardware and network topology.

  • @Lukash369

    If a customer called me with these type of end user complaints i would check a bunch of best practice settings and the use IOmeter to isolate the issue to hardware or virtulization. One you figure out which of those teo stacks you should be focused on its is easier to figure out. Or you find out there is nothing wrong and have the testing results to push an external resource to fix thier stuff.

  • No lag for me for quite some time on EU. Works fine.

  • i still have lag in eu and teleport after mid update

    my internet fine
    alt text

  • Moderator


    Also, "my internet is doing fine" is a misnomer, as that's not how the internet works. What you're saying when you say "my internet is doing fine" is "I'm able to connect to my ISP and perform various internet tasks without issue", but everything that you do on the internet goes through a series of internet nodes (computers that route traffic around the world), so the nodes that your internet traffic takes to/from reddit's servers are entirely different than the route your internet traffic takes to/from Netflix's servers (and Paladins instance match servers). Also the majority of activities on the internet such as reddit and Netflix are much more tolerant of packet loss than playing real-time online multiplayer action games.
    Some players have seen benefits from switching to a different region than the one they live in, because the issue is somewhere along the route between their home and their region's data center and they are able to sidestep that by connecting to a different region."


  • PC

    @Lukash369 @Xienen

    Let us see the net stats - something like this:


    This would help players choose a better region if there is in fact a path issue.

    Maybe some sort of test option could be added into the region selection so players could determine a suitable region before ruining matches.

  • @Lukash369 im not english speaker ^_^

    With my respect, I do not believe you, because many players complain of the same problem

  • @Vaporized that will be great

  • PC


    It would help, now get them to actually do it 🙂

    BTW, it's not a lie that what is being described can happen. It does happen and it's actually quite common.

    You can test for this problem yourself using WinMtr (Free Opensource).
    You need the IP address of the Paladins server or at least in the same data center to get the best results.

    Here is a sample test from USA to some place in Germany. Some nodes along the way don't respond to pings (100% loss) - this is normal. Notice the last point in the route has sent/received that matches the first - no packets lost at all. This is what we want.


  • Moderator

    @Vaporized Idk is it possible on UE3 engine.

    @abukhalid2011 These are not my words, but I added them because I think they make sense.

  • PC

    @Lukash369 said in Server Lag Update:

    @Vaporized Idk is it possible on UE3 engine.

    Seems it has something. Commands

    STAT NET - toggles on/off (inter)net(work) statistics display


    In system commands there is this: /stat fps Toggle the Frames-Per-Second indicator. You can also press F9 to toggle this.

    I can't try it now but does /stat net work?

    Edit: Both fps and net do not work using that method.

  • I've been playing in NA on a fantastic internet service and it genuinely feels like every third time I return to Paladins, the lag/latency/crashing/drops issues reach new lows. The other end of the spectrum is every third time I come back it's not... Stellar, but there's no noticeable problem beyond the usual online gaming experience of just a hiccup here and there. The other third is somewhere in between those two, which makes the experience.... 2/3s of the time a bad experience. Bad enough that I think this might be my first post in several years on Paladins forums, if I ever have before >.>
    If the netcode won't hold up at a large scale, you have to redesign the netcode. If it scales, then you just have to keep scaling the hardware and such accordingly. I hope y'all figure out which it is, cuz this is getting obnoxious. ESPECIALLY for those of us who actually use skill-based characters, when we're up against auto-aim characters.

  • I don't know anything about informatics but I really appreciate you all work together to solve this problem. I encourage you ♥3♥

    Have a nice day sweets

  • So for example, tonight I played from around 10:30PM EST to 11:15ish, 3 matches. NA servers. PC. I didn't time it exactly, because I was more focused on playing the game. But I CONSISTENTLY get the voicelines for the start of the match a whole second late, seeing others already going through the doors, and approximately consistently every 10 seconds or so, I lose at least a half second, up to several whole seconds, and twice, the Paladins application MAJORLY tanked and froze up at one of those seconds of freezing, to the point that Windows thought the application itself was frozen and needed to be force quit (or wait for response, which worked after a few more seconds). SOMETHING that changed with Paladins is wreaking ABSOLUTE havoc with PC installations. Did a Windows 10 network driver or service update occur that y'all need to fix up some better compatibility for? If you aren't seeing/replicating this issue on your own computers or test cluster or such things, do you have a bunch of the software that your average player might have installed? Have you tried having your most expansive-steam-library-possessing technicians or employees attempt to install and uninstall (to not need 10TB of disk space) every game in their library, to see if there's some framework or service installed by any common games that are being hugely incompatible? Can you creat a survey or poll where we can submit our SteamGauge profiles, or a list of general software installed on our PC, so that you can see the common denominator between all of these issues? I stopped playing Overwatch in part because I prefer Paladin's gameplay, and also because of the Hong Kong Protests Blizzard blatantly stood against (Blitzcheung), but this incessant lag is the kind of thing that will make me give up on your "Hero Shooter" and go back to Overwatch or try Valorant or something! Premium currency that paid for a battle pass is doubtless going far less used for all of us PC players because of how unplayable and unenjoyable this is. This kind of ongoing, escalatingly bad performance issue is what will lose your paying customers.
    Sorry it's a rant, but I really want this game to be playable rather than horrendous, and want you to consider different angles or solutions I haven't noticed mentioned in this thread.

  • The BR servers are having a ton of lag, maybe adressing that would be nice too, they probably have the same issue..

Log in to reply