PC Server Lag Improvement Released
We know the packet loss issues have been frustrating for our players, particularly PC players in Europe, but we pushed out a small change last night that should at least improve things for everyone. For those that haven't followed along, about 2 months ago, we posted saying that we believed congested internet nodes near some of our data centers were causing issues in the EU. Then about 3 weeks ago, I posted an explanation based on our continued research into the issue, which continued to point toward intermittently congested internet nodes all over Europe. The replies from you all, though, made it clear that there was something else going on, namely that some of you play multiple other games that don't have issues, that the issue also started happening during off peak times, and that it happened more often at the beginning of the match. So, I got some of the best and brightest people at our company involved in researching, but after two weeks, we couldn't find the pattern in your reports that we needed to nail down a solution. So, I posted again, asking for your help and got all sorts of good data to analyze.
The first theory that I investigated was that we made a change in the Sands of Myth update that caused the issue. So, I ran a full diff of the Sands of Myth code against the Tigron's Tale code and found nothing of interest. Undeterred, I then used Wireshark to graph the number of packets and number of bytes that we were currently sending during a match. I then spent a day standing up a server running our final update from last year, The Damned Frontier, on my personal computer that I could connect to from my work PC (thankfully we're working from home, so I always have 2 PCs sitting right next to me). I did a Wireshark capture of that client to compare against, but found that if anything we're sending less packets now than we were at the end of last year. Then one of those smart people at the company suggested, "since SMITE doesn't seem to have this issue, do a capture of it for comparison". Turns out that SMITE is sending a little over half as many packets from their clients to the server as Paladins. So that leads to the next obvious question: why?!?
We started comparing some of the shared code between SMITE and Paladins, but none of the relevant code had been changed between SMITE and Paladins. Eventually, though, we found a piece of code that has existed in the engine since before we even started making Paladins that said "if there are 10 players or less, send 90 movement packets per second, otherwise send 45 movement packets per second". Sure, fine, but SMITE also caps out at 5v5 matches, so they'd also be sending 90 movement packets per second, right? That was when that same smart person said "well, technically all of the other things in SMITE are treated as players: minions, towers, phoenixes, and titans". Okay, fine, but why are some Paladins players only experiencing packet loss at the beginning of a match and/or the beginning of each round? Oh, because Paladins also has some champions with certain abilities that are also treated as players, such as Ying's illusions. That means, most of the time, Paladins is sending 90 movement packets per second, but if any of those champions are in the match and use any of those abilities, every client would immediately shift to sending 45 movement packets per second. So, we looked at some of the old server data from known bad matches (thanks to you all sending us a mountain of bad Match IDs) and the pattern fit 100% of the time: packet loss until an 11th "player" showed up, then the packet loss disappeared until the 11th "player" was destroyed and boom, packet loss spikes right back up.
And since this code has existed in Paladins forever, we know that Paladins clearly doesn't even need to send 90 movement packets per second for players to have a smooth experience, this was just old code that was never previously problematic. Also, since the engine never sends more than 1 movement packet per frame, console players were never sending 90 movement packets per second anyway, which also explains why this appeared to only be affecting PC players. So, the PC client update that we released last night basically set the cap down to 45 movement packets per second.
Let me be clear, we don't expect this to fix every server issue that players have been encountering, but we absolutely believe this will at least improve things for the players that have been experiencing packet loss issues. So, if you're still encountering issues, please let me know and we'll look into it.
M3RC3N4RI0 last edited by
Small cause, big effect, as we say in German. Ying illusions, Luna, what other deploy is treated like a player?
Sodomantis last edited by
Okok. Sounds very interesting, lets play.
I played some training and 2 live just now. Seems there is even less damage feedback than before. I just die from what seems like nothing. I need to update my forum name to TotallyVaporized now. Seems like there is something not right with the sound too. I will have to play around with it to see if I can figure it out.
AYYDIMITRI last edited by
Probably anything with an attached HP bar, or auto tracking turret, any deployable with an AI script
In game dev they make some player like entities in order to create an independent body with its own function in game, essentially an npc body that has some base AI to carry out its role, pretty much like a lesser bot.
I would gather almost any placement ability, Luna, Ying clones, Barik Turret, maybe all deployable shields and totem objects as well.
I remember a time when grover heal would heal every npc player like ability that is shouldn't have, including the player objects such as turrets and shields.
schachmaty last edited by
Everything was fine yesterday but now it all started again
vip3rh3ro last edited by
@schachmaty there is a scope sensitivity bug for all mouse and key users that resets your scope sens to 0.0 even if you click apply
I cant play any ads charecters because my prefered scope sensitivity is 1.0
Please fix this bug paladins
321k9 last edited by
Hey, I am very thankful you put so much investigation into the issue and I appreciate the transparency. So I know you are doing your very best to understand the lag issues.
Unfortunetly, my feedback is (from Hamburg, Germany), that since 4-5 days the lag experience is in 50% of the mathces even worse for me. Partly extremly high ping like 2000-15000 ms. And even when Ping shows only 50ms, it feels like swimming in muddy lag. That is new. (I use different internet lines, one is DSL, one mobile tethering. Both connections have these problems). Any ideas left?
madhuri87 last edited by
Thanks for sharing the info...
M3RC3N4RI0 last edited by
Any ideas left?
You can try another game server, the US server for example.
U1319683871 last edited by
Haaaa. I'm experiencing this issue hardcore on a fantastic ISP between Ann Arbor and Detroit in Michigan, playing on the NA region, at any odd hour of the day. Every 10 or so seconds, last night when I played 3 matches, I'd lose a solid second AT LEAST. If I was playing something like Maeve I would have been royally screwed, thankfully Raum can just spray and pray and tank hits. Forreal though this is worse than when I played a few weeks ago, and it was pretty bad then too. You can try NA servers, but respond back if you're having the issue there too, like I am.
kr1pteya last edited by
No updates for the BR servers?
U1319683871 last edited by
Last night at like, 2AM, was shockingly smooth after the last few horrendous days. I'm noticing something that's happening upon launches is that Paladins is claiming a version mismatch and demanding I attempt to reconnect. First match, as soon as we transitioned from picking characters to beginning to load the map, crashed to desktop. Didn't have a laggy match after that event occurred this time though.
azazelecho last edited by
I'm playing in the EU region and in the past three or four days I'm experiencing the same weird lag we had at the beginning of the pandemic, the ice-skating, people teleporting around and the latency constantly going up and down. It usually starts in the evening/night or afternoon and it only stops and goes back to normal the next day in the early morning.
My network is fine, since I can play any other online game without any sort of issues. Another weird thing is that if I switch region to NA or Asia I can play Paladins without this weird lag, I only get higher ping but without the ice-skating or people teleporting around or crazy latency ups and downs, so it looks like only some regions of Europe have this weird lag at the moment.
I was wondering if anybody has any up to date information, if the devs are aware of this problem and if they are actually planning to solve it.
azazelecho last edited by
@Lukash369 Thank you, next time this happen I'll use the form.