PacketFlow - For Ashita And Windower

Now announcing, PacketFlow for both Ashita and Windower!

Players have been having persistent issues with "lag" causing messages to drop in instances for years. In reality, this "lag" can be decomposed into two separate problems:
1. Chunk prioritization is messed up in instances
2. Instances generate a lot of chunks

PacketFlow attempts to address the second problem.


For a brief overview of FFXI network dynamics and terms, I recommend reading the spoiler:

Prioritization
In non-instanced zones, SE appears to use a prioritization system to make sure that your client receives more important chunks. If you pull all of the monsters in Korroloka Tunnel at once, for instance, and cast Phalanx on yourself, you will always see the "gain the effect of Phalanx" message. More generally, chunks containing information involving your character are prioritized over more contextual chunks (like monsters moving around). Most of the time, this prioritization system does not matter because FFXI is sending you less than 2.5KB of data and so everything makes it into the UDP packet. However, sometimes you're in a very active environment and there's more than 2.5KB of data to send you.

You can experience this when by pull multiple monsters. If 5 monsters are chasing you, they typically move pretty reasonably. If 20 are chasing you, some monster start to jump between locations because the chunk that updates those monsters' positions have low priority and do not fit in the packet. However, even with 20 chasing you the messages about their attacks are generally included (and fill up your chat log). This prioritization system lets FFXI get away with a pretty low bandwidth requirement, and the fact that most of us have never really thought about it indicates that it works pretty well.

However, in instances the prioritization system is at best nonfunctional or worse is functioning in the opposite way from normal zones. You will frequently miss messages about yourself in a Dynamis - D instance (buff gain, action completed, etc.), that you would never miss in a vanilla Dynamis zone. This is all server-side, so we can only guess at the specifics of the packet prioritization system and how it's messed up in instances, but it's clearly not working as it was designed to 2 decades ago.


What are solutions here?
1. Minimize the number of chunks you are sent: fight one monster at a time, turn off party buff information, don't use GearSwap, etc. Anything that decreases the number of chunks you are sent for FFXI is going to help you avoid missing chunks, because prioritization doesn't matter unless the packet is full.
2. Increase the server -> client bandwidth.
Option #1 sucks and is very limiting, so option #2 is the interesting one.

Bandwidth is more or less max packet size * frequency. For the FFXI server -> client connection, this is (2.5 kb / 1 packet) * ( 1 packet / 0.4s) = 6.25 kb/s most of the time. We cannot change either of those parameters directly because they are part of the server software.

However, years ago I noticed that the server responds to any UDP packet it is sent with another UDP packet. Thus, we can indirectly control the incoming UDP packet frequency by adjusting the outgoing packet frequency. This idea came to nothing for the better part of a decade mostly because I had no idea how to trigger the FFXI client to send UDP packets more frequently.

I was reminded of it earlier this year when KnugenJulian started posting in the Windower Discord asking if anyone had a fix for Odyssey lag. I brought up my purely theoretical fix, and Thorny made it real by connecting it with atom0s' UDP packet handler reverse engineering.

Thus, there are now plugins (both named Packetflow) for Ashita and Windower that decrease that time-per-packet from 400ms to 250ms, effectively increasing bandwidth by 60%. They do not completely solve the problem of dropped chunks in instances, but they do make it better. Additionally, they make loading zones quite a bit faster.
So it works and that's the theory, but is it safe?
This is a major concern. As with all 3P tools, but perhaps even more here, you risk your account when you use this plugin. Packetflow is unambiguously detectable and increases the outgoing bandwidth costs for FFXI. Thus, for Windower it is in the Konami code section of the launcher.

As far as due diligence, we actually more or less finished these plugins in mid-April and I have been using the Windower one on my main and mule for the last month ~24/7. I'm not banned yet. We don't have a way to test it better than that and that's all of the testing that has been done. Caveat Emptor!


Attempts have been made to reach out to SE about these problems and to get a temperature read on the proposed solution:
* 2016 AMA - Bandwidth - basically "it would be too hard to fix"
* 2022 AMA - Prioritization
* 2022 AMA - Bandwidth
We waited a week for SE to post their hypothetical 2022 AMA forum follow-up where they answer those questions, but no dice.


Credit to:
* Thorny for the implementation and empirical testing
* atom0s for the reverse engineering that made it possible
* KnugenJulian for starting the conversation that revived the concept

Huge deal. Game changer for sure. Gj

Imma wait like a month

If only to see who gets banned for something they aren't talking about and blames it on this

Then I'll wait another month and try it

Continuing with Byrth's point about increasing SE bandwith costs, keep in mind that is something that would scale with the people using so waiting probably does nothing but create a false sense of security. It could be after 3 or 4 months enough people feel it's "safe" that its usage becomes more common and we hit a breaking point that causes SE to take notice. Obviously I have no idea what that level is or if they're even paying enough attention to ever notice.

Does this change the time per packet full time all the time, or just when in an instance?

If full time, could that be a feature? Either only upping the frequency while in an instance, or upping it when needs, if you start getting close to cap, increase, etc.

It seems to me like this would help make this slightly safer, as you'd only be displaying unusual communication behavior when it will actually benefit you, rather than all the time.

Nothing stopping you from turning it off and on. If that was a concern.

(A fine question/suggestion, just saying you can do it manually too)

Thanks Byrth, amazing guys like you are fixing the games issues without much support from SE.

Thanks for this. I'll be a guinea pig for this as if I get banned i'll be free from 11. If I don't get banned the game will be more enjoyable. Win/win

Please someone sticky this thread.
Thanks to Byrth and Thorny!


Question: can you load the addon anytime for it to work or do you have to load it before entering a zone? (or instance)

Does this mean I'm not going to be scolded anymore for not curing when people are in the red and on my screen they're still at full health?! (at least not as often)

edit: i'm still the last post so just gonna edit.
would make below less of an issue for SE.If people start using it all the time, that's a lot more traffic for SE. If by default it is only active for problematic instance zones, then even if everyone started using it, SE would probably only see a minor uptick in traffic.

This will be amazing if it makes Dynamis D actually playable… Seems very high risk though.

It changes it all the time, and that could be a feature. We would need some way to identify instances (perhaps just a list of zones that are always instances), but that seems very surmountable.

Though it was just a throwaway line in the original post, PacketFlow does also decrease the time it takes your inventory to load after zoning by about 50%. The delay that I say is 400ms in the OP is actually dynamic and resets to closer to 1.5s when you zone, decreasing over time, so when you need the packets the fastest (zoned, need to repopulate all your data structures) you're getting them the slowest.


You can load or unload it whenever you want.


Yep! That's the problem we're trying to fix.

I'm still amazed why SE has not solved this. Play style is so different now, pulling 10-30 mobs is common. And the freaking lag in dynamis is horrible.

This is wonderful! What is the consensus with multi-boxing? I assume if you want it to work on all, run it on all? Great work guys!

Yes, it will need to be run on each character you want to use it on.

Surmounted! Another version has been released that only increases bandwidth in instances. If you want the previous behavior, set the `instancesOnly` key in the plugin's settings.xml to be false.

Keep in mind that, as far as we can tell, the additional traffic consists primarily of packets that would've otherwise been discarded. This increases the frequency with which communications occur, but it won't cause the server to resend packets they wouldn't have otherwise resent. There may be edge cases where a bit more traffic occurs(if other players are continuously moving, they may end up with more position updates in a given time period because there are more opportunities to update their position). The real additional traffic will typically not be anywhere near the 60% bandwidth increase, this is only a maximum.

We had considered both the zone-based limitation and a saturation based limitation, but given the benefits to zoning and the likelihood that this is just overthinking impact on SE in the first place, we opted not to in initial release. If SE were to make a statement of concern or acknowledge in any way, then we would absolutely reconsider. But, given the very real possibility that this is not even noticeable to them, it didn't seem worth handicapping it out of the gate.

Is this increase in "flow" trackable to an IP or a specific character? How targetable does the player become?

Sorry, newb on this stuff.

It can absolutely be tracked to a character, and they can absolutely get an IP from a character. If SE were to decide they wanted to track down and ban everyone who used this, that would not be difficult for them to do. This is not a risk you should take lightly.

With that in mind, Byrth has been using it for over a month 24/7 as he stated in his post. I've used it for several day periods on various characters to do testing. Whether it becomes a problem, or a problem worth banning over, is completely up in the air. We would not release it if we believed that to be the most likely outcome, but everyone is responsible for their own risk assessments.

Thank you!

Does this also help with input lag (IE, you issue a command and it does go through, but it takes like 1-2+ seconds to initiate)?

Honestly, I've never observed 'input lag' being a thing. Is this something you've observed under controlled conditions? I'm imagining the circumstance is more along the lines of the server immediately starting your action, but taking a few seconds to display it. If that's the case, then yes, it would help.

It shouldn't change the priority(if player characters load slower than home points, they would continue to do so), but it makes zoning in general much faster. If your concern is how quickly the home points load, rest assured this will make it HUGELY faster. If you specifically want the order changed, then it won't do anything for that.

No, never under controlled conditions. For me, it only happens occasionally and in Odyssey and Dynamis-D, but it's pretty unpredictable and does not seem to be solely related to how many things are happening around you. In one run it'll happen to everyone (or almost everyone) in the party to varying degrees regardless of how much action is going on, while in another identical (from the perspective of the players) run, it'll be fine. If you were playing at the time, the initial release of Gaol had the same problem happen every single time (at least for non-JP players), except more severe.

Maybe next time it happens to me I'll keep an eye on recast or TP to see if it's delaying the actual action or just the displaying of it.

Unfortunately, recast and TP are both set by S>C packets(recast is actually the same packet that shows the action and sends the log message/etc), so if server is delaying communication but not processing, neither is a valid measure here.

The only 'real' measure is action execution relative to things that would happen without input, such as mob delay or poison tick(casting cure on someone who is going to die in 2 seconds of poison, for example). It's not an easy thing to test or verify. May be more practical to just see if it 'feels' better with the plugin than without, but I do strongly suspect it is not actual input delay.

If it was solely a display thing, I was thinking more that I might be able to see recast or TP "jump" as the server or client catches up, as input lag varies a bit in severity over time in such runs.

Input severity also varies a bit between players within the same instance, so different people would likely see it executing at different times if it's just display, but at the same time if the actual action is being delayed.

I've gotten to where in C and Dyna runs, I'll monitor what EquipViewer is doing to see if the requested action is happening. Particularly tanking in C, if I'm pulling 2 groups of mobs, it's rare that I see the casting animation triggered, but the change in gear followed by the action seems pretty consistent.

So for instance if we have two groups and I cast a cure, the casting animation rarely seems to start until the spell goes off, but I can see in EquipViewer that I'm in my SIRD set. Someone with familiarity w/ the packet handling code might be able to comment on whether or not this is actually accurate or not, but the behavior seems pretty consistent for me in most cases (e.g. I see SIRD set, spell ends up going off even if animations aren't happening). I'd assume there is some acknowledgement of these gearswaps coming from the server to indicate the swap happened, but I don't know concretely, it seems equally possible that EquipViewer is seeing the outgoing set change and assuming it's correct.

That said, I've had problems where I get stuck in sets, although it's typically when fighting Mireu. I've seen it happen in Odyssey content, though, and it appears unrelated to monster actions, but it's infrequent. All that being said, I've had a few runs lately where there is absolutely some kind of input delay: sets don't swap, action doesn't go off, and recast doesn't change. There was one run in particular where everyone seemed to have that problem, but again, it is uncommon for me at least.