In Game Macro Lines Randomly Disappearing

I've discussed about this in the past but we never came out to a consensus on what was causing it.
Recently I had a couple of friends experiencing the same issue, so I wanted to see if others experienced it and if a cause was ever found.


Description:
Sometimes whole macrolines disappear. When that happens, it's the whole line, all 20 macros (10 alt + 10 ctrl) in that line.
I've seen it happen on the slot 0, sometimes on other slots. If there's a pattern on which slots are more likely to disappear, I didn't notice one.
It doesn't happen in all of my macro books, but only in some.


Odd occurrence:
When this problem happens, if I try to download the macroset (working, with all lines okay) from the FFXI server which I uploaded weeks before, the download happens but the disappeared macro lines do not re-appear.


Solution?
The only solution I found so far is using the FFXI backup in-game data tool. Backupping the whole FFXI/user folder might work as well but I never tried.
If I restore a saved set of settings with the official config app, then all my lost macro-lines are correctly restored.


Possible causes?
I never understood what causes this, but my feeling is that it has something to do with the very very few times when I dualbox logging a friend's char.
When that happens I usually logout the friend's char before mine (and everything works) and later, when I logout my own char, I get back to Windows 10 with a black/stuck POL window that's supposed to disappear but instead stays there permanently and I have to close it manually.
This always happens 100% of the times I log a second character and logout the other char before my own.
I'm not sure this is what's causing the macro-line disappearence but I have a feeling it's somehow related.

I found this happened to me when quickly changing jobs where there was a macro set/book command in my gearswap. The fix was to remove the code from my lua and manually change macro book/sets. I never figured out the specific way to retain that automation. It was only certain luas/jobs/characters.

Like changing jobs swiftly to buff and reset JAs during odyssey at the moogle for example.

I've had this happen to me twice. Unfortunately I can't really corroborate any cause, this was months ago.

First time, my entire COR Book got wiped, or at least all 5-6 pages I was using in that book. The second time, very shortly after, I lost Page 0 only in my SCH Macro book. Dualboxing might have been the cause, I will use a defunct account an old friend dropped when they quit, but only during free plays, and both of these happened close to each other, it absolutely could have been a few months ago during the last free play. shrugs

I agree with Paulu, the few times I've had this happen it was always when doing rapid macro set changes. For me it's always been a single line that's disappeared, not a whole book. Since I added a 2-second delay between each swap, I've only seen it happen once.

You can obviously back up the macros to the server, if you remember to do it on a regular basis. Personally, I created a private GitHub repository, and back up the entire user folder any time I make a major change to my macro layout. You can automate that process as well, using a cron job/scheduler task or similar.

Had this happen a few times like others have said when quickly changing jobs/subjobs. I assumed it was something wonky with the MacroChanger addon or w/e it is. Never had it happen again after changing slower though.

I had this happen but I feel like it was after a bunch of crashes due to internet or when some addons kept crashing me until I fixed them.

I do have a code (it's a TXT script that gets executed whenever a new job lua is loaded in Gearswap, which means I just swapped main job).
This code is a simple in-game command to swap book and page. (no addon used, just a windower4 txt script with a sequence of regular in-game commands)

I'm somewhat skeptic this is the source of the issue (that each time you do that a bit too fast there's a chance some page in some books will get deleted) but then again I can't rule it out entirely.


In my experience this happened to me very few times, and each time I can remember having dualboxed a few days before.
Now this could be entirely a coincidence, but somewhat I've been led to think there was a connection between the two things.



There's also the strange thing that whenever this happened and I tried to restore macros from the FFXI server, the "deleted" pages would remain such, even though when I uploaded macros on the FFXI server those lines were perfectly there and, as such, should logically be on the server.

This pops back up every now and then, and the issue is exactly caused by changing macro book/set too fast.

What happens is that FFXI's macros live on your hard disk, and each book/page is only loaded from disk as needed, when you switch. Yes, everytime you're changing book or page, whether manually or with a command, FFXI is loading macros from disk.

When you input two macro commands too close to each other, there's a chance that the second one will fail, due to disk being slow or to the file being locked because it's currently being read. When the failure to load happens, the game client assumes that the file is corrupted, and nukes it entirely.

Yes, this even happens with ultra fast modern solid state disks, because FFXI doesn't care it's a file lock issue, not a read speed issue, it only knows that the load failed, so the file MUST be corrupted, so it needs to be nuked.

It's worse if you have a lot of addons loaded (client becomes slower) or if you have left the game running for a long time (clients becomes slower) or if you use lua bots (clients becomes bogged down, thus slower).

In some cases, the issue is external to FFXI: for example if you're doing anything in the background causing the disk to slow down, and failing to load the file immediately when FFXI tries to access it. It's a PS2 game, it assumes the system is not doing anything else at all.

Rarely, the issue is not at file load, but at file save when you shutdown the client: FFXI crashes and fails to save the macro page properly on logout. The file will then actually be corrupted, and thus will be nuked again on load.

All of this is compounded by way too many gearswap suites having commands like this one in them:

This line is present, almost verbatim, in most mote-based or selindrile-based gearswap files. See that 'wait .1' right there? That is way too short, and if your game client chokes at any point during those handful of frames, the macro set will fail loading, and get nuked entirely.
Even on systems that are fast enough where 0.1 is not a problem, spamming macro changes as a result of changing jobs too fast will also cause it - but it's not the changing of jobs, it's the inputing of /macro book and /macro set commands too close to each other. Using the jobchange addon compounds the issue, as it lets you change jobs way faster than the client does.

SOLUTION: have a longer wait between the book and set commands. I recommend a value of at least 2 seconds. 1 can also do the trick but will potentially still cause wipes if you change main job too fast. It's not like you NEED to push those buttons in those 2s after changing job, anyway.
Also, slow down when changing job. There's no reason at all to change job multiple times in a row at the speed of light.

I PR'd a change to Selindrile's repo, ages ago, setting it to 2 seconds, but it might not have propagated to everybody especially after the change in mantainers, and all mote based or bokura files still have that 0.1 value in them.

I've been talking about this issue for three years now >_>

I use my own lua, but each job lua loads up a txt script that swaps my G-keys aliases and at the end of the text there's something like this


I changed it into

Thanks Lili! Let's see if I stop seeing this issue from now on.

Yeah, same here and I know many that I had happened. All first page of every book, in my case not once, not twice but thrice. The solution was to start on page 2 of every macro book.now I loose the first page but I will never know lol

Edit 1: not a dual boxer here, I play on a laptop. Only own 1 character, started happening around June/july

At first I thought it only happened on page1 of every macro book, but then I noticed it started happening on other pages as well.
As I said before, I saw no patterns.

If we are to believe what Lili and others said (and why shouldn't we?) then the page that gets deleted is probably the one you "try" to move to.
Like take my example above. Book 11, macro set 1.
If the "delete" happens in an example like that, it would happen on page 1.
But think about another situation with Book 15, macro set 6.
If the deletion happens in this instance, it would be page 6 that gets deleted, not 1.

See what I mean?
For you it always happened on page1 because probably all your jobs start macros from page 1.
In my case most jobs start from page1, but others start at a different page and that's likely why I saw it happen even on pages that are not 1.

Is there a way to backup and re-apply macros? (I just backup the USR folder), but I would prefer a more granular approach

You can backup to server, but as I mentioned before it doesn't solve this specific issue of pages disappearing.

They added the chance to backup macros AND in-game settings in the FFXI config app that you can find inside the FFXI folder. Can only be used when the game is closed.
Allows you to put a description/comment to each config you save.
This system works all of the time even if a macro page gets deleted, unlike the Server>Client system.

It's sorta the same as backupping the whole /user/ folder but it's more granular, probably what you were looking for?

The macros are stored in the (400) files of the form mcrBBS.dat where BB is the book number minus one and S is the set number minus one (not zero-padded). You can just backup those files without the rest of the user directory if you so choose.