Whats going on with the lag ?

[mirman118]

Are you any closer to sorting this ? I heard someone mention that the DB auto backups every 5minutes on these files. And these lag spikes tend to happen every 5minutes. Is it nothing to do with that ?

 

[Easymode]

Fair play, genius! A "table level" locking sounds suspiciously familiar. That's why SQL has row-level locking support.

 

[Chriz]

Not a bad point, if these files are still flat file based could be IO locking, I dont think crystal impliemnt asyncronus operations for writing out files?

Supose it will only get worse as the data file grows.

 

[Aidonlee]

I've noticed especially today. The server lags only a small amount when there are less than 400 people on. The moment it gets a high usercount (above 400) it seems to lag.

 

[GinkoTreeValley]

Been off work couple days a lot less lag during the day then around 5:6 pm when user count hits around 400 the lag increases 10 fold

 

[MonoxideLenny]

Yer iv also noticed as soon as it hits the 400 users mark the lag kicks in

 

[kuelab]

Yer iv also noticed as soon as it hits the 400 users mark the lag kicks in

Yes, every 8 - 10 min.... big 3 sec lag

 

[Violent]

Yes, every 8 - 10 min.... big 3 sec lag

That definetly sounds like the DB save. It does an object lock while it saves all of the accounts and their objects. Which is a lot of data if there are 400 players online.

 

[forspinki]

Yeah i think they thought they fixed it but its still happening every day quite often.

 

[Xx69xX]

We are changing the frequency of the save and hoping this will help. We are testing offline. This will be added tonight

-Apocalypse Team

 

[Nyx]

We are changing the frequency of the save and hoping this will help. We are testing offline. This will be added tonight

-Apocalypse Team

Not do like Myth of Soma use to do?

Back ups were every two hours but you could type /save after getting a rare drop, levelling up etc etc and it would force write your character. As a fail safe and to prevent abuse it would perform a lookup before saving, if last save time is equal to or less than say 10 minutes you can't force save your character.

 

[Slipmatt]

Changing the frequency of the save isn't going to resolve the issue, will it? (I have absolutely no clue about these files).

Would I be right in assuming that if the frequency was changed to every hour, the lag would just happen every hour instead?

 

[DiabloVodkaShot]

We are investigating, it is not a network issue we are sure of that. As you can appreciate these things take a little of time. We are hoping to have a solution by the weekend.

-Apocalypse Team

 

[putato]

Why not have staggered saves. Divide all accounts into five sub groups or whatever works best. Save a sub group every 1 minute.

1. Divide accounts into 5 sub goups.
2. Save sub group 1
3, Wait 1 minute
4. Save sub group 2
5, Wait 1 minute
6. Save sub group 3
7, Wait 1 minute
8. Save sub group 4
9, Wait 1 minute
10. Save sub group 5
11, Wait 1 minute
12. goto 1:

 

[kuelab]

Not do like Myth of Soma use to do? 8

Oooooold school =D

 

[Chriz]

Why not have staggered saves. Divide all accounts into five sub groups or whatever works best. Save a sub group every 1 minute.

1. Divide accounts into 5 sub goups.
2. Save sub group 1
3, Wait 1 minute
4. Save sub group 2
5, Wait 1 minute
6. Save sub group 3
7, Wait 1 minute
8. Save sub group 4
9, Wait 1 minute
10. Save sub group 5
11, Wait 1 minute
12. goto 1:

This opens a window for item duplication, need to be careful with this approach.

It would also probably make no difference as the full data file would still need to be saved out, assuming that Apoc still use the base implementation.

Might be worth migrating to SQL

One thought - save out to a persistent RAM disk? Should tackle any IO bottle neck.

Edit* Please don't increase the save interval too much - If we crash and the Interval is like 1 hour, could be a big rewind. Rather have the lag tbh.

 

[IceMan]

Yea bigger save time on back up with c files means bigger rewinds on a server crash.

 

[Easymode]

flatfile doesn't scale well where as sql offers clusters/row-level locking and more that's why I've gone SQL myself.

---------- Post Merged at 01:31 PM ---------- Previous Post was at 01:31 PM ----------

Yea bigger save time on back up with c files means bigger rewinds on a server crash.

yup

 

[Hany]

Maybe you’re right, but id rather have it saving within another thread outside the main thread.

Going to sql is a no go at this point, we’re trying to stabilize not create more problems.

 

[Pixels]

server freezes at least so we do not die