[Neus]

Back up again. Please note that I have some debugging stuff added and if I manage to identify the cause of the memory leak then I will need to take the server down, so expect the possibility of quick downtime (as if you aren't already).

[Trexmaster]

Server went down around 10:17 PM EST.

[Neus]

Back up, with some changes made to patch another large memory leak. Will this be the magic fix? We will see.

[Blissey]

Server fell around 9:50 PM EST.

[Neus]

Fixed another memory leak involving summoners, added some new debug logging. If we crash again I will be implementing more frequent reboots and see if that helps (8 hours, as opposed to the current 12 hours).

[Trexmaster]

Server went down around 12:55 AM EST 4/13/2015.

[Neus]

Sorry for the downtime. The server will be up after the backup I'm making is completed.

One possibility that entered my mind is that BYOND has a 64k limit for unique turf types, and a number of things it considers 'unique' can apply, namely appearance types, which is a collection of overlays/underlays/icons/iconstates/directions/etc. I thought about how we recently added the ability to add overlays and underlays to tiles and how it might be connected. So when the game comes up, you are limited to 25 overlays and underlays (that's a collective, not individually) in your house per floor; if the game tries to load your file and there are more than that, it won't load the instances that exceed that amount. You'll receive a notification if you try to insert an overlay/underlay and it would put you over the limit, as well.

Another instance that causes unique turf changes is the creation of battle maps; I've made some adjustments there and made sure that both houses and generated battlefields are cleaning themselves up after the fact, which they weren't doing properly before.

I'm still not 100% sure this is the cause of our crashes but it seems quite likely; the limit is low enough to hit with enough activity, we had base game systems making adjustments that weren't green and could 'eventually' cause the game to crash in that fashion, a recent addition increased the amount of offending tiles, and so on. I was also able to consistently get dreamdaemon to crash just having the game loop through the first 66000 turfs it found and changing their icon state to a unique number (or by adding a unique number state to its overlays).

I've also changed the reboot timer to every 8 hours instead of every 12 hours. I'm hoping between all of these changes, the server will not find itself crashing into the ocean during peak hours. Once I have confirmed the servers have been stable and seem to be remaining that way, I will likely distribute some laplaceNET presents to say thank you for your patience.

[Neus]

As a heads up. This shouldn't affect anything considering it's just a network maintenance, but you'll probably get DC'd for a few minutes when it happens.

[Trexmaster]

Server crashed at approximately 4:50 PM EST on 5/1/2015.

[DJScias]

Rebooted the server at the 2th of May, 5:07 AM GMT+2.

Kind regards,
DJScias