Server Crashing After Last Patch

I run a dedicated MNC server on a Windows 2k8 R2 box. Before the patch, I had the server up for over 2 weeks without issues. It never crashed.

I patched yesterday, and in the last ~20 hours the server has crashed twice.

It seems like it runs for 4-6 hours or so before going down. Both times it crashed it was with a full player load (12 people).


------

Faulting application name: mncds.exe, version: 1.0.5694.0, time stamp: 0x4db1c914
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00100019
Faulting process id: 0x548
Faulting application start time: 0x01cc01cc8688b220
Faulting application path: C:\mnc\binaries\win32\mncds.exe
Faulting module path: unknown
Report Id: 2e0623b8-6de1-11e0-8e91-000c29a59b4d

uberent is aware of this, also send in my error logs. their working on it

Very glad this isn't just me - I was just commenting that the server stays up for about 5 hours before I get a Windows error closing the exe. Also I've had users report that just before it crashes everyone's ping spikes up over 800.

got like 20 minidumps already from all the crashes , going to send those over to uberent

This is (obviously) still going on.

I can provide crash info and logs if needed.

Hoping to get some feedback from uber on what they're doing to address this.

I currently have correspondence with UberGaf about the issue, i've provided him with most if not everything he asked for. It's only a matter of time now

The current status is that I can see what is happening, but it is difficult to see why it is happening. This is an area of code that has one foot in our codebase and one foot in Valve's Steam API (which we don't get to see behind the curtain).

If you guys can accumulate incidental information about these crashes - how long it took to crash, how many server instances were running, if there had been any loss of steam connectivity, if it was MNCDS or MNC, etc, etc, that would be useful at this time.

Also I have a server up called JOIN ME which I'm trying to reproduce the crash on, so if you can jump on there and play that would be helpful too.

Edit: Also did you have VIPs on your server?

There were no VIPS on the server, no VIPS are set currently.

Crashes seem to only occur when the server is full.

could there perhaps be a conflict between updating the stats at end of round and the server receiving queries to respond to? because there looks like there is a relation between the MNC population and the crash. Just after the patch when the population was high crashes occured more frequently.

In the last log i sent you, stats were being updated if i' m correct, is there maybe a server parameter we could use to disable stats as a temporary workaround?. Stats is listed in the server details in the server browser. but there seems to be no way(currently known)to disable it as far as i'm aware.

I'll try to collect more data from the crash logs i have, i already have a script that can parse a log. with some alterations it could probably list some information that could be usefull.

i will definately join your server tommorow

Update: I understand this well enough to fix the problem now; though there is still some mystery surrounding the Steam API's behavior. Unfortunately, I don't have a temporary workaround for you guys, so just sit tight and we'll get it fixed up asap.

Thanks for the information, much appreciated.

Let me know if you still need more info or answers to the questions you asked.

I did notice that the errors that kick out do seem to reference the SteamAPI. My experiences are the same as racquemis, it seems to crash only when full - it runs fine when empty or there's a low player count.

For the record, I install/update MNC using hldsupdatetool, not with a local Steam install.

As Ubergaf explained it me.The servers crash because it's writing endgame stats to steam for all the 12 players and only receiving confirmation for 11.

Any updates here? Been over a week and this is still going on.

Kind of a pain to have to constantly check my servers every ~6 hours to reset them after they crash.

To all the admins having crashes, is it after any map or just Tully's?
Out of all the crashed servers I've encountered, they only crashed at the end tally after a full round on UncleTully's, never on the other maps

Well my instances are now almost crash free.

The problem has been fixed internally for about 90%, i got a new executable some days ago from UberGaf to test and it has solved the problem mostly, there was only 1 crash in 72 hours(running 4 instances). uberent has got the log and minidump of that crash and is being looked into.

So don't worry, their working on it. and by the looks of it they are almost done. you just have to wait for the update.

Anyone who would like to get ahold of the fixed .exe I'm testing please send me a PM. Or, I think we'll probably have a patch out for this issue in the next day or two, if you want to just sit tight.

Had our server crash last night while it was full, sent you a PM requesting the .exe

the new update fixed this already no need for more crash reports i think

Our server crashed right after the last update. Uncle Tully's went fine, then halfway through Grenade III server crash.

You have a different problem.

Well that's some earth-moving info there, thanks for that.

Basically this is a Gameservers.com server and it's been fully patched and it crashed out after the last update.

I'd like some input from UberGaf on whether this patched .exe he mentioned was actually rolled out in the previous patch or if this is something new that I need to save logs for.