General Lag/Performance - Late Game? (Is this a server/internet/game/bug issue?)

Windows 7 Home Premium 64-bit SP1
CPU
AMD FX-6100 21 °C
Zambezi 32nm Technology
RAM
16.0GB Dual-Channel DDR3 @ 805MHz (9-9-9-24)
Motherboard
ASUSTeK COMPUTER INC. M5A99X EVO (Socket 942) 16 °C
Graphics
2048MB ATI AMD Radeon R9 200 Series (Sapphire/PCPartner) 29 °C
Storage
465GB SAMSUNG HD502IJ ATA Device (SATA) 10 °C
1863GB Western Digital WDC WD20EARX-00PASB0 ATA Device (SATA) 15 °C
111GB KINGSTON SV300S37A120G ATA Device (SSD) 15 °C

I have an above average upload/download internet connection.

Is there any reason why I should be getting lag spikes towards end game, is it because this game is still being developed and optimised or are people still suffering from having to just end the game when your almost at 1 hour playtime because it literally becomes unplayable, changing my graphics settings lower only helps so much, in fact turning them higher almost makes it slightly better for some reason, now I'm not saying it's completely unplayable, just the response time to click things grinds my gears, and then I get this bug where I have to keep cancelling the option to see my build queues because for some reason it just loves to keep coming on during late game.

I'd just like some one to clarify for me that these are just because the game is still young and being developed, I'm cool with that, as long as it's being worked on.

Are you playing single player or multiplayer? If the former, are you sure the game is hosting the games on your computer? How many and how big are the planets in the systems you're playing on?

Can you upload your dxdiag? - This will help troubleshoot the issue.

The game is still being developed and worked upon.

P.S. Upcoming work update!
Edit: are the unit stuttering? That would indicate a network latency issue. Otherwise it could be something else.

If it's running on your machine & not Uber's servers, the problem is likely your processor - the sim isn't multithreaded (yet, because that's actually a lot harder to implement) & AMD processors aren't exactly known for having powerful individual cores.

You can identify what is slowing the game down by pressing ctrl + p. First page is graphics performance, next is server pref, 3rd is simulation.

From what your saying it sounds like a cpu limit (the fx 6100 isn't all that fast these days, especially for single thread stuff, though pa does use threads quite well it has 2 heavy threads that could be stalling. One for the client that issues draw calls to the gpu, the other that runs the simulation).

To be honest after an hour the number of units can get out of hand, and pa can bring most systems to their knees eventually. Your best bet if playing the ai is be aggressive, and keep pressuring it. That will stop it running wild and lagging the game out with silly numbers (thousands literally) of units, especially in multi planet games.

Also performance is still being improved. The difference in large multi player games from launch (sept 14) to now is actually quite noticeable (check out the clan wars videos for examples).

I'm going to try and reply to all your replies (which I'm grateful for)

- My DxDiag.txt file is attached to this post

- 99.9% of the time I'm playing multiplayer servers, Europe and USA regions, I mainly try to use EU but I don't pay attention if I'am or not.

- I'm normally never hosting these servers

- I see every unit slow down incredibly, and then speed up and catch up with them selves, and repeat the same..

- I understand what your saying about my CPU, so gone are the days that games run fully of the graphics card and CPU is secondary or am I going mad? Bohemia Interative are the same with the ARMA series, although I can play most games on ULTRA or HIGH with no issues such as Supreme Commander 1 & 2, Total Annihilation, Starcraft 2, Dawn of War 2, Company of Heroes 2, and then Planetary Annihilation comes along with it's simplistic art style and just decides to **** **** up.

Well PA is *very* resource intensive (especially on the CPU) for a few reasons:

1: It's an OpenGL game (so to maintain cross compatibility with Mac and Linux) which isn't as well optimised for CPU load as DX11 (although even that is limited). PA uses 1 main thread + a few additional 'helper' threads for rendering which is the main bottleneck for FPS with allot of screen. The limit in question is relating to 'draw calls' and is a known issue across the industry. This is why AMD invented the Mantle API. A game with many independent objects (like PA) needs lots of draw calls, with current Open GL and DX11 they are quite limited. Mantle can result in a 10x increase, however it's limited to Windows only. This situation will improve in the future as OpenGL Next come out (as well as DX12 but again that's windows only) as both these new API's are being re-written in part based on the changes AMD made with Mantle. For RTS titles more draw calls is a big deal, here's hoping Uber will be able to make an update that can utilise all the goodness of OGL-N when it comes out

2: This game (like TA and SupCom) features simulated projectiles. That means that every missle, laser and cannon shot actually spawns an object. Most of the calculations done with this are done on the CPU which is a big reason why CPU can be the limiting factor.

3: Spherical maps make things like position and navigation calculation much more complex. Again most of this is CPU bound.

Basically PA is pushing the genre (and as a result current hardware) really hard. Whilst you can run thing's like SupCom and TA maxed out, you need to remember that *nothing on the market* could do that when those games came out (trust me I remember playing TA on my then current P100 with 32mb of ram, and it hammered the machine after about 15 minutes of play time lol). SupCom was just as bad when it came out. These games have been built with the future in mind, the game is playable now but don't expect max settings.

Out of interest, the reason you can max out most FPS games is those games are balanced to be GPU limited (and you have a good GPU). In those games, although the background scenery is complex and detailed, it's also predominantly static objects. As such all those things can be treated as a single draw call and thus the CPU hardly gets used in comparison. Now where newer games are pushing things by using many more animated AI controlled objects (Assassins Creed Unity for example), the CPU *is* much more of a bottleneck again as they're bumping into the draw call limit.

Having said all of that, the problem you describe actually sounds like a network lag one. When playing *online* the actual game simulation runs on the remote server rather than your machine. If the units pause them jump, that's lag in the data stream from the server. You can confirm this by pressing Ctrl + P and looking at your FPS, if your machine is running a decent frame rate (20+) then that isn't your issue. It could also be the server simulation lagging (press cntl + p again a couple of times to get to the sim thread). The sim should be running at 10fps to be at full speed, and the game should be ok down to about 5fps sim. The sim can get bogged down if very large numbers of units are moving all at once (e.g. area patrol of whole planet with thousands of dox), or masses of simultaneous orbital transfers are going on. Still if the sim FPS is at 10 it's probably network related.

If your using Wifi, try using a lan connection instead (seriously wifi is horrible for gaming).

Edit: Also had a look, nothing wrong with your DxDiag that I can see. If it is network lag, a quick test to prove it would be host yourself an AI skirmish (that should run by default on your machine given your specs). Play vs a few AI's on a large planet and let them expand. The FPS will slow down but if you don't see any of that jumping your describing that probably explains it.

no this is just a game that uses CPU more than anything

it's comprehensible.

it's calculations of angles and trajectories for nearly millions of projectiles in one second .... not something a GPU is built for.

Apart from the GPU side and Network side, at some point CPU is not the bottleneck at all, at least from a %usage perspective. I'm running games with local server, a single AI on a single planet. All this on my core i7 (4 cores, 8 threads), 32gigs of memory and GTX770 4gb.

With tons of unis (groups of 1000 bots), The simulation is slowing down like hell. In the mean time, checking how much each core/thread is busy result in not more than 60% of usage for each core/thread. So I neve have a single core/thread running at more than 60% each. So it might be CPU, or something else (Memory bandwidth, Memory latency) but this is a fact : from a percent usage perspective CPU is not the bottleneck.

I'm not complaining at all, just reporting facts. Now of course, i would be very happy to know where the bottleneck is and if it's possible from a programming perspective to improve the simulation speed while dealing with more than 2000 units. By comparison, it's of course obvious that PA is achieving something very impressive by managing that many single units.

Windows task manager doesn't report core usage correctly. If SIM is slow you are using 1 thread 100%, but as you have lots of cores windows bounces the thread around. A good example is watch cpu usage with a single thread app. The app will use 12.5% total cpu on an 8 thread machine like yours, but no one thread Will show full usage....

Based on the 60% usage you could limit pa to 5 threads without slow down (right click pa.exe in task manager, go to set affinity and un tick 3 cores). Then you should see fully loaded threads but performance would be the same.

Main bottlenecks for pa are the primary graphics thread (draw calls to gpu) and sim if your running local server.

I hardly understand which gain is obtained by windows, switching a single thread from another core to another one. That obviously means having to save / restore the thread context each the thread is switched from one core to another one.

Just reading through all your replies, I understand why I'm getting the issue now, thanks for all the replies, I think I've learnt a thing or two as well, I'm not quite sure what but I'll sleep on it tonight and I'm sure it will all make sense in my head, your all tech savvy boffins and I salute you a good night and well being.

Yeah I don't really understand the behaviour.

this must be ultra complicated to explain but I'm willing to believe there is a valid reason. Although I'm also willing to believe the benefit reaped is very minor.

perhaps a dev could enlighten us?

Well, I don't understand that either. But when you enter Ring-0 (*) and you are in - say - the scheduler, the user thread context is already saved. So moving it around isn't that hard.
But like all things in life it's a bit more complicated: Moving threads from one core to another will definitely mess with the L1 cache foot-print of the user thread, since the new core does not have any information about what that user thread executed on the old core. Even more worse, that exact user thread context which was written out is definitely in the L1 cache, so moving this data to a new core means at least a write back to L2 or L3 - depending on the topology of the cores. On the other side, there are some really clever people within Microsoft Research and maybe there are beneficial aspects of this I do not see currently.


(*) If I remember it correctly, this is what Intel used to call the kernel mode.

Ok, so I did a very simple test :
I launched PA with local server and changed the following affinity settings before playing the game against AI :

PA & coherent UI processes affinity set to Threads 0,1,2,3,4,5
Local server affinity set to Thread 6&7

Played the game and yes, in that case, overall CPU consumption increased up to 80% (vs 50% when not changing affinity). In addition thread 7 was ~95 busy while thread 6 was ~50/60% busy.

I don't know why i have this difference between thread 6 and 7, but at least, it confirms that Windows task scheduler either poorly reflect the exact average workload or that this specific setting has increased PA ability to take advantage of my 4/8 core/threads.

I'm actually pretty sure that PA local server is made of more than 1 thread : Neutrino had stated that the multi threading model was 1 planet / 1 thread for sim. This would make sense, otherwise i hardly understand how it would be possible to run very big games on 16 cores machine with many planets.

It looks like to me that the way windows scheduler is working is made for avoiding any possible issue while fixing affinity, but at the expense of CPU utilization efficiency.

Memories It is now nearly a decade ago, since the last time I looked into this topic. Finding the optimal schedule for a given program is in the realm of NP-hard (that is computer-science-speak for very very complicated), so it isn't a surprise that it is nearly impossible to find a scheduler that runs good for arbitrary programs, which is what the windows scheduler has to do.

After the server was released and the really big server sessions, the forum came to the conclusion that the server is multi-threaded, e.g. it uses several threads for IO and other stuff, but the simulation itself runs on a single thread.

Anyway, I think what you are seeing is some artificial artifact from the interplay between workload, scheduler and core topology. Maybe the cores enter some throttled state when PA is spread over all cores because the utilization isn't high enough, or the server causes some L2 / L1 cache thrashing and stalls the actual PA threads. Nowadays it's really hard to say when you are not sitting in front of the computer.

Thanks for the clarification ! So for games like TA, Supcom, PA, the graal and the next big jump will be multi-threaded simulation

Yeah. Looking at the current development of Intel CPUs, if they really want to put more performance into one server, for sure they have to move to multi threaded simulation at some point in the future. But the one million dollar question is when.

So because i like facts i have decided to use Process explorer in order to inspect the threads involved in server.exe.
Unfortunately, for an unknown reason PA server keeps crashing when launching the game with more than 1 planet, more than 1 AI, so currently the test is limited to 1 planet, myself and my ******* AI opponent who is really good even in front of a cheating human playing with x5 eco hahahah !

So here is the result, and this picture was taken with Massive amounts of units (4000 units on my side).

So as you can see server.exe is made of many threads. This picture shows the current avg cpu percent usage, as well as the number of cpu cycles consumed by each thread.

If you look at the threads name, you can see we have Many called "MSVCR110.dll". Some of them are active, but some of them are simply idle during the entire duration of the game.

We also have 2 threads called "server.exe" but only one was active.

This why it's unfortunate i'm not able to test with more than one planet (because of the crash).

But let's focus on the active threads.

Basically, during the game, thread 9140 cpu kept increasing up to max 12,32. And while reaching the maximum value, the sim used to slowdown, and avg cpu usage fluctuated between 9% and 12,3%. The Other threads never increased over the current displayed values.

Since I'm running PA on a 4core/8threads cpu, you can easily conclude that 100% / 8 =12,5
which confirm this very busy thread (eating up to 12,3% of cpu) is the simulation thread. I have no idea about what the other threads are. I may try 1 planet with more than 1 AI and check if i get another additional active thread.
And hopefully i will be able to fix this server crash in order to investigate what's happening when i have more than 2 planets and active battles on them

And by the way this was testing without faking thread affinity. I did not provide the information, but i've never seen so much context switches. Guess Windows and linux and really different worlds Will test later if faking thread affinity result in lower context switches and explain why i can use up to 80% of my cpu (I guess yes) while faking thread affinity.