Multithreading - The Future of PA and Titans

Discussion in 'PA: TITANS: General Discussion' started by panews, August 25, 2018.

Page 1 of 2
  1. panews

    panews Planetary News Official PA

    Messages:
    138
    Likes Received:
    500
    @Sorian is back working on server simulation performance with multithreading this month and we have a technical update to share on his work for the next update.

    Two years ago I was given the opportunity to continue work on my baby, Planetary Annihilation, for the 99337 Performance Update. My primary focus was attempting to improve performance by multithreading parts of the simulation. My initial focus was on areas of the simulation where I thought it would be easiest to do things in parallel. The most natural separation was based around planets.

    Since navigation and physics are typically the worst performance offenders, separating updates by planet seemed like a good plan. Their updates tend to only act on a single entity. Initial testing blew up spectacularly, as I expected it would. Fortunately, there were not many areas of contention between threads and a performance win was born.

    So, now that I am back, you may be wondering what I am working on now. I’ll be continuing right where I left off! This time I am digging further into the sim to see where I can split large groups of tasks over threads to speed things up. Early this week I was focused on the physics update. I figured, since we should only be looking at two unique physics objects at the same time, it should be possible to thread this.

    Well, I was right and wrong. There were the usual small issues to fix, but threading the physics update revealed a larger issue. The physics system utilizes an HGrid (described as a loose octree) to find potential entity overlaps using a function called buildPairs. It turns out that there is no guarantee that the vector of pairs that we get back does not contain duplicates. Under normal operation this isn’t a huge issue, as this is a single call that immediately returns and it was just being done thousands of times more than it needed to be. However, when you throw threading into the mix it gets ugly. Fixable, but ugly.

    I have also started work on restructuring the sim tick a bit to allow the navigation and physics updates to happen alongside other updates. Right now I have it running alongside the economy update, but the economy update is only a small part of the overall update, so not much of a win there yet. I am hoping to split some of the plan steps (our sim has plan, execute, and record steps for entities) out into a group that can also run at the same time as the navigation/physics update.

    For the navigation system, I have been mainly focusing on the epic number of allocations the navigation system does during voxel integration. I have a new system in place that saves off vectors of cost cells (the objects the integrator makes tens of thousands during voxel integration) for reuse. The integrator is also doing a better job of allocating in blocks instead of individually. On the multithreading front, the navigation system has proven to be quite difficult. There are so many little acceleration features and so much lazy loading that is simply prevents multithreading without a significant amount of work.

    So, what’s next? There is still more work to be done with navigation, there is still the sim tick stuff to be investigated, and some AI updates as well (of course), starting with a much needed performance pass. And that doesn’t even get into other fun stuff like modern toolchain and Coherent GT updates.

    Mike “Sorian” Robbins
    Last edited by a moderator: August 25, 2018
    panews, August 25, 2018
    #1
    tatsujb, Engineer1234, devoh and 36 others like this.
  2. felipec

    felipec Active Member

    Messages:
    465
    Likes Received:
    190
    That's awesome! This kind of post is very interesting to read.
    Glad you are back, man!
    felipec, August 25, 2018
    #2
    devoh and lulamae like this.
  3. doud

    doud Well-Known Member

    Messages:
    922
    Likes Received:
    568
    @Thanks for all these details Sorian.
    Let's see what you can do, i've not doubt you will come with brillant ideas :)
    To be honest i had hoped you could improved the navigation system to have larger groups of units moving alltogether without slowdown on a single planet. As i understand, fixing is not impossible but obviously not possible unless uber can allocate large amount of time to rework the system. And i guess it's all about balancing efforts and the chances of success.
    I guess nobody suspected Uber would be back on the PA front. Plus, Nothing comes for free. So are you here because you had nothing better to do at Uber, or is there something bigger coming next ? ;)
    I know it's hard to make business with RTSs today. One hadsjust to see how much delay has been introduced in "Ashes of the singularity" updates. Clearly not stardock priority. So it's not making a company business.

    I may be wrong and something new may be coming on the RTS front and i would be more than happy if it comes, let's say, from Uber and Also Chris Taylor :p
    I can still remember last King and castles video blog, it looked like so promising :(
    All the best for you new challenge ;)
    doud, August 25, 2018
    #3
  4. Quitch

    Quitch Post Master General

    Messages:
    5,885
    Likes Received:
    6,045
    Quitch, August 25, 2018
    #4
  5. doud

    doud Well-Known Member

    Messages:
    922
    Likes Received:
    568
    why make a dedicated company for this actually ? From a business perspective, a company is intended to earn money. It's not by simply improving a game made a few years back that one is going to earn money :) Hopefully there are longer term objectives ;)
    doud, August 25, 2018
    #5
  6. lulamae

    lulamae Planetary Moderator

    Messages:
    799
    Likes Received:
    307

    For me, the 2 key bits of information from that announcement are:

    My interpretation is that this isn't just a dedicated company, but a different company. A company with a different focus than the one from which PA came.
    lulamae, August 25, 2018
    #6
    tatsujb, Remy561 and Nicb1 like this.
  7. carn1x

    carn1x Active Member

    Messages:
    389
    Likes Received:
    156
    What is the current consensus on single planet AI game with regards to core utilisation? Is it a flat 100% 0% split? Or is there already some spread?
    carn1x, August 25, 2018
    #7
  8. blackwinsdk

    blackwinsdk New Member

    Messages:
    10
    Likes Received:
    6
    Hi,
    is there any change for some Nvidia RTX stuff in the PA engine? Maybe Nvidia will help or so :D ... the new AotS for NV :D

    The Tensor cores for AI/KI would be something interesting.... don't you think?

    thx
    blackwinsdk, August 25, 2018
    #8
  9. sycspysycspy

    sycspysycspy Active Member

    Messages:
    268
    Likes Received:
    80
    I doubt they would embrace any new techs since even vulkan is not planned
    sycspysycspy, August 25, 2018
    #9
  10. cdrkf

    cdrkf Post Master General

    Messages:
    5,721
    Likes Received:
    4,793
    I'm not so sure about that now.... we have @Sorian working to improve multi threading on the sim side of things- and we also know @varrak is back- he is the graphics engine programmer for PA. I dunno about nvidia RTX enhancements, but I could see a move from Open GL to Vulkan being on the cards potentially....
    cdrkf, August 25, 2018
    #10
    sevmek and tatsujb like this.
  11. Quitch

    Quitch Post Master General

    Messages:
    5,885
    Likes Received:
    6,045
    Multiple cores would be used.
    Quitch, August 25, 2018
    #11
  12. exterminans

    exterminans Post Master General

    Messages:
    1,881
    Likes Received:
    986
    Nope, all the heavy lifting is done on a single core. Before Sorian's current work, the Nav, AI and Physics are each computed in parallel per planet / AI, but the set of all 3 steps is processed sequentially. One planet and one AI means no parallelism yet.
    exterminans, August 25, 2018
    #12
    Quitch likes this.
  13. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,328
    Likes Received:
    2,125
    They are moving moving to Coherent GT though which is pretty exciting. I'm hoping for quite a speedup in the UI by that as well as improved Linux support (less black screens).
    DeathByDenim, August 25, 2018
    #13
    sevmek, tatsujb and cdrkf like this.
  14. mrobin604

    mrobin604 New Member

    Messages:
    1
    Likes Received:
    1
    Hi @Sorian,

    Have you given any thought to using a job system like the one Naughty Dog presented at GDC here?

    (Google “job system naughty dog” and click the pdf link; your website isn’t letting me post links)

    If you broke your updates up into small jobs, balancing them across the cores would be a lot more straightforward.

    This might be difficult to do if your engine is doing a lot of dependent state modifications, as it might require a lot of restructuring and protecting with mutexes to prevent multiple threads from trying to access the same data structure.

    The good part is you don’t have to move over all at once... we had a monolithic job to do game updates to start with and gradually jobified things over time when it made sense.

    Anyhow, it worked very well in our games, so I thought I’d suggest it.
    mrobin604, August 25, 2018
    #14
    Clopse likes this.
  15. shot2400

    shot2400 New Member

    Messages:
    10
    Likes Received:
    2
    Super excited to hear about the game getting love. Kinda a big question, but my game has had a bug for a while and it makes the game really hard to play. All unit icons for everyone in my game is a slight variant of Yellow. Id really love to see if it can be fixed. Love you guys
    shot2400, August 25, 2018
    #15
  16. lostmekka

    lostmekka Member

    Messages:
    69
    Likes Received:
    96
    Aah, my favourite dev returns! Truly amazing times. Onward, for honor and glory! :)
    lostmekka, August 25, 2018
    #16
    tatsujb likes this.
  17. Ralith

    Ralith Member

    Messages:
    124
    Likes Received:
    6
    What a pleasant surprise!
    Ralith, August 25, 2018
    #17
  18. Sorian

    Sorian Official PA

    Messages:
    998
    Likes Received:
    3,844
    This is correct. The current release splits the update per planet and per AI. Per planet, the nav and physics ticks run on separate threads. Additionally, each individual AI get to run on its own thread. Finally, the history system utilizes multiple threads during its update.

    In my current local copy, the heaviest parts of the physics update are multi-threaded per update, so each planet's physics update also runs on multiple threads. This should improve single planet physics perf, with more cores meaning better perf.

    Nav is a different beast and has required a different approach.

    Our engine already has a parallelfor class that utilizes background threads. We throw a big list of jobs at it and it splits them out over threads and waits for completion.

    I am also utilizing a technique used in the Unreal engine, where the physics update is run alongside other sim updates that don't call into physics, effectively "hiding" part of the perf cost.
    Sorian, August 26, 2018
    #18
    sevmek, lokiCML, NikolaMX and 14 others like this.
  19. emarkus

    emarkus Active Member

    Messages:
    312
    Likes Received:
    141
    Hiding performance problems by doing bottlenecks alongside is nice: If people spam doxes, then they must not wonder why the doexes are shoting delayed....

    I didnt understand your HGrid stuff really, but it sounded like it could cause endless loops sometimes. Which could explain why in some rare cases the simulations speed crashes down to 50, 30, 10 and then 2%.

    Would it help to do a bit pathing caching?

    There are currently separate threads for the planets. What if the planets of a system are all the same and just cloned? Can they profit by having same cache?

    Just my 1ct... Just throwing in random thoughts which might be completely useless :p

    Thank you Sorian!
    emarkus, August 26, 2018
    #19
  20. devoh

    devoh Well-Known Member

    Messages:
    445
    Likes Received:
    404
    Love reading your updates @Sorian. I hope you keep them coming.
    devoh, August 27, 2018
    #20
Page 1 of 2

Share This Page