[REL] The UberMap

Discussion in 'Released Mods' started by cola_colin, May 17, 2015.

  1. dom314

    dom314 Post Master General

    Messages:
    896
    Likes Received:
    1,196
    Do you plan on releasing the source code at all? Or is that a bit of a security risk for Uber?
  2. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    Read the OP completely it is all on github since ages :p
    The one hack I found was fixed by Uber a few weeks ago.
    The magic mainly happens in https://github.com/pamods/Minimap/b...va/info/nanodesu/reader/PaClientAccessor.java
    with a version specific config file, for example https://github.com/pamods/Minimap/b...nfo/nanodesu/reader/patches/B82098Config.java

    There certainly is a chance I'll fix it again, for example when I want to play more. Atm I have other non-PA things keeping my busy.
    dom314 likes this.
  3. dom314

    dom314 Post Master General

    Messages:
    896
    Likes Received:
    1,196
    Oh my bad xD. Thanks for that.
  4. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    with the new PTE there is an API that will mostly make this mod possible without any hackery. Well unit positions. Health and orders are missing, actually no I think at least health is also there.... unit positions = icons with correct positions = the most important feature anyway.
    Last edited: July 17, 2015
    cdrkf, stuart98 and proeleert like this.
  5. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    so generating this still takes 30-60 seconds due to the fact that I only can generate one puppet per one native call. I also can't show what kind of terrain it is. Hopefully build placement test call in the upcoming PTE will make this even better.

    [​IMG]


    [​IMG]

    The first one to guess the map correctly gets a cookie ;)
    And the person who can explain to me how the *** the clipping of d3 works gets the rest of the cookie box.
    I am planning to probably do away with d3, since rendering data pixel by pixel isn't really it's point anymore.
    That means I have to emulate the clipping that d3 somehow does myself. All I know is that it sets a svg clippath based on a sphere. Now what I don't understand is how can I effectively tell if a pixel should be clipped or not.
    The only way I can imagine is to go backwards: Iterate over a sphere and project lots of points from it. Then check if the position to test is "close" to one of those dots. But that seems.... horrible.
    the winkel triple projection I am using is part of d3 geo and has a nice invert function but it as little to no indication that it should be clipped. The really bad areas are rather easy to detect as they yield really funny numbers, but there are some areas that "wrap around" as well.
    Bigger version of the images above. Generating that took a few minutes and a LOT of poor puppets that were disposed of right after the snapped to some location.

    [​IMG]


    And some random metal planet in extra large:
    [​IMG]
    Last edited: July 26, 2015
    killerkiwijuice and tracert like this.
  6. dom314

    dom314 Post Master General

    Messages:
    896
    Likes Received:
    1,196
    Forge!
    cola_colin likes this.
  7. stuart98

    stuart98 Post Master General

    Messages:
    6,009
    Likes Received:
    3,888
    Is it forge?

    I'm thinking Meso.
  8. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    It sure is forge, I dunno why you would think that is meso.

    Also a pretty moon imho.

    [​IMG]

    Lower quality now, but this can be generated in less than 10 seconds per planet, which is probably what it'll be like.
    Last edited: July 26, 2015
    killerkiwijuice likes this.
  9. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    So this is, give or take a little quality in terms of resolution, what the final maps will probably look like.

    [​IMG]

    A bunch of things:
    - Since there is no API to get mex directly I so a planet wide "place a lot of mex" test. The placement test snaps to them, so I get results for all mex, however since atm the snapping doesn't correctly tell me to where it snapped the mex spots are displayed a bit larger, resulting in the green areas. That's basically the area where an extractor will snap to the mex spot.
    - The random black spots on some shore areas are related to this: https://forums.uberent.com/threads/layers-for-build-placement.69409/
    - Generating these images will take 5-10 seconds (hopefully there will be some API fixes that push this to 2-5 seconds instead) the first time you play a map and 2 second or less once you play it again. Especially during the initial preparation it will lag PA pretty hard. Once the image is there however it'll not create any more cpu load.
    Planets below a radius of 700 will be faster to generate, planets above that radius will have a lower image quality but won't generate slower. If the API improves and I further optimize the code I might push this quality border up a bit.
    Generally generation will be fast enough to be done within the commander spawn phase very easily.
    - You can still rotate the map, while rotating it will draw a low resolution version to keep up with your movement. 5 seconds after the rotation has ended it will redraw the high quality version.
    - There are 4 layers: Land, Sea, Mex and Blocked. For each of these a color range can be defined per biome. It will scale within this ranged based on the height of the specific location. So the color set will somewhat match the planet in question by biome at least. On this picture it uses yellow-brownish colors for land, blue for water, greenish for mex and black to gray for blocked areas.

    I want to thank @pinbender for sneaking in that build placement test API, it's the main source of information behind this picture.
    Last edited: August 5, 2015
    xankar, tatsujb, tracert and 4 others like this.
  10. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    released, see completely rewritten OP
    tracert and stuart98 like this.
  11. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    moved to the titans forums, as this mod only works in titans.

    (actually after doing it... maybe it also works in the base game after the patch the base game had yesterday).
  12. tatsujb

    tatsujb Post Master General

    Messages:
    12,902
    Likes Received:
    5,385
    omg @cola_colin !!! holy ****! this is what we've all been waiting for!
  13. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    Updated the Ubermap:

    - Now makes strategic icons a bit brighters, especially when not selected. This shadows the icons shader. However it does so with a low priority. If you use a mod with some other shader, that mod most likely will be put on top. Notice the Ubermaps don't use the shader, so they won't follow other icon mods.
    The intent is to make that stupid brown color I get all the time more visible on the maps...

    - Implemented caching of map data. This should speed up UI loading times after you played a map for the first time.

    - Reduced FPS of the rendering of the maps. Should not matter much in how it looks, but saves some CPU.

    @netpyxa You might like this ;)

    Also notice the "it will eat your computer" problem wasn't meant as being THAT bad. Please give it a try everyone and report how much of a hit it really is on your system.
    Last edited: August 21, 2015
  14. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    Nope there are no "Units died here". Though there are half transparent "you saw enemy units here, but then lost vision" ghosts of enemies. Kinda the reverse thing, but with a similar result.
    Fun use case: Fly a scout over an enemy base. Don't look at it directly. A few seconds later open the ubermap and study all the stuff your scout saw via the icons.
    Or directly know that a group of 2 dox killed your fabber when you see a ghost of 2 dox where your fabber was before.

    I can look into units died here markers, but am not sure how much sense they make with the ghosts of enemy units.
  15. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    I might not be totally overstating the "eat your computer part". Basically it seems once you reach ~1k units this creates severe game lags. It seems the handling of the API calls to get updates on all the unit positions gets pretty expensive and it seems that the API calls are not handled async to rendering. Meaning while it takes care of an API call it doesn't render anything. The whole game seems to stop.

    @pinbender can you clarify if this is the case and the API should behave like that? Once I reach these unit numbers the ingame API in fact seems to have substantially worse performance compared to my previous external memory reader. That reader really did eat one cpu core at full load, but it wasn't the core that took care of rendering the game.

    EDIT:
    So I am actually getting heavy performance drops, if I query for units states that seem to have changed. So when I have 700 dox idle it's not super great but playable.
    When they start to move it dies pretty hard, blocking the client for 500ms+
    Interesting enough this even prevails when I give them an order move and then pause the game. It keeps massively lagging, even though, since it is paused, there are no changes.
    Any explanation for this?!

    EDIT 2:
    Okay testing scenario:
    1.) Start a --nomods game on clutch vs idle AI with sandbox on. Make sure you are in the first slot (or you will need to change the army index in my example code)
    2.) Use a bunch of avatar factories to make 1k dox. Place them somewhere and let them idle.
    3.) Run this benchmark code:

    Code:
    var world = api.getWorldView(0);
    world.setServerCulling(false); // just so the test isn't influenced, I don't think this has any effects for this problem, apart from potentially creating false test results
    var tester = function() {
        console.time("getArmyUnits");
        world.getArmyUnits(0, 0).then(function(data){
            console.timeEnd("getArmyUnits");
            console.log("getArmyUnits Result", data);
    
            var unitIds = [];
            _.forEach(data, function(elem, key) {
                _.forEach(elem, function(unitId) {
                    unitIds.push(unitId);
                });
            });
    
            console.time("getUnitState");
            world.getUnitState(unitIds).then(function(states) {
                console.timeEnd("getUnitState");
                console.log("unit states results", states);
                setTimeout(tester, 3000);
            });
        });
    };
    tester();
    
    When all the dox are idle the expected result is that you'll see 10 to 30 ms for getArmyUnits and ~50 to 70ms for getUnitState. The game doesn't appear to lag, the fps might suffer a little, but clearly playable imho.

    Now move the dox.
    Expected result: getArmyUnits will still only use 10 to 30 ms. getUnitState however will use 1400 to 1500ms. Everytime this call is made the game lags for that time, no frames are rendered.

    Also notice that if you move the dox and pause the game the behavior seems to stay on "super slow"

    What is going on?
    Last edited: August 23, 2015
  16. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    Okay another find: The performance issue seems related to large groups of units that have one single order given to them.

    So:

    select 500 dox and right click somewhere -> performance implodes

    select 500 dox and use cover the line to give a line command, this means each dox gets a "private" command -> performance doesn't care at all.

    @pinbender
    So I have a guess actually. The unit info API in the info about orders of the units gives that list of unit IDs that "have the order together with this unit".
    That sounds like a good trap for quadratic runtime: For each of the 500 dox it will iterate the other 499 dox to add their ID into that list. Bam! Quadratic runtime, apparently in a place that locks out the rendering thread from doing it's rendering. Also fits very well with the fact that pausing the game doesn't matter: It's not about state changes.

    I'd say remove that list of unit IDs (part of the order objects per unit. Do not remove the full order info, just that array of IDs inside the orders) until an API is in that has orders as first class entities, so you query a list of orders and not a list of units that has orders attached. once that is in the units could simply reference the order id and interested modders could query the order. I would guess nobody uses the list of IDs at the moment, so nobody should miss it and it makes the API painful to use in some scenarios atm.

    I guess to make the runtime of the collection of IDs way better it would be possible to put all units that will be returned in a unit id -> unit state object map and then iterate the orders and their units. For each unit lookup the unit state object and link the full list of ids that the order probably holds in there. That should be O(n log n). The resulting size of the messages would still be pretty much quadratic and I don't think anybody uses those IDs, so yeah.... don't think this is necessary.

    Just removing that list of IDs inside the order should be as much as commenting out some code for now :>
    Last edited: August 23, 2015
  17. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    released an update that sets back the aggressivity with which the ubermap queries data. In case you have too big orders (see performance bug above) the map data may only update once every 1-2 seconds, but your fps should stay in the range of 20-30 with a micro lagging. Much better than what happened before. Hopefully the performance bug can be fixed soon, as it will improve performance by an order of magnitudes or two in the problem scenarios.
    tracert likes this.
  18. pinbender

    pinbender Active Member

    Messages:
    78
    Likes Received:
    137
    It's possible this is happening due to each unit having the list of all other units on the order, but I'm not in a good place to check right now. (I'd consider that somewhat unlikely, due to only increasing the number of items in the order array.) I'll add investigating that to the ever-growing todo list. The API will probably have to change to making a separate call for the order descriptions and hooking them up afterward on the JS side. It's better for memory use that way anyway.
    dom314 and cola_colin like this.
  19. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    I am talking about this:
    A result of the unit state query with one unit opened to see the orders. All other units have the same order, each with a copy of this array:
    [​IMG]
    Each single unit has that full array of all other units. So n^2 elements.
    Make n 500 or more and it'll take a while to process multiple hundred thousand elements.
    It's only IDs, but from how the performance reacts I am pretty sure there is some code in a perf critical position that iterates all these arrays one by one, effective iterating n^2 elements. When n is above 200-300 it starts to become problematic.

    Not to mention the player might give multiple orders to the group using shift. Each of them again will result in n^2 elements added.

    Thanks!
    Last edited: August 24, 2015
  20. pinbender

    pinbender Active Member

    Messages:
    78
    Likes Received:
    137
    Yes, I caught what you were talking about. I wouldn't expect serialization of 250,000 numbers to take more than about 10-20ms, though. That's definitely more than 0ms, so if you're doing it more than 1Hz, it will definitely cause some problems. (It's also possible that I'm being too optimistic about the performance of our serialization layers, of course.)
    cola_colin likes this.

Share This Page