(emphasis mine), this is the reason I'm proposing the $media approach, to avoid this. Adding a previously-unavailable link to media files has low impact, but changing the UI host behavior is a non-starter right now, both because of potential internal fallout and because of subtle breakage to existing mods for very little gain. I definitely want to find a clean solution, but I'm not moving the UI host directory. I'll keep fishing for a bit before making any final calls though.
Oh absolutely. Less facetiously: all this stuff is up in the air right now. Once we dive deeper into topics like server mods and situations where the server supplies the UI, it's bound to throw wrenches in things like this. So yes, there's absolutely a chance for this to get changed. But until we know exactly what the next stable solution should be that has a lot of tangible benefit to a lot of people, I want to avoid breaking the existing ecosystem as much as possible.
Okay folks, here's what I'm going to do. I'm not going to change any of this right now. The whole '$media' symlink thing, the JS calls for getting modded mount info, etc... not gonna rush that kind of stuff in. I have some of this stuff working locally, but I'm really uncomfortable with some of the precedents it's setting, simply because the low-level mod mounting and ui_mod_list systems aren't integrated yet. These changes we've been talking about would help to perpetuate that schism rather than bridge it, and I don't believe that's what any of us want. So if you all can get by on what's there right now a bit longer, I'll do my best to figure out a plan during the holiday break so that things can start moving The Right Way from the beginning of the new year (when I'll be dedicating the vast majority of my time to mod-related features, including stuff like this). I don't want to put in any more temporary workarounds than I have to, lest it cause everyone a lot more trouble downstream than it's worth. I'd rather think things through a bit more and make an approach that we can be happy with.
Sounds like a good plan. Modding frameworks aren't something that should be rushed Have you been able to reproduce the issue myself and @dfanz0r were having with mods not loading properly?
Haven't been able to repro that, no... was it something where you all were changing some mod info dynamically while the client was active? I know F5'ing a lot of UI stuff works fine since it's all JS/HTML, but the file-mounting mod support is currently only determined at client initialization. Just a thought...
Nothing dynamic, just creating the mod to begin with (and I restarted the client several times to make sure)
This will be a very interesting topic and I hope you'll update us on it once it starts. Since PA is build around the concept that the server owns all data and the client has none at all per default. I hope you guys will make sure that UI mods can access as much data as possible without any server mods that "push data" involved. UI mods that required a server mod will be of little use in the sense of a classical UI Mod: They cannot be used in all games a player plays. So they wont be used at all, as players tend to either use a feature all the time or not at all. Of course I am not talking about making all data available to the UI at any time without any limitations. Any players should only be allowed to access his "own" data. So nothing he cannot see anyway. Also just dumb sending all data would obviously break bandwidth limitations, probably at multiple bottlenecks, I am well aware of that. So instead there is an absolute -really I am pretty sure without such a thing UI-mods will be pretty much powerless- need to provide UI mods with a standard generic data query mechanism. Basically in my UI-modding dreams I can do things as: First exmaple: Doesn't have to be based on SQL, the main intend of the example is that a good solution should offer a way to exactly define what the mod wants. So it doesn't have to transfer more data just to throw it away. Instead the server is directly told: calculate this number and tell me whenever it changes (or at least once per second or so) Second example: I am not 100% if this is necessary if something like 1 is implemented. Depends on what data the server has I guess. of course this is probably not easy to do and it needs to be carefully implemented so no information leaks (cheat mods) result. Also it would be good if somehow the server could deny requests that ask for "too much". You guys might be well aware of these issues, I am just making sure you are, as this is really important in my eyes
Thanks cola_colin, yes this is the kind of stuff we're thinking about. A few things I can state right now: 1) There is no way we're going to (deliberately) allow a client to request, via a client-only mod, data from the server that the server does not believe the client should have. That would be a bug/exploit and we'd want to squash that kind of thing pretty fast. Thankfully our history system prevents a lot of that kind of thing; if we want clients to have a particular piece of information we kinda have to go out of our way to make sure it has it. 2) For data that a client is allowed to have, we'll certainly need a suite of events and methods of performing queries; that's a fairly open-ended problem and so might grow over time. I can tell you right now I don't plan on implementing a SQL backend, but some kind of primitive query grammar would be useful and is something I am thinking about (and is something I have implemented successfully before). 3) Re your two event examples, the second is more likely than the first, in fact general-purpose "update" events tend to be horrible performance nightmares and AFAIK most game engines regret it when they allow scripting languages to hook into such things (a major concern for us given the number of units we have). But other more discrete events can be manageable perf-wise. I'll certainly be keeping you folks in the loop as this goes forward; no worries there.
1.) Good 2.) Yeah SQL is pretty much out of scope I would guess. But I like that you are thinking about that direction. 3.) Makes sense.
I was hoping to be able to make a mod which merged the ui_mod_list with this new stuff, and the only thing currently preventing me from doing so is not having access to the data about currently installed and mounted mods. But if it gets delayed for something better that's fine. I would delay the $media stuff and try to get the ability to get information about what mods are installed out, but that's just me This is all i was planning on doing for the mod though: Add the values for scene information into modinfo.json. Use the api your going to add to get that data into the javascript code. Then where the loading of ui_mods_list.js from common.js was done, switch it to pulling all of that data from the new api. Its a pretty simple change really, and if you are going to wait on implementing things it may be worth doing on your side. I mean locally i already have replaced my ui_mods_list.js via a mod with something that loads the mods from a json file instead
Would it not be feasible to have data divided into three types for this sort of thing? Info which is sent to the client by default (i.e. everything required to run the game in vanilla) Info which absolutely cannot be sent to the client (probably the majority of info) Info which is not sent to the client by default but can be requested by mods. This would help to keep bandwidth usage under control for you guys (the majority of people will most likely play vanilla) but gives UI modders a bit of flexibility to give people with modded UIs a bit more information at their disposal. A reminder that I'm not a programmer by any means, so I'm only talking conceptually here
I think the best possible setup would not send any data at all by default. Instead all UI elements (the default ones, just like the mods) should use the same system to query data from the server. That way the data query system can be put more at the core of the system and the UI developers at Uber are the first testers for the system already.
I'd like the developers to clarify that what things the client is aware of. First some definitions of words: - Server: The part of PA that holds all information about the game and that runs the AI and stuff. - Client: The part of PA that runs on a player's machine, is written in C++ and handles the 3D visualization of the game. - UI: The part of PA that runs on top of the client, is writting in JS/HTML and handles all the 2D UI widgets and menus of the game. The question is, how much of the information that the player is entitled to does the client contain? Obviously the server won't tell the client the positions of enemy units that the player can't see, but how about the player's own units that are e.g. on another planet than what the player is currently looking at, or on the other side of the current planet? And even if the client is aware of all of the player's own units, with what granularity is it aware of them? For example is the information compressed so that the client is aware only of the rough position and movement of units that are far away, or does it know the exact state of every unit at all times? I'd hope that the client is fully aware of the player's own units and what the player is entitled to see. (At least I haven't noticed much lag in seeing my own units when moving the camera around.) In this case any querying for information by the UI would only be a matter of communication between the UI and the client - not the server. This would also have the benefit that a data-heavy mod could not overload the server with its queries - it's only a matter of how powerful the player's own machine is.
One thing i can clarify for you, is that i believe there is a server, client, and simulation. Simulation is where the game simulates the effect of all the inputs coming together and the output is the game, frame by frame. Server is the game info and receiver for client info. Clients are what players input through and also receive the view of the simulation. A few other things: -client is entirely webcode and it renders its own assets stored locally for its view. It only relies on the server to send information to render and the simulation to actually run all the data to advance the next frame with all the units and their new next position and action. -the server might be webcode or at least should be unless they have other code to mix it up and make it more protected from reverse engineer at this state of development, it contains some checks such as if a fabricator should be allowed to build something (prevents cheating), and these checks consist of communication to and from all clients, as well as its own unique data, which all collects at the server and is sent to the simulation and collected back to divide and send clients only the info it is programmed to send, which is basically all of what your ui immediately sees and nothing extra. -The simulation receives the result of server data and client input, and outputs what the clients need to render their next frame with the units and actions calculated up to that point. Last i checked it had key frames for most things at 8 a second, or is it every 8 frames... that should be searchable and it was an update either early beta or late alpha. For something almost entirely made out of code everyone knows, it is a modern marvel in my opinion. So your question is in fact answered, that the server sends only what you are viewing. It is a performance thing. If it sent you your whole status of everything you own, it would be bad for performance and for limited bandwidth availability. So yes, unfortunately the ui, which is almost all the client is anyway, has to ask the server to request additional info to be able to include it as a counter in the screen, as it does not receive a heaping ton of info you don't have eyes on by default.
I wonder if that's really so, because when moving the camera around the planet, there is no noticable popping up or jumping around of your units - which there would be if the server had not been sending updates on your units that are out of view. The network latency is orders of magnitude larger than the display's refresh rate (and there isn't really much you can do about network latency due to the speed of light), so reaching a smooth gaming experience is really hard unless all the data is kept locally. Or then PA has some amazing network code that minimizes the number of updates without any noticable degrading.
Oh that. Your game does remember last locations out of sight, but it doesn't update it so info tracked that way is wrong if a unit dies or such. It isn't tracked locally, nor is all server info sent from server. Thus getting three types of lag. One is latency where units walk in place because lack of internet, another is simulation where unit animation and everything frame by frame slows down, and last is client render where the game is choppy and the game goes full speed despite it so only your view is lagging not the game. Anyway, i wasn't as detailed as could be. Maybe someone collectively could add info to that. I'll grab some attention.
I don't think we know the specifics of when the client is sent what unit data. I would assume it is sent a large portion of it in the current version at least based on how the program behaves. (No visible lag when you quickly completely change position via camera hotkeys. Maybe it does lag when you switch between planets. I am not quite sure of it. What I think is fairly certain is that the client does cache all "curves" once it got them, so ingame lag will be gone in the replay. Anyway all this data for the rendering is not available in the UI layer.
Here's the short short version of what's going on: On the server side, we have the Sim, and the HistoryServer. The Sim is running the simulation (including AI) and is authoritative over the game state as it's updating. It's constantly feeding in new curve data to the HistoryServer. The HistoryServer knows everything that has happened in the game, with respect to the needs of Chrono Cam. This is a lot of data. If some data exists on the Sim but Chrono Cam doesn't need it (like internal AI state or whatever), the HistoryServer probably won't have it. Alongside the HistoryServer is the server-side concept of "viewpoint servers", which is the server's image of all the entity data that the server has sent to each connected client. I'm oversimplifying here, but let's just say that each client only has a subset of what's actually going on, and right now we're probably updating more than we actually need to be (there are tradeoffs here of network/memory perf vs. responsiveness, and as games get more interplanetary we'll probably be adjusting our balance for this). On the client, we have whatever the HistoryServer has sent to us (matching up with its viewpoint of us), we have a lot of internal client logic to utilize that data for rendering etc., and on top of it all there's the UI. So in terms of queries, the bulk of your query flexibility in the UI is going to be relative to whatever the client already has a view of, based on the last time that given entity was updated by the history system. That's all the renderer has access to, so if it's something you can already see on-screen, it's probably going to be something we can make queries for pretty safely. Entities which you aren't supposed to have access to, you won't. That reduces client-side cheating, and is not going to change. Now the gray area here is entities you do have access to (e.g. your army) but for which your client-side view may be a bit stale (e.g. if they're on a different planet than the one you're looking at; right now we may not optimize for this case much, but we may do so in the future, introducing a little lag for better larger multi-planet games). That's a situation where conceptually you should be able to query all of those entities, but we're tied for technical rather than conceptual reasons. In this case I'm conceptually okay with adding new client-side mod APIs to force an update of the history for one or more entities, but it'll probably be throttled pretty heavily, e.g. a limit of X non-essential entity updates per second. Now for entities in your own army, when it comes to having access to more server data than the client ever needs to see (i.e. data that we never normally send at all because it's not needed by the client even during Chrono Cam), that stuff is completely out-of-band and we try to avoid it as much as possible, because it doesn't play nice with the history system. We'll address this on a case-by-case basis, because adding new history data has real memory/bandwidth implications, and adding non-history-based out-of-band data has real complications too, so... yeah. My ideal solution would be to add a lot of extra history-capable entity fields available to the client for modding purposes, but not update them by default unless mods specifically enable those fields for update. We still wouldn't do this for any cheat-sensitive fields, but there's a range of non-cheat-sensitive fields that still aren't necessary on the client normally that this could apply to. This is something I'll be looking into early in the new year as I start building out the modding APIs. If I can get this approach to work, it would avoid the need for painful out-of-band server query stuff, and everything would be A) querying client-side history data and B) poking the server to update client-side history data for stuff that's stale or otherwise not normally updated (with the hopes that people use this feature sparingly if they want to play nice in the memory/bandwidth department; if it gets abused we'll have to dial it back). Just some thoughts.
