Large Game: Analysis and Discussion

First I would like to apologize to both uber for nearly breaking the game, as well as the community for taking away the ability to create large games (as uber said they would create a hot-fix for this issue), after I attempted to form a hundred player game. If they're any repercussions for this from uber or the community you can direct it at me. I did want to stress test uber's servers to their limits by creating an unreasonably large game, and to some extent we did what we wanted to (by getting about 67 people at one point and crashing the lobby), and seeing what the limits of the PA engine are.

And though I did want to mention the 100 slot event, I made this thread for the purpose of discussing large games in PA and the network codes potential for giant matches. It seems to me that this game already has the capacity for playing these large matches and wanted feedback from both the community and devs on what has worked in these games, what doesn't work, why it doesn't work, and what can be done to make these games more reasonable/playable.

From the large matches that i have played so far, the game runs remarkably well (to a point). The average large game (20-45 players), played on single planet, ran with little to no lag compared to the average game. This could also be that actual asset/unit numbers didn't actually increase from normal game matches, or at least that much, but it also didn't seem to effect the servers ability to relay that information to those people. We even got a few games past 50 going smoothly. Now granted based on what the dev said other games on the same server had performance hit at the cost of these games. But the large games themselves ran extraordinarily well given the circumstances. And as far as more players/commanders, the game mechanics still held up, even becoming more fun and frantic.

This wasn't replicated in 100 player attempt as even the lobby was shudder and hit by massive lag after 60 or so players. Simple things like players join on mass (at one point 20 people joining), killed the server. The other major issue seemed to be the constant spam of chat seemed to kill the performance. Even to a point that a army of 1000 units in a game had better performance than a constantly changing chat window. So there is a limit to what PAs code can do. But for what they promised us (40 man games) it is more than playable. And I would argue with more performance enhancements and better servers, 75-100 players games are indeed possible. The game is very scalable indeed.

But I do find some issues to do with these large matches. These are my thoughts in no particular order;

• Lobbies cannot support these games, and not just from a technical perspective. The game hangs on the fact that players need to join them, and there is no good support for keeping them there. No auto ready or letting the host player force other players to be ready to start the game, allowing host players to manually move other players around the teams. And among the performance of chat, the text window needs some customization such as size of the panel. Being able to easily communicate to the other players
• Teams are also an issue as the limited number of colors restricts the ability of team type games. 10 player teams can be fun, but unless they have direct communication between members, strategies quickly break down. I rather had two dozen smaller teams than three or four ten man teams that can't communicate well. These can be solved esily with a larger to color pallet, a revised icon system for enemy units, or my favorite, a army painter similar to the dawn of war series. All can be usefull for this, but as long as we can support half as many teams as max payers (two players per team) I'll be fine. This will also help with larger FFA matches as well as Alliance matches.

On a separate note that can be solved by the player being their chosen color, their allies are blue, and the enemy is red (or something).

And I do have some questions for Uber;

• When we played those large games (other than the 100) how was the performance hit on other games in that server. When we played games around 30-40?
• When you say that our larger games crippled the performance of the other games on the server, yet ours is fine? How many games are running on a server? Is there a game priority established in the server that made our games run well?
• What are your plans to increase player count in games? Will you allow 15-20 person games in the meanwhile?
• What were you guys at uber (or the community) thinking when we started up these games?

Thanks for your time to read this and feel free to post an criticisms , comments, questions or additional ideas in the thread. This is to start a discussion about the larger games.

Edit: Sorry about the poll, it posted while I was still writing it. Any way I can add options? I need; Noticeable performance hit, bad performance hit, and unplayable.

Just to reiterate what Garat said in the other thread, the lobby didn't crash from spam, he shut down the match himself. Don't want people thinking the game is so fragile that it can't handle 70 people spamming messages.

Uh. wtf poll?
The only options are "no" "not noticeable" and "slight?"
What if someone had a medium or even a large performance hit?
(Polls cannot be edited.)

You didn't need to stress test Uber's servers. They're not currently built for load balancing. They're specifically built for certain sized matches. This has already been communicated to the community from Uber. The stunt was pretty un-nessesary.

Uber has stated they will be implementing load balancing... stuff so the server load will be balanced out appropriately.

@Kranku: I did read that from garat, but it was not the 100 slot game that was shut down. We left after server bugs/lag/crash stop people from being able to join. Garat closed the second game of 60.

@Brianurkiss: Unfortunately I was writing this on my tablet and hit enter before I finished writing the poll. And now I wish I could remove it but can't find a way

As the actual 100 man discussion is in another forum, this was created to discuss the actual implementation and development of large army games. I only mention the 100 slot game as an extreme measure of the extent of the network capabilities (as least that were attempted so far).

Edit: Can a mod or admin remove the poll as the mistake seems to be irraversable user side and is confusing the discussion?

If you create large games it's of no benefit to the devs. They have set up the conditions they want players to be playing. It screws up all the data from that server. It's only for player benefit.

It might not be of immediate use, but as Uber gets closer to running larger 40 player games it could serve as a benchmark of sorts. Obviously it'd be less than ideal for Uber if ONLY these massive games were being played.

Mike

For player curiosity. It otherwise screws up all data from that server. I guess yeah, benchmark of where they were... but it works fine, and there's really no reason why it wouldn't. The only issues with it are overly fast and overly slow connections, but clearly that isn't so terrible an issue. The reason why it screws up everyone elses connection is, I imagine, because the servers are only intended for 7 10 player games, not 6 10 player games and a 60 player game.

I'd assume it has less to do with the servers and more to do with the servers connection actually, Uber has commented that they still have more optimizations they want to make for the network stuff so I imagine that it was just a coincidence that the server held up to all that traffic really.

Like I said, while maybe not immediately useful I'm sure it will be a nice reference point for Uber down the line.

Mike

As for Uber's direct comments on the games, they liked us testing out the server, multiple times telling us the office was cracking up about it, and them evening joining them at times. The problem lay when we testing the limits and pushed it too far, needing to put the breaks on us for a while untill they feel they can 'balance' server load of different games and make it dynamic. Mainly making sure that different size games can run on a server and it shifts the limit to other factors than just 8 games or so.

And though I fully admit it was blissful joy to run these large matches. I myself wanted to actually see what they're code could do. By all accounts 'IT SHOULDN'T HAVE RUN'. With the limit on number of games on the server, along with the load we were putting on the servers, our games should have seen a performance hit as well. Yet when I talk to a majority of the players directly during these games, anyone with a decent computer was getting the same performance hit as they would get in a normal game. By all accounts we should have seen an equal performance hit to the players in the other games (which I would like to have seen, but there is no way right now to find out who they are). So they obviously have a game prioritize system in the networking code, just no way to restrict the amount of games/interactions on the server yet. And that's exactly what I wanted to find out, does the code allow me to run these games? Yes to an extent. Just not stable up to a certain point. But as we're finding out the servers can be scaled up based on hardware, and they are still working on the networking code, so I see it possible in the future.

And I do want to bring the chat lag up. It was a thing, and didn't crash the game. It made it HORRIBLY LAG. Before most players left the 100 slot and join the 60 slot that garat shut down, we had player repeteadly message the same text over and over again (usually the player total we had) by a majority of the 50-60 players, the game would be nearly unresponsive. Hundreds of messages being sent withing a matter of a couple of seconds. When the chat stopped being spammed the performance got back up. This is a major problem when trying to set up large games in the future, and even might have been one of the reason we were using up so much of the server. Simply sitting in the lobby not even being able to play the game. Just as a note

@cola_colin:
Thanks for changing the poll, its all good now.

@KNight :
Thanks for making sure this stayed a discussion for both sides.

So the issue is one of load balancing the servers. I'm looking at strategies for this now.

Do you guys think getting large games going should be a priority compared to other stuff? Just thinking out loud.

We haven't done enough perf analysis to know exactly but we did build it to scale up. There are a lot of complex issues here (units vs players and how different parts of the engine scale).

We currently run about 8 games per server. The main issue is we are already fairly memory bound so the large game kills the machine because of that for the other games. CPU/Bandwidth seems reasonable but I need to spend more time looking at the data.

In my mind this experiment has increased the priority of larger games, so I'm formulating a plan for how we can roll this out.

See my other thread for what I was thinking. The summary is something along the lines of ******* AWESOME!!

Yup, we

Yup, we thought the games were ******* awesome.

How is this even close to possible? Does the curves system you set up for the chronocam really save enough bandwidth to cause the bottleneck to just be memory, or would all client-server games have this potential?

I'm really happy to see that large games is now a priority. We've had way more planet smashing in these experiments than in normal games. There was even a case of one moon chasing another moon around the solar system! (Although it got stuck once the moon it was chasing exploded.)

Oh, man, do you have video of this?

Thanks for replying uber. As for making this a priority depends on a few factors. How much this would take away from features such as revamping orbital, as well as content (new units especially) production. But it I personally would like to see it floating around the top now and seeing an incremental increase in player counts in games over the next month.

But to be viable and not just support the hack in essence. The game mechanics will have to support it as well to make it something that won't interfere with player interactions and communication within the game. This is simply to make it easier to set up these games and so it isn't a hassle to form.

• Small elements of lobby would have to be revamped; host privilages (kicking, moving players around teams, picking team colors, force start), finding what causes chat lag in extreme circumstances, revamped loading info in lobby with some sort of loading bar and percentage along with the colored dot.
• A system for more teams to be in place. I mentioned this earlier, but a majority of teams don't have voice communication and typing to 10 other team members is a hassle. The ability to have more teams and be able to display those teams needs to be in place. Players of 4 or 5 are ideals, so we need to have the ability to have create half as many teams as there are players (2 per team). This can be done by adding more colors. Making sure that enemy icons can be differentiated by friendly icons. Which fixes the problem if two player' colors are too similar, they can be differentiated. And for things like aliance games, make you your own color, firendlies blue, and enemies red. Or something like that.
  
• More teams also add the ability of large FFA or alliance games which we need for the community to support large games. We simply need the support to have more game types.

That's all I got so far. But if you guys think it won't kill production in other departments that the community feels need work, THEN YOU NEED TO DO THIS!!!

Edit: I should also show examples that inside the a large game it runs fine. Just look at these commentator videos of 40 or so player games!



Note: In this game, there seems to be a performance hit, but from my end of playing in it, and a few other realms, we didn't even notice it.

There are currently 10 different colors to chose from. The stripe colors are limited to a maximum of 3 or 4 and some background colors only allow 2 different stripe colors. If we use 10 background colors and allow all but the same color to be used for stripes we can create 90 different color combinations. Then take the stripe color and use it for the outline on the strat icons. This would allow for 90 distinct teams to be in a game.

I think Zaphod was recording it, but I'm not sure it was clear to him what was going on. My team launched a moon at theirs to stop them from using it against us, but they launched theirs as soon as they noticed. They ended up taking out our main base, causing our moon to just keep speeding up around the solar system forever. We sent our other moon at their main base not long after.

*EDIT*

Celestial view kept getting progressively laggier once we had a planet zooming around with nowhere to go. I'm really surprised the game didn't crash.

For a given definition of "distinct"

I _definitly_ vote for the priorization of larger games because of several aspects:

• It's (to state the obvious) AWESOME to see action going on everywhere for viewers and to be part of something bigger as a player - you know your mates are fighting their own fight but at the same time you are fighting together, even though you are on different planets
• I think large games will or at least could make quite a good part of played games IF they offer the "tools". Some examples what they could be
  • actions that the gameowner could be taking to "ready up" everyone, force teamswitches, kick players, start with reduced size etc.
    
  • ingame possibilty to draw landmarks or something to communicate a plan between teammates
  • possiblity to "admin-pause" the game, pause-votes or something like that because bigger games tend to take longer
  • have shared eco or not
• Its easier safer to scale down than to scale up afterwards
• This is something other RTS - games cannot offer and will result in more players joining the community - more money to make the game even AWESOME^2

Here's the game I was talking about.

But you can't see the stripes in game, and they don't fit on the icons, making a majority of the game near useless. We actually have eleven or twelve colors bu the way, we just couldn't get that many teams to work in the large games hacks because there was some sort of restriction.

We could however have some sort of Army Builder like in Dawn of War II to add more colors ourselves. Then use the enemy icons to differentiate eachother. Added with the ability to select default colors if you feel you look too similar to your opponent. Or can create multiple army skins in the army builder.