question about bandwith and datatransfer

As neutrino stated quite often that he thinks bandwith will be the limiting factor for servers and i was suggesting the replay streaming, i began to think of this.

What do you plan on sending to the client?
Everything or only changes? And only sometimes a complete new now-state.

Ok maybe not sending the destination for stuff that can change direction activly (not as by force i.e. arcs of nonhoming projectiles)?

And how are you planing the type of data? Something like XML or binary?

After all the data need to be send while the match is running should not be as big as the replay, because of vision.

The answer to this is basically a tech update blog post that hasn't been written yet.

But yes you only send what they are currently looking at in space and time.

Would this ever become a visual lag issue? If I quickly change viewpoints or switch planets is it going to take a moment to populate the units? Just curious if you guys had done any testing on this.

Neutrino means what what you the PLAYER are looking at, but more so what you CAN look at, so if you send a scout up to the enemy base, the Server needs to send the data for what the scout sees, whether to not your looking at it as it appears.

Mike

Binary, what else?

Parsing XML means an incredible overhead, not only because of the format itself, but also because it doesn't play well with binary attributes so numbers etc. would also have to be encoded in an format which can safely be embedded in XML. Decimals are easy to read for humans, but a terrible choice when transmitting data.

The format is most likely to have fixed byte-patterns and numbers etc. are going to be passed in an endian save, but still binary representation. If the packages are large enough, each package might be gzip compressed, but i don't think that would be the case.

As you have noticed, only very few of the elements in the scene are actually capable of changing their attributes. Projectiles have only creation and termination event, anything in between is expressed by the arc. Units also won't accept new commands every frame so the amount of information is limited again. However, it's slightly more complex to express unit movement in arcs, because you have also acceleration and deceleration. You need a lot of waypoints to express such movement whereby each waypoint consists of position and movement vector as well as additional details like turret orientation. Waypoints can be skipped, if turn-/acceleration rate is low because that means that the client can safely extrapolate without differing to much.

This is where an well designed vision concept comes into play, you will need to take the current viewport of the user into consideration, if the user has zoomed out far, you could group the position of units and oppress movement and projectile information until the client zooms back in.
Alternative is to choose a vision concept which generally groups/hides unimportant units and will only ever show you the most important informations in full details.

Further optimization potential lies in recomputing PARTS of the simulation for units owned by the player, this might be possible because the full state for such units (including actual waypoints, not only movement information) can be shared with a player without giving away information he should not posses. This could reduce the rate at which snapshots of own units need to be synchronized with the client as intermediate states can be approximated. This however is last resort if bandwidth becomes critical since this means that the clients simulation might differ from the server simulation and it also means that a decent computational overhead on the client.


As for replays...
Sounds strange, but since the server still performs a lot of post-processing on the data (like determining visibility) it might actually make sense to use a different format for replays and have the server recalculate the gamestates from the replay. The replay has also different requirements, as you should be able to play it backwards and alike, which means that it needs back references for events so you can roll them back. (E.g. if a projectile is destroyed, you need a reference to the creation event of the projectile, since that had all the information, especially the ark. If a unit is modified, you need the full diff, not only the new state etc.)

Wouldn't be so sure about that. That might become a lot of information to transmit in late game, although most of it isn't required at all since it is either outside your viewport or it is barely noticeable at your current zoomlevel.

Nah you dont need backwards compatibility for replays. You can use similar stuff as in video compression one full framebevery now and then in between only the changes. If you can skip backwards secondwise should be enough. Even 10 seconds or 30 are thinkable.

Also there is some stuff you dont want to send in the game but can send in the replay as you mentioned.

For instance destination of (not friendly)units, you want to only send current moving vector here. And update with changes on the fly.

Well i look foward to your blog post jon.

Edit :upssies *blog

Oh ye of little faith.

I've explained in much earlier posts how this system works. Time to go look back in time...

Yes it could result in pop-in for things if you quickly move view points. It really depends on how big the game is and how much we are limiting client bandwidth.

One thing I don't understand is that if the replay is made of game states, they should be really big no?

Do you know where to actually look for it must have been before my time.

It isn't. syox fall for the same assumption, but it's not just a set of full gamestates with forward diffs in between (what would actually be similar to video compression), but instead every unit, every projectile or any server managed entity only appears ONCE in the entire replay (and thats upon creation of the entity) and afterwards you only have references on it.

You still can't seek to a random point in time because the lack of full game states (in videos you would call the reference frames), but at least you can fast-forward and play backwards because applying the diffs is a lot faster than the actual simulation.

You could (of course) add additional snapshots to enable seeking, but that would increase the size of the replay by an significant amount. It has also to be added, that "creating a snapshot" is more difficult and requires more time than it sounds. Logging all changes into a logfile is easy and can be done on the fly. Creating additional snapshots would require to pause the simulation and to traverse across all structures of the game. That means if you were to create a full reference frame every 10 seconds, you would also need to pause the game every 10 seconds for approximately 1-5 seconds, depending on the size of the gamestate.

Uh I hate to to do this but you are making assumptions here.

Not how it works.

Making assumptions again.

I'm not aware of any current game that uses the system we have come up with so drawing on previous knowledge is going to lead you to wrong conclusions.

Yes. Size was a tradeoff for getting the features I wanted.

So what though? We are in the modern era of cheap disk space and fast transfer rates.

partially, but I hate it when I see a game take more than 10 GB space XD which happens more often lately

You could also include the changes as absolutes, So not saving the differenece and in replay addint this difference back to the starting state to get now state, rather than saving the absolute now state every time it changes (with timestamp). so you would have thee actuall state of the game every time.

But still you cant start at the end and the play backwards.

Just a quick think though.

I dont mind when it is textures though.

So a replay streaming system would be possible? :twisted:

Well, in this case, it's the replay, not the game. You can choose to delete it without affecting your game.

I'm living in a country where internet is not unlimited (ie. 20 gb / month).
I don't think it's the only one in the world, so it can be an issue if they are really big.

Now you are making me curious...

I assume you are still recording events caused by users 1:1, don't see any potential on this one.

But if you are not going to store the game state differential...

I have an idea of what you are doing now. It's just a wild guess, but could it be, that you put every entity which ever existed in the simulation into the replay and add an minimal spline (for continuous attributes) / changelog (for discrete attributes) for every attribute? Grouped and sorted by entity, attribute and then timestamp?
Assuming that you are going to reuse destroyed and often used entities, like e.g. projectiles, this would lead to an replay which is only as big as the biggest game state added pure information about changes and it eliminates lot of redundant informations. And more important: Every attribute of each entity can be looked up for every point in time while memory usage is as low as the biggest game state ever was. Also makes seeking time independent from the size of the current game state.

Did I hit it this time or am I thinking in the wrong direction again?

I hope you are wrong
As the server is 64 bits (only it seems), I can assume that stats will took over 4gb of ram, even with some good compressions, I will not be able to replay more than 20 games per month with my quota (assuming I'm only doing that with my connection)
Also HD are cheap now, but I don't really want to buy multiple big HD only to archive all my games, that will end up very costly over a year