Multi-input/multi-touch and in game replay.

Non technical summary [multi input]
I'd like to know if it would be possible for people to write mods that allows unconventional control schemes. An example would be to have buttons/gestures on your ipad in addition to the normal control scheme. Why? I'd like to experiment, but I'm sure someone would make a mini-game for their boy/girl-friend so they could help out in game [assuming you can already have more than one persons per commander].

A little more technical [multi input]
Traditionally you'll have one agent[1] that analyzes the [human] input and issue actions to the underlying framework. Would it be possible to add agents[1] though mods or the interface, with one or more of the following abilities:

• programmable interface
• reading commands from sockets[2]
• streaming [limited?] game state though socket[2]

I'm aware this makes it [a lot] easier to cheat [out of scope for this thread]. But it enables some fun/alternative ways to play the game.

Replay in game [not a video stream]
It's also worth mentioning that this can enable different mods for spectating [matches] inside the game [real-time/delayed-based]. Thus removing the need for high bandwidth, low resolution, video streams, and making it possible to a greater extent choose the part of the match you want to watch.

Observation
The in game time-rewind-review looked great, so I'm assuming there could already be a in game spectator solution [planned]. If it is could someone say something like _it's being worked on_?

What does the community and the devs think, and please only well formed arguments on topic.

[1] http://en.wikipedia.org/wiki/Intelligent_agent
[2] http://en.wikipedia.org/wiki/Unix_domain_socket


Added summary [wishlist]:

• I would like an open interface API to do crazy/interesting things.
  • Party confirmed on Mavor's blog, provided by smallcpu.
  • But not probably not until after realese.
• Interface separation
  • Come again? Two mouses + two keybords and one computer === two player.
  • I'm not asking them to implement this, I'm asking that the interface get designed in such a way that someone [me] can create it.
• High quality and flexible replay system post.
  • Reddit:"Yes we will make replays and casting a priority." is not a confirmation of this.

While what you are describing sounds interesting I'm not sure its worth spending time working on it at this point when there are so many other more pressing things that need to be created and balanced. Especially when I suspect that very few people will use "exotic" input methods.

I mean the idea of using an iPad sounds interesting but what could you really do with it? It certainly won't be able to run the PA client so I don't see what it could really offer. I don't feel that mini games have a place in a grand scale RTS. The only mini game I could possibly see making sense would be some type of FPS style tank/bot/aircraft driving sim where the player piloted one of the in game units.

However as I understand it the UI is being written in Javascript and will be fully modifiable so I'm sure someone will figure out how to do something interesting with it.

All of the replay and spectating things you are talking about are either already present or will be at the time of release.

I think he's asking if the engine can handle it.
As long as it can, then the UI skin could work?
We already know multiple viewing windows is going to be implemented.

But if the cursor/control is bound the system via the mouse, then it's more than a simple mod?

Once the game sells 600,000 copies they shall do this :mrgreen:

Lets try to clear this up, I am not asking them to spend significant additional time.

3D Mouse support?? and Controler for 3d movement is asking to use a 3D mouse. If we over simplify, I am asking for tools, so I can make that mod [only the mouse has a display as well].

I other words I would like sufficient access so we can do these things on our own.

The iPad doesn't need to run the client, it only needs to interface with a client.

How about controlling the AI settings? What about a tablet as an overview if you do not have a second or third screen?

However I am trying to make the following point:
We can not implement all good ideas now, because at some point someone will go "Would it not be great if we could do X like Z?". If we have sufficient [framework] access someone in the community can make that possible [without breaking licenses and/or laws].

The problem? How bad can this break game balance, I am hoping the game design is robust enough that additional/other interfaces can not break it [balance].

How interesting depends on how the java-script interface communicates with the underlying framework.

Could you [or someone else] provide a reference to the source that verifies this?
Edit: No, reddit:"Yes we will make replays and casting a priority." is not verification of this.

Thank you, I guess could be asking "Can we please be allowed to do more than you [the developers] are comfortable letting us?".

You are highlighting the core of what I am trying to say, only having one mouse pointer makes it hard for two peoples to play efficiently [on the same computer]. And the additional computer workaround is starting to get traditional.

I were hoping I could do this before the games sells that many copies.

On a side note, The Settlers is the first game I know of with dual mouse multiplayer [on a single computer].

I'm not really sure what exactly do you want, since you're talking about various different things that don't necessarly have any connection with each other. But take a look at the following, which is official:

Source here.


So, theoretically you should be able to do with the UI whatever you fancy. :mrgreen:

Well, initially it'll likely be fairly locked down. The UI is a webpage running in a sandbox. As people make suggestions for features to add we'll work to include them if they're not already supported, but don't think you'll be seeing too much of that till late beta if not after release.

Thank you, in the source I found something more than I hoped for.

"Mavor" or Jon says the following:

Sorry, in general I try to talk about what a good interface access could enable us to do. I am including a list of bullet points that I will be investigating.

Replays and streams in engine

• Miscellaneous unit/squad/spot highlighting in casts[1].
• Binding a single or multiple viewport(s)[2] to a cast[1].
  • Point of attachment option, useful for different aspect ratio monitors.
  • Zoom modifier, useful for different monitor resolutions.
  • Binding cast[1] spawned picture-in-picture to non-parent viewport[2].
• Set point-in-time(PIT[3]) to cast[1]. Useful for viewport(s)[2] not attached to a cast[1].
• Multiple casts[1] from the same match in different viewports[2].
  • Selectable cast[1] (audio/picture-in-picture/highlights) subscriptions.

1. cast should be understood as a replay or live/delayed-stream with added data such as highlighting, simultaneously events, voice commentary and/or more.
2. viewport should be understood as picture in picture, game window and/or monitor.
3. Point-in-time represent the clock-time(s) of the match a cast[1] is currently showing.

Other devices

• Virtual-display rendered real-time* on android tables (when running linux).
  • Possible notification and highlighting on host machine.

* Current prototype adds 80-110 ms latency while connected though 5GHz wifi.

This information causes a moral dilemma,

Is multiple mouse/touchscreen inputs on a Windows device even possible from the OS?
Obviously I think we have done 2 mice and 2 keyboard on 1 computer.
But it's not quite the same as a mouse has the fight for control.

I'm assuming anything done for mulitple mice + touchscreens would have to be built in-game?

The iPad still needs to run a client though. So you'll need to know the "APIs" (not actual programming APIs, but the network and command interfaces) to interface with the server, and then write your own iPad app to connect to the server, and then (optionally) ensure that you can communicate to the server that you wish to be treated as existing client X (i.e. the player on the PC) to be able to share information and enable whatever minigame functionality you have envisaged. That's without considering how you try to translate touch gestures into commands that the server will understand (this may be relatively simple, or could be a headache in its own right).

This is not a minor undertaking. You will basically have to write your own render engine for the iPad to interpret the information coming from the server; in essence you are writing a PA client for the iPad, even if you don't think of it as the PA client for the iPad. :shock:

Sorry, I completely forgot to attach the multi-controller part.

Input device (wishlist)

• Instead of the interface reading all input devices, it should have a list of devices it subscribes to.
  • Default action for never before seen inputs should be configurable.
  • Reconnected devices should work as before unconnected (if connected in a new usb it's a new device).
  • It is reasonable to limit device to a single interface
• Interface instances should be given a list of viewports it has access to.
  • Transactions (accessible mouse pointer area, active viewport, camera) between viewports should be programmable.
• You should be able to create new interfaces.

Disclaimer, I do not do Windows development.

You are able to pick what sound-card/audio-jack to use if you have multiple ones. Skype support click the "Checking your audio settings in Skype", you'll it it has a diffrent setting for where to play the "ringing" sound.

But more relevant and importantly, you can use multiple game-pads in Windows games without any problems i.e. Street Figther (steam) and Monaco (steam).

And there there is this .... Asus Taichi C|net video, at 2:00 they talk about the dual screen mode.

Yes, that is correct.



Note, I am chancing the name iPad to the more correct word tablet.

I'm hoping to only interface with my client interface, reverse-engineering game protocols is something I have done, but I am not looking to do it again [I would be breaking American laws]. It it would end up as a cheating mechanism, alas the opposite of enhancing game-play.

A standalone tablet interface would be competing with the original game client, a really bad situation, because we want people to buy the game.

Addition:
Sorry, I were doing two things at once. Lets break this down:

• The bottom layer is openGL, this can be rendered into a virtual-desktop and real-time encoded to a video stream that is displayed on the tablet.
• Interface is running inside a webkit as javascript, for the (non jailed) tablet you need to register all the API calls that is related to the game (worst case you do a POST/GET mappin). You could even do this in a layer outside the interface, so when a unmapped API call is done, you'll automatically map it into the computer.
• Meaning the problem is primarily related to mapping the incoming request into the sandboxed webkit, and returning the response.

There are plenty of tools that will help us doing a and b, c requires us to do injection and/or memory monitoring. Depending on what webkit they use, and how they interface with it, replacing it entirely might be easier.

If the tools are there, moving the commander with the tablet (pretty much useless) should take around a day. Beeing able to use the tablet to have the commander doge incomming fire, while he is still heading for the objective set by the [master] player, is when this is not longer minor [and thus interesting].

Addition:
In the trial example we would only need to be able to issue a move order to a know unit.

• Current selection of units can be stored one of the shorcuts.
• Commander can be selecter though the global shortcut.
• If the interface doesn't let us issue directional/vector moves, we'll need:
  • Camera orientation and position (if we can only issue move commands to pixels on the screen)
  • Center camera on commander/selected unit (if he is out of view).
  • Commander orientation and position (there is no north in space)
  • Linear algebra easily let us find a point who'm will move the commander in the correct direction.

This would be a clumsy way of doing it, and I would like something more robust.

I still don't see the point of having a tablet act as a mini interface for PA. What can it do that you couldn't achieve with a second screen or another PC running the full PA client?

"Because it can." is not a good enough reason. It needs to be able to do something that the other options absolutely cannot do to justify its existence. The strength of tablets lies in their portability and simple UIs. While these strengths translate very well to email, web browsing, and casual gaming tasks they do not really translate to the type of serious/complex gaming that a game like PA is about.

I do like the idea of being able to have two players on one PC with each one using their own screen, mouse, and keyboard. But I feel that this is going to be a very rare use case and just getting a second PC takes care of this as well. Since PA runs in a client server model the second PC doesn't even need to be particularly powerful.

This is how I chose to interpret this:

• I'm concerned you are trying to have them make another game, dumb it down into something I don't want to play or that they will spend so much time on this that it affects the quality of the game. And I am afraid that if I don't download and use this crazy interface, I'm going to end up loosing all my games.

I'm asking if we can have a complete and open (interface) API, so that they do not have to spend time making this. So if you are worried this will take a lot of time from development, please stop worrying. I'm asking because I would like to make crazy interfaces that will allow me to play with my non-gamer friends. I don't expect this will help me win, rather the opposite.

It is not supposed to do something else, it is supposed to it differently.

This is a good concern, you are probably right in the regards that dual keyboard and mouse rarely will be used. I've tried communicating [guess I'm horrible at communicating] that I would like the ability to implement this, so that they [devs] do not have to spend time on it. I have some really fond memories from The Settlers and I would like to see if that can translates to this game as well.

That is not what I mean at all.

What I mean is that since there is nothing that a tablet IU could possibly offer PA that a PC UI can't what is the point of spending time on creating a tablet UI or even making it possible to create one. There is no point in doing something "differently", only "better" is worth the effort. If better also happens to be different then good for it, it wins a cookie.

While I am all for powerful and extensible APIs on general principal, it is not always worth the time and effort/money of adding something to an API if too few of your users are going to use that particular feature.

I can see you've thought about this a hell of a lot more than I have. I suspect you'll be able to pull it off, and I know neutrino is confirmed in his desire to make PA modifiable from the word go. With luck you'll get all the hooks you need to be able to do it properly.

I'm also pretty sure that if I was going to try that particular task (getting PA in some form onto a tablet) I'd probably look at it for a few days, then just buy a Surface Pro to run the native Windows client. I don't think I could justify the amount of time and effort needed (by me, that is, since I'm a rubbish coder) versus cheating like that

There have already been a few discussions about team play turning into multiple players per commander (i.e. 3v3 with only 1 commander per team), and if that is straightforward to implement via mod, I'd definitely have a look at doing that one myself. I do like the idea of having one player concentrating on econ and keeping the commander safe, while others can leverage that infrastructure to wage battles on multiple fronts.

The human experience [of ubiquitous computing].

That is like saying you don't want mod-ability because few peoples are going to create mods. There is absolutely no way you know how many (or few) users will download and use interfaces that requires some (or all) of API interface I'm asking them to make public. And asking the developers to have a intelligent discussion on how the frameworks should interact, rarely hinders quality. Actually, if they discover that the current interface does a really bad job, this will be saving them time.

He pretty much says that they are doing, or have to do this anyway. Again I'm just asking for us to get sufficient access, it might even save them time if someone [non dev.] does a good enough job.

If you don't mind me asking, what kind of experience are you using to draw these conclusions?

I were lucky enough to do something similar in this project [paper]. So I'm just applying what we discovered/researched in that project, but thank you for the compliment.

Your solution would work better outside of your home.

This is why I love playing TA-Spring and openTTD with my brother. In openTTD he is expanding our rail system while I mange efficiency (train schedules, intersections, semaphores etc.). In TA-Spring, he tells me what he want built, and where I should "deliver" it.

I don't see what the abstract of a research paper not related to video games proves. We are talking about an RTS game. Not wearable computing devices that help me remember my grocery shopping list. However a UI mod designed to take advantage of the Oculus Rift could be pretty cool.

Not really. Its more of a statement of opportunity cost analysis. I'm totally for UI mods and making the UI easy to mod but going out of their way to allow for very specific edge cases may not be the best use of the dev team's time.

I agree that making the UI modifiable is a good thing. My only questions are:

Is making a fully network aware API for the UI worth the trouble if all it really lets you do is talk to tablets?

How many usability experts, UI engineers, or other people with more experience in game UI design than the guys at Uber are going to be modding PA?

I base these questions in my years of experience in working with application development teams in the business world. As I said I welcome the chance to mod the UI and I think its great that we will be able to do so. But asking for extra tools above and beyond what Uber uses to make the default UI might not be the best use of Uber's time.

I'll try to make it shorter this time.

I might be able to implement it on a PC, but the table makes it easier due to the sensors and built-in cameras. However I do agree with you that this aspect is turning into a corner case.

I guess you just answered "Help you remember forgotten units"?

I like this example a whole lot more than my tablet example already. Actually, I think I will start preparing this tomorrow.

Last time I checked [all] the webkits had TCP over IP support, browsing the web without is difficult. You can do POST/GET with java-script (if they let us).

More experience than Uber, probably not many. University [under-]graduates, quite a few (winter holidays).

Thank you for actually telling me your concerns, if you still think they have to make tools. Could you let me know what kind of tools, then we could discuss that?

So you are saying that games make people improvise because they don't have full knowledge of the entire situation. And that this makes them fun and/or interesting. I don't think we need a research paper to tell us that. It still doesn't say how it makes a tablet better than a PC for any particular task that doesn't expressly need mobility.

I don't think that's going to be particularly important. Any strike force large enough to matter will show up on strategic zoom. Three or four tanks on some far flung moon aren't really going to decide the outcome of a massive war. And the native PC UI could certainly handle this anyway or at least be modded to do so.

In that case all we need to do is ask Uber to not spend time crippling the API that is already there. I see nothing wrong with this on its own. But I do worry whenever I see a network aware API handed over to people who don't think about security much. This also brings up my personal paranoia about sand boxing the modifiable PA UI and its network capabilities from the rest of the OS.

I guess we will have to see if they produce anything genuinely useful.

Since you are the person asking for them I think you should know the answer to this question. However if you simply want Uber to not disable parts of the webkit API that they themselves didn't use then I am in agreement with you as long as it doesn't pose some kind of of security risk.

I am going to leave this link here.

Turns out I am going to be busy this weekend, so no proper replays until next weel.