Please Fix Replays Search

@chargrove, out of interest do you use something like s3 or a proprietary solution for your storage?

In the early livestreams there was talk about the ability to download replay files and the eventual size of said files. Neutrino said that storage space is essentially free and people shouldn't care about file sizes.
I saw that coming to be honest...

Do people even watch such old replays? If I watch a replay it was always one from the same week or so.

Very rarely. So having access to past 90 days is an excellent step forward.

I didn't know about the 90 day thing until yesterday either.

My guess is it was a communication issue. We still have the old replays; we're not deleting them, but apparently access for them from cold storage is not an automatic thing relative to the way we need to access them for replay browsing purposes, which is a much more frequent hit on our storage servers.

Not sure if this was unexpected or just lost in translation as different engineers talk about this kind of stuff amongst themselves. Maybe we'll start off with the 90 day thing and eventually add some mechanism for fetching an old replay from cold storage by special request, or something like that. I have no idea yet; this is all coming in hot.

We're going through PlayFab as an intermediary, which at the moment uses AWS/S3 for storage. PA goes through PlayFab APIs though and does not make direct assumptions about S3 (at least it shouldn't); ostensibly if PlayFab chose to change their backend in the future they could do so without affecting PA itself. With that in mind, when I say "cold storage" I'm currently talking about AWS Glacier vs. regular AWS S3 storage, but that could change in the future.

I just tested a replay from http://pastats.com/chart?gameId=96621 and it does go to "Starting replay server" but so far it stays there. For a 7min replay I would expect it to start pretty fast.
Dunno, maybe it's related to the general server slowness that it takes this long, but it definitely still does find replays more than 3 month old at least. You just need to know the lobbyid of the game. Dunno if it will launch or crash though. I would hope it can work even with replays from "cold storage"

I guess you probably should have some sort of database of replays that has the lobbyids. Then it would not really matter where you have the actual replay file.

We have a database, but some of the metadata it scrapes is in auxiliary files on S3 as it's too big to keep in the database directly.

It's all easier in theory than it is in practice.

What kind of metadata is that? I can't think of any metadata that big. Player names/ids?
The replay server is still "starting". Dunno, seems like it'll never start.
If I directly access a 3+ month old replay is it supposed to work?

Isn't everything? At least we don't get booted like before!

To make a long story short, the metadata isn't all that big (just player names, ids, army information etc.) but just big enough to exceed the 1k limit for Simple DB strings. The usage of Simple DB vs. a more heavyweight SQL server is a more complicated matter that I will not discuss here.

As for direct access working beyond 90 days, I think right now the answer is probably No, because the replay APIs are currently designed against regular S3, not Glacier. Again, I just found out about the 90 day thing yesterday, I can't tell you what this will look like going forward at this point.

1k limit on Simple DB... hmmm okay. Thanks for the info. I guess whatever reasoning exists behind that, it'll have to somewhat change to allow a fully working replay database or you need to somehow make the metadata fit into Simple DB

Yeah I gave up after ~40 minutes, the game just kept telling me that the replayserver would be started. It seems PA does not handle that case at all, fits nicely with you only found out about it yesterday ^^

So you keep replay files on glacier that nobody can launch right now

Yup, welcome to game development. Everything on fire, all the time.

Even glaciers

glacier from what i understand is either tape based or blu-ray based, hence the crazy times.

EW