Please Fix Replays Search

Discussion in 'Planetary Annihilation General Discussion' started by brianpurkiss, September 2, 2014.

  1. Bsport

    Bsport Well-Known Member

    Messages:
    268
    Likes Received:
    415
    @chargrove, out of interest do you use something like s3 or a proprietary solution for your storage?
  2. ace63

    ace63 Post Master General

    Messages:
    1,067
    Likes Received:
    826
    In the early livestreams there was talk about the ability to download replay files and the eventual size of said files. Neutrino said that storage space is essentially free and people shouldn't care about file sizes.
    I saw that coming to be honest...
    tatsujb likes this.
  3. Remy561

    Remy561 Post Master General

    Messages:
    1,016
    Likes Received:
    641
    Do people even watch such old replays? If I watch a replay it was always one from the same week or so.
  4. brianpurkiss

    brianpurkiss Post Master General

    Messages:
    7,879
    Likes Received:
    7,438
    Very rarely. So having access to past 90 days is an excellent step forward.
    Remy561 likes this.
  5. chargrove

    chargrove Uber Alumni

    Messages:
    107
    Likes Received:
    350
    I didn't know about the 90 day thing until yesterday either.

    My guess is it was a communication issue. We still have the old replays; we're not deleting them, but apparently access for them from cold storage is not an automatic thing relative to the way we need to access them for replay browsing purposes, which is a much more frequent hit on our storage servers.

    Not sure if this was unexpected or just lost in translation as different engineers talk about this kind of stuff amongst themselves. Maybe we'll start off with the 90 day thing and eventually add some mechanism for fetching an old replay from cold storage by special request, or something like that. I have no idea yet; this is all coming in hot.

    We're going through PlayFab as an intermediary, which at the moment uses AWS/S3 for storage. PA goes through PlayFab APIs though and does not make direct assumptions about S3 (at least it shouldn't); ostensibly if PlayFab chose to change their backend in the future they could do so without affecting PA itself. With that in mind, when I say "cold storage" I'm currently talking about AWS Glacier vs. regular AWS S3 storage, but that could change in the future.
    wondible likes this.
  6. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    I just tested a replay from http://pastats.com/chart?gameId=96621 and it does go to "Starting replay server" but so far it stays there. For a 7min replay I would expect it to start pretty fast.
    Dunno, maybe it's related to the general server slowness that it takes this long, but it definitely still does find replays more than 3 month old at least. You just need to know the lobbyid of the game. Dunno if it will launch or crash though. I would hope it can work even with replays from "cold storage"

    I guess you probably should have some sort of database of replays that has the lobbyids. Then it would not really matter where you have the actual replay file.
  7. chargrove

    chargrove Uber Alumni

    Messages:
    107
    Likes Received:
    350
    We have a database, but some of the metadata it scrapes is in auxiliary files on S3 as it's too big to keep in the database directly.

    It's all easier in theory than it is in practice.
    websterx01 likes this.
  8. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    What kind of metadata is that? I can't think of any metadata that big. Player names/ids?
    The replay server is still "starting". Dunno, seems like it'll never start.
    If I directly access a 3+ month old replay is it supposed to work?
  9. websterx01

    websterx01 Post Master General

    Messages:
    1,682
    Likes Received:
    1,063
    Isn't everything? At least we don't get booted like before!
  10. chargrove

    chargrove Uber Alumni

    Messages:
    107
    Likes Received:
    350
    To make a long story short, the metadata isn't all that big (just player names, ids, army information etc.) but just big enough to exceed the 1k limit for Simple DB strings. The usage of Simple DB vs. a more heavyweight SQL server is a more complicated matter that I will not discuss here.

    As for direct access working beyond 90 days, I think right now the answer is probably No, because the replay APIs are currently designed against regular S3, not Glacier. Again, I just found out about the 90 day thing yesterday, I can't tell you what this will look like going forward at this point.
    cola_colin likes this.
  11. cola_colin

    cola_colin Moderator Alumni

    Messages:
    12,074
    Likes Received:
    16,221
    1k limit on Simple DB... hmmm okay. Thanks for the info. I guess whatever reasoning exists behind that, it'll have to somewhat change to allow a fully working replay database or you need to somehow make the metadata fit into Simple DB ;)

    Yeah I gave up after ~40 minutes, the game just kept telling me that the replayserver would be started. It seems PA does not handle that case at all, fits nicely with you only found out about it yesterday ^^

    So you keep replay files on glacier that nobody can launch right now :p
  12. chargrove

    chargrove Uber Alumni

    Messages:
    107
    Likes Received:
    350
    Yup, welcome to game development. Everything on fire, all the time.
  13. Raevn

    Raevn Moderator Alumni

    Messages:
    4,226
    Likes Received:
    4,324
    Even glaciers :p
  14. Bsport

    Bsport Well-Known Member

    Messages:
    268
    Likes Received:
    415
    glacier from what i understand is either tape based or blu-ray based, hence the crazy times.
  15. websterx01

    websterx01 Post Master General

    Messages:
    1,682
    Likes Received:
    1,063
    EW

Share This Page