Backing Out, Without Ur Friends? Where Are They?

Here i am waiting for a game with a friend of mine,but then it's taking too long so what do we all do? Backout, yes we all backout and normally when you backout your party is coming allong. But they don't so you have to press your Xbox-Guide-Button and that takes time then you have to go to your Friends/Party and press invite friend and all that kinda things takes too much time we just wanna back out with the peeps u have invited and not keep doing the 'Xbox-Guide-Button' thingy it's kinda frustrating. So what would be nice is to backout with ur friends u have invited