Same Internet, Host Migration Fail?

ok, so me and my brother play MNC, on the same internet (in the same house, same hardwire internet) different xboxs. Now a few times while migrating host, it will fail for me, but he will still be in the game with other people, and vice versa, how is it only failing for one of us at a time? Shouldnt if it fails, both of us are kicked out, not just one of us?

Also something that gets annoying is when it says "Unable to migrate host" before it even tries to migrate the host.. nothing game up previous to that.. no attempting to migrate host or anything, just kicks you out of the game a few times.

Not quite the same, but my buddy and I are both on the same provider and about a mile or two away from one another, and frequently one of us will lose connection to the host or get a migration failure while the other does not.

Though I usually blame it on Comcast, but it could be a problem with the game I guess.