First know your internet connection. It's best recommended that you use at least a broadband connection for XBL.
Second, when you are in a pre-game lobby in matchmaking, you may press 'X' to choose connection types from best connection to quickest joining time. When you choose the option for best connection, it'll seek out players with the best connections but may take a while
Thirdly, it also depends on what game type and the level of players. Matchmaking tries to find people who are near your level of play. Everytime it does not find available players, it widens the range of player attributes to search. This does drag out the time it takes to find a match.