Bungie is somewhat unpredictable when it comes to their new projects. Their most recent game under the Halo name, Halo Wars, met some mixed reviews, and questions arose about how the future would affect the Halo franchise.
I can tell you a few things relating this issue:
-The games are classified in somewhat different genres. Halo 3 is a more in your face, fast paced action shooter (something that some people say Call of Duty does better), while ODST is supposed to be a more strategical, tactical shooter. Only you know which of those genres you like best.
-Halo 3 has positive and negative aspects about its online play. On the plus side, there are many many players, and you can always find a game. On the down side, the game has released many map packs, and the online modes expect you to purchase them in order to play. If you buy Halo 3, expect to spend $30 on maps right off the bat just to be able to play all the online game types.
That about sums it up. I'd recommend ODST, but you'll want to see it's reviews before you make a complete decision.