I have found that if you have an enormous contact database for it to download, you'll slow the thing right down and cause problems - and even lose recognition for many of them.
My android was loading contacts from various Google accounts, linked in, my co-workers shared contacts, twitter, there were almost 700 total.
I worked some stuff on my phone to limit the contacts down to around 200 and the voice activation is running much more quickly.
Same goes for songs - the more songs you have on your ipod, the longer it will take to decide which song matches best to what you just said. Keep it clean with only the 200-300 that you actually want to listen to on a regular basis and it will work much faster compared to if you had several thousand.