Yes, they hate it when you don't "follow the script".
I got a call once from someone claiming to be from some company reviewing a car accident I had the year before. The line of questions seemed odd and I kept telling them that I failed to understand what this was about since the claim had been paid. The caller got so frustrated with me, she got angry and said she'd have to call me back. I'm still waiting.
Two calls recently about my overdue taxes from the CRA.
Most of the time, I tell the caller that I am not home and please leave a message I can relay. A legit person will leave some sort of message.
Gotta love call display. .. I too don't answer the phone if I don't recognize the name or number and let it go to voice mail. I'll call you back.