Paulie Shore was so annoying that even though he hasn't done anything in 25 years, he still irritates me when I think about him. It was intentional, but something about that dumbed down schtick when there's nothing that's actually funny that I can't sit through for very long.
Gilbert Gottfried was purposely annoying like that and I often couldn't stand him, but then I watched some really obscene things he did and found him hilarious.
Someone mentioned Adam Sandler and I've had kind of the same issue with him. When he was on SNL, Phil Hartman, Dana Carvey, Jon Lovitz and Chris Farley were amazing and I found it irritating that Sandler was the one that ended up with big box office doing jokes from 4th grade. I don't hate him, though. When he's being himself in talk show interviews, he seems likable.
Similar to the way some of you react to Baldwin, James Woods annoys me as a Twitter troll, but like RG said about Baldwin, I respect him as an actor (though maybe not a coincidence that he excels at playing psychos.) And, he's really smart with an IQ around 190 - just ask him.
In general, the people who are massively famous with no talent at all, like the Kardashians, probably annoy me the most. But, I never watch anything about them, so I don't think about it much.