I think it's a really ungrateful job. If there's no fire, people think your job is a walk in the park. if there's a fire, you're incompetent.
I also hate being judged by colleagues, based on printer jams and other stuff that's completely out of your hands.

I seem to handle it better over the years, but every once in a while I'm still thinking of a carreer change.