AM I being unreasonable?

Yeah, I think you did. Throwing things is never an appropriate response, no matter how angry you are. And if you agree that his son's currently in a bad environment, then it isn't unreasonable for him to take the time he needs to try and figure out how to improve the situation (though granted, he shouldn't have responded to your annoyance the way he did). If you think that he should be fighting or at least asking her for primary custody, then take that up with him when you're calm enough to discuss it.