Is it my anxiety or a genuine concern?
I guess as you have found out trust is fragile and once broken is hard to regain. While I suppose there does need to be contact between your partner and his ex as children are involved it really is up to him to help put your fears to rest.
Having given you one cause to be suspicious with that incident in August he really should be going out of his way as often and as long as it takes for you to feel secure. He should do this out of simple care for you and your happiness.
You did say he had been trying. I suppose there is a possibility that he has in fact been behaving properly, and his ex has been stirring up trouble. If that were the case not mentioning might be understandable. Whatever the case if you both want the relationship to continue trust needs to be reestablished.
Can you talk with him and put the onus on him to ensure your peace of mind? His simply getting angry is no real answer. Do you think that couples counseling might be worth trying?
Being in this situation must feel as if you are very alone. do you have anyone, parent, family member or friend to support you? An outside perspective and care might be quite a help.
All this, as you say, will have made your depression and anxiety that much worse. Do you mind if I ask if you are under medical treatment at the moment? I'd think it would be a good move if you have not done so already.
Please keep in touch, and let us know what you think