I don't think either of them was really out of line.
Buffy came to make sure Angel was ok, now that wasn't her only motivation, she wanted vengence on both Faith and Angel. Faith for all the obvious reasons that I don't think we need to go into. Angel on the other hand, she was still unbeliveably pissed and hurt at his departure from Sunnydale, I don't think she really understood why he left. She had been his entire universe in Sunnydale and now she turns up in L.A. and realises it's not true anymore. I think that was a very bitter pill for her to swallow. He had Cordy and Wes, his very own Willow and Xander and at the time they seemed to be a hell of a lot closer than the Scoobies. Another blow for Buffy. Then she sees him with Faith and snaps. I think in her head Angel is very much hers and after Faith sleeping with Riley seeing her with Angel too pushed her over the edge. Now I could be pushing it with this theory but here it goes anyway, I think Buffy was also angry with Angel for Riley sleeping with Faith. In the back of her head there must be voice telling her that Angel would have known. I think that annoys her more than anything, especially since it was essentially Angel's doing that caused he relationship with Riley. More than anything I think she is confused and hurt by this new Angel. No matter what, he always had her back but now thats all changed cuz he has a job to do and for the first time since they met she is not a part of it. For anyone looking on you can see how far Angel has come since leaving Buffy and I think that breaks her heart, that he WAS right to leave, so as she is no place in his shiny brand new life she throws hers in his face to hurt him like she is hurting. Suddenly Buffy doesn't have the power in their relationship and so she hurts him the only way she knows how, with words and the life he wanted for her.
Angel was just doing his job in my opinion. Maybe he shouldn't have hit her. He didn't have time for her vendetta and he was cut up about Riley. The Day that Wasn't was still very fresh in his mind and even though he knew she would not remember I think a part of him hoped she would and was angry that she didn't. Then when she flung that bull about loving and trusting Riley, it was all he could take so he kicked her out of L.A.
Essentially I don't think it's about who was out of line, it's about two people who care deeply for each other not knowing how to exist in this new stage of their relationship, so they strike out and hurt one another to protect themselves. Like Spike said 'Your not friends, you'll never be friends'.