After Grimm lose his family he had a fight with Kill Joe (when he was still alive) but lose.

Joe made Grimm his personal toy, and made Grimm suffer a really painful despair time, also that is one of the reasons that he has a scar in his left eye, Joe just didn't kill Grimm because he wanted that Grimm despair increase.
Some time later Kill Joe died before his rematch with Grimm. Grimm feel reliefed, because he didn't wanted to see that devil again.

Grimm really had a big trauma with Joe.

Yeah, today wasn't the perfect day for me, but I am still alive, so that is what matters. Don't forgert to Hope guys!! ;)