Do You Divorce Your Friends-OPP

On Other People's Problems, a couple's best friends are divorcing. They can't agree if they should stay friends with the cheating spouse.

Mike & Kate-
My husband and I cannot agree on something and need your OPP advice.
We have been married 20 years, have three kids and have a wonderful marriage. When I met my husband his best friend was single. The two of them have been friends since they were in college. This friend met a woman about 15 years ago and they got married. They have two children.
Our two families are extremely close. We camp together most weekends in the summer. We go to concerts here in the valley. We BBQ all summer, have poker nights in the winter. We take the kids to the coast every other year. We've even gone to Disneyland together. They are our closest friends.
This summer, this couple broke up. She found out he has been having an affair for the last year. Instead of fighting for the marriage, I think he was relieved that he was caught. He has since moved out and is living with the mistress. As you can imagine, his soon to be ex wife is completely devastated. She is heartbroken and trying to keep it together for their two kids. This was a complete shock to my husband and I. We had no idea. Nothing ever seemed out of the ordinary. I feel like our extended family has broken up. We were that close.
The reason I wrote you for OPP is because my husband has continued to be friends with this man. I was shocked by the fact that he could spend time with a man who would do this to his family. He says, they were friends before the marriage and that their friendship has never been based on the marriage. I understand that, but now that he has seen what little character this man has, how can he still be friends with him? The wife is so hurt that my husband is doing this. She said it feels like a slap in the face. How could my husband not be loyal to her? I wouldn't mind so much if my husband saw him when I wasn't around. Like grabbing a beer after work or something but he is bringing this man around my house and on our camping trips. He gets the kids every other weekend so when he has them, we are still doing "family stuff" stuff with him. The only difference is that his wife is not there. She is at home crying her eyes out. Thank goodness he is not bringing the new girlfriend around...yet.
I told my husband this makes me uncomfortable. He says, "he didn't cheat on us. This has nothing to do with us." I disagree. This affair affected all of us. I feel like I'm being disloyal to my friend. And why would we want to have someone in our lives who has such little value for his family. My husband and I cannot find common ground on this. Who is right?

Thanks