Tom Sandoval and Tom Schwartz.Photo: Gabriel Olsen/Getty

Tom Schwartz, Tom Sandoval

Appearing on Wednesday’sWatch What Happens Live with Andy Cohen, Schwartz, 40, admitted that he’s not going to “abandon” his best friend and business partner.

“Tom, I know the whole world hates you, but I’m not gonna abandon you, old buddy. I’m not gonna abandon you,” he said. “He knows he’s a monster for what he did. If you see him, give him a hug. Maybe he doesn’t deserve it, but give him a hug. He’s downbad.”

Sandoval, 40, and Leviss’months-long affairwas exposed on March 3 and ultimately ended Sandoval’s nine-year relationship with his costarAriana Madix.

Since then, Bravo viewers and theVanderpump Rulescast have vocalized their support towards Madix while chastising Sandoval and Leviss on social media. And while Schwartz said onWWHLthat he understands people’s strong reaction to the affair, he noted that some have taken it too far.

“I’m not trying to evoke any sympathy for Tom, he knows what he did was disgusting,” said Schwartz. “He knows he could not have handled that affair any worse. I understand the vitriol, I get it, but the vigilantism is getting a little out of control.”

He continued, “I want to remind people to take a breath. He knows what he did was terrible, so does Raquel, but the threats, it’s getting out of control. The mob mentality is getting out of control.”

Ariana Madix, Tom Sandoval and Raquel Leviss.Tommy Garcia/Bravo (3)

VANDERPUMP RULES – Season:10 – Pictured: Ariana Madix – (Photo by: Tommy Garcia/Bravo); VANDERPUMP RULES – Season:10 – Pictured: Tom Sandoval – (Photo by: Tommy Garcia/Bravo); VANDERPUMP RULES – Season:10 – Pictured: Raquel Leviss – (Photo by: Tommy Garcia/Bravo)

That “negative blowback” translated to Schwartz and Sandoval’s Los Angeles businesses too, includingSchwartz & Sandy’s— their newest cocktail lounge and restaurant that opened in November — and TomTom, their bar.

“Tom is not realizing, and I know he’s in shock — he’s now bad, he’s like a shell of himself — but he’s not realizing the negative impact this had on his other ventures; his business, his bar, his band, his friends,” said Schwartz. “First and foremost, Ariana, of course. But his business — it’s bad, man.”

“In the beginning, it was a little rough,” the reality star added. “There was a moment where people were like, vandalizing our spot. And it’s way out of line. I get it but vandalism is too far. Take a breather — go outside, and I mean this in the least patronizing way ever — but go outside, and take three deep breaths.”

However, Schwartz also noted that the tide has since started to turn. “I got to say, every time I go in, it’s kind of thriving,” he said. “And people are coming in — you guys, if you want to talk about this, all day, I’m happy to talk about this. But it’s been great.”

Schwartz’s words of support towards Sandoval weren’t met with love from his costars, including his ex-wifeKatie Maloney, who blasted him on social media.

Sharing a clip of her former husband saying the words onWWHL, Maloney, 36, wroteon Twitter, “You’re the only that wants to hug him dude. READ THE ROOM!!!”

Maloney also had some advice for her ex following his bombshell appearance on the Bravo show. “Tom, I have two words for you … media training. Because you’re just gonna say whatever lands in your head and that’s not good,” she shared on her Instagram Stories Wednesday night.

The 32-year-old continued, “Every time I sit here and say, ‘I’m over Scandoval. I don’t want to see it anymore.’ One of those f - - - - - - people just meets another level of buffoonery. And here I am. Here I am having to school more motherf - - - - - - on life. What Schwartz said has made me need to smack my vape 500 times and I can’t f***ing find it!”

Elsewhere onWWHL, Schwartz revealed exactlywhen he found out aboutSandoval and 28-year-old Leviss' affair, and why he didn’t inform Madix of what was going on.

“I learned in August, in late August about the affair, the one-night stand — allegedly,” theSchwartz & Sandy’sco-owner said, later adding that “there was a lot of grey area there” as to how deep the infidelity ran.

HostCohen, 54, then noted that during the filming of theVanderpump Rulesseason 10 reunionlast month, Schwartz claimed that he didn’t know about the affair until early this year — a point Schwartz clarified. “In January, Tom came to me and he told me he was in love with Raquel,” Schwartz said, explaining that their romance had progressed in a linear fashion behind his back and that Leviss was like “heroin” to Sandoval. “… In my defense, I did tell Tom exactly what he needed to do. And he agreed, and then he didn’t do it.”

As for why Schwartz didn’t tell Madix about the affair, Schwartz said he was “being fed a narrative that [Tom] has broken up with Ariana, or attempted to multiple times — many, many times. And he’s not happy, they’re not healthy, they’re not intimate, blah blah blah.”

He continued, “He did a good job of placating me … And he told me he has a game plan, he’s gonna do the right thing — and then he just kind of continued to procrastinate. He kind of fed me this narrative that they weren’t happy, they were going to break up.”

RELATED VIDEO:Vanderpump Rules: Ariana Madix Confronts ‘Stranger’ Tom Sandoval After Split and Says She Wants Him ‘to Die’

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Asked where his friendship with Madix lands now, Schwartz said she is mostly “indifferent” towards him.

“Ariana, I still love you,” Schwartz said. “I know you hate me right now. I still love you.”

Watch What Happens Live withAndy Cohenairs Sunday — Thursday nights on Bravo.

source: people.com