Photo: Arturo Holmes/Getty

Ben Affleck and Jennifer Lopez attend “The Last Duel” New York Premiere at Rose Theater at Jazz at Lincoln Center’s Frederick P. Rose Hall on October 09, 2021 in New York City.

Ben Affleckgot candid about his personal life in an eye-opening interview with longtime radio show hostHoward Stern.

“Was there a hesitation on your part, because now you’re a dad?” Stern asked Affleck, noting that Lopez “brings even more attention” to everything he does.

“It crossed my mind for sure,” Affleck told Stern of getting back together with the “Let’s Get Loud” singer. “My responsibility to my children is the highest responsibility. I don’t want to do anything that is painful or destructive to them if I can help it.”

Ben Affleck and Jennifer Lopez.Vittorio Zunino Celotto/Getty

Ben Affleck and Jennifer Lopez

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Affleck shares daughters Violet, 16, and Seraphina, 12, plus son Samuel, 9, with ex-wife Garner, 49.

For more onBen Affleckand other top stories, listen below toour daily podcast on PEOPLE Every Day.

“That being said, I know that my life affects them,” Affleck continued, noting that the fact that he and Garner are celebrities is a “cross to bear” for their children.

In addition to briefly touching on his romance with the “On My Way” singer, 52, theTender Barstar spoke openly abouthis marriage to Garner, and how the couple tried to make their relationship work.

RELATED VIDEO:Ben AffleckandJennifer LopezAre All Smiles as They Cuddle Up Courtside at Lakers Game

“The truth was we took our time, we made the decision, we grew apart,” he explained to Stern. “We had a marriage that didn’t work. This happens … We tried because we had kids. We did not want it to be the model of marriage for the kids. We did our best. I knew she was a good mom, and I hoped she knew I was a good dad.”

AlthoughGarner and Affleck finalized their divorcein 2018, the couple continue to amicably co-parent their three children — Garner evenjoined her ex-husband and Lopezover Halloween to take their kids trick-or-treating together.

Ben Affleck and Jennifer Lopez.Steve Granitz/FilmMagic

Ben Affleck and Jennifer Lopez

In November,a source told PEOPLE, “They are doing really well. Jennifer loves being with Ben. She feels it’s truly meant to be.”

A separate source revealed that although the couple’s schedules are demanding, Affleck and Lopez are looking forward to spending the holidays together this year. “For Jennifer, Christmas is all about the kids,” the source shared. “She makes it very special for them. She loves Christmas.”

The source added, “This year, it will take some more planning since Ben wants to be with his kids, too. They haven’t made any specific holiday decisions yet and are still working it all out.”

source: people.com