The Brexiteer appeared on Sky News to discuss a talk he gave at Bristol University which was attended by protestors wearing balaclavas.
Video emerged of the backbencher getting involved in a scuffle in which he attempted to de-escalate the situation.
When asked if he feared for his safety, Mr Rees-Mogg replied: “What has happened is a peaceful protest got out of hand.
“I intervened because I didn’t want anything like that to happen.
“Unfortunately, it became more scuffley.
“I never felt threatened or unsafe. I am a lot more concerned about abuse of women MPs particularly on social media is much more troubling and dangerous.”
Jacob Rees-Mogg spoke at a dinner hosted by a man who advocates the repatriation of “non-indigenous” Britons.
When asked about this gaffe, he said: “That is a disgraceful organisation, I did not realise they had those views.
“They were called the Traditional Britain Group, I thought it sounded harmless.
“Clearly it was a mistake to go, I think it is a bad group.
“I made a mistake but one learns from one’s mistakes. I am very sorry I made that mistake.
“I think they have unpleasant views which I do not share.”
When asked if the activists disrupted his talk in protest against this, Mr Rees-Mogg dismissed the idea.
He added: “They didn’t mention the dinner, they just shouted. You can’t have a rational debate with someone who just shouts.”
Mr Rees-Mogg caused controversy among students who raised questions about whether to ban the politician from giving his speech at Bristol.
A video was posted on social media showing Mr Rees-Mogg involved in the fracas as a punch is thrown.
It is unclear whether the punch was in the direction of the Brexiteer, but he appears to step in front of an antagonist.
Mr Rees-Mogg can then be seen pushing the man in sunglasses and hat away, seemingly trying to ease the tension in the room.
Chloe Kaye, who posted the video on Twitter, wrote: “A huge amount of (physical) violence a Jacob Rees-Mogg speech in UWE Bristol.”
Speaking later to Express.co.uk, the politics student said: “I went into the talk to hear about Jacob Rees-Mogg and suddenly as soon as he comes in about six masked individuals run in screaming, ‘bigot, racist, sexist’.
“They’re screaming, absolutely no university security to be seen. Jacob Rees-Mogg screams, ‘I believe in free speech,’ so he runs up to them and actually wants to start talking to them.”