Mr Fallon admitted that he touched Ms Hartley-Brewer’s knee during a Conservative Party conference dinner.
But, Mr Fallon insists he apologised for the incident at the time and both consider the matter closed.
A friend of the Defence Secretary said: “Julia’s a good friend of Michael’s. He overstepped the mark when he put his hand on her knee.
“She made clear it was unwelcome and he rightly apologised 15 years ago.”
Ms Hartley-Brewer said she threatened to punch the married father-of-two if he carried on his inappropriate behaviour.
She said: “I calmly and politely explained to him that, if he did it again, I would punch him in the face. He withdrew his hand and that was the end of the matter.”
The talkRADIO presenter said she did not feel like she was a victim of sexual assault and found the incident nothing more than “mildly amusing”.
It comes after Theresa May vowed to crack down on any MP proven to have acted in a sexually inappropriate manner.
The Prime Minister wrote to Commons Speaker John Bercow yesterday demanding new rules on MP’s behaviour saying the problem “cannot be tolerated any longer”.
Mrs May has asked the Cabinet Office to investigate the conduct of international trade minister Mark Garnier who admitted to calling his secretary “sugar t**s” and sending her to buy sex toys for him.
Former cabinet minister Stephen Crabb admitted to sending “explicit” messages to a 19-year-old woman after he interviewed her for a job.
Other allegations include a current Cabinet Minister who is alleged to have put his hand on a female journalist’s thigh and said: “God I love those t**s”.
A second Cabinet Minister is alleged to have had an affair with a junior female MP who is also an MP.
A former minister is accused of propositioning his secretary by asking her to “feel the length of my ****”.
The Guido Fawkes website released a spreadsheet of Tory MPs which includes two Cabinet ministers, 18 other serving ministers, 12 other MPs accused of inappropriate behaviour towards women and four accused of inappropriate behaviour towards men.
Some of the allegations on the spreadsheet include “handsy at parties,” “impregnated former researcher and made her have an abortion and had other female researchers” and “video exists of three males urinating on him”.
Three Labour MPs and an MP representing another party have also been accused of harassment.
Labour suspended Jared O’Mara MP last week over misogynistic behaviour.