The MP often worked only a two-day week as a justice minister, as opposed to his work as an MP, sources in his former department have claimed.
His office declined to comment on the two-day a week allegation.
He was also ridiculed for a plan to give 900 of the UK’s most violent young offenders martial arts training, an idea dubbed “Kung Fu Kids” by bemused colleagues.
The Justice Minister quit on Monday in protest at Theresa May’s Brexit policy.
A Ministry of Justice source said: “He would only work two days a week. Many in the department thought he was lazy.”
However, a friend of Dr Lee described the claims as “ridiculous”, and added: “My experience of him is he works incredibly hard. He used to be a GP. I don’t think you can accuse someone who used to be a doctor of being lazy.”
The friend said he was a great believer in the lessons taught by sport, like teamwork, respect, discipline and control, which was why he backed allowing more prisoners and youth offenders to access its benefits.
Dr Lee quit because he said his “first responsibility” was to his constituents.