And he said if the competing wings of the Conservative Party did not manage to come to an accommodation over Brexit, they risked letting Labour leader Jeremy Corbyn into Number 10.
Lord Lamont, who served as Chancellor under Prime Minister John Major for four years, from 1990 to 1993, is a keen Brexiteer and a member of the Leave Means Leave campaigning organisation.
Writing in a column for the Daily Mail, he said nobody wanted to undermine the Good Friday agreement which is widely believed to have brought peace to the province, nor did they want a hard border.
He added: “Yet is the issue really so difficult?
“Why is the question of a border dividing northern and southern Ireland so different from Switzerland’s border with Germany, France, Italy or Austria?
“Switzerland is not in the EU, yet the border is barely noticed by the passing traffic.”
Most customs and other checks were carried out in warehouses, while 99 percent of customs declarations were electronic, he pointed out.
He added: “To me, the so-called issue of the Irish border seems a ploy by the EU and Ireland to keep us in the customs union or half in the EU.
“Even so, the Prime Minister has ruled out ‘Canada-plus precisely because of it
“Had the Irish border not been made into such a problem, it is difficult to see how anyone could possibly have objected.
“Moreover, technology helping overcome these issues is advancing all the time.”
Mr Lamont also said some Brexiteers also needed to “give some ground”.
He added: “The stakes are high after all.
“The public is exasperated. And, however much we like to reassure ourselves that he could never be elected, a government led by Jeremy Corbyn is not impossible, despite all the horrors that would inevitably follow.”
Mr Lamont is the latest senior politician to suggest the EU is using the issue of the border to frustrate UK efforts to quit the bloc.
However, speaking to Express.co.uk earlier this month, Dr Katy Hayward, a reader in Sociology at Queens University Belfast said it was important not to gloss over the problems it presented.
She warned: ““People have forgotten what it used to be like – those living near the border are highly sensitive to any change in the arrangements.
“I’ve heard this described as ‘reopening an old wound’.
“Checks away from the border are still checks.
“They are still symbolically contentious and would be a source of ‘friction’, both in terms of administrative demands and in terms of political sensitivities.