Mr Swann and Ulster Unionist MEP Jim Nicholson delivered their pointed messaged during a meeting with the EU’s chief Brexit negotiator in Strasbourg this morning.
The UUP backed the Remain campaign ahead of the 2016 referendum, in which Northern Ireland as a region voted by 56per cent to 44per cent to stay within the EU.
However, the party’s position since the vote is that the overall result should be respected and Brexit implemented.
Mr Swann said the discussions with Mr Barnier were “constructive and business-like”.
He said: “We were keen to address a number of the misconceptions currently being peddled by some.
“For instance, despite what others say, unionists who may have voted remain, definitely did not vote to leave the UK.
“It was a democratic vote which all unionists respect. Unionists are unionists for a reason.
“They want to remain part of the United Kingdom of Great Britain and Northern Ireland.
“We wanted Mr Barnier to be absolutely clear about that.
“The tone of the language being used by some, more specifically the Irish government, is less than helpful.
“The Irish Government in particular seem intent on using Brexit as a Trojan Horse to pursue Irish Unity by other means.”
Mr Swann accused the Irish Government of undermining the key principle of the Good Friday Agreement – that it is for the people of Northern Ireland to decide their constitutional future.
He said: “We also told Mr Barnier that Northern Ireland welcomed the help and assistance of the EU over the years through its peace programmes.
“It is widely recognised that the EU did great work in this area, but that good work is now being undermined because there is a growing perception that the EU is now siding with nationalists against unionists.
“We impressed upon Mr Barnier of the potential downside of doing so.
“We have a relative peace in Northern Ireland but we don’t want that equilibrium being upset by Northern Ireland being used as a bargaining chip in Brexit negotiations.
“We invited Mr Barnier to come to Northern Ireland and speak directly to unionists to get their views.”