The eight-strong group, who were escorting a junior diplomat, has now left the country bound for Malta on board HMS Cumberland.
However, Foreign Secretary William Hague said the government intended to send further diplomatic personnel soon to "strengthen dialogue" with rebel leaders.
"I can confirm that a small British diplomatic team has been in Benghazi," Mr Hague said. "The team went to Libya to initiate contacts with the opposition. They experienced difficulties, which have now been satisfactorily resolved. They have now left Libya.
"We intend, in consultation with the opposition, to send a further team to strengthen our dialogue in due course. This diplomatic effort is part of the UK's wider work on Libya, including our ongoing humanitarian support.
"We continue to press for Gaddafi to step down and we will work with the international community to support the legitimate ambitions of the Libyan people.
According to reports, guards challenged the SAS team when they arrived at an agricultural compound in the eastern city of Benghazi.
They were detained after a search of their bags revealed ammunition, explosives, maps and fake passports.
The news came as the battle for control of Libya raged, with both sides entrenched in key locations and a drawn-out civil war looking increasingly likely.
Airstrikes by Gaddafi's forces reportedly hit the strategic oil port of Ras Lanuf today, but failed to reclaim it.
They fared better in the town of Bin Jawwad - about 110 miles east of Gaddafi's birthplace of Sirte - and are apparently back in control.
In the capital Tripoli, residents awoke before dawn to the crackle of unusually heavy and sustained gunfire that lasted for at least two hours.
The US has moved military forces closer to Libya's shores to put military muscle behind its demand for Gaddafi to step down immediately.
But Washington has expressed caution over the idea of imposing a "no fly" zone over the North African nation so the Libyan leader cannot use his warplanes to attack the population.
Speaking earlier, Defence Secretary Liam Fox indicated that a no-fly zone was still on the cards, saying the issue would be discussed at a meeting of Nato defence ministers later this week.
"If we (Nato) were to decide on a no fly zone, there would be a number of options. Would it be the whole of Libya, would it just be the population centres that we focused on?" he added.
Dr Fox refused to comment on the activities of UK special forces in Libya.
But he said the government was trying to "build a picture" of the situation.
"We want to clearly understand what the dynamic is here because we want to be able to work with them to ensure the demise of the Gaddafi regime, to see a transition to greater stability in Libya and ultimately to more representative government," he said.
Meanwhile, David Cameron reiterated the UK government's demand for Gaddafi to step down immediately.
In a speech to the Conservative Spring forum in Cardiff, he also launched a stinging attack on Labour for doing "dodgy deals in the desert" - a reference to Tony Blair's controversial agreement with Gaddafi in 2004.
The Prime Minister's comments came despite Mr Hague previously saying the former government's level of engagement with the dictator had been "right" because it persuaded him to abandon weapons programmes.
Mr Cameron said: "While Labour were doing dodgy deals with dictators in the desert, remember, we - the people of this party - were out volunteering in Rwanda, building schools and teaching English."
He also urged "solidarity" with those "out on the streets of North Africa and the Middle East today".
"When Margaret Thatcher was Prime Minister, this party stood by those who wanted to reject Communism and embrace freedom," the premier said.
"And today, this party stands by those reaching for that same freedom in the Arab world.
"On Libya, our strategy is clear.
"We will continue to intensify pressure on the regime.
"We will continue to state clearly that international justice has a long reach and a long memory, and that those who commit crimes against humanity will not go unpunished.
"We will continue to provide humanitarian assistance to those affected by this crisis, and continue to demand access for aid agencies to reach those in need.
"And we will continue to plan, with our allies, for every eventuality.
"But let me repeat, it is time for Gaddafi to go."
Comments