The 15 council members expect the Syrian president to detain and hand over to UN investigators members of his family who have been implicated in the killing of Rafik Hariri last February.
The council unanimously adopted a resolution calling on Syria to co-operate unconditionally with the UN investigators or face unspecified further action.
But, in the face of stiff resistance from Russia and China, which feared the resolution was being used to politicise the UN investigation, the resolution was watered down to drop the threat of economic sanctions.
Several foreign ministers, including Condoleezza Rice and Jack Straw, flew to New York to adopt the resolution, drafted mainly by the US and France, which also took the lead at the UN in ensuring the withdrawal of Syrian troops from Lebanon after 30 years.
Fulfilling the terms of the resolution will be an extremely delicate matter for President Bashar al-Assad. German prosecutor Detlev Mehlis' report implicated the Syrian leader's brother-in-law, Assef Shawkat, the chief of military intelligence, and his brother, Maher Assad, head of the powerful Republican Guards.
The resolution provides for a financial freeze and travel ban on individual suspects designated by the UN commission, headed by Mr Mehlis, or by the Lebanese government. However, any Council member can object to a name on the list.
The resolution expressed "extreme concern" at the Mehlis report's conclusion that "there is converging evidence pointing at the involvement of both Lebanese and Syrian officials in this terrorist act".
Mr Hariri was killed on 14 February, with 22 other people, when his car was blown up in Beirut.
The resolution also picked up the complaint in the Mehlis report that Syrian officials had deliberately misled the UN team.
Speaking after the vote, Ms Rice made it clear the death of Mr Hariri appeared linked to his support for the departure of Syrian forces from his country. "The Syrian government has actively and consistently worked to break the will of the Lebanese people - and to thwart the will of the international community," she said. "With our decision today, we show that Syria has isolated itself from the international community - through its false statements, its support for terrorism, its interference in the affairs of its neighbours, and its destabilising behaviour in the Middle East."
She said the Syrian government it "must make a strategic decision to fundamentally change its behaviour". Jack Straw said the Security Council is "putting the government of Syria on notice that our patience has limits". He added: "Failure to co-operate fully and now will oblige us to consider further actions to ensure that the Security Council ... can play its part in the Lebanese government's determination to see justice done."
Syria's Foreign Minister Farouk al-Shara, also in New York, said Mr Mehlis' report contained "no evidence" to prove Syrian involvement in Mr Hariri's death. He offered to demonstrate Syria's co-operation with the inquiry in a future closed session of the Security Council.
In letters sent to Britain, the US and France, President Assad pledged that "any Syrian" could be brought to trial who could be proved "by concrete evidence" to have played a role in Mr Hariri's killing.
After meeting Egypt's President Hosni Mubarak over the weekend, the Syrian president ordered a state-run investigation into suspects, which official sources pledged would meet the demands of the UN inquiry.
In Damascus, a leading opposition figure called for the resignation of President Assad. "I ask the President and the Syrian government to present their resignations," said veteran politician and opposition activist Riad al-Turk. "We need a transitional government to lead the country into democratic elections. This is the only way out our current dilemma."
The demand by Mr Turk, who is seen as the godfather of the Syrian opposition, is the first to directly challenge the president's power.