According to a survey by the Pew Research Center, in eight of the ten European countries surveyed at least half are concerned about the terror threat.
The majority of people in five nations say refugees will be an economic burden and take away jobs and social benefits.
In Hungary, Italy, Poland and Greece, the majority of respondents say they have an unfavourable opinion of the Muslims. At least a quarter of every nation had the same belief.
The United Nations High Commissioner for Refugees told The Independent: "Refugees are people who are fleeing conflict, persecution and/or abuses of their human rights and they have a right under international law to seek asylum in other countries.
"Many will have encountered gruelling dangers as they fled for safety. All they want is to rebuild their lives in peace and safety.
Pointing to safeguards built into the 1951 Refugee Convention, they added: "Combatants/fighters are not entitled to refugee status, those involved in war crimes, crimes against humanity or other serious crimes may be excluded from refugee status, and refugees may be expelled under certain circumstances.
"The growing and misguided focus on the threat from refugees and asylum-seekers has enormous and damaging repercussions."
"This report points to the need to further create awareness in Europe (and elsewhere) about who refugees are and why some are coming to Europe. It is of concern to UNHCR the findings included in this report point to negative perception of refugees which feed into mistrust, but also discrimination and xenophobia."
The survey, which was conducted between April and May this year, covered the European countries of Germany, Sweden, the Netherlands, Britain, France, Spain, Italy, Greece, Hungary and Poland.
Underlining the divisive nature of the refugee crisis, the survey notes: “Across the EU nations surveyed, the refugee crisis has brought into sharp relief deep ideological divides over views of minorities and diversity.
“On nearly all of the questions analyzed in this report, people on the ideological right express more concerns about refugees, more negative attitudes toward minorities and less enthusiasm for a diverse society.”
More than one million refugees arrived in Europe in 2015, the majority coming from warn-torn Muslim countries such as Syria, Iraq and Afghanistan.
Anti-refugee populism has increased hugely, with political parties such as the National Front in France, the UK Independence Party, Alternative for Germany and Austria's Freedom Party making huge gains.
Europe has recently suffered several major terrorist attacks, including the assaults by the Isis group on Paris and Brussels that killed scores of people. Many of the attackers were European-born, but some are believed to have traveled to Syria to join Isis before returning to carry out the attacks.
Some 76 per cent of people surveyed in Hungary said they're concerned that refugees will increase the likelihood of terrorism in their country, followed by the Polish with 71 per cent.
Germany and Sweden — which took in the second most asylum seekers in 2015 — are the only countries where at least half say refugees make their nations stronger because of their work and talent.
The number of refugees in the world has reached the highest level ever recorded, according to figures published by the United Nations (UN), at an estimated 65.3 million by the end of 2015.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies