A quarter of agencies that provide care to people in their own homes do not meet all five national standards of quality and safety, regulators said.
The Care Quality Commission (CQC) found that 26% of agencies did not meet all the criteria set out to ensure people are being cared for properly.
Charities said that more needs to be done to ensure that elderly people are treated with the "dignity and respect they deserve" following the new CQC report which flagged up some areas of concern in the sector.
The CQC said that some people who rely on carers do not feel valued when visits are cancelled or delayed. And some people were "resigned to accepting a level of unreliability within the service," the report states.
Inspectors also raised concerns about about unsupported staff, poor care planning and lack of continuity of care workers.
The report, based on inspections of 250 domiciliary care agencies providing at-home support and care to around 26,500 people, also identified failures of providers to listen to people using services and their families and carers.
People who used the providers had a "reluctance to complain" if care standards slipped, with some people worried about getting their regular care workers into trouble and others may be concerned about reprisals, the report adds.
The CQC measured the agencies on respecting and involving people who use services; the care and welfare of people who use services; how the agencies safeguarded people who use services from abuse; how providers support their staff and how they monitor the quality of the services they deliver.
Inspectors found 184 of 250 providers were meeting all five standards and are delivering a good service.
David Behan, chief executive of the CQC, said: "People have a right to expect to be treated as an individual, to be able to exercise choice, and to make sure their carers are aware of their specific care needs.
"We found plenty of evidence of this however we also found elements of poor care which happen too often."
The National Pensioners Convention said the Government and local authorities need to do more to improve standards of care.
Dot Gibson, general secretary of the pensioner organisation, said: "Around 80% of home care is now provided by the private sector, yet a significant number of them are failing to treat older people with the dignity and respect they deserve.
"Local authorities have a responsibility to start commissioning services on the grounds of quality - rather than for the lowest price. Contracts should only be awarded to those who can guarantee that staff are properly trained and qualified to do the job - and the idea that services can be provided in blocks of 15 minutes at a time has got to stop.
"It is virtually impossible to maintain someone's dignity in a 15 minute visit and expecting staff with compassion to behave in this way is barbaric, as much as it is for the resident.
"The system as it stands it set up to fail.
"The Government needs to recognise that services should be funded in a way that we can ensure care workers feel able to take time, without fear of having to rush."
Jeremy Hughes, chief executive of the Alzheimer's Society, said: "Half a million people with dementia live in the community and as the brain gradually shuts down many will need help with everyday tasks such as eating meals, washing or going to the toilet.
"However, patchy quality means that some people are being forced to struggle with poor and in some cases undignified care.
"Good quality home care not only maintains dignity but also prevents people reaching crisis point and going into hospital unnecessarily or care homes early. If people are to be supported to live at home for as long as possible we must ensure that the good care highlighted in this report is being seen across the board."
Equality and Human Rights Commission chief executive officer Mark Hammond said: "Very many more people receive home care than residential care. Many older care users are reluctant to complain, so home care needs to be better managed and closely and carefully regulated, to protect their human rights.
"We are pleased that the CQC has put home care under scrutiny through this review. The CQC needs to continue this increased regulatory attention on providers and listen closely to the voices of home care service users."
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies