Jeremy Wright said the former foreign secretary suggesting Muslim women who wear the garment "look like letterboxes" was not "helpful".
Mr Johnson triggered a furious backlash after using a newspaper column to call the niqab "absolutely ridiculous" and suggest wearers "look like letterboxes" and "a bank robber".
Theresa May and Brandon Lewis, the Tory chairman, both called on him to apologise but he has so far refused to do so.
Mr Wright said it was important for people to "choose our language with care".
He told the BBC Radio 4's Today programme: "When you are discussing a subject such as this, then I think describing it as people looking like letterboxes isn't helpful.
"I think we should all choose our language with care.
"Boris and others are perfect entitled to have their views on this subject but we should all be careful how we express ourselves and not cause offence if we can avoid it.
Asked about Mr Johnson's suggestion that the niqab is "oppressive" and "absolutely ridiculous", he said "That's the sort of language I think we should try and avoid using."
But he added: "We're not disagreeing that it's an important conversation to have, neither are we disagreeing that there are circumstances in which the burqa should be removed."
Debate about the niqab was "perfectly legitimate", he said. Mr Johnson criticised the wearing of the veil but said it should not be banned.
Mr Wright said he was sure that, "on reflection", the Uxbridge and South Ruislip MP would want to reconsider the language he had used.
But asked whether he thought Mr Johnson should have the Conservative whip removed, he said: "That's not a decision for me."
It comes after a number of Conservatives, including Theresa May, joined widespread criticism of Mr Johnson.
The prime minister said: "I do think that we all have to be very careful about the language and terms we use.
"And some of the terms Boris used describing people's appearance obviously have offended. So I agree with Brandon Lewis."
Mr Lewis had earlier called on Mr Johnson to apologise.
Other Tories said Mr Johnson should have the party whip removed.
Lord Sheikh, a Conservative peer and the founder of the Conservative Muslim Forum, told BBC Newsnight the party should take "severe action".
He said: "Take the whip from him. Why not? He's not a super human being, he's a member of the party.
"The party chairman, the prime minister has the right to take the whip... that's the thing I'd like to see.
Mr Johnson has been reported to the equalities watchdog by Labour MP Jess Phillips, and to the police by a member in the public who accused him of "spreading hate crime".
However, a source close to the former London mayor said: "It is ridiculous that these views are being attacked - we must not fall into the trap of shutting down the debate on difficult issues," said the source.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies