The danger we face could arrive sooner were they to go quiet and become complacent.
This leads me to the recent statements made to the BBC in Davos by JP Morgan boss Jamie Dimon.
The bank had warned in the run up to the EU referendum that up to 4,000 jobs could go in the event of the UK voting to leave the EU.
Since that happened, the number has been revised downwards to between 500 and 1,000. Brexit supporters have pointed to the original warning as another example of “project fear” that hasn’t come to pass.
Well, up to a point. Up to 1,000 jobs will still go, and if that is reflected at banks across the city it still amounts to an awful lot of highly paid people, who pay an awful lot of tax, relocating and helping to fund other countries’ schools and hospitals by so doing. We are talking hundreds of millions, perhaps billions, of pounds here.
But there’s more. Mr Dimon also warned that that 4,000 is a number still very much on the table. It all depends on the rules and the regulations here and in the EU, how they all work together going forward, and the effect it all has on how banks like his do what they do.
They will be aligned at the start of the process, but that might not last. If it doesn't, and if bankers based in London can’t do their jobs in the City, then a lot more of them will be hunting for property, probably in or around Frankfurt. Probably to their chagrin. Being paid great wads of cash is a lot less fun if you can’t find fun things to do with it, and there’s not a lot of fun to be had in Frankfurt.
When the City types I know tell me they’re taking a trip there they usually wrap up the conversation by saying “ugh”. Or something like that.
Britain’s loss will come through the steady accumulation of those “ughs” expressed as a result of permanent relocation over a number of years from JP Morgan, and Goldman Sachs, and Morgan Stanley, and UBS, and Deutsche Bank and the rest of them.
Good riddance some might say, and I can understand that. I have no great love for bankers myself.
But when their numbers are totted up in five, or 10 years’ time, the loss to the British economy resulting from people leaving, will be enormous.
The drip feed of departures, and the lower tax revenues stemming from them, will be hard to notice at first. But it will eventually be felt one way or another.
This is true of many of the deeply malign consequences of Brexit, which will also only be expressed over time, and perhaps a long period of time. They will be made manifest in a succession of winters like the one the NHS has been experiencing, in schools dealing with the sort of difficult budgetary issues the one I am a governor of is facing, or just in the build up of potholes in our roads.
With relocation of the 4,000 at JP, and more at similar companies, set to occur over a long period, the economic impact will be gradual. This makes highlighting its link to Brexit difficult, even though the link will be there. Of course, there have been consequences already. Inflation has shot up. Living standards have fallen. Britain's economy has gone from the front of the G7 pack to its back. But the real damage is yet to come.
Fortunately we have the swivel eyed loons to remind us of the calibre of the people who are driving this process, and the way they make the toxic mix of cynicism and paranoia that motivates them more obvious by the day. It’s one of those small mercies the sensible half of Britain almost has to feel thankful for.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies