Carlo played a few shows with the Stones before Charlie Watts arrived on the scene - but decided there were better prospects working with Screaming Lord Sutch.
Nowadays Carlo, who is 59, sells hot dogs and burgers from a catering trailer in the shadow of Wembley Stadium.
But the man who once gave the young Keith Moon some drumming lessons just before he sprang to fame with The Who, remains philosophical about his split with the Stones. "It was the right thing to do at the time," says Carlo, who was a military drummer during his National Service period.
Carlo came to the fore as a founder member of Screaming Lord Sutch and the Savages. He later reached further prominence with Cyril Davies and his Rhythm and Blues All-Stars. But, right at the end of 1962, he was invited to play three or four dates with a new group calling themselves The Rolling Stones.
"People came up to me and said `what the hell are you doing playing with this mob?'," he recalls. "They were seeing me - the pro-drummer, well- respected - playing with young lads who, at that time, didn't really know what they were doing."
Carlo remembers turning down an offer to stay on for some more dates - seeing a chance to return to the Sutch band as a better option. "More gigs were coming in with Sutch, and the money was about three times as much. I suggested that they have a look at a drummer called Charlie Watts. I said `try Charles - because I can't do it'. And that was the most stupid thing I've ever said!"
Since early 1963, the nearest Carlo has come to any of the Rolling Stones was the time he saw Mick Jagger backstage at a Wembley show in the Seventies. He was too embarrassed to go up to him and say hello.
There has been much in the way of irony in the Carlo Little story. It was after a Sutch gig at Wembley Town Hall that a local youngster called Keith Moon approached him and asked for some drumming lessons. Carlo remembers Keith coming round to his home nearby for about half a dozen two-hour lessons at 10 shillings (50p) a time.
"I never saw him again until I caught him on the telly with The Who and they were in the charts. I went `bloody hell - I'm still farting about and they're on the telly and in the charts'."
Meanwhile Carlo became a delivery driver and then a telephone engineer, before running a greengrocer's shop with Nick Simper, who'd left Deep Purple after three albums. Eventually Carlo spent 15 years as a bread salesman, getting up in the early hours to drive a bakery van.
Carlo still keeps his drums, packed in their cases, in a six-bedroom, detached house at Wembley Park, and he admits he does have a few regrets.
"I'm a little bit sorry about it, because I know I was well- regarded as a drummer among musicians. It comes a little hard, and you think `if I was that well-respected, how come I never got anywhere?' How do you think I feel when I see Charlie up there tapping away?"
But Keith Moon is dead, and many others have fallen victim to rock and roll excesses. Carlo says: "That's one thing about all this. I've got a nice life, I don't want for anything, and I'm alive. I'm happy with my life."