The OED is unhelpful about its etymology, telling us only that it comes from the Old French rif et raf; 'riff nor raff' meant 'nothing' in the 14th century. It should not be confused with the modern word riff, which is an abbreviation of riffle and means, of course, a familiar kind of repetitive backing used by jazz musicians, as well as a comedian's catch-phrase.
What can be said is that for more than half a millennium riff-raff has given pleasure and satisfaction to countless users. This can only be something to do with the short '-i-' and '-a-', which, combined in a half-rhyme, seem for some reason to make a good vehicle for resentment or contempt. Consider flimflam and fiddle-faddle, and the tittle-tattle that can give you the jim-jams, and a whole mishmash of others. Even knick-knack and bric-a-brac are mildly dismissive. (Tick-tack, chiffchaff and zig-zag can be classed lexicographically as imitative and don't really count.)
Other half-rhymes don't have the same ring. Nig-nog is highly offensive, but flip-flops can be tip-top if properly fitted, or shipshape. Straight rhymes, though also expressive, come in several different flavours and it would be rash to generalise about them. Though it is obviously not pleasant to be described as arty-farty (or niminy-piminy, to use the old expression), no one blames those who, willy-nilly perhaps, find they have to hobnob with bigwigs on a picnic (preferably not next a golf course).