Jimmy Scott: Singer with a remarkable contralto voice who spent years away from music but returned in triumph

Scott at the Montreux Jazz Festival in 2001; he was championed by Lou Reed and Ray Charles, among others (EPA)

Lou Reed, no sentimentalist in the normal run of things, said that Jimmy Scott's voice was the only human sound that could make him cry. The qualities of that voice were unique: a male contralto with a tight, accurate vibrato, a sense of phrasing that was almost instrumental and a range of dramatic expression reminiscent of Billie Holiday and Sarah Vaughan.

Its origins were medical rather than the result of training, but Scott was not a freak-show act. He was an artist of remarkable sensibility and one of that small but significant cohort who disprove Scott Fitzgerald's famous but slightly perverse insistence that there are no second acts in American creative lives.

Scott became a singing sensation with the Lionel Hampton band – the vibraphonist gave him his "Little Jimmy Scott" tag – and went on to work with most of the heavyweights of jazz and r'n'b. He was admired by Miles Davis, another second-acter, and by Charlie Parker, who failed to get past the first curtain, but his muse was Billie Holiday.

He started out professionally touring the Midwest with Estelle Young. After some early success, albeit tinged by business naivety, multiple bad luck and downright insult, Scott spent many years out of the music business in the 1960s before making a remarkable comeback in his own right and as a crossover singer who performed on Reed's 1992 Magic and Loss and, an unforgettable sequence, in the finale to Twin Peaks, where his rendition of "Sycamore Trees" helped clinch the series' odd blend of beauty and dread.

James Victor Scott was born in July 1925 in Cleveland, Ohio – "the most musical town in America" he told a British journalist, "though you mightn't believe it" – third in a family of 10. His mother, Justine Stanard Scott, taught him piano and got him singing in church. She was killed in a motor accident when Jimmy was 13.

By his late teens, it became evident that he had not passed through puberty. Though known since the mid-19th century, Kallmann's syndrome, whose effects include reduced stature, hypogonadism and a significantly altered and reduced sense of smell, was not fully identified until 1944. It remains exceedingly rare (though now treatable) and was never spoken of in Scott's family. At the turn of his 20s he was still only four feet tall (he grew another few inches in his late 30s) and had a high, unbroken voice, similar to that of a castrato.

Its qualities were unmistakable, but inevitably ambiguous. One of the greatest humiliations of Scott's life was to be credited as "Chubby Newsome" (who was believed to be a female vocalist) on the issued recording of Charlie Parker's and Fats Navarro's June 1950 performance at Birdland. He wasn't credited at all on that year's hit single "Everybody's Somebody's Fool" with the Hampton band.

Because of his stature and the unusual placing of his voice, Scott found it difficult to build a career. After leaving Hampton, he worked with Paul Guyten's r'n'b band. In 1955 he made If You Only Knew for Herman Lubinsky's Savoy label, setting out an emotional terrain otherwise unexplored in American jazz.

On songs like "Address Unknown", "Recess In Heaven" and the Rose Marie McCoy title track, Scott seemed to be speaking from some distant place, marked by hurt and yearning, redeemed by impeccable musicianship; it's rarely noted that the record is one of the best accompanied of the time, featuring players of unusually thoughtful musicality, a match for Scott's own.

Though dismissed as a novelty by some critics, Scott had influential supporters, including the blues singer Doc Pomus and the influential Ray Charles, and it was Charles in 1962 who brought him over to the newly formed Tangerine label, where he recast Scott as a ballad singer. Falling In Love Is Wonderful featured string and softened horn arrangements by Marty Paich and Gerald Wilson.

It saw the Scott persona evolve in dramatic ways, from the almost childlike pettishness of "Why Try To Change Me Now?" to the aching hopefulness of "Someone To Watch Over Me". The record is a high-point in American vocal jazz, but it was withdrawn from sale after only a month, when Lubinsky claimed that Scott was still under contract to him.

Jimmy heard the news while on honeymoon. He was married five times in all, most recently to Jeanie McCarthy, who he wed in 2003. There were, of course, no children. In interviews, Jimmy made as light as possible of his medical condition. "It's something God gave to me, and that's all there is to it. It made difficulties in the marriages, of course, but..."

Disillusioned, he drifted away from music by the end of the 1960s, working as a hospital orderly and lift attendant. In 1991 he was persuaded to sing again at Doc Pomus's funeral. Word spread. Lou Reed, Madonna and Nick Cave all declared themselves admirers and Scott became hero and muse to a new generation, from the Red Hot Chili Peppers to Antony Hegarty of Antony and the Johnsons.

In addition to singing Angelo Badalmenti's music on Twin Peaks: Fire Walk With Me, and making guest appearances with Reed ("Power and Glory" is the stand-out song on Magic and Loss) and others, Scott began to record again under his own name, first on All The Way for Sire and most notably on Holding Back The Years (1998) where he covered contemporary material by Simply Red, Elton John and Elvis Costello as well as Prince's "Nothing Compares 2U". After the millennium he made a further sequence of albums for Milestone, all critically admired. Falling In Love Is Wonderful eventually reappeared as a modern classic.

Scott sang "Why Was I Born?" at Dwight Eisenhower's inauguration in 1953 and the same song at Bill Clinton's 40 years later. In between, much had changed in his life and in American life. He recognised that his "gift" came with a terrible price in terms of emotional isolation and a measure of bitterness, but he embraced it sufficiently to create a body of music that stands out for quality – possibly even greatness – and not mere oddity. He is survived by his fifth wife Jeanie McCarthy Scott.

James Victor Scott, singer: born Cleveland, Ohio 17 July 1925; married five times; died Las Vegas 12 June 2014.