

He returned to music in 1989 when manager Alan Eichler arranged for him to share a late-night bill with Johnnie Ray at New York's Ballroom. Scott's career faded by the late 1960s, and he returned to his native Cleveland to work as a hospital orderly, shipping clerk, and elevator operator. Another album, The Source, was recorded in 1969, released in 1970, but due to another Lubinsky threat of breach of contract, it was not promoted by Atlantic and quickly went out of print. Scott disputed the contract he had with Lubinsky, who had loaned him to Syd Nathan at King for 45 recordings in 1957–58. The album was withdrawn while Scott was on his honeymoon because he had signed a contract with Herman Lubinsky it would be 40 years before the album was reissued. In 1963 his girlfriend, Mary Ann Fisher, who sang with Ray Charles, helped him sign with Tangerine, Charles's label, and record the album Falling in Love is Wonderful. A similar event occurred several years later when his vocal on " Embraceable You" with Charlie Parker, on the album One Night in Birdland, was credited to the female vocalist Chubby Newsom. Credit on the label went to "Lionel Hampton and vocalists" Scott received no credit on any of the songs. He rose to prominence as Little Jimmy Scott in the Lionel Hampton band as lead singer on " Everybody's Somebody's Fool", recorded in December 1949. His phrasing made him a favorite of artists including Billie Holiday, Ray Charles, Frankie Valli, Dinah Washington, and Nancy Wilson.

Lionel Hampton gave him the nickname "Little Jimmy Scott" because he looked young and was short and of slight build. At 13, he was orphaned when his mother was killed by a drunk driver. As a child he got his first singing experience by his mother's side at the family piano and later in church choir.

The son of Arthur and Justine Stanard Scott, he was the third child in a family of 10. James Victor Scott was born on July 17, 1925, in Cleveland, Ohio, United States.
