Conley was born in McIntosh County, Georgia and grew up in Atlanta. He first recorded in 1959 as the lead singer of Arthur & the Corvets. With this group, he released three singles in 1963 and 1964 (“Poor Girl”, “I Believe”, and “Flossie Mae”) on the Atlanta based record label, National Recording Company.
In 1964, he moved to a new label (Baltimore‘s Ru-Jac Records) and released “I’m a Lonely Stranger”. When Otis Redding heard this, he asked Conley to record a new version, which was released on Redding’s own fledgling label Jotis Records, as only its second release. Conley met Redding in 1967, but after this meeting Redding took Conley under his care, and taught him the finer points of the music industry. Together they re-wrote the Sam Cooke song “Yeah Man” into “Sweet Soul Music”, which, at Redding’s insistence, was released on the Atco-distributed label Fame Records, and was recorded at FAME studios in Muscle Shoals, Alabama. It proved to be a massive hit, going to the number two position on the U.S. charts and the Top Ten across much of Europe. “Sweet Soul Music” sold over one million copies, and was awarded a gold disc.