This is Soul Town.
Here's what we know about The Intruders from Joel Whitburn's Top Pop Singles 1955-1990.
Soul group formed in Philadelphia in 1960. Consisted of Sam "Little Sonny" Brown, Eugene "Bird" Daughtry, Phil Terry and Robert "Big Sonny" Edwards. First recorded for Gowen in 1961.
This song deserved better than #49.
This is why I have such fond memories of listening to CKLW.