Abercrombie attended the Berklee College of Music from 1962 to 1966 and graduated from in 1967. He later studied at the University of North Texas College of Music. In 1998, the Berklee College of Music presented him Music Distinguished Alumni Award.
The ongoing groups he has led or co-led include:
- Gateway, with bassist Dave Holland and drummer Jack DeJohnette (mid-1970s, and again mid-1990s)
- duo with guitarist Ralph Towner (starting mid-1970s, occasionally through mid-1990s)
- quartet with pianist Richie Beirach, bassist George Mraz, and drummer Peter Donald (late 1970s to early 1980s)
- trio with bassist Marc Johnson and drummer Peter Erskine (mid-1980s to early 1990s)
- trio with organist Dan Wall and drummer Adam Nussbaum (through 1990s)
- quartet with violinist Mark Feldman, bassist Marc Johnson, and drummer Joey Baron (starting 2000)