Graffman was born in New York City to Russian-Jewish parents. Having started piano at age 3, Graffman entered the Curtis Institute of Music at age 7 in 1936 as a piano student of Isabelle Vengerova.

After graduating from Curtis in 1946, he made his professional solo debut with conductor Eugene Ormandy and the Philadelphia Orchestra. By the age of 20, Gary Graffman had made a name for himself worldwide as a classical pianist. In 1949, he won the prestigious Leventritt Competition. He then furthered his piano studies with Rudolf Serkin at the Marlboro Music Festival and informally with Vladimir Horowitz.

Over the next three decades, he toured and recorded extensively, performing solo and with orchestras around the globe. In 1964, he recorded Rachmaninoff's Rhapsody on a Theme of Paganini with Leonard Bernstein conducting the New York Philharmonic. He also made a classic recording of Prokofiev's Third Piano Concerto with George Szell and the Cleveland Orchestra in 1966. It was reissued on CD as part of Sony Classical's "Great Performances" series in 2006.

Probably Graffman's most famous recorded performance was for the soundtrack of the 1979 Woody Allen movie Manhattan in which he played George Gershwin's Rhapsody In Blue, accompanied by the New York Philharmonic. Portions of the Philharmonic/Graffman version have been featured countless times in TV and movies over the last quarter century.

In 1977, he sprained the ring finger of his right hand. Because of this injury he began re-fingering some passages for that hand in such a way as to avoid using the affected finger. Unfortunately, this altered technique exacerbated the injury rather than ameliorating it, ultimately forcing him to stop using his right hand altogether by around 1979.

This setback encouraged him to pursue other interests such as writing, photography, and Oriental art. In 1980, he joined the faculty at the Curtis Institute where his career had begun. He took over as the school's director in 1986, and added the title of President in 1995, serving in both capacities through May, 2006. He continues as a faculty member at Curtis teaching piano.

In 1985 he gave the UK premiere of Erich Wolfgang Korngold's Piano Concerto in C-sharp for the Left Hand. Paul Wittgenstein commissioned the work in the 1920s and played it many times, but it later slipped from the repertoire.[3]

Seven left-hand works have been commissioned for Graffman. In 1993, for example, he performed the world premiere of Ned Rorem's Piano Concerto No. 4, written specifically for the left hand, and in 2001 he premiered Daron Hagen's concerto Seven Last Words. The American composer William Bolcom composed Gaea, a concerto for two pianos, left hand for Graffman and Leon Fleisher. It received its first performance in Baltimore in April 1996.

Through his longstanding service and devotion to music, Graffman has received honorary doctoral degrees, was honored by the cities of Philadelphia and New York, and received the Governor's Arts Award by the Commonwealth of Pennsylvania. In addition to his administrative responsibilities, Graffman remains active as a teacher and coach of piano and chamber music. His notable students include the piano virtuosos Lydia Artymiw, Lang Lang, Yuja Wang, and Haochen Zhang.