Abraham was born in Pittsburgh, Pennsylvania, the son of Josephine, a housewife, and Fahrid Abraham, an auto mechanic. His father was an Assyrian Christian who immigrated from Syria during the 1920s famine; his paternal grandfather was a cantor in the Syriac Orthodox Church. Abraham's mother, one of 14 children, was Italian American and the daughter of an immigrant who worked in the coal mines of Western Pennsylvania. Abraham was raised in El Paso, Texas, near the Mexican border. He attended Vilas Grammar School, graduated from El Paso High School in 1958. He was a gang member during his teenage years. He attended Texas Western College (later named University of Texas at El Paso), where he was given the best actor award by Alpha Psi Omega for his portrayal of the Indian Nocona in Comanche Eagle during the 1959-60 season. He attended the University of Texas at Austin, then studied acting under Uta Hagen in New York City. He began his acting career on the stage, debuting in a Los Angeles production of Ray Bradbury's The Wonderful Ice Cream Suit.

Abraham made his screen debut as a taxi driver in Neil Simon's The Prisoner of Second Avenue. Abraham can be seen as one of the undercover police officers along with Al Pacino in Sidney Lumet's Serpico (1973). He also appears very early in All the President's Men as one of the police officers who arrests the Watergate burglars in the offices of the Democratic National Headquarters.

Until his acclaimed role in Amadeus, Abraham was perhaps best known to audiences as a talking bunch of grapes in a series of television commercials for Fruit of the Loom underwear. He worked with Pacino again in the gangster film Scarface in 1983, playing drug dealer Omar Suárez. Abraham won the Academy Award for Best Actor for his role as Antonio Salieri in Amadeus (1984). After Amadeus he has mainly focused on classical theatre, and has starred in many Shakespearean productions such as Othello and Richard III, as well as many other plays by the likes of Samuel Beckett and Gilbert and Sullivan. He is also known for his roles in The Name of the Rose (1986), in which he played Bernardo Gui, nemesis to Sean Connery's William of Baskerville, Woody Allen's Mighty Aphrodite (1995), Ahdar Ru'afo in Star Trek: Insurrection (1998), and Gus Van Sant's Finding Forrester (2000), where he once again played nemesis to Connery.

Abraham has focused on stage work throughout his career, giving notable performances as Pozzo in Mike Nichols's production of Waiting for Godot, Malvolio in Twelfth Night for the New York Shakespeare Festival, and Shylock in The Merchant of Venice, for the Off Broadway Theatre For A New Audience (TFANA) in March 2007, which was performed at the Duke Theatre in New York and also at The Swan Theatre, part of the Royal Shakespeare Company.

Abraham most recently made a guest appearance on the popular television series Saving Grace, on which he played an angel, Matthew and the Law & Order: Criminal Intent, on which he played Dr. Theodore Nichols, father of Det. Zach Nichols.

Abraham's relatively low-profile film career subsequent to his Academy Award has been widely considered an example of the Oscar Jinx; according to film critic Leonard Maltin professional failure following an early success is referred to in Hollywood circles as the "F. Murray Abraham syndrome". Abraham rejects this notion and once told an interviewer:

The Oscar is the single most important event of my career. I have dined with kings, shared equal billing with my idols, lectured at Harvard and Columbia. If this is a jinx, I'll take two.

In the same interview, Abraham said:

Even though I won the Oscar, I can still take the subway in New York, and nobody recognizes me. Some actors might find that disconcerting, but I find it refreshing.

Abraham has been married to Kate Hannan since 1962; they have two children. He taught Theater at Brooklyn College.