Born in White Plains, New York, to Florence Belle (née Crane) and James Allen Whitmore, Sr., a park commission official, Whitmore attended Amherst Central High School in Snyder, New York, before graduating from The Choate School (now Choate Rosemary Hall) in Wallingford, Connecticut. He went on to study at Yale University, where he was a member of Skull and Bones and had his first taste of radio drama as part of other broadcasting experience as a member of the student-run WOCD-AM, later renamed WYBC-AM. He later was commissioned a Second Lieutenant and served in the United States Marine Corps in the Panama Canal Zone during World War II.
Following World War II, Whitmore appeared on Broadway in the role of the Sergeant in Command Decision. MGM hired Whitmore on contract, but his role in the film adaptation was played by Van Johnson. Whitmore's first major picture was Battleground, in a role that was turned down by Spencer Tracy, and for which Whitmore was nominated for the Academy Award for Best Supporting Actor. Other major films included Angels in the Outfield (1951 film), The Asphalt Jungle, The Next Voice You Hear, Above and Beyond, Kiss Me, Kate, Them!, Oklahoma!, Black Like Me, Guns of the Magnificent Seven, Tora! Tora! Tora!, and Give 'em Hell, Harry!, for which he was nominated for the Academy Award for Best Actor for his portrayal of former U.S. President Harry S Truman. Whitmore never one the Academy Award, but he did win an Emmy and a Tony.