Lee was born in Belgravia, London, the son of Lieutenant Colonel Geoffrey Trollope Lee (1879–1941) of the 60th King's Royal Rifle Corps, and his wife, Countess Estelle Marie (née Carandini di Sarzano; 1889–1981). Lee's father fought in the Boer War and First World War, and his mother was an Edwardian beauty who was painted by Sir John Lavery, Oswald Birley, and Olive Snell, and sculpted by Clare Sheridan; her lineage can be traced to Charlemagne. Lee's maternal great-grandfather was an Italian political refugee, whose wife, Lee's great-grandmother, was English-born opera singer Marie Carandini (née Burgess). He had one sister, Xandra Carandini Lee (1917–2002).
Lee died at the Chelsea and Westminster Hospital on 7 June 2015 at 8:30 am after being admitted for respiratory problems and heart failure, shortly after celebrating his 93rd birthday. His wife delayed the public announcement until 11 June, in order to break the news to their family. He was cremated and his ashes were scattered on the hills of Surrey.