The son of Sir William Eliott, sixth baronet (d. 1812); in 1826 he married Theresa Boswell, daughter of Sir Alexander Boswell, first baronet. Sir Walter Scott described him as “that daintiest of dandies.”