The son of the Rev. James Scott and brother of Lady Oxford; he was educated at St. Mary Hall, Oxford, and the Middle Temple. In 1811 he was unsuccessfully sued by Francis Burdett who attempted to recovery a £5000 legacy to Lady Oxford.