He was the son of Sir Francis Godolphin Osborne, fifth duke of Leeds by his second marriage, the first having ended in divorce after Amelia Darcy (mother of Augusta Leigh) eloped with Byron's father.