Sir Robert's father was Sir Thomas Clifford, later created 1st Baron Clifford of Chudleigh. His mother was Lady Elizabeth Bourchier, daughter of Henry Bourchier, 2nd Earl of Essex.
This lineage places Sir Robert within the English aristocracy of the 17th century, with connections to prominent noble families.