Most of the castles I've ever seen were 'slighted' by Oliver Cromwell's order during the Civil War. Unless they were later converted to country homes, they were never rebuilt after being slighted because the age of castles was over. They were architectural dinosaurs. I've always been annoyed that Cromwell ruined so many castles. It seemed petty to me.
However, Dover Castle wasn't slighted. I still don't know why, except that it was held by the King, not a noble family resisting Cromwell's rule. It was apparently taken by a 'Parliamentarian Trick' without a shot being fired. Don't know what the 'trick' was.
Thus, today it is not only the largest castle in England, but also the only castle I've ever seen that is intact. There may be others, but I don't know where they might be.
I found out another interesting fact - the Anglo-Saxon church and the Roman Pharos are on a hill or mound within the castle grounds, but apart from the castle keep. Excavators have found that the mound is bronze age, which I htink is pretty interesting.
As to whether the Anglo-Saxon church at Dover Castle is on the site of a Roman Temple, I can't find anything either way. It might have been, or it might even have been a much older sacred site.