Now the UKIndependent reviews archeological evidence that there was indeed a castle dating from that time: it was quite large and had evidence of trade from the Mediterranean.
(I should note that Cornwall was one of the sources of tin so was on the trade routes since ancient times).
alas, it wasn't just the Saxons who caused the decline:
The whole complex appears to have come into existence some time in the 5th or the early 6th century AD – but was probably in decline by the early 7th. So far, no evidence of any catastrophic destruction has been found. However, the latter half of the 6th century and the 7th century were notorious for a terrible plague pandemic (an early version of the later medieval Black Death) which almost certainly devastated parts of Britain after having killed millions throughout the Mediterranean world.
that would be the Plague of Justinian which decimated the Mediterranean world in 541
The Anglo Saxons had started working as mercenaries after the Romans left in 410, but it wasn't like today, a general takeover but a gradual takeover of the south west and then spread.
The UK, or England for that matter, is not a monolith, which is why some areas wanted Brexit and some areas did not.
Although eastern and much of central Britain had been taken over by Germanic (ie, Anglo-Saxon) conquerors and settlers from what is now Germany and Denmark, much of the west of Britain (including Cornwall) remained under native British control.