Source of this report - see above link.
"The Castle was probably built by William Revel, one of Bernard de Newmarch's knights.
Matilda de Braose is said to have built the stone keep in c1200, but it is perhaps more likely that she added the gateway arch to a tower built in the 1180s.
She died of starvation at the command of King John, who burnt the castle and town of Hay in 1216 while attempting to suppress the rebellion of Giles and Reginald de Braose.
They were burnt again by Llywelyn Fawr in 1231 and had to be rebuilt by Henry III. In 1232 and 1237 he granted the townsfolk of Hay the right to collect a special toll to pay for walling in the town with stone. The castle was captured by Prince Edward in 1264 and by Simon de Montfort's forces in 1265.
Both town and castle suffered damage by Owain Glyndwr's forces in 1400, but the castle was listed as defensible against the Welsh in 1403.
The castle had passed to the Earls of Stafford, later Dukes of Buckingham, and is said to have suffered further damage during the conflicts of the 1460s.
The last Duke, executed by Henry VIII in 1521, remodelled the keep.
Whatever apartments then adjoined it were swept away in the 1660s when James Boyle of Hereford built a new mansion. Most of the curtain wall was either demolished during the Civil War or later, to improve the views from the mansion.
In the early 19th century, the house was occupied by the Wellington family who purchased it from Glyn heiresses. The house was restored c1910 but the eastern part was gutted by fire in 1939. The western part was gutted by a second fire in 1979, but has been restored.
It and various outbuildings are now used for second-hand bookselling."