The setting of this church must be one of the strangest in Norfolk, for it sits in one corner of the great outline of the Roman town of Venta Icenorum. Little remains, apart from the ramparts, which are at their highest near to the church. There is some walling on the north side, and a tower down by the river, but otherwise this is a wild and open place, especially on this gloomy, drizzly day in early Spring. There is some evidence that the church sits on one of the main roads of the town, which was not much smaller than Roman Colchester, where so much has survived. Remains of a fourth century church have been found in Colchester, and so it is not unlikely that St Edmund sits on the site of a predecessor that predated its patron Saint's life and death by several centuries.

Not surprisingly, a fair amount of Roman brick has been used in repairs and reconstructions over the centuries, and these create delightful outlines and fillings to doorways and windows. The tower also contains a fair amount of brick, and there seems to have been a fairly late medieval restoration to the top of the tower.

This is not a church which seems to be used very often, and we had some difficulty getting access to the inside. You step through a doorway flanked by big corbel heads at shoulder height, and then down into what is a substantially Victorianised interior, with a slightly sad feeling, as if a busy former life had now passed it by. The ceiling is very unusual, in the shape of, but slightly lower than, the roof, with some beams showing through the plaster, but no cross or tie beams visible.

There are wall paintings, but they are so faint you would have no idea what they showed unless you were told; opposite the north doorway is a large St Christopher, suggesting that the north side has always been the main way in. You can just make out the familiar shape of the Saint. On the north side of the chancel arch there is another figure, its haloed head just discernible. In the past, enough of the poisoned chalice he was holding survived to tell that this is St John the Evangelist. Perhaps he was matched on the south side by another figure, the Blessed Virgin perhaps, or St John the Baptist as at Weston Longueville.

In this small space the font is imposing, a typical deep cut East Anglian design with Evangelistic symbols and angels holding shields, familiar lions supporting the stem. Interestingly, there is a dedicatory inscription in Latin around the base asking for prayers, not for an individual, but for the Bothers and Sisters, Benefactors of the Guild of St John the Baptist at Caistor.

Leaving the church on the south side, you have to climb the Roman ramparts to leave the graveyard, enhancing the sense of remoteness, of an ancient and wild space. From here, you can walk clockwise around the ramparts. This is a round trip of about twenty minutes, longer if you amble and dawdle to take in the atmosphere. You can stand and look back at the church from the far west, and see how it might once indeed have been the site of a Roman temple, a Roman city sprawled around it, an awesome thought. The centuries roll away, until an express train bursts through not fifty metres behind you, hurtling on its way to London and destroying the illusion. Coming back to the graveyard, I found to the east of the church the grave of Georgiana Docking, which has a massive oak tree bursting through it. I wonder if she was buried with an acorn in her pocket?