The north Norfolk coast became a popular holiday destination, thanks to Scott's writing and the opening up of the area by the railways. Perhaps a characterless Victorian church did not fit in with Scott's vision of what Poppyland was, and what people might find there. The medieval parish church at neighbouring Sidestrand had been rebuilt on a new site in an entirely medieval round-towered style, and so it was that Christ Church was demolished, and the ruin of St Martin restored to something approaching its former glory. The architects were Cecil Upcher and AJ Lacey, and the church was opened on the eve of the First World War.

Tragedy haunted Buxton and his wife Hannah: four of their children died during an outbreak of whooping cough in the early spring of 1820. Buxton himself never came to terms with his failure to eradicate slavery from Africa; he died in 1845, and was buried here. His memorial is inside the church, but he is also to be found on a recent British five pound note, where he is the spectacled figure standing to the left of his sister-in-law, Elizabeth Fry.