Cricket, perhaps more than other sports, is characterised by the beauty of its venues. Newlands in Cape Town rests against the towering majesty of Table Mountain. And who can forget Brian Lara hooking his way to a world record in Antigua, with the Caribbean Sea shimmering behind him? Galle, with its 17th century fort and Indian Ocean views, ranks with the best settings.

The high ramparts of Galle's Dutch Fort loom behind the ground, a giant lithic spectator. The fort, replete with terracotta-tiled villas and half-hidden eateries, is as wondrous an environ for cricket as you could wish for. Within its thick hiri gal walls, peddle-carts trundle door to door, bringing freshly caught fish from the nearby harbour. Wiry men weave along the streets on bicycles laden with heavy bunches of king coconut. The English Gate, the main entrance to the fort, is a two-minute amble from the ground. When cricket's in town, the fort is overrun.

Galle has a special place in Sri Lankan hearts. Muttiah Muralitharan and latterly Rangana Herath have conjured improbable victories on its turning pitches. Murali concocted his own bit of magic, taking the eight wickets he needed to reach a world-record 800 Test dismissals in his last match. He even managed to prolong the drama, as India's obdurate tailenders and Lasith Malinga's searing yorkers seemed fated to leave him on a not-too-dusty 799 wickets, before Mahela Jayawardene swooped on an edge extracted from Pragyan Ojha.

The other reason Galle lives in our hearts is the horrific events of December 2004. The Indian Ocean tsunami battered Sri Lanka's coastline. Galle harbour's narrow channel forced sea waters inland. In some places, shops and houses on the seafront escaped relatively unharmed, while buildings two roads back were obliterated. It's hard to describe the impact of seeing dusty piles of rubble where people's homes used to be. That day, I was fortunate to be travelling between Kandy and Nuwara Eliya, the two highest cities in the country. Others were not so lucky. Over 40,000 lives were lost across the island.

The cricket ground was not spared. Sea silt turned the pitch into a swamp, and the stadium was so badly damaged, it had to be rebuilt from scratch. A frenetic effort was made to prepare the ground in time for England's tour in December 2007. Right up to the last moment, there were serious doubts around the stadium being ready. An estimated million and a half people were made homeless by the tsunami; national priorities were focused elsewhere.

The ground was playable but not perfect by the time of the Test. The England team were gracious in their acceptance of the imperfections they found. The trip from Colombo, described by Paul Collingwood as "sickening", took England past a scarred coastline. For mile upon mile, only shells of houses remained. You might see a concrete staircase leading to a wide-open space where the upper storey of a house had been; or a red cement slab, the only remnant of someone's home. Despite the self-evident difficulties, England demonstrated an admirable commitment to "bring cricket back to Galle".

Jayawardene was also mindful of the greater significance of the game. "A lot of people who probably saw us play here four years ago are not with us, so it is emotional," he said. He played his part in lifting Sri Lankan spirits, giving England a masterclass in Test match batting while compiling a characteristically elegant and unhurried 213. He was abetted by a swashbuckling Tillakaratne Dilshan - has there ever been a more piratical cricketer? - and the underrated Chaminda Vaas. England contrived to get themselves all out for 81 in their first innings. The follow-on was enforced, but despite Murali's best efforts, the combination of an Alastair Cook century and tropical rains saw England scramble to a draw.

If the ramparts of the fort are visible from the ground, the ground is equally visible from the ramparts. I watched England's first innings collapse from crenellations in the rampart walls, which the Dutch conveniently situated behind the bowler's arm. The height of Galle Fort allows a view over the sightscreen and straight down the wicket. The green hills of the Galle hinterland rise up behind the ground, and the Indian Ocean embraces it on either side; it's a breathtaking place to watch cricket.

The views, good enough to allow Jonathan Agnew to use them to commentate from the ramparts when the BBC was locked out of the ground, are giving the administrators headaches. In 2012, the travelling English support expressed their displeasure at high ticket prices by camping out on the ramparts, thus depriving the administrators of much needed gate revenue.

The Fort itself also faces challenges. The huge spike in its popularity brought about by the end of the war and the opening of the Southern Expressway, Sri Lanka's first motorway, has also brought development. The ugly march of modernity will not, it seems, be checked. There is talk of supermarkets and high-end fashion outlets in the sleepy streets, which threaten the Fort's UNESCO World Heritage Site status. But for now, Galle remains a peerless Test match setting, and a repository of many memories.

Janaka Malwatta is a poet, doctor and cricket lover who lives in Brisbane. He tweets here