Sir William Hillary moved to the Isle of Man in 1808 and began to know the dangerous waters of the Irish Sea which surrounded Douglas Bay. In around 1824 he conceived the concept of a life boat service manned by a trained crew. The idea was new, but soon prompted the creation of the Royal National Lifeboat Institution.
It was not till Hillary took part in a rescue for a Steam Packet vessel and was washed overboard that he realised the coast was too far to swim to, yet a place was needed for sailors to wait for rescue. Soon after an idea Sir William Hillary had became a reality. Designed by John Welch the Tower of Refuge, as it came to be called, was completed in 1832. Situated upon Conister Rock in Douglas Bay a granite haven in the form of a small castle took its place as a lifesaver for sailors. Built at a cost of which almost half was paid by Hillary himself and the other by subscription. The tower was kept well stocked with fresh water and bread, ready to offered shelter from the weather and sea.