National Wildlife Refuge. The earliest documentation of an island in the vicinity of Fisherman Island is from an 1815 navigational chart of the Chesapeake Bay. Two small islands, named the Bird Islands are shown on the chart just south of Cape Charles. Maps prior to 1815 show only shoals in the area and it is probable that Fisherman Island did not become permanently exposed until around this time.
The first accurate map of Fisherman Island and the Isaacs Islands to the east is from a Coast Survey of 1852, which shows Fisherman to be about 25 acres. While all of Virginia's other barrier islands are shrinking in size and giving way to the constant battering of the sea, Fisherman Island continues to grow, reaching over 1850 acres at the present.
Human history is almost as long as the existence of the island itself. Early residents used Fisherman as a hunting and fishing spot. Later in history it was used as a quarantine station for European immigrants heading to Baltimore and a military installation for harbor defense during World War I and II. Now, it is a National Wildlife Refuge for protection of coastal species such as Piping Plovers and American Oystercatchers.