Silkeborg's oldest history  is bound up with Silkeborg Castle, which was located at Slotsholmen, northeast of Silkeborg center where the river Gudenåen runs into Silkeborg Langsø. The first Silkeborg Castle was built in 1385. It was a primitive wooden building with palisades around. It was later replaced by a stone house and later again in the early 1500-century, by a main building in stone, a courtyard and a barn yard. Silkeborg Castle had several royal visits. Both Frederick 2nd  and Christran 4th  stayed regularly in the main building "The Great Stone House", for instance during their hunting trips in Central Jutland.
The castle was badly damaged during the wars with Sweden  in the 1600s, where Silkeborg 3 times was occupied by Swedish troops that ravaged the castle buildings. The last remains of the castle was demolished in 1726 after having been in disrepair for years.

Today is the floor plan of both the main building and the courtyard  is marked by ramparts. There is access to the area, both via a bridge over the river Gudenåen at Silkeborg Langsø and from the Paper Factory area.