Magnetic fluctuations, also referred to as magnetic noise, in very small (sub-micron) magnetic systems are important both in studying the fundamental physics of statistical mechanics and in technology. Thermal fluctuations of the magnetization define the ultimate limit of magnetic storage densities and sensing technologies but may be useful in some biomedical applications. At high frequencies (>100 kHz), fluctuations of the magnetization about the internal field are the dominant form of magnetic noise. At lower frequencies, 1/f and random telegraph noise have been observed in many magnetic systems. Yet these noise sources are challenging to reproduce due to their origin in defects and, thus, identification of the physical mechanism which produces them is difficult. Further progress in studying magnetic noise requires a model system where the fluctuations are reproducible and the physical origin is known. In this thesis, random telegraph noise is identified in square magnetic dots and shown to originate from a configurational anisotropy associated with the square dot geometry. The square dots were fabricated from thin (10 nm) Permalloy films with side lengths ranging from 200 nm to 1000 nm, and the magnetization was measured via the anisotropic magnetoresistance. It is first shown, through measurements unaffected by the noise in these samples, that the square dot geometry exhibits a preference for the magnetization to align parallel to an edge of the square. A model of this four-fold configurational anisotropy explains the behavior of the magnetization and provides two methods to characterize the strength of the anisotropy.It is then shown that when a field is applied along the dot diagonal, the configurational anisotropy barrier in this direction is lowered, which allows thermal switching of the magnetization between low-energy magnetic states. The telegraph state lifetimes are quantified and shown to vary with applied field magnitude, field direction, and temperature as expected. The switching rate obeys an Arrhenius law and the energy barriers measured in the noise data agree well with those measurements independent from the noise. In addition, micromagnetic simulations of the Landau-Lifshitz-Gilbert equation reproduce the observed behavior and confirm the explanation of the magnetic noise in these samples.