There were a few different programs for that, but probably the most popular was Unsanity's Shapeshifter, which allowed theming without replacing any system files (it instead hooked app launches and patched those files in memory).
Those themes were pretty good, but suffered some significant limitations, especially with dark themes and carbon apps. The best OS X themes really looked great though, since they supported full color and alpha.
SIMBL, ShapeShifter, ThemePark, artFileTool (ArtFile.bin, SArtFile.bin, Extras.rsrc, and Extras2.rsrc), and, more recently, ThemeEngine (/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/*.car) are all of the OS X/macOS customization tools I remember using over the years.
