Swap files appear to contain some basic information about your computer (try opening the swap file in vim! I found the vim version, pc name, file path&name, utf encoding and some other "settings" stored) as well as a whole lot of "nothing".
The "nothing" is actually a significant amount of padded null characters ^@
(which I presume is used as a placeholder for the swap file to include other relevant information).
As such, swap files aren't actually "big", they just have some base information making them a bit bigger than the original file. As your files get larger, the % difference between the original file and the swap file becomes less significant.