In addition to .exe files being MS-Windows executables, as @linux-aarhus says, the icons will be located in your ~/Desktop folder, and thus you should be able to remove them from there using dolphin, or by way of the command line.
If the above seems too difficult for you, then please read the following short essay…
