If you didn’t changed the PREFIX nearly every file should be somewhere in /usr/local
There is no other way as manual deleting all files create by the make install command. One of the major downsides of manual installing software this way. Even with a remove option, there is no guarantee that all files have been removed.
You also could try to reinstall and hope that the make install command prints all installed files. Manual removing the installed files is a little bit easier with this list.
You could also check the installed file list of the normal python package(s) to see what you are looking for.