Okay, found the solution:
xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s false
google “xfce4-session - Advanced Topics”
Unfortunately this still prevents the required variables to be passed trough. google “Allowing xfce4-session to use external ssh-agent”
Currently , there is no way to get this running with XFCE. I switch to KDE to work around this. (Also for its better HiDPI scaling support.)
Thanks everybody who invested their time.