You can not remove Avahi without removing a lot of other packages. If you don’t need a GUI, you can remove Avahi. But probably need to set up your networking manually.
If you don’t want to use it, disable and mask the services provided by the Avahi package. It is the closest thing to remove it.
An Alterativ would be to build your own packages without a Avahi dependency.