Does an affiliate program interfere with SEO efforts?

i want to start an affiliate program on my site. However, this could entice affiliates to spam my site, compete with me on my keywords, register similar domain names, frame my site etc etc which will all damage SEO efforts. So should I wait until my site is established before starting an affiliate program, or can it still work out favourably if I start one straight away? thanks in advance