You know, it is hard for anyone to speak definitively on behalf of Google, but look at it this way: The affiliate marketing business is huge and growing and is used by MAJOR advertisers to cost effectively market their goods and services. AdSense by Google is effectively their affiliate program. I think it is highly unlikely that there is any penalty applied to any link that carries an affiliate ID.
On an affiliate marketing forum I belong to this has been discussed before. The "evidence" is strictly anecdotal but some people (I'm one of them) believe they've seen fluctuations in their Google traffic for pages as CJ links are added or removed. I work with very few CJ merchants for this reason, among others.
I prefer to work with independent affiliate programs that are run in-house when they've proven themselves to be responsive and trustworthy over tmie.
There was a big to-do over blog comment spam a couple of years ago, and Google was recommending the nofollow attribute to combat it. Basically it meant "I'm not endorsing this site with a link - the link isn't mine."
The syntax was < a href="URL" rel="nofollow" >
I know they were using it heavily for rankings in blogsearch, but there was a lot of argument over whether the same held true for web searches. I used it on my blogs (which quickly made pr6) but never used it on web pages.
What an interesting idea that is, Carl. I have never heard of it nor thought of it before, but since I use lots and lots of image links, it might actually be a good thing for me to do to minimize the danger of looking like a link farm operation. The links out work, but the SEs ignore them -- I like it.