Average Karma is based on a window of comments. Something like the last 60 IIRC, so if you have highly rated comments that are moving out of the window because you are posting new comments it will decrease your score. I also suspect it is weighted towards comments (perhaps even 0 weight on posts) as my own 60some point submission had almost no impact on my average.