I think you are disabling the caching too often, you should check if the cache you are disabling is the xhtml cache as otherwise you also disable the instruction cache which is a lot worse in terms of performance than just the renderer cache.
I think you could also simply set $this->info['cache'] to false in the renderer whenever the link function is called, that way you can make sure that you only disable the cache when the plugin's renderer is actually used.
[edit] I added this as
issue #1 to the issue tracker.