Meanwhile @michaelsy was nice enough to do what @tzjinzo couldn't do and did some resizing checks using libGD, imagemagick and Photoshop. It shows that there is no obvious problem in our resizing algorithm and that the results are all similar in quality. Thanks @michaelsy
That is what I wanted to know in the first place. Unfortunately responses where either "you don't need to know anything, it's obviously shitty looking - now go fix it!" or "let me explain how to display large images smaller using CSS, because you obviously don't know shit about the web".
It's also great how I asked two things and never got the answer on my second question. But @michaelsy's research shows that there isn't really a quality difference between imagemagick and libGD, so we can ignore it.
Now, we finally have enough data to actually talk about the problem @tzjinzo has: He wants images to be downloaded larger than they are displayed for better image quality. A thing we oldfarts used to call Retina-Support (even though it is not limited to Apple's Retina(tm) Display technology).
In the post by michaelsy we can see how that could look like as the end result. However this needs proper support in DokuWiki and not just some custom CSS. Implementing this is not as simple as you might think because you have also to think about the drawbacks (as mentioned by virk).
There are mechanisms to give browsers the choice of multiple image sizes and let the browser pick the one it thinks is best, but this can lead to bad experience as well. Because ironically these days phones are the devices with the best displays but with the shittiest network connections. For display purposes a 4K resolution pic will look best on a modern iPhone, but downloading that 15MB image is no fun when you're on Edge...
So yeah. I know what the "problem" is, I kinda know in what direction a "solution" is, but I don't know how to exactly implement it, yet.
And because we're all having so much fun in this thread, I will close it.
I will open an issue to keep track of the feature later.