It's a lot but not unreasonable. You may want to look into the xsendfile option (
https://www.dokuwiki.org/config:xsendfile) to improve static file delivery.
Again, the more details you can give us the better we may help you. Eg. are you using fpm? Is it a linux host? What PHP version is this? Are there any other specialities (virtualizing, docker, NAS, proxies, firewalls, ...)?