It turns out that designing a really good headphone amplifier is not trivial (kinda seems like it ought to be eh?)
Years ago I worked for a company that was designing an AVB based personal monitoring system. One of my tasks was to design a headphone amplifier that would "make a drummers ears bleed". Now they were just kidding, but the headphones had to be able to cut through noise the drummer was already making.
Fortunately, for me, I had also designed the power supply, which made designing the headphone amplifier a tiny bit easier.
Unfortunately, for me, it had to be able to drive any headphone design. Among other factors, that means handling impedances from around 8 ohms to as high as 600 ohms, maybe higher? And we did not want to hamper low end response, so it had to be direct-coupled. And it could not burn up if you shorted the output by mistake. It was a fun challenge!
I probably spent more time on the headphone amplifier than any other aspect of the design. In the end it turned out well. I wouldn't put it up against a high end, stand-alone device, but it didn't stink.
Disclosure - as much as I am proud of what we built, I still use a (now discontinued sadly) Rane HC-6 headphone amplifier, by far the best sounding, most reliable headphone amplifier for under $1k.