>
With normal headphone sensitivity (1mW for 100 dB spl)
That's high. I usually assume 92dB SPL/mW +/-10dB.
>
you need about 11mA current for 100 dB sound pressure
But you "need" peaks to 105dB SPL for "realistic" sound level, and more for rock-n-roll, fine monitoring, or deaf musicians.
I happened to have 87dB SPL @ 1W speakers, a 50W stereo, and some piano recordings, in the same room with a piano. The live piano played louder than the stereo could without clipping.
You need 10mW-50mW power in the cans or you won't be happy. I design for over 250mW (but won't expose myself to that for more than a minute per lifetime; it is about "knowing" that any clipping is not in the HP amp).
50mW in 32 ohms means 55mA peak. Few "common" chips will do that, or not at low distortion, even if you build-out to the several hundred ohm minimum load for good gain performance.
Then you need a second jack for your asst. engineer or girlfriend, 100mA.
People do get-by with single opamp phone drivers. Works good for >32 ohms and modest demand. But if you are going to work around live music, and make critical decisions, on live music, you really-really want "ample" power, not "just adequate" power. The buffer, in quantity production, is about a buck of jellybeans and PCB space. On anything more than a $99 nano-Mixer, the cost is justifiable for fewer customer complaints. At DIY pricing it may be $5 of parts and $10 labor-equivalent, still cheap in context.
If you want
Really Ample Power For Any Likely Headphone, ponder
this essay. It assumes headphones are made to be driven to MAX rated power (else why build them so rugged?) and that you might sometimes
want that max output. In many cases, this gives ~~120dB SPL, which is LOUD. But not out of line with control room speaker systems, which typically approach the output of an A-7 (117dB SPL max) with hopefully smoother sound. If your speaker tastes run more to Minimus 7 and 10W amps, then you could design your headphone amp to 20dB less, 1/10th the voltage/current. However the "cost of headroom", at HP power levels, is so much lower than the cost of loudspeaker gear, that we prefer to be a bit more generous.