The GPU Helps Before the Queue Gets Long

Image resize requests do not arrive in streamlined batches. They arrive when people upload pictures, open pages, and miss the cache.

Thrice in my career, for three different employers, I've found myself behind the keyboard crafting an image pipeline. In each case, the target group of viewers was large enough that I not only had to consider the obvious effects—image size and download time—but also the queueing implications of processing incoming user requests on a multicore server. What I implemented in the past varied depending on the employer's specific image characteristics, performance and image quality tradeoffs, and budgets, but the theme was consistent: it was the anticipated queueing behavior, not the fully loaded batch story, that was the measure of success.

In the past, I considered encoding choices to be a knob that could be turned in queue management, but in this paper, I ask a different question. What happens if we change where the images are optimized in the system and move the burden to the GPU?

This weekend I decided to revisit this from that direction: can a GPU-forward reimplementation of JPEGli meaningfully move the needle on processing queue depth?

A healthy on-demand server spends most of its time with only a small portion of its capacity utilized. That is not wasted capacity. That is the headroom that keeps a burst from becoming a queue and a queue from becoming latency.

It is with this in mind that I conceived this experiment.

I added an AMD Vulkan path to JPEGli for progressive encoding with no overlap. On a Ryzen 7 PRO 8700GE with its integrated Radeon 780M, encoding the 30 CLIC 2025 test images at quality 75 with progressive 4:2:0 output fell from 21.9 ms to 15.8 ms. The p95 fell from 32.0 ms to 20.8 ms.

The byte streams remained the same. I was not interested in getting a faster encoder by changing the scan script or producing files that cost more to store and deliver. Every matched CPU and GPU run produced the same byte count and checksum.

That is also why I use JPEGli specifically rather than merely JPEG syntax. Bruse et al. found that users preferred JPEGli over same-sized libjpeg-turbo or MozJPEG images [1]. Better output at the same byte count makes the encoder worth using. This project is about reducing its compute cost without changing the bytes users prefer.

What runs where

The CPU does input conversion, the color transform, chroma downsampling, adaptive decisions, JPEGli's serial state, Huffman optimization, entropy coding, and final bit packing.

The GPU takes the block work after that: level shift, 8×8 DCT, quantization, zigzag ordering, and progressive token formation.

The CPU then stitches the remaining cross-block state and writes the exact same entropy stream it would have written without the GPU.

This is a direct Vulkan backend for the AMD APU I am deploying. It uses wave64 operations and mapped, coherent system memory. The CPU and integrated GPU see the same memory, so there is no PCIe copy to a discrete card. I did not put a cross-vendor abstraction layer in the latency path.

The endpoint is synchronous. One request arrives, the GPU helps encode it, and that request completes. It does not wait for a batch or another image behind it. There is a separate pipelined API for workloads that already have a backlog, but it is not the design in this test.

Earlier decoder work found two useful CPU/GPU splits. Sodsong et al. left serial Huffman decoding on the CPU, then partitioned IDCT, upsampling, and color conversion across the CPU and GPU using image entropy, width, and height [2]. Weißenberger and Schmidt went farther: they used the self-synchronizing behavior of Huffman codes to recover correct boundaries after starting at arbitrary bit positions, allowing fine-grained GPU work inside JPEG bitstreams [3].

The second design matters because it does more than send each image in a batch to a different GPU thread. It attacks serial-looking work inside an image. It also implements a subset of JPEG and was measured on batches of 200 to 2,400 images on V100 and A100 hardware. Our exact progressive decoder on this smaller 780M still lost to the CPU. Decode stays on the CPU here because that was faster, not because decoder entropy is impossible to parallelize.

The GPU helps while the server has headroom

I next kept 1, 2, 3, and eventually all 16 worker processes busy. Each worker encoded the same resident 30-image corpus at quality 75 using progressive 4:2:0 output and optimized Huffman tables. Every worker count ran three times, and the chart uses the median.

Workers 1 through 8 were pinned to separate physical cores. Workers 9 through 16 added the second hardware thread on each core.

With one active worker, the GPU path delivers 37.4% more request capacity and uses 35.1% less energy per request. With four workers, the gains are still 24.7% and 19.7%. At eight workers, they are 10.8% and 9.7%.

At fifteen workers, the two paths are even. At sixteen, CPU-only is 1.2% faster and uses 1.3% less energy per request.

The power readings explain this. With one or two active workers, the GPU path draws less package power and finishes sooner. From three workers onward, both designs spend most of the run near the APU's shared 42 W package limit. The GPU path can still save energy by completing work sooner, but that margin shrinks as more CPU and GPU work compete inside the same package.

To check whether quality 75 was a special case, I repeated the one-worker run from quality 25 through 100 in steps of five. It was not. CPU JPEGli drew 27.4–28.9 W across the sweep. The GPU path drew 23.4–24.5 W, or 10.6–16.1% less, at every quality.

Power alone is not energy. The GPU path also finished each fixed workload 17.6–38.3% sooner, so package energy per image fell by 28.7–46.0% across the same quality sweep.

The energy figures come from Linux's package counter, not a wall-plug meter.

Zhang et al. found the same general trap across 42 programs on AMD and Intel integrated CPU/GPU systems. Only eight ran fastest by co-running on both devices. Architecture differences and shared memory bandwidth made most of the others better on the CPU or GPU alone. In their AMD co-run-friendly example, power changed little across partition ratios, so energy mostly followed completion time [4]. That does not prove which shared resource bends this JPEG curve. It does explain why a one-worker gain cannot simply be multiplied by sixteen.

This makes a poor bulk-processing story, but it does make a useful request-latency story.

Headroom keeps requests out of the queue

A server does not jump from idle to sixteen permanent encodes just because it owns sixteen logical CPUs. Requests arrive independently. Their service times overlap sometimes; then the machine becomes quiet again.

Dean and Barroso call out shared CPU, cache, memory, and network contention as sources of latency variation, then show how layers of queueing amplify it. One of their practical rules is to keep low-level queues short [5]. Their subject is a much larger distributed service. The rule carries down to one box: spare capacity is what stops a burst from becoming a wait.

I used the measured completion rate at each active-worker count to build a simple queue model. Arrivals are Poisson. Up to sixteen requests can be encoded at once; later arrivals wait. This is an estimate based on the measured 1–16 worker curve, not a second traffic-generator benchmark.

Incoming requests/s	CPU active	GPU active	CPU mean response	GPU mean response	Average queued, CPU / GPU
50	1.15	0.85	23.0 ms	17.0 ms	0.000 / 0.000
100	2.42	1.82	24.2 ms	18.2 ms	0.000 / 0.000
200	5.65	4.45	28.3 ms	22.3 ms	0.005 / 0.002
250	8.00	6.58	32.4 ms	26.6 ms	0.100 / 0.062
300	11.17	9.96	41.3 ms	36.8 ms	1.229 / 1.069
330	13.47	12.89	58.7 ms	59.1 ms	5.909 / 6.604

At 200 incoming requests per second, CPU-only averages 5.65 active encodes and the GPU path averages 4.45. Both queues are effectively empty. The GPU path cuts mean response from 28.3 ms to 22.3 ms.

At 300 requests per second, bursts fill all sixteen workers about 25% of the time on CPU and 20% on the GPU path. The GPU still saves 4.5 ms, but average response has already climbed to 37–41 ms.

Around 330 requests per second, both designs are full more than half the time. Each has about six requests waiting on average, and their mean response times meet near 59 ms. At 340 requests per second, CPU-only wins because its maximum sixteen-worker throughput is slightly higher.

If an on-demand resize server regularly has fifteen or sixteen encodes active, the lesson is not to celebrate full utilization. The queue is already turning spare milliseconds into response time. Add another instance before that becomes the steady state.

The dispatch rule

The crossover depends on how much coefficient work an image creates. For noisy, coefficient-heavy synthetic images, the plotted crossover is 151×151 at quality 50 and 131×131 at quality 90. For a simple gradient, it is 480×480 and 481×481.

A simple gradient changes slowly from pixel to pixel. After the DCT and quantization, most of its AC coefficients are zero. That leaves little ordering and progressive-token work, so the CPU finishes small gradients before GPU setup pays off. Noise changes sharply everywhere. It leaves many nonzero coefficients in every block, giving the GPU enough parallel work at a much smaller image size. These synthetic images mark two ends of the workload; a production dispatcher uses the measured coefficient density, not a “gradient” or “noise” label.

The size sweep used square synthetic images with progressive 4:2:0 output at qualities 50 and 90, with 11 encodes per point. The chart plots CPU median time divided by GPU median time, so values above 1.0× favor the GPU.

The dispatch rule is the encoder-side version of Sodsong et al.'s model [2]. Dimensions estimate how many blocks exist. Quantized-coefficient density estimates how much progressive token work those blocks contain. The rule uses both because two images with the same dimensions can give the GPU very different amounts of work.

My current rule is:

Progressive 4:2:0 output:
    coefficient-heavy and at least 256x256 -> synchronous GPU path
    simple and at least 512x512            -> synchronous GPU path
    smaller                                -> CPU JPEGli

Baseline JPEG                              -> CPU JPEGli
Decode and resize                          -> CPU

A production dispatcher can make the content decision from the quantized-coefficient density that JPEGli already has. It should not delay a request to form a batch.

The claim is deliberately small. On this Radeon 780M, the GPU cuts about six milliseconds and one-third of the package energy from the progressive encode leg of a typical 2–4 MP on-demand request, while producing the same JPEG bytes. The advantage fades as active work approaches the machine's limit. That is acceptable because a latency-sensitive service should scale out before it lives there.

The backend also recognizes the Ryzen AI Max+ 395 and Radeon 8060S target, but I have not measured that hardware yet. The source is on the amd-vulkan-progressive-encode branch.

References

1. Martin Bruse, Luca Versari, Zoltan Szabadka, and Jyrki Alakuijala. “Users Prefer Jpegli over Same-sized libjpeg-turbo or MozJPEG.” arXiv:2403.18589, 2024.
2. Wasuwee Sodsong, Jingun Hong, Seongwook Chung, Yeong-Kyu Lim, Shin-Dug Kim, and Bernd Burgstaller. “Dynamic Partitioning-based JPEG Decompression on Heterogeneous Multicore Architectures.” arXiv:1311.5304, 2013.
3. André Weißenberger and Bertil Schmidt. “Accelerating JPEG Decompression on GPUs.” 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC), 121–130, 2021.
4. Feng Zhang, Jidong Zhai, Bingsheng He, Shuhao Zhang, and Wenguang Chen. “Understanding Co-Running Behaviors on Integrated CPU/GPU Architectures.” IEEE Transactions on Parallel and Distributed Systems 28(3), 905–918, 2017. DOI: 10.1109/TPDS.2016.2586074.
5. Jeffrey Dean and Luiz André Barroso. “The Tail at Scale.” Communications of the ACM 56(2), 74–80, 2013. DOI: 10.1145/2408776.2408794.