AI Foundry · Thermal camera hub

Seeing millikelvin heat with the P3

Switch the load on a schedule and average the first seconds after each switch, and the P3 shows heat leaving the CPU area within 2 seconds, spreading over the next 20, at 7–15 mK noise. Per frame it resolves about 4 mK on a small patch once the jitter shared by the whole image is removed. The biggest error is the camera itself: hung from the laptop it watches, it warms by degrees with the load, and every pixel moves with it.

Thermal Master P3 hung from the top of an M5 MacBook's screen, looking down at the keyboard, recorded overnight on while the Mac's CPU was switched on and off on fixed schedules: 14 recordings, 311,000 frames (3.5 hours at 25 Hz). Builds on Reverse-engineered Thermal Master P3.

2 s
until heat shows at the keyboard after the load starts
3.7 mK
16×16 patch per frame, frame jitter removed (was 13.5)
3.1 mK
per pixel, lock-in over 29 load cycles of 20 s
+3.8 °C
the camera's own warming in 5 min of full load
The first 20 s after the load starts, averaged over 7 starts: red where the keyboard warms, blue where it cools relative to the rest, grey where the change stays under 3× the noise (15 mK).

The setup: a keyboard, a camera above it, a load on a timer

Thermal image of the MacBook keyboard: warm keys on the left, a bright band and a cold strip in the middle, cooler keys on the right
What the P3 sees: about 18 × 14 cm of keyboard, rotated.

The camera is clipped to the top of the screen bezel, lens down, about 25 cm from the keys. A script on the Mac ran float32 matrix multiplies on 16 of its 18 cores (or on one) on a square-wave schedule, logging every switch. The P3 streamed its lossless 16-bit frames from a Realme phone over Wi-Fi to the same Mac.

Two kinds of output were compared: the calibrated temperature plane (62.5 mK steps), and the raw 14-bit counts that skip the camera's lookup table (38.6 mK steps). Every recording logged the camera's focal-plane temperature, the Mac's CPU use each second, and the phone's battery.

Heat leaves the CPU area within 2 seconds and spreads for 20

Six thermal images of the keyboard at 2, 4, 8, 12, 16 and 19 seconds after the load starts; a red warm region grows along the central band and four spots at the top light up
Calibrated mode, 7 load starts averaged, 1 s per picture, 3σ threshold.

Heat first shows along the middle of the bright band and at its right edge, 1–2 s after the cores start. By 8 s a warm region fills the band and keeps spreading. Keys to the left turn faintly blue only because each picture is relative to its own median, which rises as the warm region grows.

Four small spots at the top of the band are something else. They change within a second of the switch, by about as much for one core as for 18, and they cool in a 20 s schedule but warm in a 120 s one. Heat conducted from the CPU would scale with its power and take seconds to arrive, so these are probably the Mac's fans or a reflection (not yet tested). Read them as a landmark, not as heat.

The trick that makes this clean is timing. The camera's own body is slow: for about 20 s after each start its temperature keeps drifting the way it was going (here down 0.3 °C), so the only fast change in those seconds is the keyboard's. Averaging that window over every start of a run (7 here) and subtracting each moment's image median (the camera's drift, since the early heat covers a few percent of the view) leaves the heat alone.

A shutter calibration just before each start makes it 2–3 times cleaner

Two columns of thermal images at 4, 8 and 19 seconds after the load starts, with the shutter calibrated 5 s before each start. Left, 18 cores: a warm region covers the central band by 4 s and spreads right across the keys by 8 s. Right, the same timing with no load: only faint pale patches
Shutter calibrated 5 s before each of 7 starts, both columns on one colour scale (±1 K). Left: 18 cores (σ ≈ 7 mK). Right: the same schedule with no load, the control (σ ≈ 3 mK).

In run T the camera recalibrated with its shutter 5 s before every load start, so each onset window began on a fresh calibration, before the camera could drift. The noise per 1 s picture fell to 6.9 mK over 7 starts, against 15 and 24 mK in two runs without it and 14 mK with factory settings. Heat now shows spreading from the band to the right across the keys within 8 s. Since the shutter's own settling is now in step with the load too, a control ran the same schedule with no load (right): its largest changes stay under 40 mK, against about 530 mK with the load, so at least nine tenths of the left column is heat.

Most of a frame's noise is shared by the whole image, and is removable

On a 16×16-pixel patch, three quarters of the frame-to-frame noise moves the whole image at once. Subtracting the image's own mean each frame (or a patch away from the heat) removes it. About 6 mK of that shared part is the camera's thermistor: the P3 corrects every frame with its focal-plane reading, so each flicker of that reading's last bit (about 15 mK per step) shifts all pixels together.

Noise per frame, 16×16 patch, raw mode (mK): each strategy alone, then combined
as recorded
13.5
thermistor flicker removed
12.1
frames averaged in even+odd pairs
8.8
thermistor removed + whole-image mean subtracted
3.7
One pixel: 26 mK per frame, 20 mK with the shared part removed. A 48×48 patch: 3.2 mK.
Allan deviation of a 16x16 patch against averaging time: lowest, about 2.5 mK, near 0.1 s, then rising; lower at long times with the auto-shutter on
Averaging a 16×16 patch helps only to about 0.1 s (2.5 mK). With the camera's auto-shutter off, the drift beyond 10 s is 2.5–3.5 times larger. No load, 5–10 min each.

The rise after 0.1 s is not camera noise: the Mac never stops working (indexing, sync, other agents), so the keys wander by about 10 mK within a second and 100 mK within a minute. That is the useful lesson for the lab too: a slow average can only be as steady as the thing being watched. To see a small change, make it happen on a schedule, and measure in step with it.

The camera heats with the laptop, and every pixel follows the camera

Two panels, raw and calibrated, each over 14 minutes with 5 minutes of load: the camera's own temperature rises 2 to 3.5 °C; in raw mode the keys appear to cool by 5.7 K, in calibrated mode they rise and then fall with the camera
The same 5-minute full load, recorded twice. Raw counts: the keys "cool" 5.7 K while the camera warms 3.5 °C. Calibrated: the keys rise and fall almost in step with the camera.

The P3 hangs from the screen bezel, which warms when the M5 works hard, so the camera's own temperature follows the load: up 2.4–3.8 °C in 5 minutes, and back down by 4–6 °C afterwards. Raw counts move 400–700 mK per degree, opposite to real heat. The calibrated output was supposed to fix that; with the auto-shutter off it moves about +0.4–0.6 K per degree instead, now in the same direction as real heat, which is worse because it can't be told apart. The camera's correction only works through its shutter, and even the factory auto-shutter left most of the artifact in place (below). The fix is mechanical: keep the camera's temperature still.

Lock-in to the load finds the heat; a short period keeps the camera out of it

Each pixel's series was fitted with a sine and cosine at the load's frequency, plus a slow trend in the same fit (never a high-pass filter first). At a 120 s period the camera's focal plane swings ±0.79 °C, peaking 80 s after each load-on, and most of what the pixels show is that. At a 20 s period it moves only ±0.004–0.025 °C: the camera body is too slow to follow, and what remains is mostly the keyboard.

Load period, cyclesCamera swing at that periodMedian pixel responseNoise per pixelPixels above 5σ
120 s × 9 (raw)±0.79 °C483 mK, mostly the camera30 mK100%
20 s × 29 (raw)±0.004 °C12.8 mK4.5 mK13% (42% above 3σ)
20 s × 29 (calibrated)±0.025 °C17.6 mK*3.1 mK65%
* In calibrated mode even the camera's ±0.025 °C could be worth up to 15 mK, and at 20 s its timing is too close to the heat's to split them.
Lock-in amplitude at a 20 s period over the keyboard on a log colour scale: over 100 mK at the top of the central band, tens of mK across the keys
Where each 20 s load cycle shows up: over 100 mK at the top of the bright band, tens of mK across the keys; grey where it stays under 3σ. Raw mode, 29 cycles. The strongest stripe at the top is the four spots, which are probably not CPU heat (see above).

Phase separates real heat from the camera's own

At the slow period the artifact can still be removed. The camera's warming reaches every pixel with one timing, the focal plane's, only its size and sign differ. Keeping the part of each pixel's response that is perpendicular to the focal-plane phase removes it whatever its size; real heat, which arrives with its own timing, survives. That turns the 120 s run's map from a wash into a few hundred mK on the keys, plus the four spots at the top, which stand out at every period and load and are probably not CPU heat.

Map from the 120 s run with the camera's own warming rejected: near zero almost everywhere, two hot stripes at the top centre reaching about 2 K
120 s cycles, raw mode, with the part in step with the camera's own temperature removed: the stripes at the top (the four spots) swing by up to 2 K each cycle; the keys stay within a few hundred mK.

The second campaign: calibrated mode, shutter timing, factory settings, one core

The camera's calibrated output is meant to correct for its own temperature. Without its shutter it does not: in a 10-minute no-load baseline its focal plane warmed 2.1 °C and the whole image rose with it (about +0.4–0.6 K per degree). At a 120 s load period its swing (±0.77 °C, peaking 75 s after each start) still dominates the pixels, and rejecting it by phase again leaves the four spots at the top. Its clear wins are the onset views above and the lowest per-pixel lock-in noise at 20 s.

Run (load cycles)Camera swingMedian response, camera part / restLock-in noise per pixelOnset noise, 1 s picture
Calibrated, shutter off, 120 s (7; recorded twice)±0.77–0.79 °C376–383 / 69–96 mK26–41 mK15–24 mK
Calibrated, shutter 5 s before each start, 120 s (7)±0.83 °C419 / 115 mK25 mK6.9 mK
Factory: auto-shutter and noise filters, 120 s (5)±0.69 °C321 / 72 mK22 mK14 mK
Calibrated, one core instead of 18, 120 s (7)±0.11 °C111 / 57 mK30 mK7.8 mK†
Control: shutter 5 s before each start, no load, 120 s (7)±0.05 °C31 / 6 mK14 mK2.8 mK
Calibrated, 20 s (29)±0.025 °C17.6 mK in all*3.1 mK9.9 mK
* Too close in timing to split at 20 s. † Nothing in the first 8 s repeats from start to start.
Four maps of the 120 s response with the camera's own warming removed: calibrated with the shutter off, with the shutter timed to the load, factory settings, and one core. The first three look alike, with the four spots at the top strongest; the one-core map is weak and noisy
What is left at a 120 s period once the part in step with the camera's own temperature is removed. Each map has its own scale; the one-core map's is seven times smaller.

What to do, in order

#StrategyWhat it buys hereCost
1Mount the camera off the thing it watchesRemoves the largest error: 0.4–0.7 K per °C of camera warming, in every modeA stand or bracket
2Drive the heat on a schedule and lock in to it3.1–4.5 mK per pixel in 10 min; rejects everything not in stepA controllable load
3Average the first seconds after each startHeat flow as pictures, at 7–15 mK, before the camera's body reactsMany repeats
4Calibrate the shutter just before each startOnset noise 15–24 → 7 mKA shutter click per cycle
5Modulate faster than the camera body followsCamera swing falls from ±0.79 °C (120 s) to ±0.004–0.025 °C (20 s)Less depth: shallower heat
6Subtract the whole-image jitter each frame13.5 → 3.7 mK per frame on a 16×16 patchNeeds pixels away from the heat
7Reject the camera by phaseTurns a slow, camera-dominated map into the real hot spotNeeds the camera's temperature log
8Average patches, not just frames26 mK per pixel → 3.2 mK on 48×48, per frameResolution
9Drop glitch frames before statisticsThe P3 sends rare frames several kelvin off; they dominate any mean or Allan curveA robust threshold
10Raw 14-bit countsFiner steps (38.6 vs 62.5 mK), but no correction for the camera's temperatureOnly for fast, differential work

The phone app got sturdier along the way (1.3.0, deployed over Wi-Fi)

Next

  1. Mount the camera on its own stand, then repeat the 120 s lock-in: the camera artifact should vanish.
  2. Identify the four spots: log the fan speed alongside a run (needs sudo powermetrics), or cover them with matte tape and see whether they move.
  3. Matte tape on the area of interest (shiny metal reflects the room), and a passive reference block in view.
  4. For the board in the lab: modulate slowly enough to get through 1.6 mm of FR-4 (around 0.003–0.03 Hz); lateral blur through the board is then about 2 mm.
  5. The hub gives the phone only 500 mA; a powered PD supply is still the real fix for the battery.
Methods, code and sources

Code: ~/git/thermal-android/experiments/sensitivity/ (recorder, load generator, campaign runner, analysis, figures). Recordings are lossless 16-bit frames with the camera's InfoLine, timed by the camera's frame counter; lock-in fits include a quadratic trend; noise beside the drive frequency estimates each pixel's error. Allan deviation is overlapping. Every run's settings were set explicitly before it and watched during it; runs disturbed by app restarts or other agents were flagged.

Sources: Breitenstein et al. on lock-in thermography (MPI Halle, QIRT 2014); principles of lock-in detection (Zurich Instruments); Allan deviation (arXiv 0908.4484); 3D noise for FPAs (ImageJ); common-mode templates with per-pixel gains (SysRem); dither and quantization (Lipshitz 1992).