Back from the North, I've spent the last couple days using some new lenses to reduce the beam size. So far so good, and I've been able to reduce the un-molested beam to around 5mm in diameter, half what it was before. With this change almost the entire beam fits into the WFS measurement area I'm using (around 1000x800 px, usually sub-sampled).
Its taken some tinkering, but I've been able to get a recognizable poke matrix out of this configuration.


Notice how nearly the full extent of the modes are in the frame, much closer to the theoretical version. Its not all rainbows and puppies though, as you can see in this comparison between actuator influence functions (for the same actuator) using the small and large beams

Notice the depression surrounding the peak when the smaller beam is used. I would expect the peak to be narrower since the beam is condensed, but this is unexplained. Maybe related to that is the ring that often appears around the modes in the modal poke matrix.
I think this crap shows up because the beam diameter really changes when commands are applied, and the effect is more noticeable when the whole beam is in the WFS frame. Essentially, the problem is the assumption that each WFS sub-aperture measures the same area of the beam, independent of the command. In reality, the beam changes shape with a changed wavefront, so sub-apertures are actually measuring different parts of the beam each time a new command is generated. For small commands, or when the beam is much larger than the measurement area this isn't really a problem, but I think its what's causing the distortions here.
Surely someone must have noticed this before, but as far as I can tell from a 30 second Google scholar search ignorance is bliss, and no one's has the stones to address it. Its a tough problem since I have a suspicion the distortion isn't easy to predict analytically. I'm looking up a few papers on beam shaping based on deformable mirrors, so maybe there's a way to do some kind of pre-warping to the beam as a function of the command, sort of like pre-warping in the bilinear transform. Maybe then the sub-aperture measurements could be linked to a specific part of the beam and used to interpolate a wavefront profile. Maybe its possible to linearize this and make it just a series of simple transformations. Maybe I've had a few too many rusty nails while writing this.
Either way, el hefe thinks this might be a reason we couldn't generate a decent poke matrix from the HEL simulation we have and instead had to rely on a theoretical version. I shall vocalize my pre-warping idea to him tomorrow and report back.
Over and out.