I don't know if it's what you are looking for exactly, but the instance function create a transformation matrix. You just need to multiply to @P.
https://www.sidefx.com/docs/houdini/vex/functions/instance.html
Here a simple vex code to rotate all points with an axis and pivot attributes previously defined.
// rotate
vector scale, postrotation;
vector4 orient;
float angle = ch("ang");
matrix3 rotm = ident();
rotate(rotm, radians(angle), v@axis);
orient = quaternion(rotm);
scale = 1;
postrotation = 0;
matrix xform;
xform = instance(v@pivot, @N, scale, postrotation , orient, v@pivot);
@P *= xform;
EDIT: You can also take a look with quaternion, eulertoquaternion, qrotate, etc.. for rotation