My current suspicion about the laser being wonky is that the arm it is mounted on isn't particularly strong. It isn't meant to hold much weight, it was originally holding a pen.
So it can bend just a little bit and that may be enough to throw off the beam angle after it cuts into the material just a little bit.
This is solvable, but may require a new frame.