I have no idea what I'm talking about, but two things come to mind. First is Gimbal Lock, which I barely understand (that is, not at all). But my crappy understanding is that if you're using Euler angles you've got to be careful in some situations of you'll get some odd results. As I don't really understand it I have no idea if that's what you're looking for.

Personally, given the description I would wonder if you weren't getting a loss of precision due to the limitations of the IEE design of floating point numbers. Even if you're using doubles, you might be losing precision quicker than you realise.

You can test your understanding of doubles with a simple piece of java code:

public boolean testDoublePrecision(){    
   return Double.MaxValue == (Double.MaxValue - 1000000d);
}

There are quite a few non-obvious things relating to floating point numbers that even seasoned devs aren't aware of. Sorry about the poor formatting, but here are some other potential 'gotchas' with Double floats.

    System.out.println("Nan/PositiveInfinity = "+String.format("%a", Double.NaN/Double.POSITIVE_INFINITY));
    System.out.println("PositiveInfinity/NaN = "+String.format("%a", Double.POSITIVE_INFINITY/Double.NaN));
    System.out.println("PositiveInfinity/NegativeInfinity = "+String.format("%a", Double.POSITIVE_INFINITY/Double.NEGATIVE_INFINITY));
    System.out.println("PositiveInfinity/PositiveInfinity = "+String.format("%a", Double.POSITIVE_INFINITY/Double.POSITIVE_INFINITY));
    System.out.println("PositiveInfinity/0 = "+String.format("%a", Double.POSITIVE_INFINITY/0d));
    System.out.println("0/0 = "+String.format("%a", 0d/0d));
    System.out.println("1/0 = "+String.format("%a", 1d/0d));
    System.out.println("-1/0 = "+String.format("%a", -1d/0d));
    System.out.println("MaxValue/0 = "+String.format("%a", Double.MAX_VALUE/0d));
    System.out.println("NaN/0 = "+String.format("%a", Double.NaN/0d));
    System.out.println("0/Nan = "+String.format("%a", 0d/Double.POSITIVE_INFINITY));
    System.out.println("0/PositiveInfinity = "+String.format("%a", 0d/Double.POSITIVE_INFINITY));

    System.out.println("0*-0 = "+String.format("%a", 0d*-0d));
    System.out.println("0*PositiveInfinity = "+String.format("%a", 0d*Double.POSITIVE_INFINITY));
    System.out.println("0*NegativeInfinity = "+String.format("%a", 0d*Double.NEGATIVE_INFINITY));
    System.out.println("0*NaN = "+String.format("%a", 0d*Double.NaN));

    System.out.println("NaN==NaN = "+String.format("%b", Double.NaN==Double.NaN));
    System.out.println("PositiveInfinity==NaN = "+String.format("%b", Double.POSITIVE_INFINITY==Double.NaN));
    System.out.println("PositiveInfinity==PositiveInfinity = "+String.format("%b", Double.POSITIVE_INFINITY==Double.POSITIVE_INFINITY));
    System.out.println("PositiveInfinity==NegativeInfinity = "+String.format("%b", Double.POSITIVE_INFINITY==Double.NEGATIVE_INFINITY));
    System.out.println("0==NegativeInfinity = "+String.format("%b", Double.POSITIVE_INFINITY==Double.NEGATIVE_INFINITY));

It would seem strange for a rounding error to cause everything to be 0 for that vertex, but as I don't know your code it seems like it could be possible. I'm sorry if this isn't much help, just throwing out some ideas.