c# - How can I find the closest vector3 to a given target vector among all vectors orthogonal to a given input vector3?

Question

Suppose I have two input vectors with nonzero magnitudes:

Vector3 t;
Vector3 o;

I need to find the normalized vector v such that out of all normalized vectors that are orthogonal to o, v is the one that is closest to this t "target" vector. I also need to know if t and o are colinear in which case no one such single v exists.

Illustration of this relationship:

For a concrete example, suppose I have a game where the player will occasionally see a certain compass-like object on a surface in the environment. The player will have to point the needle on this compass towards a pictured object that was placed randomly somewhere in the game world. When the player selects the needle's position, I need to know how far off the player's selection is from the "true" direction of the object.

The compass needle is fixed to rotate around an axis with world direction o (perhaps this is the compass's transform.forward), and the direction of the pictured game object is t.

When t and o are not orthogonal, the compass needle will not be able to point directly in t, so I want to know: How can I get the closest possible direction the compass needle can point at, which is v? This way, I can measure the angle between this goal direction and the player's selected direction. And, when that angle is below some threshold, the player is successful.

Answer 1

You can use vector cross products to solve for this. Note that the order of the parameters of Vector3.Cross matters--inconsistently ordering the parameters can produce unexpected results.

1. Take the cross product of t and o.

   Vector3 x = Vector3.Cross(t, o);
   

2. If o and t are colinear, x will be equal to Vector3.zero.

   if (x == Vector3.zero) 
   {
       // Handle colinear situation
   }
   

3. Otherwise, x will be a vector that is orthogonal to both o and t and therefore also orthogonal to v:

   else
   {
   

4. Now that we have two vectors that are orthogonal to v (that is, o and x), we can do another cross product to find a vector that is colinear with v. And if we are careful about the ordering of this second cross product compared to the cross product we used to calculate x, the output of the second cross product will point in the same direction as v as well. All that is then left is to normalize it:

       Vector3 v = Vector3.Cross(o, x).normalized;
   
       // Use v
   }
   

Altogether:

Vector3 x = Vector3.Cross(t, o);

if (x == Vector3.zero) 
{
    // Handle colinear situation
}
else
{
    Vector3 v = Vector3.Cross(o, x).normalized;

    // Use v
}