Let's assume we have an object that is moved, rotated and scaled, and we define a transformation matrix as follows:
glm::mat4 objTrans ...; // translation
glm::mat4 objRot ...; // roation
glm::mat4 objScale ...; // scaling
glm::mat4 objMat = objTrans * objRot * objScale;
And we have rotation matrix that we want to run on the object. In this case we have rotation around the Z-axis:
foat angle ...; // rotation angle
glm::mat4 rotMat = glm::rotate( angle, glm::vec3( 0.0, 0.0, 1.0 ) );
We have several rotations we can do with this information.
First we want to rotate the object on its local axis:
glm::mat4 modelMat = objMat * rotMat;
A Rotation around the worlds origin can be performed like this:
glm::mat4 modelMat = rotMat * objMat;
In order to rotate around the origin of the object in the world coordinate system, we must eliminate the rotation of the object:
glm::mat4 modelMat = objMat * (glm::inverse(objRot) * rotMat * objRot);
A Rotation around the worlds origin in relation to the object you have to do the opposite:
glm::mat4 modelMat = (objRot * rotMat * glm::inverse(objRot)) * objMat;
If you have a complete transformations matrix for an object and you do not know the rotation part, then it can be easily determined.
Note that a transformation matrix usually looks like this:
( X-axis.x, X-axis.y, X-axis.z, 0 )
( Y-axis.x, Y-axis.y, Y-axis.z, 0 )
( Z-axis.x, Z-axis.y, Z-axis.z, 0 )
( trans.x, trans.y, trans.z, 1 )
To generate a rotation only matrix you have to extract the normalized axis vectors:
glm::mat4 a ...; // any matrix
glm::vec3 x = glm::normalize( a[0][0], a[0][1], a[0][2] );
glm::vec3 y = glm::normalize( a[1][0], a[1][1], a[1][2] );
glm::vec3 z = glm::normalize( a[2][0], a[2][1], a[2][2] );
glm::mat4 r;
r[0][0] = x[0]; r[0][1] = x[1]; r[0][2] = x[2]; r[0][3] = 0.0f;
r[1][0] = y[0]; r[1][1] = y[1]; r[1][2] = y[2]; r[0][3] = 0.0f;
r[2][0] = z[0]; r[2][1] = z[1]; r[2][2] = z[2]; r[0][3] = 0.0f;
r[3][0] = 0.0f; r[3][1] = 0.0f; r[3][2] = 0.0f; r[0][3] = 1.0f;