Let x be a vector in Rn. If it has coordinates x1 x2 … xn, then x is mapped to x1×M1 + x2×M2 + … + xn×Mn, a linear combination of the rows of M. Using matrix notation, write this as x*M.
Consider two vectors x and y. They are mapped to x*M and y*M respectively. Compute the dot product of their images. The first coordinate of the image of x is the dot product of x with the first column of M. The first coordinate of the image of y is the dot product of y with the first column of M. Multiply these two expressions together and expand. For i ≠ j, we find terms like xiyjMi,1Mj,1. This looks like a mess, especially when you repeat it for all the other coordinates of the images of x and y. But with proper regrouping, it all clears up. The common factor xiyj is multiplied by the sum of Mi,kMj,k as k runs from 1 to n. This is the dot product of rows i and j in the matrix M. Since M is orthonormal, that drops to 0. We can forget about i ≠ j.
When i = j, xiyi is multiplied by the sum of Mi,k2 as k runs from 1 to n. This is the ith row dotted with itself, and in an orthonormal basis, that is 1. The dot product of the image of x with the image of y is the sum of xiyi, which happens to be the dot product of x and y. Therefore M preserves dot product.
Since length is determined by dot product, an orthonormal transformation preserves lengths. Line segments don't grow or shrink as you apply the transformation. Furthermore, angle is a function of dot product and lengths, so angles are preserved as well. A shape is not bent or twisted as it is remapped; it is simply moved to a new location. This is called a rigid rotation.
Conversely, let a matrix M implement a linear transformation that preserves lengths and angles. In other words, it preserves the dot product. Premultiply M by the ith coordinate vector to extract the ith row. This must have length one, just like the unit vector that produced it, hence the rows of M all have unit length. The dot product of any two coordinate vectors is 0, hence the dot product of any two rows in M is 0. A matrix is orthonormal iff it defines a rigid rotation.
(The transpose of an orthogonal matrix need not be orthogonal, as shown by [4,2|-1,2]. Nor is its inverse orthogonal, nor its square.)
If the orthonormal matrix has determinant -1 then the adjoint is minus the transpose.
Let's see why the group is nonabelian. Take the standard x y z basis, rotate it 90° around the z axis, then 90° around the x axis. The orientation is different if you apply these rotations in reverse.
Ok, we have some more work to do. The determinant of an orthonormal matrix is ±1. Let the rigid rotations be the subgroup of orthonormal matrices that have determinant 1. The other matrices, with determinant -1, are called reflections. Since determinant is a continuous function, there is no continuous path of rigid rotations connecting the identity matrix to a reflection. Such a path would see the determinant jump from 1 to -1. However, we can establish a path to any matrix with determinant 1. This is the "rotation" part of rigid rotation. The coordinate system gradually moves, intact, towards its destination, until it finally coinsides with the basis of M. Let's try to build such a path of matrices.
In one dimension the only rigid rotation is [1]; the unit vector just sits there. The single reflection maps 1 to -1.
In two dimensions the rigid rotations spin the plane about the origin. The vector [1,0] is mapped to a point on the unit circle, and [0,1] is mapped to a point 90° farther around. The matrix looks like (s,t|-t,s), where s,t is any point on the unit circle. All these linear maps are determined by some angle θ, and functions are composed by adding angles. Thus the group of rigid rotations in two dimensions is abelian.
If a matrix looks like (s,t|t,-s), it is certainly orthonormal, but its determinant is -1. It is a reflection. The y axis is 90° behind the x axis, instead of 90° ahead. Still, any reflection can be continuously transformed into another reflection by rotating the basis through an appropriate angle. The group of rotations and reflections defines the dihedral group on the circle.
Interestingly, each reflection really is a simple mirror image, if you tilt the mirror properly. Start with the standard basis and map y to -y. This is a reflection through the x axis. All other reflections are rotations of this one. Take the standard reflection and rotate it counterclockwise through an angle of θ. You can accomplish the same thing by reflecting the plane through a line that intersects the origin and meets the x axis at an angle of θ/2. (This is our tilted mirror.) Choose any point on the unit circle, representing an angle of w. The first transformation takes w to -w+θ, while the second takes w to (w-θ/2)×(-1)+θ/2. These are algebraically equivalent.
We've found a connecting path of matrices for rotations, or reflections, in 2 dimensions; proceed by induction on n. The top row of M, call it v, must map continuously onto w, the top row of the identity matrix. Let a linear map fix everything perpendicular to v and w, and gradually spin v around until it equals w. The rest of M is mangled in some way, but the top row has become w, and the remaining rows are perpendicular to w. Except for a 1 in the upper left, the first row and column of M are all zeros. The rest of M is orthonormal, and spans the subspace perpendicular to w. Furthermore, its determinant is still 1. By induction one can map the rest of M onto the identity matrix. Therefore every rotation can be continuously transformed into the identity map, and every pair of rotations is path connected.
If the matrix is a reflection, apply the same procedure. We can always morph to the identity matrix with -1 in the lower right. At the base of this inductive argument, any reflection in 2 dimensions can be continuously transformed into [1,0|0,-1]. Thus all reflections are path connected.
In higher dimensions reflections are no longer simple mirror images. You can't just tilt the mirror; a pre or post rotation is necessary. To illustrate, reflect 3 space through the xy plane, then spin it 90° around the z axis. (These operations commute, so do them in either order.) If this were a simple reflection the positive z vector would be reflected through a mirror, and would point down the negative z axis. The mirror has to be the xy plane. At the same time the positive x axis has become the positive y axis, hence the mirror is the plane x=y. It can't be "done with mirrors", as was the case with 2 dimensions.