Suppose H is a normal subgroup of order 2. If c is the involution, then xc/x is either c or 1. It can't be 1, else c = 1. So xc/x = c, and c is in the center. O has no center, so try a normal subgroup of order 60 and index 2. Indeed, a computer program verifies this, and same for other small primes. O appears to have a normal subgroup O′ of index 2.
A5 within S5 makes you think of permutations, does it not? Let M be the antidiagonal matrix of -1. (Note that det(M) = 1.) According to my computer program, M is not in O′, so perhaps M induces an odd permutation on a set. The matrices of O are nonsingular; hence they act on vectors of dimension 3. There are p3 such vectors. Our matrix M is an involution, so it either leaves vectors fixed or it induces transpositions. The permutation is odd iff there are an odd number of transpositions. A vector is fixed iff it is of the form [x,0,-x]. There are p such vectors. Subtract this from p3, and divide by p (which won't change the parity). The result is p2-1. There are (p2-1)/2 transpositions, and this is an even number. So we're not there yet.
Happily, O permutes unit vectors, and the product of matrices in O induces the composition of the corresponding permutations. In other words, a group homomorphism maps O into the symmetric group on unit vectors. Before we proceed, let's prove that O embeds.
Suppose a matrix in O fixes everything. It maps [1,0,0] to itself, and that means the top row has to be [1,0,0]. A similar result holds for [0,1,0] and [0,0,1], hence we have the identity matrix. This generalizes to all dimensions. On permutes the unit vectors of length n.
Now return to M, which is antidiagonal, with entries of -1. If M fixes a unit vector [x,0,-x], then 2x2 = 1. Either x is ± the square root of ½, or it can't be done at all. Let p be 5 mod 8. (In our example, p was 5.) By quadratic reciprocity, 2 is not a square, and neither is ½. Nothing is fixed by M. In an earlier section we found p2+p unit vectors. This is 6 mod 8, so divide by 2 and find an odd number, hence an odd number of transpositions, and an odd permutation. O′ is a proper subgroup of index 2, and a normal subgroup of O.
When p = 3 mod 8 the analysis is similar to that above. M fixes nothing, and there are (p2-p)/2 transpositions. Once again the permutation is odd.
Since quadratic reciprocity works for all finite fields, the above analysis holds for any finite field F of size w, where w is 3 or 5 mod 8.
The case of w = ±1 mod 8 is considerably more complicated. Here is an outline. Find a matrix U in O2 that seeds the cycle of length w-1 or w+1 (when w is 1 or 3 mod 4). Place U in the upper left and 1 in the lower right. This induces an odd permutation on unit vectors, and proves that O′ is a proper subgroup of O. This works for all values of w, so we didn't really need to construct the aforementioned matrices, for w = 3 or 5 mod 8, though that was a good exercise. The proof is rather technical, and it falls naturally under the scope of the next section, so please turn the page.