If xj = 1 then xj-1 is the inverse of x, and x is nonsingular.
Let y be similar to x. In other words, y = qx/q, where q is an invertible matrix. Verify that yj = qxj/q. At the same time, xj = (1/q)yjq. Thus xj = 1 iff yj = 1. Similar matrices have the same order.
If x is a large matrix, it is often worth while looking for a nonsingular matrix q that diagonalizes x. In other words, x = qd/q, where d is diagonal. Now xj = qdj/q, and dj is easily calculated.
If x is similar to d, as described above, then the order of x is j iff all the entries in the diagonal matrix d are roots of 1, and the least common multiple of these exponents yields j.
Let x have an eigen value of e. Apply x twice and find an eigen value of e2. If e is not a root of 1, then the powers of x can never produce the identity matrix. If x has a positive order than all its eigen values are roots of 1. This is affirmed by the diagonalization argument given above. The elements of d are the eigen values, and they are all roots of 1.
If R is the complex numbers, and the trace of x exceeds n, then at least one eigen value exceeds 1, and x has order 0. The same holds if the absolute value of the trace of x exceeds n.
Let x be block diagonal. Raising x to the j raises each block to the j. Thus x has order j iff each block has an order dividing j, and the lcm of these orders yields j. This confirms the earlier results when x was diagonalizable. Each entry is its own block, and a root of 1, and together these elements must be raised to the jth power to reach 1.
For the rest of this page, and the pages to come, I will assume R is an integral domain. Therefore x is similar to a matrix y in jordan canonical form. Concentrate on a jordan block that is not pure diagonal. It therefore has l down the diagonal, and ones on the subdiagonal. If l is zero then this block eventually becomes zero, and x has order 0, so assume l is nonzero. Write the block as the sum of the diagonal matrix and the subdiagonal matrix. Raise this to the jth power by applying the binomial theorem. (The binomial theorem is valid for any commutative ring, and since the diagonal and subdiagonal pieces commute, we're ok.) The diagonal matrix is raised to the j, then the next term is the diagonal raised to the j-1, times the subdiagonal, times j. Since R is an integral domain, this is always nonzero. The entries all lie on the subdiagonal. The remaining terms bring in entries below the subdiagonal. The result is a block that is never 1. Therefore x has order 0.
In conclusion, x has a positive order iff it is diagonalizable, and its eigen values are all roots of 1. The order of x is the lcm of these exponents.