Let |G| = pq, where p is the larger prime. By the third sylow theorem, the number of subgroups of order p equals 1 mod p, and divides q, hence there is one such subgroup, and Zp is normal in G, with factor group Zq.
Since G contains an instance of Zq, and q and p are relatively prime, G is a semidirect product. The elements of Zq acurately represent the factor group G/Zp.
suppose q does not divide p-1. The automorphisms of Zp carry one generator onto another. There are p-1 such automorphisms, including the trivial automorphism that fixes Zp. A semidirect product implies a group homomorphism from Zq into the automorphisms of Zp. If the image of this map is nontrivial then it becomes a q-cycle inside a group of order p-1, which is impossible. Therefore Zq induces trivial automorphisms on Zp. Our semidirect product has become a direct product, namely ZpZq.
Now assume q divides p-1, and G is nonabelian. It is now possible for the members of Zq to induce nontrivial automorphisms on Zp.
Since Zp and Zq are cyclic, let's use additive notation. Their identity elements are 0, and their generators are 1. Now 1 (in Zq) carries 1 to s in Zp, where s is between 2 and p-1 inclusive. This defines the entire map from Zq into the automorphisms of Zp. If 1→s is applied q times, we are back to start; hence sq = 1 mod p.
Suppose a different nonabelian pq group maps 1 (in Zq) to the automorphism 1→t. Once again tq = 1 mod p. Both s and t are qth roots of 1.
Remember that q is prime. Any qth root can be raised to a certain power to get any other qth root, in any field. You can see this on the unit circle in the complex plane. Plot out the seven 7th roots of 1. The third such root, raised to the fifth power, gives the first root, and so on. for purposes of illustration, say s = t3. Now 1 in Zq leads to the automorphism that we call t. This means 3 leads to the automorphism that we call s. Yet 3 is a perfectly good generator for Zq. Relabel the elements so that 3 becomes 1. Now both groups are semidirect products of Zp by Zq, such that 1 in Zq induces the automorphism 1→s in Zp. The two groups are isomorphic.
In summary, a pq group could be the direct product ZpZq, or, if q divides p-1, it could be the semidirect product of Zp by Zq, where 1 in Zq induces the automorphism 1→s, where s is some predesignated qth root of 1. The latter (nonabelian) group is called the metacyclic group of order pq.
When q = 2, the metacyclic group is the same as the dihedral group. The only nontrivial automorphism of order 2 caries 1 to -1, and is a reflection of Zp. This gives the reflections and rotations of the p-gon, which is the dihedral group.
The metacyclic group is equivalent to upper triangular 2×2 matrices mod p, where each matrix has 1 in the upper left and a qth root of 1 mod p in the lower right. There are indeed pq such matrices. Since some of these matrices don't commute, we have a nonabelian pq group, and it has to be metacyclic.
Once again G is a semidirect product, and 1 in Zm maps to 1→s in Zp. This map is trivial (making G a direct product), or s is a nontrivial mth root of 1 mod p.
If m goes into p-1, we can characterize these groups, up to isomorphism. Consider two such groups. We can connect 1→s and 1→t iff gcd(s,m) = gcd(t,m). Find an exponent e such that s = te. We can relabel the elements of Zm, with e acting as generator, iff e and m are coprime. Yet e preserves the gcd relative to m, so e cannot introduce any new factors of m, and e and m are indeed coprime. Make e the generator of Zm and build the isomorphism.
When m divides p-1, there is a distinct group of order pm for each factor of m. The factor 1 leads to the abelian group ZpZm.
As a special case, let m be the prime q, whence there is but one nontrivial factor, namely q, and that leads to the metacyclic group described above.
The sylow subgroup Z5 is normal, and G also contains a sylow subgroup H of order 4. Since 5 and 4 are relatively prime, G is a semidirect product of Z5 by H.
Let H be cyclic, namely Z4, and apply the results shown above. There are three possible groups, according to the factors of 4. One is the direct product, one mapps 1 to 1→2, and one maps 1 to 1→4.
If H is Z2Z2, then 0 or 2 elelements reflect Z5. That's another 2 groups, hence there are 5 groups of order 20, up to isomorphism.