Is it an insult to say that a particular referee is the "worst ever"? If we were to rank all the referees Australia and New Zealand, in order of quality, or height, or age, or any other criterion, what does it prove? In fact, when it comes to quality, they are already ranked. What is wrong with players, spectators, administrators, whoever, having an opinion about the rankings?
Some referees are clearly more competent than others, the most competent referees are selected for the most important matches. The least competent/experienced/whatever are probably running around refereeing subbies - but even these officials are presumably good enough to do their job.
So Digby might be implying that the referee in question is less competent than a subbies referee, although it could be argued that he has never seen subbies referees, so could not be referring to them.
It is a bit like ranking schools. Just because a particular school is at the bottom of the rankings does not mean that it is not operating acceptably. It just means that the schools above it are operating more effectively.
If Digby believes that this referee is at the bottom of the list of referees that he has played under, or watched, surely that is an acceptable opinion to hold. It is not as though he has accused him of cheating, showing bias, or anything reprehensible.