Okay, here's an example of a sport that regularly moves from one ref to three. (Not gridiron, but they make it work too.)
The wrestling World Championships recently finished -- freestyle, greco and women's freestyle. I'm in the States, an in collegiate wrestling, there's only one ref on the mat (except for finals of major tournaments, but that may have changed). At the international level, there's the ref, mat judge (scorekeeper), and mat chairman (timekeeper) -- and a TMO, for just two competitors. So the North American wrestlers have to switch from having one ref to three depending on the competition.
The way it works is once the ref makes a call, the mat judge has to agree with the call before the points are scored. If there's a scramble and the scoring isn't immediately clear, the mat chairman gets involved. It all moves along really smoothly -- you never notice when a call is confirmed, because happens almost immediately. Throughout the hundreds of matches at the Worlds, I only saw maybe three or four times where the officials had to go to the video because they weren't sure how to make the call.
But the TMO can get involved, and they have interesting rules for that. If a coach or competitor doesn't agree with a call, they can challenge it withing 5 seconds of the call. A wrestler gets one challenge per match, so they have to be judicious, and if a coach throws the challenge, the wrestler can turn it down -- but can only turn it down once per match, to keep it from being used as a stalling technique. A separate group of refs (called a jury-of-appeal -- basically the TMO) reviews the call. If the wrestler who called for the challenge wins the appeal, the score is changed. If the wrestler who called for the challenge loses the appeal, the other wrestler receives a point.
Of course this isn't a one-to-one parallel with rugby, and I'm not saying there should be three refs on the pitch. I'm just using it as an example where the sport regularly moves from 1 ref to 3 without much issue. The model of a call being confirmed is interesting; you could see something like that for scrums, where there's always two or three refs watching it, and whatever call is made has to be confirmed. Maybe also allowing a limited number of captain's challenges if they feel a ref got a call wrong, but doing something like losing possession if they review it and the call was right.
Because lets face it, as long as the ref is on the side of the feed, the LHP on the opposite side can apparently angle all day without consequence. If the ref blew up a scrum for one side wheeling, and two others could immediately say "No, it was the other side boring in, and that's why it turned," you'd get the right call, and eventually packs would stop cheating because they'd be putting their side at a disadvantage rather than earning a penalty.
It's something I could imagine being trialed at the club or NRC level, and would take some time to bed in. There'd be growing pains, but it doesn't seem impossible; it'd probably take a while before teams started to police themselves a little better because they couldn't get away with as much, but right now there's little incentive to do so. The other problem I can see is apparently there aren't enough referees of quality or the necessary standard to fulfill such a need, so who knows. And I'm not sure some current sanctions are enough to limit something like cheating at a scrum, but stiffer sanctions wouldn't matter if the ref isn't getting the call right in the first place. All I'm saying is I've seen multiple refs work in another sport, and work well.
(The challenge in wrestling makes for great sponsorship opportunities. Normally a red or blue foam cube is thrown out onto the mat. But in some sponsored tournaments, I've seen red or blue stuffed Angry Birds used instead of cubes, and recently Minions wearing red or blue overalls. So the ref has to pick up this stuffed animal and carry it around while being the stoic official in charge.)
