I'd be far more worried if I were a Brumbies fan. I can think of a few reasons why they would be the first to go.
National footprint. The Wallabies are a national team and the ARU prize highly the national footprint of Super Rugby to undergird it. In short that means they need to prioritize both Perth and Melbourne over Canberra.
Financial reasons. Why have the Force struggled? Because they've been on the bottom of the ladder or there abouts for 6 of the last 7 seasons. In contrast the Brumbies have struggled in the midst of on field success. The crowd attendance at their home finals in the last few years has been appalling. The Brumbies won't make money at the gate until Canberra build a roofed stadium which isn't happening for another 15 years. While cutting down to 4 teams will solve the shortage of quality players that is the most pressing issue for the Force. The Force did pretty well until the Rebels came along.
The Brumbies board have been allegedly caught up in some dodgy property sale to the University of Canberra. The board have been nicknamed the Canberra mafia. Why keep an organisation run by these people afloat?
NRC. It would be much easier and cheaper to continue an NRC team out of Canberra without a Super Rugby team than it would Perth. They could even continue the Brumbies trademark in this competition and stop fighting over the name and colours of their NRC team.
Perth allows for a later time zone which creates a better product for TV particularly abroad.
Perth have doubled their rugby player numbers since the inception of the Force and if they pulled out now the ARU will never convince Sanzaar to give it back and would have all been a waste of an investment at a time where it just seems they might start to produce Wallabies. If the ARU believe that Perth will one day produce more Wallabies than Canberra then the decision becomes very easy.
In contrast the ARU could make a very good argument to SANZAAR, once the roofed stadium is built, that Canberra deserves a Super Rugby Franchise, because it has in the past, and get back that 5th licence.