I wish it were always that easy.
Some internet connections experience severe packet loss and jitter as utilization increases. I had a user tell me the other day that he experienced 20% packet loss (!!) at 100% utilization. Most symptoms are not that severe, but are prevalent enough that audio quality problems frequently occur when the internet connection is heavily loaded. If that does happen, the problem can be solved by using QoS to rate limit your internet connection to ~80% of its
If you never have problems, that's great! You are not the majority. Audio quality issues with VoIP are incredibly common, and a large proportion of the problems are caused by poor quality bandwidth.