Quality of Service Management
April 15, 2013 - These Policies are Currently Under Review
With our backhaul upgrade approaching completion, we plan to liberalize these rules substantially in the near future. These are NOT the policies we plan to apply to the Wicked Fiber network. We plan to have these polocies updated and new rules deployed by late July 2013.
This document is intended to outline the quality of service management for the Wicked Broadband network. The document is intended to help network administrators and Freenet members understand how quality of service is managed on the Freenet network and outline the basic settings for the various levels of QoS management.
The audience for this document should have a decent understanding of Layer3 protocols, Linux and wireless signaling architecture.
Quality of Service Overview
The goal of Wicked Broadband's quality of service mechanism is to ensure that end users of the system get the bandwidth they need, when they need it. The network is operated according to network neutrality standards as set forth in the Broadband USA NOFA published by the USDA and NTIA on July 1st, 2009.
Though our ideal solution for bandwidth management would be to install a 1Gbps fiber-optic connection at each member's home, currently available monetary resources do not make this possible. As a result we are trying to make the best use of available network resources.
The first layer of our QoS strategy is not QoS at all, it is simply a routing mechanism that ensures that our various Points of Presence (POPs) are continuously connected to the network. To accomplish this, the company uses OSPF routing on its network core. This routing scheme uses constantly updating dynamic tables to determine the "lowest cost" route between one POP site and another.
The basic setup of this system is that each major site has at least 2 points of egress. If any link is severed, OSPF uses the next available route to send data to the end user. The QoS part comes in when costs are assigned to the various links. Wicked Broadband manages inter-tower traffic by assigning the highest cost to the lowest capacity links. This ensures that traffic is running through the fastest, highest capacity link at any given time.
Another layer of the company's QoS/Routing strategy is the use of mesh networking technology. This technology is very similar to OSPF, but uses various wireless characteristics including signal strength, latency and interference to generate the best path between the end user and the network core. Once again, a failure of any given link generally results in a new path to the internet, provided the mesh unit can see at least two neighbors.
Traffic on the network core is managed by a payment gateway system. This Linux based system is responsible for allowing end users to access the Internet. It also handles DHCP address assignment and Quality of Service.
Quality of Service on the Payment Gateway makes use of bandwidth shaping to ensure that the most critical packets get priority on the network. The technical team has defined critical in the following order:
1. Protocols that are responsible for network connectivity. (ICMP, DNS, ACK Packets)
2. Protocols that are related to network management. (SNMP, SSH, TELNET)
3. Protocols that have high processor latency and interact with users. (SSL)
4. Protocols that are commonly used by end users to interact. (HTTP)
5. Other common protocols that are often interactive. (Traffic on Ports 0-1024)
6. Protocols known to abuse network resources. (Bittorrent, Gnutella, Limewire)
7. Any traffic not classified above.
To manage this traffic, the payment gateway makes use of iptables and tc to create packet queues. These queues are emptied in order and allow end users making use of interactive protocols to receive better latency on the network.
Traffic is transported from the network core to the mesh network via an intermediate transport system. Currently this system consists primarily of Ubiquiti AirMAX equipment, but it is being replaced with fiber-optic cable where possible.
The company uses a "bucket" queuing mechanism to enable burst speeds on these links. Settings for the various equipment are as follows:
- 5 GHz Mesh Backhaul - Uplink 300 Mbps, Downlink 300 Mbps, Burst (Reflective) 0
- 900 MHz Mesh Backhaul - Uplink 6,800, Downlink 33,200, Burst (Reflective) 500,000
- 5 GHz Rural - Uplink 512, Downlink 2048, Burst Up 0, Burst Down 50,000
- 900 MHz Rural - Uplink 200, Downlink 976, Burst Up 0, Burst Down 50,000
Rural rules are still rather tight primarily because rural members are on the intermediate transport system and contend with mesh backhauls for bandwidth. This means that 5 municipal members behind a mesh gateway would contend on an equal basis with a single rural member directly connected to the intermediate transport system. As more of the mesh comes up on fiber-optic cable, these rules will be relaxed.
Bandwidth in this system is allocated based on a burst system. The uplink and downlink queues fill up at the uplink and downlink rates. When an end user gets online, his traffic draws from the queue at the maximum available speed (up to 300Mbps at 5GHz, 4Mbps at 900MHz). Once the bucket is depleted, the end user's uplink and downlink speed are capped at their respective uplink and downlink rates. If the end user is trying to send or receive more than this data, the data is queued into memory and dropped once the queue is full.
The mesh layer also makes use of a QoS rule set. This rule set is built to identify users who are using more than normal and reduce their speed for a pre defined period. At this time the company is also experimenting with protocol priority on this network using the same rules as identified on the payment gateway, however, this system is not currently deployed.
This system simply accounts for user traffic over a specific time segment. Freenet's default segment is 5 min or 300 seconds. This means that during the first 5 min, traffic is unlimited. Users making use of most Internet protocols will not hit the cap, this means that traffic speed is essentially unregulated at this layer.
This can result in up and down speeds of 22 Mbps+ on other mesh nodes it can be up to 5Mbps. Your mileage may vary, however, depending on how many members are on your cluster, the current wireless conditions the materials between you and the node and, most importantly, your distance from the nearest node.
Also, if your WiFi card or antenna sucks (See T-Mobile G-1) you may get less from your Freenet connection that you might like. We suggest a high power modem or bridge from a company like the Ubiquiti Power-N. You can buy these modems from us, lease them from us, or buy them from Wisp-Router at http://www.wisp-router.com. Since they are in Pittsburg, KS UPS ground shipping usually results in overnight delivery.
Rate Limiting - Wireless Backhaul Mesh
This policy applies to nodes attached to backhauls that make use of wireless technologies.
If the end user moves more than 614 Mbits of down traffic or 307 Mbits of up traffic through the network during the 600 second timeslot, the end user will get marked egregious by the rate limiting device. This will then throttle the connection to 1,536 Kbps down, 384 Kbps up. This throttle will remain in place for 60 min, at which point the cap will be removed.
To avoid hitting the egregious cap, make sure that abusive applications (napster, bittorent, gnutella) have their bandwidth settings set to the lowest reasonable speed. We suggest 10Kbps down, 5Kbps up. This will ensure that your bandwidth is available for interactive applications like video, vo-ip, etc.
Mesh Protocol Blocking
It should be noted that Freenet does block "for your own good" protocols such as DHCP, SMB and others traffic that end users often misunderstand or misuse. This is to protect end users and from themselves and other members. Three comments on this matter:
1. It is not a good idea to share your "shared folder" with the rest of earth.
2. The cable plugs into the WAN port of your router.
3. All of your bases are belong to us!
The goal of the network team is to provide the best possible service to all of our members. The team therefore reserves the right to manage bandwidth in any way necessary to ensure the viability of the network. This includes throttling specific users, protocols, ports, traffic, etc. In other words if a member or a member's computer are damaging the network or soaking up communal resources, we reserve the right to throttle you, NAT you, turn your service off or take any other action necessary to keep network resources available for everyone.