Installation Problems
Check the following:
- The ISO should have been copied to a USB drive with MBR partitions and DD mode.
- The server BIOS should have its factory default boot mode (for example, DUAL).
- If there is a RAID controller, a logical drive must be configured and marked as bootable.
- Check if server used in the installation meets BQN hardware requirements.
- Check that the installation was done in the hard disk and not overwrote the USB drive.
No Access to the Management IP Address
The BQN uses a dedicated network interface for management. The management interface supports both the SSH and WEB (HTTPs) services. In case of problems accessing the configured management IP check the following:
- Ensure that the management network interface port is connected to the appropriate network.
- Verify that the link state of the management network interface is up. If the management interface is connected to a network switch, verify that the port in the switch is up and its attributes match the properties shown by the show interface command.
- If accessing the management IP address from a different network, make sure that static routing is configured to the access network, as explained in the Network Interface section in the User Guide.
- If there are firewalls in the management network, allow access to TCP port 22 for the SSH service and TCP port 443 for the WEB service.
- Verify using the system console that the management IP address and network prefix are correct. Connect a monitor and a keyboard to the server and login as root:
- If you suspect the OAM IP settings are incorrect or unknown, connect a monitor and a keyboard to the server and login as root to change it using the bta wizard in interactive mode. For example, to change the management interface to en0o1 with IP address 10.10.10.12/24 (press enter to accept suggested response):
If the interface is not available at the time of the change (for example, participated in a wire), a message will request a reboot. After the reboot, the BQN should have the new IP and management network interface.
- The BQN management interface may be protected by its own firewall. The problem could be that your source IP address is not included in that firewall white list. This could happen even for addresses from the same subnet of the BQN management IP if the subnet is not part of the firewall rules. You can disable the firewall temporarily until the connection to the management port is restored. Connect a monitor and a keyboard to the server and login as root:
Once the management IP is reachable, you can define the new white list of allowed source IP ranges.
- If the command <span class="character-highlight">wizard bta interactive</span> fails, connect with a console and a keyboard and try changing the configuration directly. In the following example, the management interface is in en0o2 and we will move it to interface en0o1 and from IP address 192.168.0.121 to IP address 10.0.0.121, with default gateway 10.0.0.1:
Web not Accessible
- Check that the management IP address is accessible using SSH,
- Check that you are using HTTPS in the access (HTTP is not supported). Example of URL: https://192.168.0.121
- Check that you are using bqnadm user (root cannot be used in GUI access).
- When installing from scratch, make sure the command wizard bta was run (otherwise, the GUI web service will not be active a no bqnadm user created).
- Verify that the SSH port of the BQN server has not been modified. To access the BQN using a port other than 22, you can define port forwarding rules in router on the access path, but the BQN SSH port cannot be changed. Logging to the server as root, you can verify that the SSH port is 22 as follows:
If needed, comment the line that specifies a port other than 22.
- Check that your browser is supported (Edge, Firefox, Chrome). MS Explorer, for example, is not supported.
Network Interface Down
If the Network Interfaces icon in the Dashboard is not in green.
Go to Configuration->Interfaces->Data Wires
In Red (Critical)
- If there is no wire configured, create one.
- If there are wires configured but their interfaces are not in UP state, this most likely indicates that the interfaces are not Intel compatible. Go to Configuration->Interfaces->Data Wires and click on the down interface to see the model (PCI vendor ID) and confirm it is not Intel. Remove the wire and create a new one with both interfaces in pcap mode. This should place the interfaces in UP state, but with much lower throughput capacity (less than 1Gbps).
- If there are wires configured, with interfaces in UP state but with the LINK down, there is a problem in the connection with the other equipment. Connect both interface ports to one another in a loop using a suitable cable/fiber. If both interfaces are in up, then the problem is on the other equipment.
If the link is still down and optic ports are used, verify that the transceivers:
- are Intel-compatible
- are supported (see Supported Network Cards for a list).
- of the type required by the installation (e.g.,SFP+-LR in an installation with monomode fiber and SFP+-LR on the other side).
In Yellow (Notice)
- If the wires that appear as down are supposed tobe with traffic, follow the steps of the previous section (Critical)
- If the wires that appear as down are not in use and you want to remove the notice signal, you can delete the unused wires. Consider that changes in the wire configuration will stop the traffic for some seconds.
Inverted Traffic
If the Inverted Traffic icon in the Dashboard is in orange (Warning), it is indication that traffic throughput in the uplink direction is bigger than in the downlink direction. This can be normal in small deployments (less than a hundred subscribers, like in a BQN in a lab) but in a network deployment most likely indicates that some of the wires have been connected incorrectly, with the access port connected to the Internet side and vice versa.
To verify that this is the case, go to Statistics->Throughput->Interfaces and select the wire interface configured in the access side. If it shows more received throughput that sent throughput, it is indication that the wire is inverted. This can be confirm selecting the throughput of the Internet-side interface to see that its sent traffic is bigger than its received traffic.
To fix the issue, go to Configuration->Interfaces->Data Wires and, in the inverted wire, press the Swap interfaces icon.
Low Traffic
If the Low Traffic icon in the Dashboard is not in green.
In Yellow (Notice)
Hover the mouse over the icon to confirm that Traffic-low notice is shown. It indicates that there are very little traffic going through the BQN server. This is normal if the system is still waiting for traffic being routed through it. However, in a system in production can be an indication that some failure elsewhere in the network is preventing the traffic to reach the BQN server.
In Orange (Warning)
Hover the mouse over the icon. Either Traffic-uplink or Traffic-downlink should be in warning. This is because there is no traffic going through the BQN in that direction and therefore the traffic is asymmetric. You should go to Configuration->TCPO/ACM Settings and set the Overall TCP Optimization to Off while the issue is being delt with.
Fix the traffic routing so the BQN sees both directions (uplink and downlink). When done and after the icon returns to green, go to Configuration->TCPO/ACM Settings to enable the overall TCP optimization.
License Manager
If the icon License Manager in the dashboard is in Yellow and the text says “license-mgr-connection:notice”, this is because the BQN server cannot reach the license manager. The license manager is responsible of validating BQN SW licenses and also helps Bequant to provide a more proactive support by reporting server problems.
Make sure the BQN server can initiate outgoing connections to the License Manager IP (contact Bequant support for details).
To verify that an outgoing connection is possible, log as root and a telnet to the provided IP and port should work:
License not OK
If the License icon in the Dashboard is Red, and the text says “license-available:critical”or "license-expiration: critical", there is no valid license. This could be due to several reasons:
- There is no license defined in the node
- The license is not valid
- The license is no longer valid (its final date has expired).
Contact your distributor for a valid license.
You can check the license state in Administration->License.
When there is no valid license, the BQN will forward all traffic transparently: the service will not be affected, but none of the BQN advanced processing will be applied to the traffic.
License Limit Exceeded
If the License icon in the Dashboard is Orange, , andthe text says “license-usage: warning”, the maximum capacity of the license is being exceeded (the traffic throughout in the BQN server is above the license limit).
Contact your distributor for a license upgrade.
You can check the license capacity in Administration->License.
In Statistics->Throughput->Overview, a red line will show the license limit along with recent throughput levels.
Exceeding license limit should be temporal, while the license is upgraded to the right capacity.
When the license limit is exceeded, the QoE will not drop any packets; it will simply bridge them.
The effect on the traffic is different at flow and at subscriber session level.
Effects at flow level:
- While the license limit is being exceeded, new flows will have no license.
- Existing flows prior to the license being exceeded are unaffected: if they were being optimized remain so and likewise if without a license.
- Once the optimized traffic falls below the limit, new flows are optimized again.
- Existing flows prior to the license going below the limit are unaffected: if they had no license remain so and likewise if they were being optimized.
- A flow without a license has no TCPO, no shaping and it does not generate metrics (retransmissions, latencies, DPI, etc.).
Effects at subscriber session level:
- While the license limit is being exceeded, new subscriber sessions will have no license.
- Existing subscriber sessions prior to the license being exceeded remain initially in their current state: if they were being optimized remain so and likewise if without a license.
- If the license remains above the limit, subscriber sessions being optimized may transition to no license if they accumulate a big number of non-optimized flows.
- Once the optimized traffic falls below the limit, new subscriber sessions are optimized again.
- Exiting subscriber sessions prior to the license going below the limit remain initially in their current state: if they had no license remain so and likewise if they were being optimized.
- If the license remains below the limit, subscriber sessions without a license may transition to being optimized if they accumulate a small number of optimized flows.
- A subscriber session without a license has no ACM, no rate limiting and it continues generating traffic volume totals. Quota volume reporting is suspended. Blocking at quota exhaustion will not take place. The other metrics, like retransmissions, latencies, etc. will be reduced to those from the flows being optimized.
- A subscriber session being optimized will be affected by the fact that some of his flows may not have a license and therefore the subscriber may have a reduced set of measurements in comparison with normal operation. This affects the ACM and also the metrics generated for that subscriber.
The effect of exceeding the licensed limit is that more and more traffic will no longer get QoE functionality, and, conversely, the amount of traffic getting QoE functionality gets lower. Once the amount of traffic getting QoE functionality is below the license limit, new flows and subscriber sessions will regain QoE functionality. This continues until the license limit is exceeded, after which the effect repeats again. With this oscillating behavior, the QoE keeps providing functionality to traffic up to the license limit. The BQN is continuously checking the level of throughput vs the license limits (every minute or less), so the adaptation is pretty quick.
High CPU Load
If the CPU icon in the Dashboard is not in green, some CPUs are running at abnormally high levels. This is normally due to unbalanced traffic (concentrated in a few subscriber IPs) or to too much traffic being proceeded by the BQN server.
The BQN has internal mechanisms to mitigate this situation, trying to prevent traffic losses by reducing the amount of optimized traffic.
The throughput level can be verified in Statistics->Throughput->Overview and the CPU levels in Statistics->System->CPU.
There are two alarm types, depending on the CPU load levels:
- Orange if some CPU cores are at high load (above 80% usage).
- Red if some CPU cores are at very high load (above 90% usage).
Follow these steps:
- If you are using NAT between the BQN and the end subscribers, increment the number of IPs used by the NAT, so the BQN can distribute the traffic among more addresses. You can see how traffic is distributed among IPs in Statistics->Subscribers->Top By Time.
- Enable the bypass path, or, if not possible, reduce the amount of traffic being routed through the BQN
- Disable TCP optimization in Configuration->TCPO/ACM Settings.
- A hardware upgrade may be needed. Contact Bequant support.
If the BQN server is used only for TCPO and/or per-flow speed limits, the CPU load distribution can be improved enabling per-flow steeting. Per-flow steering distributes the traffic load across CPU cores per individual flow, instead of the default per-subscriber traffic distribution. This improves CPU load balance, but because per-subscriber control is done per core, does not allow subscriber-level control, such as ACM, policy rates or per subscriber flow limits.
High Memory Load
If the Memory icon in the Dashboard is not in green, some processes are running out of memory. This is normally due to unbalanced traffic (traffic concentrated in a few subscriber IPs) or to too much traffic being processed by the BQN server.
The BQN has internal mechanisms to mitigate this situation, trying to prevent traffic losses by reducing the amount of optimized traffic.
The throughput level can be verified in Statistics->Throughput->Overview and the Memory levels in Statistics->System->Memory.
There are two alarm types, depending on the memory load levels:
- Orange if some processes reach high usage (above 90% usage).
- Red if some processes reach very high usage (above 95% usage).
Follow these steps:
- If you are using NAT between the BQN and the end subscribers, increment the number of IPs used by the NAT, so the BQN can distribute the traffic among more addresses. You can see how traffic is distributed among IPs in Statistics->Subscribers->Top By Time.
- Enable the bypass path, or, if not possible, reduce the amount of traffic being routed through the BQN.
- Disable TCP optimization in Configuration->TCPO/ACM Settings.
- A hardware upgrade may be needed. Contact Bequant support.
No RADIUS messages are received
If no RADIUS information is shown in the BQN GUI, to check if RADIUS messages are incoming, log in to BQN shell and run the following command:
In this example, the management interface is eno1. Check the one of your BQN server in Configuration->Interfaces->Management (the 0 added by BQN configuration must be removed, for example, en0s1f0 in BQN GUI is ens1f0 in Linux).
If the BQN firewall is configured (Configuration->Interfaces->Management Firewall), all RADIUS client IPs must be added (in this example, 10.10.10.10 and 10.10.10.11).
And now the RADIUS messages should be received:
If RADIUS messages are still no received, the rest of the traffic jumps need verification. In our example, RADIUS clients are in subnet 10.10.10.0/24 and BQN in subnet 192.168.0.0/24. It has to be verified if there are valid routes between the two subnets and that no intermediate firewall is blocking UDP port 1813 (RADIUS Accounting). You can start the verification at the RADIUS client; for example, if it is a Mikrotik, use Tools->Tourch with port radius-acct.
If the RADIUS messages are received, but the information is not reflected as expected, enable the event log entering the BQN server shell via ssh:
Traffic captures are also possible from the GUI, in Status->Interfaces->Link State selecting the management interface and using “udp 1812 and udp 1813” as filter:
Now, you can see the event log of the received RADIUS requests:
No REST messages are received
If the REST information is not shown in the GUI, the reception of REST messages can be checked in the BQN logging to the BQN shell and executing the following command:
In this example, eno1 is the management interface.
Check the one in your server going to Configuration->Interfaces->Management (remove the 0 added by BQN configuration, e.g., en0s1f0 in the BQN GUI is ens1f0 in UNIX).
If the BQN firewall is configured (Configuration->Interfaces->Management Firewall), the IPs of all the REST clients must be added (in our example, 10.10.10.10 and 10.10.10.11).
And now the messages should be received:
If the REST messages are not yet received, check the rest of traffic steps. In this example, the REST clients are in 10.10.10.0/24 and the BQN in 192.168.0.0/24. Check that there are valid routes between both subnets and that no intermediate firewall is blocking the TCP port 3443.
If the REST messages are received, but the information is not reflected as expected, enable the event log and traces entering the BQN server shell via ssh:
Now, you can see the event log of the received REST requests. In the following example, the BQN receives some POSTs assigning subscribers to rate policies:
And the traces of the last requests and responses can be found in the directory /opt/bqn/var/trace:
REST messages are rejected
If a REST API requests gets an error related to certificateverification, for example:
The reason is that the BQN certificate is self-signed and the client must disable the certificate validation. For example, using curl:
And using python:
BQN uses TLS 1.2. Some new versions of python reject TLS 1.2 by default. To make it accept it, use the following code:
A full example of this can be found here.
No Billing messages are received
If the Billing system is configured and the Billing icon in the Dashboard is red, the access to the Billing system is failing.
Click on the billing icon to go to the billing status page:
In this example, there was a successful synchronization at 12:13:52 retrieving 1000 subscribers, but there was one failed attempt afterwards. You can force a synchronization attempt pressing the Sync now button.
If the failure remains, follow these steps:
- Check that the billing configuration is correct (direction and credentials).
- Check that the billing IP address is reachable from the BQN server:
- Enable the API logs entering the BQN server shell via ssh:
Now, you can see the event log between the BQN and the billing system. In the following example, the BQN connects to an Azotel billing are retrieves three policies associated to ten subscribers:
If there is an error, the log should give some indication of what is happening.
- It is also possible to extend the logging to detailed traces of the requests and responses exchanged between the BQN and the billing system:
And the BQN will generate traces of the last 5 request and responses between BQN and the billing system. The traces can be found in directory /opt/bqn/var/trace:
- As a last resort, you can send a request directly from the BQN to the billing system using UNIX curl command, following the billing system API conventions. For example, to send a query to an Azotel billing:
No NTP Servers Synchronized
If the NTP icon is yellow, the NTP is not configured. Click on the icon to go to the configuration page.
If the NTP servers are configured but the BQN cannot synchronize with any of them, the NTP icon in the dash will be in orange:
An abrupt change in the system time because of lack of NTP synchronization, may lead to brief service losses while the system adjusts. To avoid this from happening, have always at least one NTP server in sync.
You can check the list of configured NTP servers in Administration->System Date->NTP Servers.
At least one NTP server should be synchronized. In the example above, the NTP server 145.238.203.14 has been chosen for clock synchronization (indicated by the * next to the server IP address) and has been contacted 36 seconds ago (column WHEN).
If no NTP server is available, the BQN will show the warning message NTP not synchronized, as in the window below:
To solve the issue, if you have a local NTP server, add it to the list clicking the <i class="fa-solid fa-ellipsis-vertical"></i> menu icon and selecting Add Server…
If you have no local NTP servers, make sure the UDP port 123 is open from the BQN management IP to the Internet, including in the BQN firewall, if activated.
Disk Issues
To check the status of the system disk, click on the DISK icon of the dashboard:
The disk icon will be in orange (warning state) when less than 15% of the disk storage is free.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.