wiki:WARPLab/Debugging

WARPLab 7: Debugging Errors

This page shows various common error messages when using WARPLab 7 and offers suggestions on resolving the problem.


ERROR: Boot problems

During the boot of a WARP node, the hex display and the LEDs provide information about the state of the boot process. Here is a list of the various hex display / LED states to help debug boot issues:

Hex Display

LED State

Boot State

OFF

OFF

If power is on, then board is waiting for bitstream to be downloaded or bitstream is currently being downloaded to the board

8.8.

OFF

If CM-MMCX or CM-PLL modules are present, then board is waiting for a valid reference clock. You can confirm this by attaching the UART and looking for the following message:

***********************************************
     ***** WARP v3 Clock Config Core *****
      Program Assembly Date: 24 Jan 2015

No config data in EEPROM - Using Defaults
Detected CM-PLL Module
Loading configuration B
Waiting for valid PLL ref clk......

To resolve this, either provide a valid clock or change the clock module settings.

00

OFF

Boot of the node is proceeding. Wait for a different condition.

E0

Red LEDs Flashing

Right shift operator not working correctly. This is an issue with the 14.7 Xilinx tools when using the -Os optimization. Use the 14.4 Xilinx tools or use a different optimization level.

E1

Red LEDs Flashing

Node initialization error. This could be due to a mismatch between the hardware and software used in the bitstream or a failure of one of the hardware components. Please see the UART output for more debug information.

E2

Red LEDs Flashing

Sub-system initialization error. This is generally due to an issue in the RF interfaces (for example an RF interface failed to lock or there is a mismatch between the number of RF interfaces on the board and the number specified by the bitstream) but could be caused by any of the Trigger Manager, Baseband, Interface, or User sub-systems. Please see the UART output for more debug information.

E3

Red LEDs Flashing

Transport initialization error. Please see the UART output for more debug information.

E4

Red LEDs Flashing

Linker command file issue. The linker command file does not place the global transport data structures into BRAM that is accessible by the DMA. Please use the linker command file provided by the reference design and only make hand modifications to the linker command file to meet your project needs.

E5

Red LEDs Flashing

Interrupt initialization error. Please see the UART output for more debug information.

E6

Red LEDs Flashing

Interrupt enable error. Please see the UART output for more debug information.

01 to 99

Green LED Flashing

Waiting for a valid Ethernet connection. UART output should be similar to:

WARPLab vX.Y.Z (compiled Sep  1 2015 15:29:29)
Configured for 2 RF Interfaces - Using WARP v3 on-board RF interfaces
No clock module detected - selecting on-board clocks

NODE: W3-a-AAAAA using Node ID: 1
DRAM SODIMM detected ...
  Contents not cleared
Configuring baseband ...
  Using DDR for buffers
  Rx samples:       32768 ( 134217728 max)
  Tx samples:       32768 ( 117440512 max)
Configuring transport ...
  ETH A MAC Address: 40:D8:55:NN:NN:NN
  ETH A IP  Address: 10.0.0.2
  Configuring ETH A with 9024 byte buffers (2 receive, 2 send)
  ETH A speed 1000 Mbps (default)
  Listening on UDP ports 9000 (unicast) and 10000 (broadcast)

Waiting for Ethernet link ...

where X.Y.Z shows your WARPLab version; AAAAA is your hardware serial number; and NN:NN:NN are the last six digits of your MAC address.

A successful boot will have the Hex display indicate the last octet of the IP address of the node and the green LEDs will be used to indicate that valid WARPLab commands / packets are being received by the node. The UART output will be:

WARPLab vX.Y.Z (compiled Sep  1 2015 15:29:29)
Configured for 2 RF Interfaces - Using WARP v3 on-board RF interfaces
No clock module detected - selecting on-board clocks

NODE: W3-a-AAAAA using Node ID: 1
DRAM SODIMM detected ...
  Contents not cleared
Configuring baseband ...
  Using DDR for buffers
  Rx samples:       32768 ( 134217728 max)
  Tx samples:       32768 ( 117440512 max)
Configuring transport ...
  ETH A MAC Address: 40:D8:55:NN:NN:NN
  ETH A IP  Address: 10.0.0.2
  Configuring ETH A with 9024 byte buffers (2 receive, 2 send)
  ETH A speed 1000 Mbps (default)
  Listening on UDP ports 9000 (unicast) and 10000 (broadcast)

Waiting for Ethernet link ...

Initialization Successful - Waiting for Commands from MATLAB

where X.Y.Z shows your WARPLab version; AAAAA is your hardware serial number; and NN:NN:NN are the last six digits of your MAC address.


ERROR: Maximum number of transmissions

Error using wl_transport_eth_udp_pnet/send (line 228)
maximum number of retransmissions met without reply from node

Error in wl_node/sendCmd (line 677)
            resp = obj.transport.send(cmd.serialize());

Error in wl_node/procCmd (line 644)
                    node.sendCmd(myCmd);

Error in wl_node/wl_nodeCmd (line 310)
                out(n) = currNode.procCmd(n, currNode, varargin{:});

Error in wl_node/applyConfiguration (line 160)
            obj.wl_nodeCmd('initialize');

Error in wl_initNodes (line 106)
        currNode.applyConfiguration(nodeIDs(n));

This error indicates that a WARP node didn't respond to repeated communication attempts from the wl_initNodes script.

Things to check:

  • Did all WARP nodes boot successfully? On each node a green LED will blink while waiting for the Ethernet link, then all LEDs will turn off and the hex display will show the node's IP address suffix.
  • Did you specify an invalid number of nodes or invalid node IDs in the arguments to wl_initNodes?
  • Does each WARP node have a unique ID? In the reference design IDs are assigned by setting the user DIP switch on the WARP node.
  • Are all WARP nodes connected to a 1Gb Ethernet switch?

Debugging steps: To isolate which node is failing to communicate, you can use the optional wl_initNodes syntax to set node IDs explicitly:

wl_initNodes(1, 0); %initialize one node with ID 0
wl_initNodes(1, 1); %initialize one node with ID 1
wl_initNodes(1, 2); %initialize one node with ID 2
%continue until a node fails to initialize

ERROR: Broadcast Triggers

Error using wl_initNodes (line 154)
Broadcast triggers are not working. Please verify your ARP table has an entry for the broadcast
address on your WARPLab subnet

This error is printed by wl_initNodes when it is able to communicate with individual nodes, but nodes failed to receive a trigger packet.

Things to check:

  • Are all WARP nodes and the host PC on a common Ethernet switch?
  • Is the host PC NIC IP address in the same subnet as the WARP nodes (10.0.0.x by default)?

Debug steps: some PCs require a manual ARP table entry to allow broadcast IP packets. On Windows, open a command prompt as an admin user and run arp -s 10.0.0.255 ff-ff-ff-ff-ff-ff. Run arp -a to confirm the new entry was accepted. The output should look like:

Interface: 10.0.0.250 --- 0xd
  Internet Address    Physical Address     Type
  10.0.0.255          ff-ff-ff-ff-ff-ff    Static

ERROR: DMA transfers

If you make changes to the WARPLab software project, you may see this error printed via the UART:

*** ERROR in RX DMA transfer: 0x400
*** Resetting DMA

This error is caused by the Ethernet DMA being unable to access the network packet buffers in the C code. This is due to the linker script assigning code data sections to the DLMB (data local memory bus). The LMB memories are not accessible to the AXI interconnect where the DMA is attached.

To resolve this error update your linker script to assign data sections to a memory area accessible by the AXI interconnect. The axi_bram (listed as axi_bram_0_S_AXI_BASEADDR in the current design) is a good choice.


ERROR: MEX Transport out of memory

When using large buffers, it is possible for Matlab to run out of memory when trying to process a Read IQ request:

Error using wl_mex_udp_transport
Out of memory. Type HELP MEMORY for your options.

Error in wl_transport_eth_udp_mex/read_buffers (line 545)
                    [num_rcvd_samples, cmds_used, rx_samples]  =
                    wl_mex_udp_transport('read_iq', obj.sock, data8,
                    length(data8), obj.address, obj.port, num_samples,
                    buffer_ids, start_sample, obj.maxS
Error in wl_baseband_buffers>readIQ (line 1393)
        rxSamples_IQ = node.transport.read_buffers('IQ', numSamps, buffSel,
        offset, myCmd, 0);

Error in wl_baseband_buffers/procCmd (line 576)
                    out = readIQ(obj, node, buffSel, cmdStr, varargin{:});

Each IQ sample is stored in Matlab as a complex double. This means that each sample requires 16 bytes of memory to be stored in Matlab. Now, if a Read IQ request of a single buffer is transferring 225 samples, then this implies that an array of those samples requires 229 bytes (i.e. 512 MB). Similarly, if a Read IQ request of 2 buffers is transferring 225 samples per buffer, then this implies that an array of those samples requires 230 bytes (i.e. 1 GB). By using the 'memory' command in Matlab, you can see the maximum possible array size for your installation:

Maximum possible array:              XXXXX MB (Y.YYYe+ZZZ bytes)

This will allow you to understand the maximum number of samples you can request in a single Read IQ command.


ERROR: Transport only supports X samples

When using large buffers, even though WARPLab allows for up to 227 samples (ie 128M samples), requesting greater than 225 samples when using the MEX transport or greater than 220 samples when using the Java transport will result in the following error messages:

??? Error using ==> wl_baseband_buffers>readIQ at 1989
read_iq: Requested 67108864 samples.  Due to Matlab memory limitations, the mex transport only supports 33554432 samples.
    If your computer has enough physical memory, you can adjust this limit using node.baseband.MEX_TRANSPORT_MAX_IQ.
??? Error using ==> wl_baseband_buffers>readIQ at 2003
read_iq: Requested 67108864 samples.  Due to performance reasons, the java transport only supports 1048576 samples.
    Please use the MEX transport for larger requests.

The error in the Java transport is issued because, as seen in the benchmarks, the Java transport is much slower than the MEX transport so larger requests can take a "long time". Therefore, it is recommended to move to the MEX transport for larger requests.

The error in the MEX transport is issued because many host computers do not have enough memory to handle longer transfers. As discussed above, it requires 229 bytes (ie 512 MB) to hold 225 samples. As noted in the error message, if your host computer has more memory (can be seen using the Matlab 'memory' command), then you can adjust the maximum using the MEX_TRANSPORT_MAX_IQ variable.


WARNING: Read IQ / Read RSSI request timed out

When performing a Read IQ or Read RSSI command, you might see the following warning:

WARNING:  Read IQ / Read RSSI request timed out.  Retrying remaining samples. 
          If this message occurs frequently, please adjust the Read IQ 
          maximum request size (in bytes) for the transport using the 
          M code function:  
              wl_mex_udp_transport('read_iq_set_max_request_size', size)  
          Defaults to 80 percent of the receive buffer allocated by the OS. 

          To suppress all IQ warnings for the transport use the M code function: 
              wl_mex_udp_transport('suppress_iq_warnings')

When Matlab initializes the socket during initNodes, the code requests an Ethernet receive buffer from the OS for 222 bytes, ie 4 MB (see REQUESTED_BUF_SIZE in the open() function of the transport, for example wl_transport_eth_udp_mex.m or wl_transport_eth_udp_java.m). This value was chosen because it gives good performance and most OSes are able to allocate an Ethernet receive buffer that large. Then, by default, WARPLab only allows requests to the node that are 80% of that receive buffer (~3.35 MB) so that the node cannot overwhelm the host with data.

However, if you see the warning above, this means that packets are being dropped during the Read IQ / Read RSSI command. This could come from a number of sources:

  1. Intermittent / flaky Ethernet hardware such as a bad Ethernet cable, Ethernet switch, or NIC. These can be tested by swapping components and seeing if that resolves the warning.
  2. Even with the built in buffer overhead, the host still cannot keep up with the flow of data packets from the node and begins dropping packets. This could be due to the hardware specifications of the host. In most systems, if WARPLab requests a 4 MB socket buffer from the OS, and the OS cannot provide that much buffer space, then the OS will allocate a smaller buffer and tell WARPLab how much buffer space it can actually use. Then, the maximum request to the node that transport issues will be adjusted accordingly and there are no issues. However, if the OS tells WARPLab that a 4 MB socket buffer is available but in reality there is much less than 4 MB available (i.e. the OS begins to drop Ethernet packets way before 4 MB of data is transferred), then the transport can run into problems. When a situation like this occurs, you can use the command wl_mex_udp_transport('read_iq_set_max_request_size', size) to set the size of the request, in bytes. This should be less that (0.8 * 222) bytes, or (0.8 * <value OS reduced recv buffer printed during initNodes>), since you want to reduce the maximum size of the request.

As long as you just receive the warning and there are no Read IQ / Read RSSI errors, then all this does is slow down the commands (ie you will not be getting the best performance). If you are ok with the reduced performance and wish to remove the warning, you can use the command wl_mex_udp_transport('suppress_iq_warnings').

Also, if using Ubuntu 12.04 and you see a message during initNodes OS reduced recv buffer size to 131071 then you can adjust the receive buffer size by using the following command: sysctl -w net.core.rmem_max=8388608

Last modified 9 years ago Last modified on Nov 18, 2015, 3:59:33 PM