WARPLab 7: Debugging Errors
This page shows various common error messages when using WARPLab 7 and offers suggestions on resolving the problem.
ERROR: Boot problems
During the boot of a WARP node, the hex display and the LEDs provide information about the state of the boot process. Here is a list of the various hex display / LED states to help debug boot issues:
Hex Display | LED State | Boot State |
OFF | OFF | If power is on, then board is waiting for bitstream to be downloaded or bitstream is currently being downloaded to the board |
8.8. | OFF | If CM-MMCX or CM-PLL modules are present, then board is waiting for a valid reference clock. You can confirm this by attaching the UART and looking for the following message: *********************************************** ***** WARP v3 Clock Config Core ***** Program Assembly Date: 24 Jan 2015 No config data in EEPROM - Using Defaults Detected CM-PLL Module Loading configuration B Waiting for valid PLL ref clk...... To resolve this, either provide a valid clock or change the clock module settings. |
00 | OFF | Boot of the node is proceeding. Wait for a different condition. |
E0 | Red LEDs Flashing | Right shift operator not working correctly. This is an issue with the 14.7 Xilinx tools when using the -Os optimization. Use the 14.4 Xilinx tools or use a different optimization level. |
E1 | Red LEDs Flashing | Node initialization error. This could be due to a mismatch between the hardware and software used in the bitstream or a failure of one of the hardware components. Please see the UART output for more debug information. |
E2 | Red LEDs Flashing | Sub-system initialization error. This is generally due to an issue in the RF interfaces (for example an RF interface failed to lock or there is a mismatch between the number of RF interfaces on the board and the number specified by the bitstream) but could be caused by any of the Trigger Manager, Baseband, Interface, or User sub-systems. Please see the UART output for more debug information. |
E3 | Red LEDs Flashing | Transport initialization error. Please see the UART output for more debug information. |
E4 | Red LEDs Flashing | Linker command file issue. The linker command file does not place the global transport data structures into BRAM that is accessible by the DMA. Please use the linker command file provided by the reference design and only make hand modifications to the linker command file to meet your project needs. |
E5 | Red LEDs Flashing | Interrupt initialization error. Please see the UART output for more debug information. |
E6 | Red LEDs Flashing | Interrupt enable error. Please see the UART output for more debug information. |
01 to 99 | Green LED Flashing | Waiting for a valid Ethernet connection. UART output should be similar to: WARPLab vX.Y.Z (compiled Sep 1 2015 15:29:29) Configured for 2 RF Interfaces - Using WARP v3 on-board RF interfaces No clock module detected - selecting on-board clocks NODE: W3-a-AAAAA using Node ID: 1 DRAM SODIMM detected ... Contents not cleared Configuring baseband ... Using DDR for buffers Rx samples: 32768 ( 134217728 max) Tx samples: 32768 ( 117440512 max) Configuring transport ... ETH A MAC Address: 40:D8:55:NN:NN:NN ETH A IP Address: 10.0.0.2 Configuring ETH A with 9024 byte buffers (2 receive, 2 send) ETH A speed 1000 Mbps (default) Listening on UDP ports 9000 (unicast) and 10000 (broadcast) Waiting for Ethernet link ... where X.Y.Z shows your WARPLab version; AAAAA is your hardware serial number; and NN:NN:NN are the last six digits of your MAC address. |
A successful boot will have the Hex display indicate the last octet of the IP address of the node and the green LEDs will be used to indicate that valid WARPLab commands / packets are being received by the node. The UART output will be:
WARPLab vX.Y.Z (compiled Sep 1 2015 15:29:29) Configured for 2 RF Interfaces - Using WARP v3 on-board RF interfaces No clock module detected - selecting on-board clocks NODE: W3-a-AAAAA using Node ID: 1 DRAM SODIMM detected ... Contents not cleared Configuring baseband ... Using DDR for buffers Rx samples: 32768 ( 134217728 max) Tx samples: 32768 ( 117440512 max) Configuring transport ... ETH A MAC Address: 40:D8:55:NN:NN:NN ETH A IP Address: 10.0.0.2 Configuring ETH A with 9024 byte buffers (2 receive, 2 send) ETH A speed 1000 Mbps (default) Listening on UDP ports 9000 (unicast) and 10000 (broadcast) Waiting for Ethernet link ... Initialization Successful - Waiting for Commands from MATLAB
where X.Y.Z shows your WARPLab version; AAAAA is your hardware serial number; and NN:NN:NN are the last six digits of your MAC address.
ERROR: Maximum number of transmissions
Error using wl_transport_eth_udp_pnet/send (line 228) maximum number of retransmissions met without reply from node Error in wl_node/sendCmd (line 677) resp = obj.transport.send(cmd.serialize()); Error in wl_node/procCmd (line 644) node.sendCmd(myCmd); Error in wl_node/wl_nodeCmd (line 310) out(n) = currNode.procCmd(n, currNode, varargin{:}); Error in wl_node/applyConfiguration (line 160) obj.wl_nodeCmd('initialize'); Error in wl_initNodes (line 106) currNode.applyConfiguration(nodeIDs(n));
This error indicates that a WARP node didn't respond to repeated communication attempts from the wl_initNodes script.
Things to check:
- Did all WARP nodes boot successfully? On each node a green LED will blink while waiting for the Ethernet link, then all LEDs will turn off and the hex display will show the node's IP address suffix.
- Did you specify an invalid number of nodes or invalid node IDs in the arguments to wl_initNodes?
- Does each WARP node have a unique ID? In the reference design IDs are assigned by setting the user DIP switch on the WARP node.
- Are all WARP nodes connected to a 1Gb Ethernet switch?
Debugging steps: To isolate which node is failing to communicate, you can use the optional wl_initNodes syntax to set node IDs explicitly:
wl_initNodes(1, 0); %initialize one node with ID 0 wl_initNodes(1, 1); %initialize one node with ID 1 wl_initNodes(1, 2); %initialize one node with ID 2 %continue until a node fails to initialize
ERROR: Broadcast Triggers
Error using wl_initNodes (line 154) Broadcast triggers are not working. Please verify your ARP table has an entry for the broadcast address on your WARPLab subnet
This error is printed by wl_initNodes when it is able to communicate with individual nodes, but nodes failed to receive a trigger packet.
Things to check:
- Are all WARP nodes and the host PC on a common Ethernet switch?
- Is the host PC NIC IP address in the same subnet as the WARP nodes (10.0.0.x by default)?
Debug steps: some PCs require a manual ARP table entry to allow broadcast IP packets. On Windows, open a command prompt as an admin user and run arp -s 10.0.0.255 ff-ff-ff-ff-ff-ff. Run arp -a to confirm the new entry was accepted. The output should look like:
Interface: 10.0.0.250 --- 0xd Internet Address Physical Address Type 10.0.0.255 ff-ff-ff-ff-ff-ff Static
ERROR: DMA transfers
If you make changes to the WARPLab software project, you may see this error printed via the UART:
*** ERROR in RX DMA transfer: 0x400 *** Resetting DMA
This error is caused by the Ethernet DMA being unable to access the network packet buffers in the C code. This is due to the linker script assigning code data sections to the DLMB (data local memory bus). The LMB memories are not accessible to the AXI interconnect where the DMA is attached.
To resolve this error update your linker script to assign data sections to a memory area accessible by the AXI interconnect. The axi_bram (listed as axi_bram_0_S_AXI_BASEADDR in the current design) is a good choice.
ERROR: MEX Transport out of memory
When using large buffers, it is possible for Matlab to run out of memory when trying to process a Read IQ request:
Error using wl_mex_udp_transport Out of memory. Type HELP MEMORY for your options. Error in wl_transport_eth_udp_mex/read_buffers (line 545) [num_rcvd_samples, cmds_used, rx_samples] = wl_mex_udp_transport('read_iq', obj.sock, data8, length(data8), obj.address, obj.port, num_samples, buffer_ids, start_sample, obj.maxS Error in wl_baseband_buffers>readIQ (line 1393) rxSamples_IQ = node.transport.read_buffers('IQ', numSamps, buffSel, offset, myCmd, 0); Error in wl_baseband_buffers/procCmd (line 576) out = readIQ(obj, node, buffSel, cmdStr, varargin{:});
Each IQ sample is stored in Matlab as a complex double. This means that each sample requires 16 bytes of memory to be stored in Matlab. Now, if a Read IQ request of a single buffer is transferring 225 samples, then this implies that an array of those samples requires 229 bytes (i.e. 512 MB). Similarly, if a Read IQ request of 2 buffers is transferring 225 samples per buffer, then this implies that an array of those samples requires 230 bytes (i.e. 1 GB). By using the 'memory' command in Matlab, you can see the maximum possible array size for your installation:
Maximum possible array: XXXXX MB (Y.YYYe+ZZZ bytes)
This will allow you to understand the maximum number of samples you can request in a single Read IQ command.
ERROR: Transport only supports X samples
When using large buffers, even though WARPLab allows for up to 227 samples (ie 128M samples), requesting greater than 225 samples when using the MEX transport or greater than 220 samples when using the Java transport will result in the following error messages:
??? Error using ==> wl_baseband_buffers>readIQ at 1989 read_iq: Requested 67108864 samples. Due to Matlab memory limitations, the mex transport only supports 33554432 samples. If your computer has enough physical memory, you can adjust this limit using node.baseband.MEX_TRANSPORT_MAX_IQ.
??? Error using ==> wl_baseband_buffers>readIQ at 2003 read_iq: Requested 67108864 samples. Due to performance reasons, the java transport only supports 1048576 samples. Please use the MEX transport for larger requests.
The error in the Java transport is issued because, as seen in the benchmarks, the Java transport is much slower than the MEX transport so larger requests can take a "long time". Therefore, it is recommended to move to the MEX transport for larger requests.
The error in the MEX transport is issued because many host computers do not have enough memory to handle longer transfers. As discussed above, it requires 229 bytes (ie 512 MB) to hold 225 samples. As noted in the error message, if your host computer has more memory (can be seen using the Matlab 'memory' command), then you can adjust the maximum using the MEX_TRANSPORT_MAX_IQ variable.
WARNING: Read IQ / Read RSSI request timed out
When performing a Read IQ or Read RSSI command, you might see the following warning:
WARNING: Read IQ / Read RSSI request timed out. Retrying remaining samples. If this message occurs frequently, please adjust the Read IQ maximum request size (in bytes) for the transport using the M code function: wl_mex_udp_transport('read_iq_set_max_request_size', size) Defaults to 80 percent of the receive buffer allocated by the OS. To suppress all IQ warnings for the transport use the M code function: wl_mex_udp_transport('suppress_iq_warnings')
When Matlab initializes the socket during initNodes, the code requests an Ethernet receive buffer from the OS for 222 bytes, ie 4 MB (see REQUESTED_BUF_SIZE in the open() function of the transport, for example wl_transport_eth_udp_mex.m or wl_transport_eth_udp_java.m). This value was chosen because it gives good performance and most OSes are able to allocate an Ethernet receive buffer that large. Then, by default, WARPLab only allows requests to the node that are 80% of that receive buffer (~3.35 MB) so that the node cannot overwhelm the host with data.
However, if you see the warning above, this means that packets are being dropped during the Read IQ / Read RSSI command. This could come from a number of sources:
- Intermittent / flaky Ethernet hardware such as a bad Ethernet cable, Ethernet switch, or NIC. These can be tested by swapping components and seeing if that resolves the warning.
- Even with the built in buffer overhead, the host still cannot keep up with the flow of data packets from the node and begins dropping packets. This could be due to the hardware specifications of the host. In most systems, if WARPLab requests a 4 MB socket buffer from the OS, and the OS cannot provide that much buffer space, then the OS will allocate a smaller buffer and tell WARPLab how much buffer space it can actually use. Then, the maximum request to the node that transport issues will be adjusted accordingly and there are no issues. However, if the OS tells WARPLab that a 4 MB socket buffer is available but in reality there is much less than 4 MB available (i.e. the OS begins to drop Ethernet packets way before 4 MB of data is transferred), then the transport can run into problems. When a situation like this occurs, you can use the command wl_mex_udp_transport('read_iq_set_max_request_size', size) to set the size of the request, in bytes. This should be less that (0.8 * 222) bytes, or (0.8 * <value OS reduced recv buffer printed during initNodes>), since you want to reduce the maximum size of the request.
As long as you just receive the warning and there are no Read IQ / Read RSSI errors, then all this does is slow down the commands (ie you will not be getting the best performance). If you are ok with the reduced performance and wish to remove the warning, you can use the command wl_mex_udp_transport('suppress_iq_warnings').
Also, if using Ubuntu 12.04 and you see a message during initNodes OS reduced recv buffer size to 131071 then you can adjust the receive buffer size by using the following command: sysctl -w net.core.rmem_max=8388608