WARP Project Forums - Wireless Open-Access Research Platform

You are not logged in.

#1 2015-Apr-27 01:56:25

Registered: 2015-Mar-05
Posts: 18

How to Enable two Ethernet Ports


We are using the latest WARP 7.5.1 for MIMO-OFDM system now. We are planning to enable two Ethernet ports: ETHA and ETHB for data transmission simultaneously in order to reduce the delay. Is there have any guidance on how to implement it?




#2 2015-Apr-27 14:47:22

From: Mango Communications
Registered: 2013-May-15
Posts: 612

Re: How to Enable two Ethernet Ports

First, can you elaborate on the "delay" you are trying to reduce?  While enabling two Ethernet ports will theoretically double your throughput (i.e. you will have 2 Gbps of bandwidth vs 1 Gbps of bandwidth), it will not reduce the latency (i.e. time to process a packet) associated with transfers to / from the host. 

That being said, there are a number of different levels for "enabling two Ethernet ports" with different associated complexity.  Depending on how far you want / need to go, this could involve re-writing the entire transport.  Before you get started, you need to understand how the transport works: 

First, the WARPLab transport is a polling based transport that will process packets when they are detected.  As of WARPLab 7.5.1 for WARP v3, both Ethernet interfaces have the same hardware (i.e. both have an AXI DMA in front of the Ethernet peripheral; see the FPGA architecture for more details).  The low-level hardware interaction and basic Ethernet packet processing occurs in the WARPxilnet library that is part of the BSP.  The WARPxilnet library allows for multiple receive buffers to reduce the probability of dropping a packet when receiving multiple small packets since there is non-zero overhead when processing a packet.  By default, two buffers are allocated for a given Ethernet device and we have not seen any issues with this level of buffering in the current WARPLab implementation.  While there are multiple receive buffers, there is only a single send buffer allocated for a given Ethernet device.  All data to be sent on a given Ethernet device must be copied into this buffer.  While this limitation in the send process has not been an issue, we are looking at adding more flexibility to the send process so that reference designs will not be required to copy data as many times (e.g. in the Read IQ process, we copy the IQ data from the DDR to the Ethernet buffer and the copy the data from the Ethernet buffer into the Ethernet peripheral); we do not currently have an ETA on this update.  Please note that buffers are allocated for a given Ethernet interface so there are two receive and one send buffer allocated per Ethernet device, by default.

The WARPxilinet library handles the basic Ethernet processing and socket management (e.g. this is where ARP and IMCP messages are processed).  The main thing to notice is that when a valid UDP message is received for a given socket, the WARPxilnet library calls the associated callback for the socket (see xilnet_udp.c).  This is how the transport in the reference design is informed that it should process a given Ethernet packet.  One thing to note is that means the entire processing of a command, including all responses, happen during that callback context.  This means that the receive packet is valid the entire time since it will not be "freed" (i.e. returned to the DMA to be filled again) until the callback is finished so any delays in the processing of packets mean that the transport cannot process additional packets (this is why for commands like NODE_IDENTIFY, the M script must wait until the node is finished blinking before sending additional commands). 

While all of the C files are involved in the transport (i.e. wl_baseband.c, wl_interface.c, wl_node.c, wl_transport.c, wl_trigger_manager.c and wi_user.c) since each file processes commands destined for its subsystem, wl_transport.c and wl_node.c are the main files that you need to study.  First in main() of wl_node.c, you can see that the transport is initialized and that the transport_set_host_msg_receive_callback is set.  During the transport initialization, the transport creates and initializes two sockets (see code here).  This sets up the UDP callback for each socket.  Also as part of the initialization, the IP and MAC address of the Ethernet device is set. 

In terms of packet flow:
  1) Main infinite loop calls transport_poll() on the given Ethernet device.
  2) transport_poll() calls the low level WARPxilnet function to check if an Ethernet packet is available. 
  3) If a packet is available, the WARPxilinet library processes that packet and if it is a valid UDP packet to an open socket, it will call the appropriate callback;  this has be initialized to the transport_receive_callback().
  4) The transport_receive_callback() processes packets with valid WARPLab Transport Headers and if this is a Host-to-Node message, will then call the host_msg_receive_callback() which was initialized to node_rx_from_transport()function.
  5) The node_rx_from_transport() function then distributes the WARPLab command to the various sub-systems for further processing
  6) Once the processing is done in the callback, then, based on the return value, the functions will send a response back from the node.  Note:  if a command, for example Read IQ, needs to send multiple responses to one command, then the node_send_early_response() function is used.

In general, WARPLab commands follow a sequential Request-Acknowledgement packet flow, whereby the Host will send a WARPLab command and then, based on the command, expect one or more responses from the node before the next command is sent.  From the node perspective, it can only process one command at at time and therefore does not get out of sync between commands and responses (though commands can be dropped if sent while the node is processing a packet and the node does not have enough buffer space).  However, from a host perspective, things are more difficult since there is an OS between the Ethernet device and the transport processing that occurs in Java or MEX.  Therefore, the transport tries to be very rigid in that each command needs a response and that a new command is not sent until a response is received (i.e. everything must be done in-order).  The exception to this is Read IQ where the sample header encodes enough additional information to handle IQ data received out of order. 

Now, that you have a basic understanding of how the transport works, there are a couple of levels of "enabling both Ethernet ports"

  1) At the very basic level, you can change the Ethernet port being used (i.e. move from Eth A to Eth B) by modifying the WL_ETH define in node.c.  This will still only use one Ethernet port but it shows that the infrastructure will support both Ethernet ports equally.

  2) To enable both ports to process commands while on different switches (i.e. Eth A and Eth B cannot be on the same switch because they have the same IP address), you would need to change main() in wl_node.c to initialize both Ethernet devices and then poll both Ethernet devices.  While the Ethernet devices will have different MAC addresses, the IP addresses will be the same.  Also, both Ethernet devices will process packets in the same way (i.e. both use the same callback, node_rx_from_transport(). 

  3) To enable both ports to be on the same switch, you need to update transport_get_hw_info() to provide different IP addresses for each interface.  Also, you need to make sure that you will not conflict with other nodes on the network.  For example, you could have the last octet be: 


ip_addr[3] = (node + ((eth_dev_num - 1) * 64) + 1);  

instead of 

ip_addr[3] = (node + 1);

which would work as long as you did not have a node with an ID >= 64.  In that case, for node 0, Eth A would have an IP address of and Eth B would have an IP address of  You just have to make sure your IP address assignment will not result in conflicting IP addresses.

At this point, you would have each Ethernet port able to process its own command-response stream from a host.  From a host perspective in Matlab, you could consider a single physical WARP board to behave like two node objects (i.e. one "node" object able to process commands on Eth A and one "node" object able to process commands on Eth B).  However, these two "nodes" would timeshare the single physical board in that only one Ethernet interface would be processed at a time.  One issue you would run in to would be that, depending on the commands, the transport might timeout for a given "node" while the board is processing a command for the other "node".  Therefore, you would need to increase the transport timeout from its current default of 1 second. 

Any further merging of the transports (e.g. having both Ethernet ports return data from a single command) is more complicated.  Since you are working in a single threaded, bare-metal C environment using the node_send_early_resp() function to push packets out over both ethernet ports is straight forward.  Also, having each Ethernet port perform different processing steps is also pretty straight forward. You just have to update the [url=http://warpproject.org/trac/browser/ResearchApps/PHY/WARPLAB/WARPLab7/C_Code_Reference/wl_transport.c?rev=4487#L78]transport_receive_callback()[url]to use different host_msg_receive_callbacks depending on the Ethernet device number (eth_dev_num).  However, on the host side, you will need to extend the node transport in Matlab to have two sockets.  Then you would need to check and process packets from both sockets.  Also, you have to be careful that the OS doesn't filter any packets before they are transferred to the socket.  We have run in to issues in the past with firewalls filtering packets so you would need to confirm that you can actually receive packets on both interfaces from a single command.  We have not tried this and do not know all the issues associated with doing this.

Also, some caveats to consider:  Currently a host can overwhelm a node with small packets due to the overhead of processing a packet on the node.  Basically, the over-the-wire time of a small Ethernet frame is smaller than the time required to process the Ethernet frame so the node cannot process more than a couple of back-to-back Ethernet frames since the node will run out of buffering.  This is why in the Write IQ process in MEX, we add a delay between small packets.  Otherwise, the node will drop IQ packets and the transport will have to revert to "slow write".  This is not the case for large Ethernet frames (i.e. jumbo frames).  For jumbo frames, the over-the-wire time is such that the node can process back to back Write IQ Ethernet frames continuously.



Board footer