1. Purpose[edit | edit source]
This article describes how to configure TSN (Time-Sensitive Networking) endpoint.
Throughout this article, IEEE standards is mentioned. To get more information about it, refer to TSN standards introduction.
This article provides different steps to set up the TSN standards on Ethernet standalone:
- PTP (Precision Time Protocol) [IEEE-1588, IEEE-802.1AS]
- FQTSS / CBS (Credit Based Shaper) [ IEEE-802.1Qav ]
- EST (Enhancements for Scheduled Traffic) [ IEEE-802.1Qbv ]
2. IEEE 802.1AS-Rev or gPTP synchronization[edit | edit source]
Testing using "ptp4l" tool from "linuxptp" project
To check the "ptp" clock adjustment with PTP protocol, a PTP with ordinary clock (client) and a PTP Grandmaster (server) applications are needed to run on separate devices. Open source application package "linuxptp" can be used.
Create file /etc/gPTP.cfg with content as below:
[global] gmCapable 1 priority1 248 priority2 248 logAnnounceInterval 0 logSyncInterval -3 syncReceiptTimeout 3 neighborPropDelayThresh 800 min_neighbor_prop_delay -20000000 assume_two_step 1 path_trace_enabled 1 follow_up_info 1 transportSpecific 0x1 ptp_dst_mac 01:80:C2:00:00:0E network_transport L2 delay_mechanism P2P
Hence TX timestamp generation can be delayed (especially with low speed links) the "ptp4l" “tx_timestamp_timeout” parameter need to be set for "ptp4l" to work. The default value for "tx_timestamp_timeout" is 5 seconds To change the value of "tx_timestamp_timeout", you can add or modify the following line in the configuration file:
tx_timestamp_timeout <timeout_value>
PTP synchro between two boards:
ptp4l -i end0 -p /dev/ptp0 -m -f /etc/gPTP.cfg
PPS (Pulse Per Second) is a signal that is commonly used to check the synchronization of a system clock with a reference clock, such as a PTP grandmaster clock. To check the synchronization of the system clock with the reference clock using PPS, you can output signal PPS on gpio pin to control it on oscilloscope. More details in Ethernet_device_tree_configuration#How_to_configure_PTP_PPS_out_-3F
3. CBS (Credit Based Shaper) [IEEE-802.1Qav][edit | edit source]
TSN 802.1Qav is an IEEE specification that allows stream reservation for networking traffic queues. It adds enhancements to traffic shaping with the introduction of stream reservation. This article demonstrates how TSN 802.1Qav enhances traffic shaping, using Ethernet standalone.
This is an example of configuration with 4 reserved bandwidth given below:
- Uses 4 Queues (Q0-Q3). Each Q has a bandwidth reserved in h/w. Maximum 4 Queues supported
- Queue 0 has No Bandwidth Restriction, general traffic
- Queue 1 has 10% of Maximum bandwidth reserved (100Mbps)
- Queue 2 has 20% of Maximum bandwidth reserved (200Mbps)
- Queue 3 has 30% of Maximum bandwidth reserved (300Mbps)
- Q3 is the highest priority Queue and Q0 is the lowest priority
Here are the steps to configure this use case.
Bring up end0 interface
ip link add link end0 name end0.200 type vlan id 200 ip link set end0.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 ifconfig end0.200 192.168.200.60 netmask 255.255.255.0
CBS is used to enforce a Quality of Service by limiting the data rate of a traffic class, to separate packets into traffic classes the user may choose mqprio(8) https://man7.org/linux/man-pages/man8/tc-mqprio.8.html, and configure it like this:
tc qdisc add dev end0 clsact tc qdisc add dev end0 handle 100: parent root mqprio \ num_tc 4 \ map 0 0 1 1 2 2 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0
Configure Bandwidth allocated for each queue (https://man7.org/linux/man-pages/man8/tc-cbs.8.html)
// Set bandwidth of queue 1 to be 100Mbps tc qdisc replace dev end0 parent 100:2 cbs \ idleslope 100000 \ sendslope -900000 \ hicredit 150 \ locredit -1350 \ offload 1
// Set bandwidth of queue 2 to be 200Mbps tc qdisc replace dev end0 parent 100:3 cbs \ idleslope 200000 \ sendslope -800000 \ hicredit 300 \ locredit -1200 \ offload 1
// Set bandwidth of queue 3 to be 300Mbps tc qdisc replace dev end0 parent 100:4 cbs \ idleslope 300000 \ sendslope -700000 \ hicredit 450 \ locredit -1050 \ offload 1
So for queue 0 the bandwidth should be around 400Mbps (port_transmit_rate - sum of all bandwidth of each queue)
sendslope = idleslope - port_transmit_rate
hicredit = max_interference_size * (idleslope / port_transmit_rate)
locredit = max_frame_size * (sendslope / port_transmit_rate)
with max_interference_size = max_frame_size = 1500
For more details: https://en.wikipedia.org/wiki/Time-Sensitive_Networking#IEEE_802.1Qav_Forwarding_and_queuing_enhancements_for_time-sensitive_streams
Network core and Driver uses the skb priority to deliver frames to specific h/w queues. In the above case, priority 6 to 15 SKB (packet) goes to Q3 (4th entry in map in the tc qdisc command), priority 4 to 5 SKB goes to Q2 (3rd entry in map) priority 2 to 3 SKB goes to Q1 (2nd entry in map) and priority 0 to 1 SKB goes to Q0 (1st entry in map)
Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to end0 with IP address 192.168.200.30
iperf3 -s -p 5003& iperf3 -s -p 5002& iperf3 -s -p 5001& iperf3 -s -p 5000&
At DUT, start transmission of stream using iperf3
//Stream with SKB priority 6 : TOS=0x14 == SO_PRIORITY=6 // https:man7.org/linux/man-pages/man8/tc-prio.8.html iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4 iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &
//Stream with SKB priority 2 iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort) iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60 &
Result without tc rule apply :
For Traffic with priority 6:
[ 5] 0.00-60.00 sec 1.40 GBytes 200 Mbits/sec 0.046 ms 11841/1031880 (1.1%) receiver
For Traffic with priority 4:
[ 5] 0.00-60.00 sec 40.1 MBytes 5.60 Mbits/sec 39.651 ms 936/29490 (3.2%) receiver
For Traffic with priority 2:
[ 5] 0.00-60.00 sec 4.94 GBytes 707 Mbits/sec 0.035 ms 164701/3768910 (4.4%) receiver
For Traffic with priority 0 (Best effort):
[ 5] 0.00-60.00 sec 106 MBytes 14.8 Mbits/sec 0.017 ms 1765/76956 (2.3%) receiver
Without CBS TSN rules in place, multiple traffic streams may contend for network resources, causing congestion and possible packet loss. This can lead to poor network performance and lower quality of service for users. Furthermore, without CBS TSN rules, there no prioritization of important traffic, such as real-time audio or video streams, which can cause delays and interruptions in these applications. In summary, the absence of CBS TSN rules can create an unstable and undependable network environment.
Result with tc rule apply:
For Queue 3 with priority 6 and with BW of 300Mbps:
[ 5] 0.00-60.00 sec 1.72 GBytes 246 Mbits/sec 0.052 ms 189322/1444777 (13%) receiver
For Queue 2 with priority 4 and with BW of 200Mbps:
[ 5] 0.00-60.00 sec 1.16 GBytes 167 Mbits/sec 0.064 ms 110342/959769 (11%) receiver
For Queue 1 with priority 2 and with BW of 100Mbps:
[ 5] 0.00-60.00 sec 641 MBytes 89.6 Mbits/sec 0.266 ms 32740/489287 (6.7%) receiver
For Queue0 with priority 0 (Best effort):
[ 5] 0.00-60.00 sec 2.31 GBytes 331 Mbits/sec 0.006 ms 234299/1921970 (12%) receiver
When TSN rules are implemented, the network can provide guaranteed bandwidth allocation and prioritization for different traffic streams. CBS TSN rules allow for the allocation of credits to different traffic streams, which can be used to transmit data over the network. This ensures that critical traffic, such as real-time audio or video streams, receive the necessary bandwidth and are not affected by other traffic streams. CBS TSN rules also prevent congestion and packet loss by limiting the amount of data that can be transmitted by each traffic stream. This results in a more predictable and reliable network environment, with improved quality of service for users.
4. EST (Enhancements for Scheduled Traffic) [IEEE-802.1Qbv][edit | edit source]
TSN 802.1Qbv is an IEEE specification that allows to define transmission time slots for networking traffic queues. It adds enhancements to traffic scheduling with the introduction of time-aware shaper. This article demonstrates how TSN 802.1Qbv enhances traffic scheduling, using Ethernet standalone.
An example configuration with 4 schedule entries given below:
- Uses 4 Queues (Q0-Q3). Each Q has a Gate associated in h/w. Maximum 4 Queues/Gates supported
- 3 higher priority Gates open for 125usec (Q1, Q2 and Q3)
- Q3 is the higest priority Queue and Q0 is the lowest priority
- 1 lower priority (Q0) opens remaining gates for 250 usec
Here are the steps to configure this schedule.
Bring up end0 interface
ip link add link end0 name end0.200 type vlan id 200 ip link set end0.200 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 ifconfig end0.200 192.168.200.60 netmask 255.255.255.0
Setup EST schedule with 4 Gates (Q0-Q3). For description of Command parameters, see manual page for taprio. https://man7.org/linux/man-pages/man8/tc-taprio.8.html TC0 <-> Q0, TC1 <-> Q1, TC2 <-> Q2 and TC3 <-> Q3
tc qdisc replace dev end0 parent root handle 100 taprio \ num_tc 4 \ map 0 0 1 1 2 2 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 \ base-time 0000 \ sched-entry S 8 125000 \ sched-entry S 4 125000 \ sched-entry S 2 125000 \ sched-entry S 1 250000 \ sched-entry S 0 375000 \ flags 2
Where num_tc is same as number of queues = 4, maps 16 priorities to one of 4 TCs, queues specify the Queue associated with each TC, TC0 - One queue @0, TC1 - One queue @1, TC2 - One queue @2 and TC3 - One queue @3
sched-entry S 8 125000 S - SetGateStates operation 8 - Bit mask showing bit 4 set (Q3/TC3) 125000 - 125000 nsecs (125 usecs ) duration of Gate open The cycle-time is 1 msec (which is the sum of the intervals of individual schedule)
Enable classifier. Classifier is used to mark the packet based on packet meta data.
tc qdisc add dev end0 clsact
Network core and Driver uses the skb priority to deliver frames to specific h/w queues. In the above case, priority 6 to 15 SKB (packet) goes to Q3 (4th entry in map in the tc qdisc command), priority 4 to 5 SKB goes to Q2 (3rd entry in map) priority 2 to 3 SKB goes to Q1 (2nd entry in map) and priority 0 to 1 SKB goes to Q0 (1st entry in map)
Run 4 iperf sessions to generate several streams with different priorities Remote PC or other device connected to end0 with IP address 192.168.200.30
iperf3 -s -p 5003& iperf3 -s -p 5002& iperf3 -s -p 5001& iperf3 -s -p 5000&
At DUT, start transmission of stream using iperf3
//Stream with SKB priority 6 : TOS=0x14 == SO_PRIORITY=6 // https:man7.org/linux/man-pages/man8/tc-prio.8.html iperf3 -c 192.168.200.30 -u -b 0 -p 5003 -l1472 -t60 --tos 0x14 &
//Stream with SKB priority 4 iperf3 -c 192.168.200.30 -u -b 0 -p 5002 -l1472 -t60 --tos 0x1a &
//Stream with SKB priority 2 iperf3 -c 192.168.200.30 -u -b 0 -p 5001 -l1472 -t60 --tos 0xa &
//Stream with SKB priority 0 (Best effort) iperf3 -c 192.168.200.30 -u -b 0 -p 5000 -l1472 -t60 &
Result without tc rule apply :
Result with tc rule apply:
5. References[edit | edit source]