DTrace Network Providers
The following is a design proposal for a collection of DTrace Networking Providers. These aim to provide networking observability and troubleshooting information for Solaris users. A prototype TCP provider was demonstrated at CEC 2006.
I originally posted this at http://www.opensolaris.org/os/community/dtrace/NetworkProvider/ . This project was later completed and merged into the Solaris kernel.
This document will list the probes that may be made available, their arguments, and examples of their proposed usage. Feedback is welcome, please post to dtrace-discuss. NOTE: this is a work in progress, and this network provider is not yet available! The purpose of this site is to flesh out ideas for a future network provider.
Who is talking to my web server?
# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
@pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C
192.168.1.8 9
fe80::214:4fff:fe3b:76c8 12
192.168.1.51 32
10.1.70.16 83
192.168.7.3 121
192.168.101.101 192
What ports are people connecting to?
# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C
79 2
22 14
80 327
Author: Brendan Gregg and others, 16-Oct-2006
Contents
Aims
The main aims of the network providers are:
- Support for common network protocols: TCP, UDP, IP.
- Everyday observability: connections/packets/bytes by address/port.
- Performance observability: throughputs, network latencies.
- Socket level tracing: associate events to PIDs.
- Further troubleshooting support: TCP flag analysis, header info.
Future enhancements and additions to the providers include:
- Other protocol support: ICMP, SCTP, ...
- Kernel code path latency probes.
Providers
The Network provider is a collection of several DTrace providers, such as ip and tcp (there is no monolithic net provider). They are:
| provider | description |
|---|---|
| gld | This is the generic LAN device layer, and shows link layer activity such as Ethernet frames. The probes allow frame by frame tracing. |
| arp | This allows tracing of ARP and RARP packets. |
| icmp | This allows tracing of ICMP packets, and provides the type and code from the ICMP header. |
| ip | This is the IP layer, which provides tracing for IPv4 and IPv6. The probes allow packet by packet tracing, and are placed near the interface to GLD. |
| tcp | This is the TCP layer, and shows what TCP activity is occuring. The probes are placed close to the IP interface, so that TCP events passed down to or received from IP can be traced. |
| udp | This allows tracing of UDP events. |
| sctp | This allows tracing of SCTP events. |
| socket | This allows tracing at the socket layer, close to the application. These probes fire in the same context as the corresponding process, and show data sent to or received from sockets. |
Other providers such as icmp and igmp may also be added (and many more).
Modules
The module and function names are from the kernel code, and are usually ignored unless deeper code-based analysis is needed.
Probes
tcp:::accept-established tcp:::accept-refused tcp:::connect-request tcp:::connect-established tcp:::connect-refused tcp:::state-bound tcp:::state-close-wait tcp:::state-closed tcp:::state-closing tcp:::state-established tcp:::state-fin-wait1 tcp:::state-fin-wait2 tcp:::state-idle tcp:::state-last-ack tcp:::state-listen tcp:::state-syn-received tcp:::state-syn-sent tcp:::state-time-wait tcp:::drop-saturation tcp:::data-send tcp:::data-resend tcp:::data-receive tcp:::ack-send tcp:::ack-receive tcp:::fuse-send tcp:::fuse-receive tcp:::send tcp:::receive |
udp:::send udp:::receive sctp:::send sctp:::receive ip:::send ip:::receive arp:::send arp:::receive icmp:::send icmp:::receive gld:::send gld:::receive |
Send and receive probes fire whenever that layer sends or receives. They can be used for observability and measuring network latency, but their placement is not sufficient for kernel code path latency measurements.
TCP fuse probes fire for localhost traffic (TCP fusion), IP probes do not.
drop-saturation probes fire when packets are dropped due to load; other varients may include drop-error.
TCP Probes
To start with, send and receive fire for all traffic that is TCP; this includes the TCP handshake, TCP ACKs, and also for invalid packets such as those with corrupt flags. Since the TCP flags are available in the probe arguments, any particular types of TCP traffic can be matched using DTrace predicates.
Additional probes have been provided for convienence in addition to send and receive. This includes the data-* probes for matching on TCP packets containing data, connect-* probes for client connections, accept-* probes for servers accepting connections, state-* for RFC 793 state changes, and so on.
Outbound TCP Connections
For outbound connections, connect-request fires before the first SYN, connect-established fires after the SYN-ACK as been received, and the final ACK is being sent.
In summary:
| TCP event | DTrace probe |
|---|---|
| SYN sent | tcp:::connect-request |
| SYN-ACK received, ACK sent | tcp:::connect-established |
| SYN send, RST received | tcp:::connect-refused |
An outbound connection to an open port will fire a connect-request followed by a connect-established. An outbound connection to a closed port will fire a connect-refused.
A possible connect-closed probe (normal close) has been dropped from this design due to difficulties encountered in the tcp code; the state-closed probe may serve a similar purpose.
Inbound TCP Connections
For inbound connections, accept-established fires after the final ACK was received in the three way handshake in response to an inbound SYN. accept-refused fires if a connection was refused, such as a SYN to a closed port.
In summary:
| TCP event | DTrace probe |
|---|---|
| SYN-ACK sent, ACK received | tcp:::accept-established |
| RST sent, port closed | tcp:::accept-refused |
An inbound connection to an open port will fire an accept-established, and an inbound connection to a closed port will fire an accept-refused.
A possible accept-closed probe (normal close) has been dropped from the design due to difficulties encountered in the tcp code; the state-closed probe may serve a similar purpose.
Note: TCP events such as the three way handshake can also be traced using tcp:::receive and tcp:::send, and examination of the TCP flags (args[2]->tcp_flags).
Arguments
The following information is made available for the probes, where possbile and appropriate:
| probes | args[0] | args[1] | args[2] | args[3] |
|---|---|---|---|---|
|
gld:::send gld:::receive | gldstateinfo_t * | ipinfo_t * | gldinfo_t * | NULL |
|
ip:::send ip:::receive | ipstateinfo_t * | ipinfo_t * | ipv4info_t * | ipv6info_t |
|
tcp:::send tcp:::receive | tcpstateinfo_t * | ipinfo_t * | tcpinfo_t * | NULL |
|
tcp:::accept-* tcp:::connect-* | tcpstateinfo_t * | ipinfo_t * | tcpinfo_t * | NULL |
|
tcp:::state-* | tcpstateinfo_t * | NULL | NULL | NULL |
|
tcp:::fuse-* | tcpstateinfo_t * | tcpfuseinfo_t * | size_t | NULL |
|
udp:::send udp:::receive | udpstateinfo_t * | ipinfo_t * | udpinfo_t * | NULL |
|
udp:::stream-* | udpstateinfo_t * | NULL | NULL | NULL |
|
icmp:::send icmp:::receive | icmpstateinfo_t * | ipinfo_t * | icmpinfo_t * | NULL |
For IPv4 traffic, the ip probes provide header details in args[2] and args[3] is NULL; for IPv6 traffic, the header details are in args[3] and args[2] is NULL. For both traffic, summary header details are in args[1] - ipinfo_t.
Argument Details
The arguments contain the following members:
| arg | type | contents | types | notes |
|---|---|---|---|---|
| args[0] | tcpstateinfo_t * | args[0]->tcps_cid args[0]->tcps_loopback args[0]->tcps_active args[0]->tcps_state args[0]->tcps_statestr |
uint64_t int int int string |
tcps_sid is a pointer to a unique connection-ID. tcps_loopback is a boolean to identify loopback traffic. tcps_active is a boolean for active opens - host initiated connections. tcps_state is the numeric TCP state. tcps_statestr is a string form of the TCP state. |
| args[1] | ipinfo_t * |
args[1]->ip_ver args[1]->ip_plength args[1]->ip_saddr args[1]->ip_daddr |
uint8_t uint16_t string string |
These fields are valid for both IPv4 and IPv6, and have been provided for convienence. ip_saddr and ip_daddr are stringified versions of the IPv4 or IPv6 addresses. |
| args[2] | etherinfo_t * | args[2]->e_source args[2]->e_dest args[2]->e_type args[2]->e_length args[2]->e_ifname |
ether_addr_t ether_addr_t uint16_t uint16_t char * |
e_ifname is the interface name, such as hme0. |
| args[2] | ipv4info_t * | args[2]->ipv4_ver args[2]->ipv4_tos args[2]->ipv4_length args[2]->ipv4_ident args[2]->ipv4_flags args[2]->ipv4_offset args[2]->ipv4_ttl args[2]->ipv4_protocol args[2]->ipv4_src args[2]->ipv4_dst args[2]->ipv4_hdr |
uint8_t uint8_t uint16_t uint16_t uint8_t uint16_t uint8_t uint8_t ipaddr_t ipaddr_t ipha_t |
See RFC 791 IPv4 traffic only, /args[1]->ip_ver == 4/ |
| args[3] | ipv6info_t * | args[3]->ipv6_ver args[3]->ipv6_tclass args[3]->ipv6_flow args[3]->ipv6_plen args[3]->ipv6_next args[3]->ipv6_hlim args[3]->ipv6_src args[3]->ipv6_dst args[3]->ipv6_hdr |
uint8_t uint8_t uint32_t uint16_t uint8_t uint8_t in6_addr_t in6_addr_t struct ip6_hdr |
See RFC 2460 IPv6 traffic only, /args[1]->ip_ver == 6/ |
| args[2] | tcpinfo_t * | args[2]->tcp_sport args[2]->tcp_dport args[2]->tcp_seq args[2]->tcp_ack args[2]->tcp_offset args[2]->tcp_flags args[2]->tcp_window args[2]->tcp_data |
in_port_t in_port_t uint32_t uint32_t uint8_t uint8_t uint16_t uintptr_t |
See RFC 793 |
| args[2] | udpinfo_t * | args[2]->udp_sport args[2]->udp_dport args[2]->udp_length args[2]->udp_data |
in_port_t in_port_t uint16_t uintptr_t |
See RFC 768 |
| args[2] | icmpinfo_t * | args[2]->icmp_type args[2]->icmp_code |
uchar_t uchar_t |
See RFC 792 |
The info structures are new, and will be provided by DTrace. They have been based on the RFCs so that their contents are widely understood.
Example Usage
The following are demonstrations how the net provider could be used to answer various common observability questions.
- New inbound TCP connections by port
- New inbound TCP connections by source IP address
- Inbound TCP packets by local port
- Inbound TCP packets by remote address
- Inbound TCP packets by remote address, for a specific port, eg: port 80 (HTTP)
- Sent TCP bytes by local port
- TCP bytes by address and port
- TCP bytes by address and port, per second
- Detect TCP connect() scan by IP address
- Detect TCP SYN stealth scan by IP address
- Detect TCP null scan by IP address
- Detect TCP FIN stealth scan by IP address
- Detect TCP Xmas scan by IP address
- Network Utilization by TCP Port
- Connect Latency
- 1st Byte Latency
- Throughput Latency
- Connection Lifespan
- TCP Saturation Drops by IP address
Examples: General
New inbound TCP connections by port
# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C
79 1
22 2
515 5
80 94
This one-liner shows new inbound TCP connections by port. The sample output shows 94 new connections occured on port 80, HTTP.
New inbound TCP connections by source IP address
# dtrace -n 'tcp:::accept-established { @[args[1]->ip_saddr] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C
192.168.1.31 1
fe80::214:4fff:fe3b:76c8 2
192.168.1.2 7
192.168.1.51 9
192.168.1.8 11
192.168.7.100 17
192.168.101.101 31
This one-liner shows new inbound connections by source ip address, which is the string form of either an IPv4 or IPv6 address. Here the host 192.168.101.101 has made 31 successful connections to our server while DTrace was tracing.
Inbound TCP packets by local port
# dtrace -n 'tcp:::receive { @pkts[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::receive' matched 1 probe
^C
79 2
515 9
22 32
2049 121
80 8462
This one-liner shows inbound TCP packets by port. The sample output shows that 8462 packets arrived for port 80, HTTP.
Inbound TCP packets by remote address
# dtrace -n 'tcp:::receive { @pkts[args[1]->ip_daddr] = count(); }'
dtrace: description 'tcp:::receive' matched 1 probe
^C
192.168.1.8 9
fe80::214:4fff:fe3b:76c8 12
192.168.1.51 32
192.168.1.8 54
192.168.7.3 121
192.168.101.101 192
This one-liner shows which hosts TCP packets have been received from.
Inbound TCP packets by remote address, for a specific port, eg: port 80 (HTTP)
# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
@pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C
192.168.1.8 9
fe80::214:4fff:fe3b:76c8 12
192.168.1.51 32
192.168.1.8 54
192.168.7.3 121
192.168.101.101 192
This short script shows which hosts are sending packets to TCP port 80, the web server. This script can easily be tweaked to show which hosts are sending packets to any of the local services, such as DNS, NFS, etc. RPC services can be resolved by comparing output with an "rpcinfo -p".
Sent TCP bytes by local port
# dtrace -n 'tcp:::send {
@bytes[args[2]->tcp_sport] = sum(args[1]->ip_plength - args[2]->tcp_offset);
}'
dtrace: description 'tcp:::sent' matched 1 probe
^C
515 16
79 204
22 427
2049 8264
80 108268
The DTrace oneliner subtracts the TCP header length from the IP payload length, which provides is with the TCP payload length. The output shows that 108,268 bytes (106 Kbytes) were sent from port 80, HTTP, during the time DTrace was tracing.
TCP bytes by address and port
# dtrace -n '
tcp:::receive {
@bytes[args[1]->ip_saddr, args[2]->tcp_dport] =
sum(args[1]->ip_plength - args[2]->tcp_offset);
}
tcp:::send {
@bytes[args[1]->ip_daddr, args[2]->tcp_sport] =
sum(args[1]->ip_plength - args[2]->tcp_offset);
}'
dtrace: description 'tcp:::receive' matched 2 probes
^C
192.168.1.8 22 68
fe80::214:4fff:fe3b:76c8 22 582
192.168.1.6 80 1216
192.168.101.101 512 2082
192.168.1.1 22 2187
192.168.1.51 80 3763
192.168.3.5 109 4080
192.168.1.8 80 8039
192.168.7.3 80 10921
192.168.1.1 2049 54080
192.168.101.101 80 287623
This provides a report of IP address, TCP port and bytes transferred. This provides an immediate view of which hosts are communicating with our web server, and which ports they are using. During the above sample, host 192.168.101.101 transferred 287623 bytes with our port 80, HTTP.
TCP bytes by address and port, per second
# dtrace -qn '
tcp:::receive {
@bytes[args[1]->ip_saddr, args[2]->tcp_dport] =
sum(args[1]->ip_plength - args[2]->tcp_offset);
}
tcp:::send {
@bytes[args[1]->ip_daddr, args[2]->tcp_sport] =
sum(args[1]->ip_plength - args[2]->tcp_offset);
}
profile:::tick-1sec {
printf("\n %-32s %16s\n", "HOST", "BYTES/s");
printa(" %-32s %@16d\n", @bytes);
trunc(@bytes);
}
'
dtrace: description 'tcp:::receive' matched 3 probes
HOST BYTES/s
192.168.1.8 22
fe80::214:4fff:fe3b:76c8 54
192.168.3.5 516
192.168.2.11 8142
HOST BYTES/s
fe80::214:4fff:fe3b:76c8 54
192.168.2.11 148
192.168.3.5 686
192.168.101.101 60872
HOST BYTES/s
192.168.1.8 6
192.168.3.5 8044
192.168.101.101 826306
...
This DTrace script provides a rolling summary of which hosts have transferred how many bytes by TCP, per second. This makes for a useful tool to immediately identify which host is responsible for consuming network bandwidth. Various enhancements to this script would include adding a field for the destination port, and printing in Kbytes instead of bytes.
Examples: Security Monitoring
While other options such as installing a NIDS (such as snort) would usually offer a better long term traffic monitoring solution, DTrace can be handy for short term traffic analysis.
Detect TCP connect() scan by IP address
# dtrace -n 'tcp:::accept-refused { @num[args[1]->ip_saddr] = count(); }'
dtrace: description 'tcp:::accept-refused' matched 1 probe
^C
192.168.1.1 1
220.241.6.37 307
The accept-refused probe fires when a connection is attempted to a closed port. Host 220.241.6.37 attempted 307 times to connect to closed ports, which is likely to be malicious activity such as port scanning. Host 192.168.1.1 had only 1 refused connection, which may be normal activity depending on what their OS is doing.
Detect TCP SYN stealth scan by IP address
# dtrace -n '
tcp:::receive
/syn[args[0]->tcps_cid] && (args[2]->tcp_flags & TH_RST)/
{
@num["TCP_stealth_scan", args[1]->ip_saddr] = sum(1);
}
tcp:::receive
{
syn[args[0]->tcps_cid] = 0;
}
tcp:::receive
/args[2]->tcp_flags & TH_SYN && ! (args[2]->tcp_flags & TH_ACK)/
{
syn[args[0]->tcps_cid]++;
}
'
dtrace: description 'tcp:::receive' matched 2 probes
^C
192.168.3.5 1
220.241.6.37 4
This script (or one similar to this) detects half connections - SYNs that are followed by a RST, a sign of a TCP SYN stealth scan. In the above output, host 220.241.6.37 has made 4 half connections, and is most probably scanning our host (or this is a spoofed IP address). The host 192.168.3.5 made 1 half connection, which is not suspicious as this may have completed had we traced for longer. If you are being stealth scanned, you will see many more accept-refused events that appear like a connect scan, and only a few half connections for the ports you have open.
Detect TCP null scan by IP address
# dtrace -n '
tcp:::receive
/args[2]->tcp_flags == 0/
{
@num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C
220.241.6.37 197
The net provider can be easily used to analyse esoteric TCP traffic, such as null scans. Here null scan packets by host are reported, identifying 220.241.6.37 as sending 197 TCP null packets.
Detect TCP FIN stealth scan by IP address
# dtrace -n '
tcp:::receive
/args[0]->tcp_state != TCPS_ESTABLISHED && args[2]->tcp_flags & TH_FIN/
{
@num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C
220.241.6.37 45
Here stealth FIN scans are identified by matching on packets to non-established connections (using the TCP state from args[0]), and also matching on the TCP FIN flag. Our evil host sent 45 of these packets. There may certainly be other ways available to identify these scans.
Detect TCP Xmas scan by IP address
# dtrace -n '
tcp:::receive
/args[2]->tcp_flags == (TH_URG + TH_PUSH + TH_FIN)/
{
@num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C
220.241.6.37 167
There are variations on the Xmas scan, and many different ways to write predicates to match the TCP flags. Here we found 167 likely Xmas packets.
It would be straightforward to write a DTrace script to monitor for many types of malicious TCP traffic, and produce a report of host and traffic type; and as such can serve as a temporary Host based NIDS (Network Intrusion Detection System). Producing such reports without the DTrace net provider usually involves installing additional software, which either sets the network interface to promiscuous mode and analysip_saddr)] = sum(1); }'s every packet, such as snort; or processes the output of network capture files, such as snoop's RFC1761 or libpcap.
Examples: Performance Monitoring
Network Utilization by TCP Port
# dtrace -n '
tcp:::receive {
/* assume 1 packet with standard ether + IP + TCP header */
@util[args[2]->tcp_dport] = sum(args[1]->ip_plength + 14);
}
tcp:::send {
/* assume 1 packet with standard ether + IP + TCP header */
@util[args[2]->tcp_sport] = sum(args[1]->ip_plength + 14);
}
profile:::tick-1sec {
/* many assumptions, including 1 NIC @ 100 Mbps */
normalize(@util, 10000000)
printf(" %12s %12s\n", "PORT", "%UTIL");
printa(" %12d %@12d\n", @util);
trunc(@util);
}'
dtrace: description 'tcp:::receive' matched 3 probes
^C
PORT %UTIL
22 0
2049 2
80 12
PORT %UTIL
2049 0
22 1
80 23
...
This is a short and crude script based on numerous assumptions, however it will serve to provide an approximation for a single 100 Mbps NIC's utilization by port. While we prefer things to be 100% accurate, this is far better than nothing at all - so long as we bear in mind that the values are an approximation.
Connect Latency
# dtrace -n '
tcp:::connect-request
{
start[args[0]->tcps_cid] = timestamp;
}
tcp:::connect-established
/start[args[0]->tcps_cid]/
{
@latency["Connect Latency (ns)", args[1]->ip_daddr] =
quantize(timestamp - start[args[0]->tcps_cid]);
start[args[0]->tcps_cid] = 0;
}'
dtrace: description 'tcp:::connect-request' matched 2 probes
^C
Connect Latency (ns) 192.168.101.101
value ------------- Distribution ------------- count
32768 | 0
65536 |@@@@@@@ 2
131072 |@@@@@@@@@@@@@@@ 4
262144 |@@@@ 1
524288 |@@@@@@@@@@@ 3
1048576 |@@@@ 1
2097152 | 0
Connect Latency (ns) 10.1.1.7
value ------------- Distribution ------------- count
8388608 | 0
16777216 |@@@@@ 3
33554432 |@@@@@@@@@@@@@@@@ 9
67108864 |@@@@@@@@@@@@@@@ 8
134217728 |@@@@ 2
268435456 | 0
This script measures outbound connection latency by IP address, and presents this as a distribution plot by nanosecond and count of occurances. We define this to be the time from the SYN to the SYN-ACK (matched by the first receive on the same conn_s *), so most of the TCP handshake costs. The output shows that one IP address is slower to connect than the other. This metric includes network latency and destination kernel overheads.
1st byte Latency
# dtrace -n '
tcp:::connect-established
{
start[args[0]->tcps_cid] = timestamp;
}
tcp:::receive
/start[args[0]->tcps_cid] && (args[1]->ip_plength - args[2]->tcp_offset) > 0/
{
@latency["1st Byte Latency (ns)", args[1]->ip_saddr] =
quantize(timestamp - start[args[0]->tcps_cid]);
start[args[0]->tcps_cid] = 0;
}'
dtrace: description 'tcp:::send' matched 2 probes
^C
1st Byte Latency (ns) 192.168.101.101
value ------------- Distribution ------------- count
32768 | 0
65536 |@@@@@@ 2
131072 |@@@@@ 1
262144 |@@@@@@@@@@@@@@@ 4
524288 |@@@@@@@@@@@ 3
1048576 |@@@@ 1
2097152 | 0
This measures a 1st byte latency on an outbound connection, defined as the time from the TCP handshake completing, to the time that the 1st application byte is returned. This metric includes network latency, destination kernel scheduling latency, and application latency.
Throughput Latency
...Connection Lifespan
The following script measures the lifespan of a connection in milliseconds.
# cat tcplifespan1.d
#!/usr/sbin/dtrace -qs
dtrace:::BEGIN
{
printf("Tracing... Hit Ctrl-C to end.\n");
}
tcp:::accept-established
{
raddr[args[0]->tcps_cid] = args[1]->ip_saddr;
lport[args[0]->tcps_cid] = args[2]->tcp_dport;
stime[args[0]->tcps_cid] = timestamp;
}
tcp:::state-closed,
tcp:::state-time-wait
/stime[args[0]->tcps_cid]/
{
@lifespan[raddr[args[0]->tcps_cid], lport[args[0]->tcps_cid]] =
quantize((timestamp - stime[args[0]->tcps_cid]) / 1000000);
raddr[args[0]->tcps_cid] = 0;
lport[args[0]->tcps_cid] = 0;
stime[args[0]->tcps_cid] = 0;
}
dtrace:::END
{
printf("Inbound Connection Lifespan Report (ms),\n\n");
printa(" %-32I %21P %@d\n", @lifespan);
}
And now running this script:
# ./tcplifespan1.d
^C
Inbound Connection Lifespan Report (ms),
deimos http
value ------------- Distribution ------------- count
1 | 0
2 |@@@@@@@@@@@@@@@@@@@@@@@ 7
4 |@@@@@@@@@@ 3
8 |@@@ 1
16 | 0
32 |@@@ 1
64 | 0
deimos finger
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4
32 |@@@@@@@@ 1
64 | 0
deimos ssh
value ------------- Distribution ------------- count
2048 | 0
4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
8192 | 0
tcplifespan1.d measures the lifespan of an inbound TCP connection and prints a distribution plot. In the above output, we captured a dozen HTTP connections, five finger connections and an SSH connection. One HTTP connection was much slower than the others as it was in the 32 to 63 ms bucket, most of the rest were from 2 to 3 ms. The SSH connection was the slowest (naturally, as this was an interactive session), which was in the 4 to 8 second bucket.
TCP Saturation Drops by IP address
# dtrace -n 'tcp:::drop-saturation {
@drops[args[1]->ip_saddr, args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::drop-saturation' matched 1 probe
^C
10.1.1.17 192.168.1.1 1
192.168.1.1 192.168.6.2 5
192.168.6.2 192.168.1.1 9
This measures TCP drops caused by saturation, by IP address. Nine packets from 192.168.6.2 to 192.168.1.1 were dropped.
Solved Problems
Some of the (numerous) network obserability problems that this provide will solve include:
- Who is connecting to my web server? (TCP accepts by IP address)
- Which services are busy? (bytes by destination port)
- Which hosts are responsible for network load? (bytes by IP address)
- Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
- Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)