Chaosreader

A open source tool to trace TCP/UDP/... sessions and fetch application data from snoop or tcpdump logs. This is a type of "any-snarf" program, as it will fetch telnet sessions, FTP files, HTTP transfers (HTML, GIF, JPEG, ...), SMTP emails, ... from the captured data inside network traffic logs. A html index file is created that links to all the session details, including realtime replay programs for telnet, rlogin, IRC, X11 or VNC sessions; and reports such as image reports and HTTP GET/POST content reports. Chaosreader can also run in standalone mode - where it invokes tcpdump or snoop (if they are available) to create the log files and then processes them.

For a look at the command line options see the readme printed by the program (chaosreader --help2).

Screenshots

• Chaosreader example output In this example, a snoop file was created while I loaded a website, used telnet to login, and ftp to transfer files; Chaosreader has managed to extract the HTTP sections, follow the telnet session, grab the FTP files, and create an Image Report from the snoop log. It has also created a replay program to playback the telnet session.
• Chaosreader example output 2 This demonstrates Chaosreader processing IPv6. A few telnet sessions were captured that used site local IPv6 addresses.
• Chaosreader example output 3 This is a large example to demonstrate the standalone mode of chaosreader - where it does not require a log file, instead running tcpdump or snoop and generating it's own logs. In this example it was run using "-S 1,5" which generates five samples of one minute each. (While this runs, reports can be viewed by reloading the index.html file - which is updated as new samples complete). I was viewing websites and using telnet and ftp while chaosreader collected and processed it's five samples. This is the final index.html generated, which links to all the collected data.
• Chaosreader example output 4 This demonstrates Chaosreader capturing HTTP GET and POST queries, and presenting them in the GET/POST Report. The processed log file captured a search on google, slashdot, and imdb. This reporting feature could be used to highlight potential vunerabilities.
• Chaosreader example output 5 Here the options "-ve" were used, where it extracts everything it can from a log. This includes hex dumps and files containing the raw traffic (usually three - ".raw" is 2-way, ".raw1" is A to B, ".raw2" is B to A).
• Chaosreader example output 6 This demonstrates processing X11 traffic - currently an experimental feature. Here I run "xeyes", "xclock" and "xterm" - and run a few commands from the xterm. The following is created from the X11 "xterm" session: a red/blue html report containing keystrokes and displayed text from the session; a text playback perl program to replay the displayed text; and an X11 playback perl program to redisplay the X11 session in realtime or by a factor - although the most interesting feature it is currently the least robust (this example requires a 24 bit display). See troubleshooting below.
• Chaosreader example output 7 This demonstrates processing VNC traffic. Here I connect to a VNC session and type in a few commands. Chaosreader creates the following: a html report containing the keystrokes; a keystroke playback program. But neither are too interesting as they lack the reply text (it's been pixelated). What is interesting is the VNC playback perl program that redisplays the VNC session in realtime or by a factor. The program is used in conjunction with vncviewer and is experimental. This example requires 24 bit colour support, an 8 bit example is here. See troubleshooting below.
• Chaosreader example output 8 This demonstrates processing SSH traffic. Three connections are captured: an SSH command line session, an X11 session forwarded over SSH, and an SCP copy. From the command session a html report, a playback program, and a keydata file has been created. The encryption is not broken, instead we focus on the data that is available - transfer size and timing. See sshanalysis for a demonstration of further analysis.

chaosreader - latest version (github)
chaosreader0.94 - version 0.94, stable, 29-May-2004


This started life as a small handy perl program, and as it grew I was reluctant to break it into libraries. It still is a handy perl program although the code is massive. It currently remains a single program for ease of use.

• Chaosreader on github - for the latest development version.
• http://www.brendangregg.com/chaosreader.html - (this) Latest version, documentation, screenshots, troubleshooting.
• http://sourceforge.net/projects/chaosreader - Latest version, older versions, change log.
• http://chaosreader.sourceforge.net - Summary site.

This has been tested on Solaris 9, RedHat 9 Linux and Windows 98. Perl version 5.6 is required.

Feature List

(New features are in red)

* Reads Solaris snoop logs and four versions of tcpdump/libpcap logs
* Standalone mode generates a series of logs and then processes those
* Processes HTTP, FTP, telnet, SMTP, IRC, ... application protocols
* Processes any TCP and UDP traffic
* Processes 802.11b wireless traffic
* Processes PPPoE traffic, tun device traffic
* Retrieves transferred files from FTP and HTTP traffic
* Creates HTML and text reports to list contents of the log
* Creates realtime replay programs for telnet or IRC sessions
* Creates red/blue coloured HTML reports for 2-way sessions such as telnet and FTP
* Creates red/blue coloured HTML reports for any TCP, UDP or ICMP traffic
* Creates image reports from HTTP, FTP transfers
* Creates HTTP GET and POST reports from queries
* Creates red/blue coloured HTML hex dumps for any TCP, UDP or ICMP traffic
* Creates plain text hex dumps for any TCP, UDP or ICMP traffic
* Creates HTTP proxy logs based on observed HTTP traffic, using the squid log format
* Creates X11 realtime replay programs to playback an X11 session. (experimental).
* Creates red/blue coloured HTML reports for X11 text and keystrokes.
* Creates realtime replay programs for X11 text communication.
* Creates VNC realtime replay programs to playback a VNC session. (experimental).
* Creates HTML reports for VNC keystrokes.
* Creates realtime replay programs for VNC keystrokes.
* SSH content analysis. reports, replays and keystroke delay data files.
* Creates raw data files from TCP or UDP transfers
* Supports TCP out of sequence number delivery
* Supports IP fragmentation
* Supports IPv4 and IPv6
* Processes ICMP and ICMPv6
* Very configurable (including filtering on IPs and ports)
* Can sort data based on time, size, type or IP.
* Can skip sessions smaller than a min size.
* Runs on Solaris, RedHat, Windows, ...

I plan to add many more protocols, especially freeware/opensource protocols.

This temporary news section is here to explain what may appear in future versions.
I'm currently working on wireless protocols, reducing memory usage, and usability features.
You could help by emailing a small sample snoop/tcpdump dump of wireless (or any odd) traffic to [email protected] .

Troubleshooting

Just as a note, the most reliable client I've used to playback X11 has been RedHat 9 Linux; The most reliable client to playback VNC has been Solaris 9;
The most reliable playbacks in general are those on the same OS and hardware as the original.

1. X11 playbacks, although these are experimental here are some things to check,

• Nothing Happens. - Did you capture the start of the connection? This is necessary to draw the initial windows.
• Too big? - Large X11 sessions are problematic. Experiment by capturing and playing back something simple first, eg an "xterm", "xclock" or "xeyes". Check the size of the replay script, I've yet to see one larger than 500 Kb actually work. Check for newer versions of chaosreader that may improve on this.
• Authorised to connect? - Check your xhost settings on your target server. eg, "xhost +hostname" can authorise the hostname.
• Colour depth ok? - A 24 bit colour playback won't display on an 8 bit screen, and visa versa. Read the text the playback script prints during connection to see both the server and the playback depth. It will help if you can change your screen depth to match. I use an extra VNC window as the target screen, eg "/usr/local/bin/Xvnc :2 -geometry 700x600 -depth 16 & DISPLAY=mars:2 dtwm -display :2 &" and then direct the script to port 6002 ("-p 6002") while vncviewer is displaying it.
• Can't connect to server? - try using the options "-d" and "-p", eg, "./session_0001.X11.replay -d mars -p 6000"

• Red text garbeled? - If the red text (keystrokes) are garbeled, then the playback contains a strange keymap. You can view the keymap using "xmodmap -pke" on the client host. Check for newer versions of chaosreader.

• vncviewer prompts for a password? - Just hit 1 character (anything) and then enter. The playback will continue.
• the colours are a little odd - Sometimes the playback will appear to have a tinge of green or blue, this usually happens when a session from one desktop is replayed on another with different video settings. This is normal (so far).
• The colours are VERY odd - if you just get a mess of ranbow colours that slowly write from top down, you are probably playing back a 8 bit or 24 bit depth session on a video display with a different depth. One solution can be to change your display depth. Or try forcing the depth in vncviewer using "-depth", new versions have extra options such as "-8bit" (for 8 bit playbacks) which can definately help. If you are on a unix style vncviewer, you can read the requested and delivered colour depth values during the connection.
• vncviewer crashes - "Rect too large". I found this was a depth negotiation problem, as written above.
• The screen draws okay then blinks white - This usually happens when the vnc session window is larger than your current screen (eg replaying a 1024x768 session on a 800x600 screen). Try increasing your screen resolution.
• vncviewer crashes - "Runtime Error". Try upgrading to the latest version of vncviewer. You may have tried to display a newer session on an older client.
• "Please Wait - initial screen loading" - is all I see, it never seems to load. I've found this was due to a newer version of vncviewer autonegotiating "Format and Encoding" incorrectly. In the vncviewer options, disable "Auto select" and manually try encoding, eg "Hextile".

• Try running chaosreader without the extra report options such as "-ve", as these use much more memory to process.

• Try the minsize option, eg "-m 1k" to only save sessions that are at least 1 Kb.
• Try to exclude data you are not interested in using the include/exclude port options or the include/exclude IP address options. See "chaosreader --help".