squidmagic - Analyze a Web-Based Network Traffic to Detect Central Command and Control (C&C) Servers and Malicious Site

squidmagic is a tool designed to analyze a web-based network traffic to detect central command and control (C&C) servers and Malicious site, using Squid proxy server and Spamhaus.

usage

squidmagic # python squidmagic.py /var/log/squid3/access.log

                 _     _                       _      
                (_)   | |                     (_)     
 ___  __ _ _   _ _  __| |_ __ ___   __ _  __ _ _  ___ 
/ __|/ _` | | | | |/ _` | '_ ` _ \ / _` |/ _` | |/ __|
\__ \ (_| | |_| | | (_| | | | | | | (_| | (_| | | (__ 
|___/\__, |\__,_|_|\__,_|_| |_| |_|\__,_|\__, |_|\___|
        | |                               __/ |       
        |_|                              |___/        
     Analyzing...

Analyzing by SBL Advisory...
    Spam server detected, ip is 65.182.101.221
Analyzing by SBL_CSS Advisory...
    safe server detected, host or ip is 65.182.101.221
Analyzing by PBL Advisory...
    safe server detected, host or ip is 65.182.101.221

Run server

<?php
namespace SquidApp\Core;

require dirname(__DIR__) . '/lib/vendor/autoload.php';

$banner = new \SquidApp\Squid();
$squidmagic = new FileSystem();

// output banner
echo $banner->bannerAction();

// Scans a directory for files
$squidmagic->scandirs('squidmagic/Collector path');

// Checks if file exists in certain location 
$squidmagic->fileExists('Collector Path/server.php');

// run server
$squidmagic->openInBackground('Collector Path/lib/bin/');

squidmagic/lib # php squidmagic.php 


                              | |                          
         ___  __ _ _   _ _  __| |_ __ ___   __ _  __ _ _  ___ 
        / __|/ _` | | | | |/ _` | '_ ` _ \ / _` |/ _` | |/ __|
        \__ \ (_| | |_| | | (_| | | | | | | (_| | (_| | | (__ 
        |___/\__, |\__,_|_|\__,_|_| |_| |_|\__,_|\__, |_|\___|
                | |                               __/ |       
                |_|                              |___/ 
                    squidmagic collector started   

Download squidmagic

Read More

Dshell - Network Forensic Analysis Framework

An extensible network forensic analysis framework. Enables rapid development of plugins to support the dissection of network packet captures.

Key features:

• Robust stream reassembly
• IPv4 and IPv6 support
• Custom output handlers
• Chainable decoders

Prerequisites

• Linux (developed on Ubuntu 12.04)
• Python 2.7
• pygeoip, GNU Lesser GPL
  • MaxMind GeoIP Legacy datasets
• PyCrypto, custom license
• dpkt, New BSD License
• IPy, BSD 2-Clause License
• pypcap, New BSD License

Installation

1. Install all of the necessary Python modules listed above. Many of them are available via pip and/or apt-get. Pygeoip is not yet available as a package and must be installed with pip or manually. All except dpkt are available with pip.
   1. sudo apt-get install python-crypto python-dpkt python-ipy python-pypcap
   2. sudo pip install pygeoip
2. Configure pygeoip by moving the MaxMind data files (GeoIP.dat, GeoIPv6.dat, GeoIPASNum.dat, GeoIPASNumv6.dat) to /share/GeoIP/
3. Run make. This will build Dshell.
4. Run ./dshell. This is Dshell. If you get a Dshell> prompt, you're good to go!

Basic usage

• decode -l
  • This will list all available decoders alongside basic information about them
• decode -h
  • Show generic command-line flags available to most decoders
• decode -d <decoder>
  • Display information about a decoder, including available command-line flags
• decode -d <decoder> <pcap>
  • Run the selected decoder on a pcap file

Usage Examples

Showing DNS lookups in sample traffic

Dshell> decode -d dns ~/pcap/dns.cap
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 39867 PTR? 66.192.9.104 / PTR: 66-192-9-104.gen.twtelecom.net **
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 30144 A? www.netbsd.org / A: 204.152.190.12 (ttl 82159s) **
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 61652 AAAA? www.netbsd.org / AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86400s) **
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 32569 AAAA? www.netbsd.org / AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86340s) **
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 36275 AAAA? www.google.com / CNAME: www.l.google.com **
dns 2005-03-30 03:47:46    192.168.170.8:32795 ->   192.168.170.20:53    ** 9837 AAAA? www.example.notginh / NXDOMAIN **
dns 2005-03-30 03:52:17    192.168.170.8:32796 <-   192.168.170.20:53    ** 23123 PTR? 127.0.0.1 / PTR: localhost **
dns 2005-03-30 03:52:25   192.168.170.56:1711  <-      217.13.4.24:53    ** 30307 A? GRIMM.utelsystems.local / NXDOMAIN **
dns 2005-03-30 03:52:17   192.168.170.56:1710  <-      217.13.4.24:53    ** 53344 A? GRIMM.utelsystems.local / NXDOMAIN **

Following and reassembling a stream in sample traffic

Dshell> decode -d followstream ~/pcap/v6-http.cap
Connection 1 (TCP)
Start: 2007-08-05 19:16:44.189852 UTC
  End: 2007-08-05 19:16:44.204687 UTC
2001:6f8:102d:0:2d0:9ff:fee3:e8de:59201 -> 2001:6f8:900:7c0::2:80 (240 bytes)
2001:6f8:900:7c0::2:80 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de:59201 (2259 bytes)

GET / HTTP/1.0
Host: cl-1985.ham-01.de.sixxs.net
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Encoding: gzip, bzip2
Accept-Language: en
User-Agent: Lynx/2.8.6rel.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.8b

HTTP/1.1 200 OK
Date: Sun, 05 Aug 2007 19:16:44 GMT
Server: Apache
Content-Length: 2121
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
<pre><img src="/icons/blank.gif" alt="Icon "> <a href="?C=N;O=D">Name</a>                    <a href="?C=M;O=A">Last modified</a>      <a href="?C=S;O=A">Size</a>  <a href="?C=D;O=A">Description</a><hr><img src="/icons/folder.gif" alt="[DIR]"> <a href="202-vorbereitung/">202-vorbereitung/</a>       06-Jul-2007 14:31    -   
<img src="/icons/layout.gif" alt="[   ]"> <a href="Efficient_Video_on_demand_over_Multicast.pdf">Efficient_Video_on_d..&gt;</a> 19-Dec-2006 03:17  291K  
<img src="/icons/unknown.gif" alt="[   ]"> <a href="Welcome%20Stranger!!!">Welcome Stranger!!!</a>     28-Dec-2006 03:46    0   
<img src="/icons/text.gif" alt="[TXT]"> <a href="barschel.htm">barschel.htm</a>            31-Jul-2007 02:21   44K  
<img src="/icons/folder.gif" alt="[DIR]"> <a href="bnd/">bnd/</a>                    30-Dec-2006 08:59    -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="cia/">cia/</a>                    28-Jun-2007 00:04    -   
<img src="/icons/layout.gif" alt="[   ]"> <a href="cisco_ccna_640-801_command_reference_guide.pdf">cisco_ccna_640-801_c..&gt;</a> 28-Dec-2006 03:48  236K  
<img src="/icons/folder.gif" alt="[DIR]"> <a href="doc/">doc/</a>                    19-Sep-2006 01:43    -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="freenetproto/">freenetproto/</a>           06-Dec-2006 09:00    -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="korrupt/">korrupt/</a>                03-Jul-2007 11:57    -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="mp3_technosets/">mp3_technosets/</a>         04-Jul-2007 08:56    -   
<img src="/icons/text.gif" alt="[TXT]"> <a href="neues_von_rainald_goetz.htm">neues_von_rainald_go..&gt;</a> 21-Mar-2007 23:27   31K  
<img src="/icons/text.gif" alt="[TXT]"> <a href="neues_von_rainald_goetz0.htm">neues_von_rainald_go..&gt;</a> 21-Mar-2007 23:29   36K  
<img src="/icons/layout.gif" alt="[   ]"> <a href="pruef.pdf">pruef.pdf</a>               28-Dec-2006 07:48   88K  
<hr></pre>
</body></html>

Chaining decoders to view flow data for a specific country code in sample traffic (note: TCP handshakes are not included in the packet count)

Dshell> decode -d country+netflow --country_code=JP ~/pcap/SkypeIRC.cap
2006-08-25 19:32:20.651502       192.168.1.2 ->  202.232.205.123  (-- -> JP)  UDP   60583   33436     1      0       36        0  0.0000s
2006-08-25 19:32:20.766761       192.168.1.2 ->  202.232.205.123  (-- -> JP)  UDP   60583   33438     1      0       36        0  0.0000s
2006-08-25 19:32:20.634046       192.168.1.2 ->  202.232.205.123  (-- -> JP)  UDP   60583   33435     1      0       36        0  0.0000s
2006-08-25 19:32:20.747503       192.168.1.2 ->  202.232.205.123  (-- -> JP)  UDP   60583   33437     1      0       36        0  0.0000s

Collecting netflow data for sample traffic with vlan headers, then tracking the connection to a specific IP address

Dshell> decode -d netflow ~/pcap/vlan.cap
1999-11-05 18:20:43.170500    131.151.20.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:42.063074     131.151.32.71 ->   131.151.32.255  (US -> US)  UDP     138     138     1      0      201        0  0.0000s
1999-11-05 18:20:43.096540     131.151.1.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.079765     131.151.5.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:41.521798    131.151.104.96 ->  131.151.107.255  (US -> US)  UDP     137     137     3      0      150        0  1.5020s
1999-11-05 18:20:43.087010     131.151.6.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.368210   131.151.111.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.250410    131.151.32.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.115330    131.151.10.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.375145   131.151.115.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:43.363348   131.151.107.254 ->  255.255.255.255  (US -> --)  UDP     520     520     1      0       24        0  0.0000s
1999-11-05 18:20:40.112031      131.151.5.55 ->    131.151.5.255  (US -> US)  UDP     138     138     1      0      201        0  0.0000s
1999-11-05 18:20:43.183825     131.151.32.79 ->   131.151.32.255  (US -> US)  UDP     138     138     1      0      201        0  0.0000s

Download Dshell

Read More

CapTipper - Malicious HTTP traffic explorer tool

CapTipper is a python tool to analyze, explore and revive HTTP malicious traffic.

CapTipper sets up a web server that acts exactly as the server in the PCAP file, and contains internal tools, with a powerful interactive console, for analysis and inspection of the hosts, objects and conversations found.

The tool provides the security researcher with easy access to the files and the understanding of the network flow,and is useful when trying to research exploits, pre-conditions, versions, obfuscations, plugins and shellcodes.

Feeding CapTipper with a drive-by traffic capture (e.g of an exploit kit) displays the user with the requests URI's that were sent and responses meta-data.

The user can at this point browse to http://127.0.0.1/[URI] and receive the response back to the browser.

In addition, an interactive shell is launched for deeper investigation using various commands such as: hosts, hexdump, info, ungzip, body, client, dump and more...

Download CapTipper

Read More

NetHogs - Small 'net top' tool

NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. NetHogs does not rely on a special kernel module to be loaded. If there's suddenly a lot of network traffic, you can fire up NetHogs and immediately see which PID is causing this. This makes it easy to indentify programs that have gone wild and are suddenly taking up your bandwidth.

Since NetHogs heavily relies on /proc, it currently runs on Linux only.

 Now supported:

• Shows TCP download- and upload-speed per process
• Supports both IPv4 and IPv6
• Supports both Ethernet and PPP

Ideas/ToDo for new releases:

• Incoming UDP packets?
• Sort the output by other values than network usage
• Monitor specific processes
• Make it work correctly on machines with multiple IP addresses
• Integrate into another tool?
• gui? 

Download NetHogs

Read More

Bro - Passive Open-Source Network Traffic Analyzer

While focusing on network security monitoring, Bro provides a comprehensive platform for more general network traffic analysis as well. Well grounded in more than 15 years of research, Bro has successfully bridged the traditional gap between academia and operations since its inception. Today, it is relied upon operationally in particular by many scientific environments for securing their cyberinfrastructure. Bro’s user community includes major universities, research labs, supercomputing centers, and open-science communities.

Bro is a passive, open-source network traffic analyzer. It is primarily a security monitor that inspects all traffic on a link in depth for signs of suspicious activity. More generally, however, Bro supports a wide range of traffic analysis tasks even outside of the security domain, including performance measurements and helping with trouble-shooting.

Features

• Deployment
• Runs on commodity hardware on standard UNIX-style systems (including Linux, FreeBSD, and MacOS).
• Fully passive traffic analysis off a network tap or monitoring port.
• Standard libpcap interface for capturing packets.
• Real-time and offline analysis.
• Cluster-support for large-scale deployments.
• Unified management framework for operating both standalone and cluster setups.
• Open-source under a BSD license.
• Analysis
• Comprehensive logging of activity for offline analysis and forensics.
• Port-independent analysis of application-layer protocols.
• Support for many application-layer protocols (including DNS, FTP, HTTP, IRC, SMTP, SSH, SSL).
• Analysis of file content exchanged over application-layer protocols, including MD5/SHA1 computation for fingerprinting.
• Comprehensive IPv6 support.
• Tunnel detection and analysis (including Ayiya, Teredo, GTPv1). Bro decapsulates the tunnels and then proceeds to analyze their content as if no tunnel was in place.
• Extensive sanity checks during protocol analysis.
• Support for IDS-style pattern matching.
• Scripting Language
• Turing-complete language for expression arbitrary analysis tasks.
• Event-based programming model.
• Domain-specific data types such as IP addresses (transparently handling both IPv4 and IPv6), port numbers, and timers.
• Extensive support for tracking and managing network state over time.
• Interfacing
• Default output to well-structured ASCII logs.
• Alternative backends for ElasticSearch and DataSeries. Further database interfaces in preparation.
• Real-time integration of external input into analyses. Live database input in preparation.
• External C library for exchanging Bro events with external programs. Comes with Perl, Python, and Ruby bindings.
• Ability to trigger arbitrary external processes from within the scripting language.

Download Bro

Read More