NAME

PandoraFMS::NmapParser - parse nmap scan data with perl

SYNOPSIS

  use PandoraFMS::NmapParser;
  my $np = new PandoraFMS::NmapParser;
  
  $np->parsescan($nmap_path, $nmap_args, @ips);
    #or
  $np->parsefile($file_xml);
  
  my $session    = $np->get_session();
    #a PandoraFMS::NmapParser::Session object
    
  my $host       = $np->get_host($ip_addr);
    #a PandoraFMS::NmapParser::Host object
    
  my $service = $host->tcp_service(80);
    #a PandoraFMS::NmapParser::Host::Service object
    
  my $os         = $host->os_sig();
    #a PandoraFMS::NmapParser::Host::OS object
 
 #---------------------------------------
 
 my $np2 = new PandoraFMS::NmapParser;
 
 $np2->callback(\&my_callback);
 
 $np2->parsefile($file_xml);
    #or
 $np2->parsescan($nmap_path, $nmap_args, @ips);
    
 sub my_callback {
 
   my $host = shift;
    #PandoraFMS::NmapParser::Host object
    #.. see documentation for all methods ...

 }

For a full listing of methods see the documentation corresponding to each object.

DESCRIPTION

This module implements a interface to the information contained in an nmap scan. It is implemented by parsing the xml scan data that is generated by nmap. This will enable anyone who utilizes nmap to quickly create fast and robust security scripts that utilize the powerful port scanning abilities of nmap.

The latest version of this module can be found on here <http://apersaud.github.com/Nmap-Parser/>

OVERVIEW

This module has an internal framework to make it easy to retrieve the desired information of a scan. Every nmap scan is based on two main sections of informations: the scan session, and the scan information of all hosts. The session information will be stored as a PandoraFMS::NmapParser::Session object. This object will contain its own methods to obtain the desired information. The same is true for any hosts that were scanned using the PandoraFMS::NmapParser::Host object. There are two sub objects under PandoraFMS::NmapParser::Host. One is the PandoraFMS::NmapParser::Host::Service object which will be used to obtain information of a given service running on a given port. The second is the PandoraFMS::NmapParser::Host::OS object which contains the operating system signature information (OS guessed names, classes, osfamily..etc).

  PandoraFMS::NmapParser                        -- Core parser
     |
     +--PandoraFMS::NmapParser::Session         -- Nmap scan session information
     |  
     +--PandoraFMS::NmapParser::Host            -- General host information
     |  |
     |  |-PandoraFMS::NmapParser::Host::Service -- Port service information
     |  |
     |  |-PandoraFMS::NmapParser::Host::OS      -- Operating system signature information

METHODS

PandoraFMS::NmapParser

The main idea behind the core module is, you will first parse the information and then extract data. Therefore, all parse*() methods should be executed before any get_*() methods.

parse($string)

parse($filehandle)

Parses the nmap scan information in $string. Note that is usually only used if you have the whole xml scan information in $string or if you are piping the scan information.

parsefile($xml_file)

Parses the nmap scan data in $xml_file. This file can be generated from an nmap scan by using the '-oX filename.xml' option with nmap. If you get an error or your program dies due to parsing, please check that the xml information is compliant. The file is closed no matter how "parsefile()" returns.

parsescan($nmap,$args,@ips)

This method runs an nmap scan where $nmap is the path to the nmap executable or binary, $args are the nmap command line parameters, and @ips are the list of IP addresses to scan. parsescan() will automagically run the nmap scan and parse the information.

If you wish to save the xml output from parsescan(), you must call cache_scan() method BEFORE you start the parsescan() process. This is done to conserve memory while parsing. cache_scan() will let PandoraFMS::NmapParser know to save the output before parsing the xml since PandoraFMS::NmapParser purges everything that has been parsed by the script to conserve memory and increase speed.

See section EXAMPLES for a short tutorial

Note: You cannot have one of the nmap options to be '-oX', '-oN' or '-oG'. Your program will die if you try and pass any of these options because it decides the type of output nmap will generate. The IP addresses can be nmap-formatted addresses see nmap(1)

If you get an error or your program dies due to parsing, please check that the xml information is compliant. If you are using parsescan() or an open filehandle , make sure that the nmap scan that you are performing is successful in returning xml information. (Sometimes using loopback addresses causes nmap to fail).

cache_scan($filename)

This function allows you to save the output of a parsescan() (or nmap scan) to the disk. $filename is the name of the file you wish to save the nmap scan information to. It defaults to nmap-parser-cache.xml It returns the name of the file to be used as the cache.

 #Must be called before parsescan().
 $np->cache_scan($filename); #output set to nmap-parser-cache.xml

 #.. do other stuff to prepare for parsescan(), ex. setup callbacks

 $np->parsescan('/usr/bin/nmap',$args,@IPS);

purge()

Cleans the xml scan data from memory. This is useful if you have a program where you are parsing lots of nmap scan data files with persistent variables.

callback(\&code_ref)

Sets the parsing mode to be done using the callback function. It takes the parameter of a code reference or a reference to a function. If no code reference is given, it resets the mode to normal (no callback).

 $np->callback(\&my_function); #sets callback, my_function() will be called
 $np->callback(); #resets it, no callback function called. Back to normal.

get_session()

Obtains the PandoraFMS::NmapParser::Session object which contains the session scan information.

get_host($ip_addr)

Obtains the PandoraFMS::NmapParser::Host object for the given $ip_addr.

del_host($ip_addr)

Deletes the stored PandoraFMS::NmapParser::Host object whose IP is $ip_addr.

all_hosts()

all_hosts($status)

Returns an array of all the PandoraFMS::NmapParser::Host objects for the scan. If the optional status is given, it will only return those hosts that match that status. The status can be any of the following: "(up|down|unknown|skipped)"

get_ips()

get_ips($status)

Returns the list of IP addresses that were scanned in this nmap session. They are sorted using addr_sort. If the optional status is given, it will only return those IP addresses that match that status. The status can be any of the following: "(up|down|unknown|skipped)"

addr_sort(@ips)

This function takes a list of IP addresses and returns the correctly sorted version of the list.

PandoraFMS::NmapParser::Session

This object contains the scan session information of the nmap scan.

finish_time(): Returns the numeric time that the nmap scan finished.
nmap_version(): Returns the version of nmap used for the scan.
numservices()
numservices($type): If numservices is called without argument, it returns the total number of services that were scanned for all types. If $type is given, it returns the number of services for that given scan type. See scan_types() for more info.
scan_args(): Returns a string which contains the nmap executed command line used to run the scan.
scan_type_proto($type): Returns the protocol type of the given scan type (provided by $type). See scan_types() for more info.
scan_types(): Returns the list of scan types that were performed. It can be any of the following: "(syn|ack|bounce|connect|null|xmas|window|maimon|fin|udp|ipproto)".
start_str(): Returns the human readable format of the start time.
start_time(): Returns the numeric form of the time the nmap scan started.
time_str(): Returns the human readable format of the finish time.
xml_version(): Returns the version of nmap xml file.
prescripts()
prescripts($name): A basic call to prescripts() returns a list of the names of the NSE scripts run in the pre-scanning phase. If $name is given, it returns the text output of the a reference to a hash with "output" and "content" keys for the script with that name, or undef if that script was not run. The value of the "output" key is the text output of the script. The value of the "content" key is a data structure based on the XML output of the NSE script.
postscripts()
postscripts($name): A basic call to postscripts() returns a list of the names of the NSE scripts run in the post-scaning phase. If $name is given, it returns the text output of the a reference to a hash with "output" and "content" keys for the script with that name, or undef if that script was not run. The value of the "output" key is the text output of the script. The value of the "content" key is a data structure based on the XML output of the NSE script.

PandoraFMS::NmapParser::Host

This object represents the information collected from a scanned host.

status()

Returns the state of the host. It is usually one of these "(up|down|unknown|skipped)".

addr()

Returns the main IP address of the host. This is usually the IPv4 address. If there is no IPv4 address, the IPv6 is returned (hopefully there is one).

addrtype()

Returns the address type of the address given by addr() .

all_hostnames()

Returns a list of all hostnames found for the given host.

extraports_count()

Returns the number of extraports found.

extraports_state()

Returns the state of all the extraports found.

hostname()

hostname($index)

As a basic call, hostname() returns the first hostname obtained for the given host. If there exists more than one hostname, you can provide a number, which is used as the location in the array. The index starts at 0;

 #in the case that there are only 2 hostnames
 hostname() eq hostname(0);
 hostname(1); #second hostname found
 hostname(400) eq hostname(1) #nothing at 400; return the name at the last index

ipv4_addr()

Explicitly return the IPv4 address.

ipv6_addr()

Explicitly return the IPv6 address.

mac_addr()

Explicitly return the MAC address.

mac_vendor()

Return the vendor information of the MAC.

distance()

Return the distance (in hops) of the target machine from the machine that performed the scan.

trace_error()

Returns a true value (usually a meaningful error message) if the traceroute was performed but could not reach the destination. In this case "all_trace_hops()" contains only the part of the path that could be determined.

all_trace_hops()

Returns an array of PandoraFMS::NmapParser::Host::TraceHop objects representing the path to the target host. This array may be empty if Nmap did not perform the traceroute for some reason (same network, for example).

Some hops may be missing if Nmap could not figure out information about them. In this case there is a gap between the "ttl()" values of consecutive returned hops. See also "trace_error()".

trace_proto()

Returns the name of the protocol used to perform the traceroute.

trace_port()

Returns the port used to perform the traceroute.

os_sig()

Returns an PandoraFMS::NmapParser::Host::OS object that can be used to obtain all the Operating System signature (fingerprint) information. See PandoraFMS::NmapParser::Host::OS for more details.

 $os = $host->os_sig;
 $os->name;
 $os->osfamily;

tcpsequence_class()

tcpsequence_index()

tcpsequence_values()

Returns the class, index and values information respectively of the tcp sequence.

ipidsequence_class()

ipidsequence_values()

Returns the class and values information respectively of the ipid sequence.

tcptssequence_class()

tcptssequence_values()

Returns the class and values information respectively of the tcpts sequence.

uptime_lastboot()

Returns the human readable format of the timestamp of when the host had last rebooted.

uptime_seconds()

Returns the number of seconds that have passed since the host's last boot from when the scan was performed.

hostscripts()

hostscripts($name)

A basic call to hostscripts() returns a list of the names of the host scripts run. If $name is given, it returns the text output of the a reference to a hash with "output" and "content" keys for the script with that name, or undef if that script was not run. The value of the "output" key is the text output of the script. The value of the "content" key is a data structure based on the XML output of the NSE script.

tcp_ports()

udp_ports()

Returns the sorted list of TCP|UDP ports respectively that were scanned on this host. Optionally a string argument can be given to these functions to filter the list.

 $host->tcp_ports('open') #returns all only 'open' ports (even 'open|filtered')
 $host->udp_ports('open|filtered'); #matches exactly ports with 'open|filtered'

Note that if a port state is set to 'open|filtered' (or any combination), it will be counted as an 'open' port as well as a 'filtered' one.

tcp_port_count()

udp_port_count()

Returns the total of TCP|UDP ports scanned respectively.

tcp_del_ports($portid, [$portid, ...])

udp_del_ports($portid, [ $portid, ...])

Deletes the current $portid from the list of ports for given protocol.

tcp_port_state($portid)

udp_port_state($portid)

Returns the state of the given port, provided by the port number in $portid.

tcp_open_ports()

udp_open_ports()

Returns the list of open TCP|UDP ports respectively. Note that if a port state is for example, 'open|filtered', it will appear on this list as well.

tcp_filtered_ports()

udp_filtered_ports()

Returns the list of filtered TCP|UDP ports respectively. Note that if a port state is for example, 'open|filtered', it will appear on this list as well.

tcp_closed_ports()

udp_closed_ports()

Returns the list of closed TCP|UDP ports respectively. Note that if a port state is for example, 'closed|filtered', it will appear on this list as well.

tcp_service($portid)

udp_service($portid)

Returns the PandoraFMS::NmapParser::Host::Service object of a given service running on port, provided by $portid. See PandoraFMS::NmapParser::Host::Service for more info.

 $svc = $host->tcp_service(80);
 $svc->name;
 $svc->proto;

PandoraFMS::NmapParser::Host::Service

This object represents the service running on a given port in a given host. This object is obtained by using the tcp_service($portid) or udp_service($portid) method from the PandoraFMS::NmapParser::Host object. If a portid is given that does not exist on the given host, these functions will still return an object (so your script doesn't die). Its good to use tcp_ports() or udp_ports() to see what ports were collected.

confidence(): Returns the confidence level in service detection.
extrainfo(): Returns any additional information nmap knows about the service.
method(): Returns the detection method.
name(): Returns the service name.
owner(): Returns the process owner of the given service. (If available)
port(): Returns the port number where the service is running on.
product(): Returns the product information of the service.
proto(): Returns the protocol type of the service.
rpcnum(): Returns the RPC number.
tunnel(): Returns the tunnel value. (If available)
fingerprint(): Returns the service fingerprint. (If available)
version(): Returns the version of the given product of the running service.
scripts()
scripts($name): A basic call to scripts() returns a list of the names of the NSE scripts run for this port. If $name is given, it returns a reference to a hash with "output" and "content" keys for the script with that name, or undef if that script was not run. The value of the "output" key is the text output of the script. The value of the "content" key is a data structure based on the XML output of the NSE script.

PandoraFMS::NmapParser::Host::OS

This object represents the Operating System signature (fingerprint) information of the given host. This object is obtained from an PandoraFMS::NmapParser::Host object using the "os_sig()" method. One important thing to note is that the order of OS names and classes are sorted by DECREASING ACCURACY. This is more important than alphabetical ordering. Therefore, a basic call to any of these functions will return the record with the highest accuracy. (Which is probably the one you want anyways).

all_names(): Returns the list of all the guessed OS names for the given host.
class_accuracy()
class_accuracy($index): A basic call to class_accuracy() returns the osclass accuracy of the first record. If $index is given, it returns the osclass accuracy for the given record. The index starts at 0.
class_count(): Returns the total number of OS class records obtained from the nmap scan.
name()
name($index)
names()
names($index): A basic call to name() returns the OS name of the first record which is the name with the highest accuracy. If $index is given, it returns the name for the given record. The index starts at 0.
name_accuracy()
name_accuracy($index): A basic call to name_accuracy() returns the OS name accuracy of the first record. If $index is given, it returns the name for the given record. The index starts at 0.
name_count(): Returns the total number of OS names (records) for the given host.
osfamily()
osfamily($index): A basic call to osfamily() returns the OS family information of the first record. If $index is given, it returns the OS family information for the given record. The index starts at 0.
osgen()
osgen($index): A basic call to osgen() returns the OS generation information of the first record. If $index is given, it returns the OS generation information for the given record. The index starts at 0.
portused_closed(): Returns the closed port number used to help identify the OS signatures. This might not be available for all hosts.
portused_open(): Returns the open port number used to help identify the OS signatures. This might not be available for all hosts.
os_fingerprint(): Returns the OS fingerprint used to help identify the OS signatures. This might not be available for all hosts.
type()
type($index): A basic call to type() returns the OS type information of the first record. If $index is given, it returns the OS type information for the given record. The index starts at 0.
vendor()
vendor($index): A basic call to vendor() returns the OS vendor information of the first record. If $index is given, it returns the OS vendor information for the given record. The index starts at 0.

PandoraFMS::NmapParser::Host::TraceHop

This object represents a router on the IP path towards the destination or the destination itself. This is similar to what the "traceroute" command outputs.

PandoraFMS::NmapParser::Host::TraceHop objects are obtained through the "all_trace_hops()" and "trace_hop()" PandoraFMS::NmapParser::Host methods.

ttl(): The Time To Live is the network distance of this hop.
rtt(): The Round Trip Time is roughly equivalent to the "ping" time towards this hop. It is not always available (in which case it will be undef).
ipaddr(): The known IP address of this hop.
host(): The host name of this hop, if known.

EXAMPLES

I think some of us best learn from examples. These are a couple of examples to help create custom security audit tools using some of the nice features of the PandoraFMS::NmapParser module. Hopefully this can double as a tutorial. More tutorials (articles) can be found at <http://apersaud.github.com/Nmap-Parser/>

Real-Time Scanning

You can run a nmap scan and have the parser parse the information automagically. The only constraint is that you cannot use '-oX', '-oN', or '-oG' as one of your arguments for nmap command line parameters passed to parsescan().

 use PandoraFMS::NmapParser;

 my $np = new PandoraFMS::NmapParser;
 my @hosts = @ARGV; #get hosts from cmd line

 #runs the nmap command with hosts and parses it automagically
 $np->parsescan('/usr/bin/nmap','-sS O -p 1-1023',@hosts);

 for my $host ($np->all_hosts()){
        print $host->hostname."\n";
        #do mor stuff...
 }

If you would like to run the scan using parsescan() but also save the scan xml output, you can use cache_scan(). You must call cache_scan() BEFORE you initiate the parsescan() method.

 use PandoraFMS::NmapParser;
 my $np = new PandoraFMS::NmapParser;

 #telling np to save output
 $np->cache_scan('nmap.localhost.xml'); 
 $np->parsescan('/usr/bin/nmap','-F','localhost');
 #do other stuff...

Callbacks

This is probably the easiest way to write a script with using PandoraFMS::NmapParser, if you don't need the general scan session information. During the parsing process, the parser will obtain information of every host. The callback function (in this case 'booyah()') is called after the parsing of every host (sequentially). When the callback returns, the parser will delete all information of the host it had sent to the callback. This callback function is called for every host that the parser encounters. The callback function must be setup before parsing

 use PandoraFMS::NmapParser;
 my $np = new PandoraFMS::NmapParser;

 
 $np->callback( \&booyah );
 
 $np->parsefile('nmap_results.xml');
    # or use parsescan()

 sub booyah {
    my $host = shift; #PandoraFMS::NmapParser::Host object, just parsed
    print 'IP: ',$host->addr,"\n";
         # ... do more stuff with $host ...

    #when it returns, host object will be deleted from memory
    #(good for processing VERY LARGE files or scans)
 }

Multiple Instances - ("no less 'of'; my $self")

Using multiple instances of PandoraFMS::NmapParser is extremely useful in helping audit/monitor the network Policy (ohh noo! its that 'P' word!). In this example, we have a set of hosts that had been scanned previously for tcp services where the image was saved in base_image.xml. We now will scan the same hosts, and compare if any new tcp have been open since then (good way to look for suspicious new services). Easy security Compliance detection. (ooh noo! The 'C' word too!).

 use PandoraFMS::NmapParser;
 use vars qw($nmap_exe $nmap_args @ips);
 my $base = new PandoraFMS::NmapParser;
 my $curr = new PandoraFMS::NmapParser;
 
 
 $base->parsefile('base_image.xml'); #load previous state
 $curr->parsescan($nmap_exe, $nmap_args, @ips); #scan current hosts
 
 for my $ip ($curr->get_ips ) 
 {
        #assume that IPs in base == IPs in curr scan
        my $ip_base = $base->get_host($ip);
        my $ip_curr = $curr->get_host($ip);
        my %port = ();
        
        #find ports that are open that were not open before
        #by finding the difference in port lists
        my @diff =  grep { $port{$_} < 2} 
                   (map {$port{$_}++; $_} 
                   ( $ip_curr->tcp_open_ports , $ip_base->tcp_open_ports ));
        
        print "$ip has these new ports open: ".join(',',@diff) if(scalar @diff);
        
        for (@diff){print "$_ seems to be ",$ip_curr->tcp_service($_)->name,"\n";}
                
 }

SUPPORT

Discussion Forum

If you have questions about how to use the module, or any of its features, you can post messages to the PandoraFMS::NmapParser module forum on CPAN::Forum. <https://github.com/apersaud/Nmap-Parser/issues>

Bug Reports and Enhancements

Please submit any bugs or feature requests to: <https://github.com/apersaud/Nmap-Parser/issues>

Please make sure that you submit the xml-output file of the scan which you are having trouble with. This can be done by running your scan with the -oX filename.xml nmap switch. Please remove any important IP addresses for security reasons. It saves time in reproducing issues.

AUTHORS

Anthony G Persaud <http://anthonypersaud.com> . Please see Changes file and CONTRIBUTORS file for a list of other great contributors.

Additional Contributors: * Robin Bowes <http://robinbowes.com> * Daniel Miller <https://github.com/bonsaiviking> * See Changes file for other contributors.

COPYRIGHT

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.