rwpmaplookup(1) SiLK Tool Suite rwpmaplookup(1)NAMErwpmaplookup - Map keys to prefix map entries
SYNOPSISrwpmaplookup { --map-file=MAP_FILE | --address-types[=MAP_FILE]
| --country-codes[=MAP_FILE] }
[--fields=FIELDS] [--ipset-files] [--no-errors]
[--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
[--no-titles] [--no-columns] [--column-separator=CHAR]
[--no-final-delimiter] [{--delimited | --delimited=CHAR}]
[{--output-path=PATH | --pager=PAGER_PROG}]
[--no-files ARG [ARGS...] | --xargs[=FILE] | FILE [FILES...]]
rwpmaplookup--help
rwpmaplookup--version
DESCRIPTIONrwpmaplookup finds keys in a binary prefix map file and prints the key
and its value in a textual, bar (|) delimited format.
By default, rwpmaplookup expects its arguments to be the names of text
files containing keys---one key per line. When the --ipset-files
switch is given, rwpmaplookup takes IPset files as arguments and uses
the IPs as the keys. The --no-files switch causes rwpmaplookup to
treat each command line argument itself as a key to find in the prefix
map.
When --no-files is not specified, rwpmaplookup reads the keys from the
files named on the command line or from the standard input when no file
names are specified and neither --xargs nor --no-files is present. To
read the standard input in addition to the named files, use "-" or
"stdin" as a file name. When the --xargs switch is provided,
rwpmaplookup will read the names of the files to process from the named
text file, or from the standard input if no file name argument is
provided to the switch. The input to --xargs must contain one file
name per line.
You must tell rwpmaplookup the prefix map to use for look-ups using one
of three switches:
· To use an arbitrary prefix map, use the --map-file switch.
· If you want to map IP addresses to country codes (see ccfilter(3)),
use the --country-codes switch. To use the default country code
prefix map, do not provide an argument to the switch. To use a
specific country code mapping file, specify the file as the
argument.
· If you want to map IP addresses to address types (see addrtype(3)),
use the --address-types switch. To use the default address types
prefix map, do not provide an argument to the switch. To use a
specific address types mapping file, specify the file as the
argument.
If the --map-file switch specifies a prefix map containing
protocol/port pairs, each input file should contain one protocol/port
pair per line in the form PROTOCOL/PORT, where PROTOCOL is a number
between 0 and 255 inclusive, and PORT is a number between 0 and 65535
inclusive. When the --ipset-files switch is specified, it is an error
if the --map-file switch specifies a prefix map containing
protocol/port pairs.
When querying any other type of prefix map and the --ipset-files switch
is not present, each textual input file should contain one IP address
per line, where the IP is a single IP address (not a CIDR block) in
canonical form or the integer representation of an IPv4 address.
The --fields switch allows you to specify which columns appear in the
output. The default columns are the key and the value, where the key
is the IP address or protocol/port pair, and the value is the textual
label for that key.
If the prefix map contains IPv6 addresses, any IPv4 address in the
input is mapped into the ::ffff:0:0/96 netblock when searching.
If the prefix map contains IPv4 addresses only, any IPv6 address in the
::ffff:0:0/96 netblock is converted to IPv4 when searching. Any other
IPv6 address is ignored, and it is not printed in the output unless the
"input" field is requested.
Prefix map files are created by the rwpmapbuild(1) and rwgeoip2ccmap(1)
utilities. IPset files are created most often by rwset(1) and
rwsetbuild(1).
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
One of --map-file, --address-types, or --country-codes is required.
--map-file=PMAP_FILE
Find the IP addresses or protocol/port pairs in the prefix map file
PMAP_FILE.
--address-types
Find the IP addresses in the address types (see addrtype(3))
mapping file specified by the SILK_ADDRESS_TYPES environment
variable, or in the default address types mapping file if that
environment variable is not set.
--address-types=ADDRTYPE_FILE
Find the IP addresses in the address types mapping file specified
by ADDRTYPE_FILE.
--country-codes
Find the IP addresses in the country code (see ccfilter(3)) mapping
file specified by the SILK_COUNTRY_CODES environment variable, or
in the default country code mapping file if that environment
variable is not set.
--country-codes=COUNTRY_CODE_FILE
Find the IP addresses in the country code mapping file specified by
COUNTRY_CODE_FILE.
--fields=FIELDS
Specify the columns to include in the output. The columns will be
displayed in the order the fields are specified. FIELDS is a comma
separated list of field-names. Field-names are case-insensitive.
When this switch is not provided, the default fields are
"key,value". The list of available fields are:
key The key used to search the prefix map.
value
The label returned from the prefix map for the key.
block
The block in the prefix map that contains the key. For a
prefix map file that contains IPv4 addresses, the result will
be a CIDR block such as 10.18.26.32/27.
start-block
The value at the start of the block in the prefix map that
contains the key.
end-block
The value at the end of the block in the prefix map that
contains the key.
input
The text read from the input file that rwpmaplookup attempted
to parse. Note that blank lines, lines containing only
whitespace and comments, and lines longer than 2048 characters
will not be printed. In addition, any comments appearing after
the text are stripped. When --ipset-files is specified, this
field contains the IP address in its canonical form.
--no-files
Causes rwpmaplookup to treat the command line arguments as the text
to be parsed. This allows one to look up a handful of values
without having to create a temporary file. Use of the --no-files
switch disables paging of the output. This switch may not be
combined with --ipset-files.
--no-errors
Disables printing of errors when the input cannot be parsed as an
IP address or a protocol/port pair. This switch is ignored when
--ipset-files is specified.
--ipset-files
Causes rwpmaplookup to treat the command line arguments as the
names of IPset files to read and use as keys into the prefix map.
It is an error to use this switch when --map-file specifies a
protocol/port prefix map. When --ipset-files is active, the
"input" column of --fields contains the IP in its canonical form,
regardless of the --ip-format switch. This switch may not be
combined with --no-files.
--ip-format=FORMAT
When printing the key of an prefix map containing IP addresses,
specify how IP addresses are printed. When this switch is not
specified, the SILK_IP_FORMAT environment variable is checked for a
format. If it is empty or contains an invalid format, IPs are
printed in the canonical format. The FORMAT is one of:
canonical
Print IP addresses in their canonical form: dotted quad for
IPv4 (127.0.0.1) and hexadectet for IPv6 ("2001:db8::1"). Note
that IPv6 addresses in ::ffff:0:0/96 and some IPv6 addresses in
::/96 will be printed as a mixture of IPv6 and IPv4.
zero-padded
Print IP addresses in their canonical form, but add zeros to
the output so it fully fills the width of column. The
addresses 127.0.0.1 and "2001:db8::1" are printed as
127.000.000.001 and "2001:0db8:0000:0000:0000:0000:0000:0001",
respectively.
decimal
Print IP addresses as integers in decimal format. The
addresses 127.0.0.1 and "2001:db8::1" are printed as 2130706433
and 42540766411282592856903984951653826561, respectively.
hexadecimal
Print IP addresses as integers in hexadecimal format. The
addresses 127.0.0.1 and "2001:db8::1" are printed as "7f000001"
and "20010db8000000000000000000000001", respectively.
force-ipv6
Print all IP addresses in the canonical form for IPv6 without
using any IPv4 notation. Any IPv4 address is mapped into the
::ffff:0:0/96 netblock. The addresses 127.0.0.1 and
"2001:db8::1" are printed as "::ffff:7f00:1" and "2001:db8::1",
respectively.
--integer-ips
Print IP addresses as integers. This switch is equivalent to
--ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will
be removed in the SiLK 4.0 release.
--zero-pad-ips
Print IP addresses as fully-expanded, zero-padded values in their
canonical form. This switch is equivalent to
--ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it
will be removed in the SiLK 4.0 release.
--no-titles
Turn off column titles. By default, titles are printed.
--no-columns
Disable fixed-width columnar output.
--column-separator=C
Use specified character between columns and after the final column.
When this switch is not specified, the default of '|' is used.
--no-final-delimiter
Do not print the column separator after the final column. Normally
a delimiter is printed.
--delimited
--delimited=C
Run as if --no-columns --no-final-delimiter --column-sep=C had been
specified. That is, disable fixed-width columnar output; if
character C is provided, it is used as the delimiter between
columns instead of the default '|'.
--output-path=PATH
Determines where the output of rwpmaplookup is written. If this
option is not given, output is written to the standard output.
--pager=PAGER_PROG
When the --no-files switch has not been specified and output is to
a terminal, invoke the program PAGER_PROG to view the output one
screen full at a time. This switch overrides the SILK_PAGER
environment variable, which in turn overrides the PAGER variable.
If the value of the pager is determined to be the empty string, no
paging will be performed and all output will be printed to the
terminal.
--xargs
--xargs=FILENAME
Causes rwpmaplookup to read file names from FILENAME or from the
standard input if FILENAME is not provided. The input should have
one file name per line. rwpmaplookup will open each file in turn
and read the IPset, textual IP addresses, or textual protocol/port
pairs from it, as if the files had been listed on the command line.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Lines have been wrapped for improved readability, and the back slash
("\") is used to indicate a wrapped line.
Country code examples
Print the country code for a list of addresses read from the standard
input.
$ cat my-addrs.txt
128.2.0.0
128.2.0.1
$ cat my-addrs.txt | rwpmaplookup--country-codes
key| value|
128.2.0.0| us|
128.2.0.1| us|
Use --no-files to list the address on the command line.
$ rwpmaplookup--country-codes 128.2.0.0 128.2.0.1
key| value|
128.2.0.0| us|
128.2.0.1| us|
Use --ipset-files to read the addresses from an IPset file.
$ rwsetbuild my-addrs.txt my-addrs.set
$ rwpmaplookup--country-codes --ipset-files my-addrs.set
key| value|
128.2.0.0| us|
128.2.0.1| us|
Use the --fields switch to control which columns are printed.
$ rwpmaplookup--country-codes --fields=value my-addrs.txt
value|
us|
us|
Add the --delimited and --no-titles switches so the output only
contains the value column. Print the country code for a single address
using the default country code prefix map.
$ rwpmaplookup--country-codes --fields=value --delimited \
--no-titles --no-files 128.2.0.0
us
Alternatively
$ echo 128.2.0.0 \
| rwpmaplookup--country-codes --fields=value --delim --no-title
us
To use a different country code mapping file, provide that file as the
argument to the --country-codes switch.
$ rwpmaplookup --country-code=old-address-map.pmap --no-files 128.2.0.0
key|value|
128.2.0.0| us|
CIDR block input
Note that rwpmaplookup does not parse text that contains CIDR blocks.
$ echo '128.2.0.0/31' \
| rwpmaplookup--country-codes
key|value|
rwpmaplookup: Invalid IP '128.2.0.1/31' at -:1: Extra text follows value
For this case, use the IPset tool rwsetbuild(1) to parse the CIDR block
list and create a binary IPset stream, and pipe the IPset to
rwpmaplookup.
$ echo '128.2.0.0/31' \
| rwsetbuild \
| rwpmaplookup--country-code --ipset-files
key|value|
128.2.0.0| --|
128.2.0.1| --|
For versions of rwpmaplookup that do not have the --ipset-files switch,
you can have rwsetcat(1) read the binary IPset stream and print the IP
addresses as text, and pipe that into rwpmaplookup. Be sure to include
the "--cidr-blocks=0" switch to rwsetcat which forces individual IP
addresses to be printed.
$ echo '128.2.0.0/31' \
| rwsetbuild \
| rwsetcat --cidr-blocks=0 \
| rwpmaplookup--country-code
key|value|
128.2.0.0| --|
128.2.0.1| --|
General prefix map usage
Consider a user-defined prefix map, assigned-slash-8s.pmap, that maps
each /8 in the IPv4 address space to its assignment.
$ rwpmapcat assigned-slash-8s.pmap | head -4
ipBlock| label|
0.0.0.0/8| IANA - Local Identification|
1.0.0.0/8| APNIC|
2.0.0.0/8| RIPE NCC|
Use the --map-file switch to map from IPs to labels using this prefix
map.
$ cat my-addrs.txt
17.17.17.17
9.9.9.9
$ cat my-addrs.txt | rwpmaplookup --map-file=assigned-slash-8s.pmap
key| value|
17.17.17.17| Apple Computer Inc.|
9.9.9.9| IBM|
Use --ip-format=decimal to print the output as integers.
$ cat my-addrs.txt \
| rwpmaplookup --ip-format=decimal --map-file=assigned-slash-8s.pmap
key| value|
286331153| Apple Computer Inc.|
151587081| IBM|
Add the "input" field to see the input as well.
$ cat my-addrs.txt \
| rwpmaplookup --ip-format=decimal --fields=key,value,input \
--map-file=assigned-slash-8s.pmap
key| value| input|
286331153| Apple Computer Inc.| 17.17.17.17|
151587081| IBM| 9.9.9.9|
Combine the "input" field with the --no-errors switch to see a row for
each key.
$ rwpmaplookup --fields=key,value,input --no-errors --no-files \
--map-file=assigned-slash-8s.pmap 9.9.9.9 17.1717.17
key| value| input|
9.9.9.9| Apple Computer Inc.| 9.9.9.9|
| | 17.1717.17|
The input can contain integer values.
$ echo 151587081 \
| rwpmaplookup --fields=key,value,input --delimited=, \
--map-file=assigned-slash-8s.pmap
key,value,input
9.9.9.9,IBM,151587081
Block output
Specifying "block" in the --fields switch causes rwpmaplookup to print
the CIDR block that contains the address key.
$ cat my-addrs.txt
9.8.7.6
9.10.11.12
17.16.15.14
17.18.19.20
$ rwpmaplookup --map-file=assigned-slash-8s.pmap \
--fields=key,value,block my-addrs.txt
key| value| block|
9.8.7.6| IBM| 9.0.0.0/8|
9.10.11.12| IBM| 9.0.0.0/8|
17.16.15.14| Apple Computer Inc.| 17.0.0.0/8|
17.18.19.20| Apple Computer Inc.| 17.0.0.0/8|
To break the CIDR block into its starting and ending value, specify the
"start-block" and "end-block" fields.
$ rwpmaplookup --map-file=assigned-slash-8s.pmap \
--fields=key,value,start-block,end-block my-addrs.txt
key| value| start-block| end-block|
9.8.7.6| IBM| 9.0.0.0| 9.255.255.255|
9.10.11.12| IBM| 9.0.0.0| 9.255.255.255|
17.16.15.14| Apple Computer Inc.| 17.0.0.0| 17.255.255.255|
17.18.19.20| Apple Computer Inc.| 17.0.0.0| 17.255.255.255|
To get a unique list of blocks for the input keys, do not output the
"key" field and pipe the output of rwpmaplookup to the uniq(1) command.
(This works as long as the input data is sorted).
$ cat my-addrs.txt \
| rwpmaplookup --map-file=assigned-slash-8s.pmap \
--fields=block,value \
| uniq
block| value|
9.0.0.0/8| IBM|
17.0.0.0/8| Apple Computer Inc.|
The values printed in the "block" column corresponds to the CIDR block
that were used when the prefix map file was created.
$ rwpmaplookup --map=assigned-slash-8s.pmap --fields=block,value \
--no-files 128.2.0.1 129.0.0.1
block| value|
128.0.0.0/8|Administered by ARIN|
129.0.0.0/8|Administered by ARIN|
In the output from rwpmapcat(1), those two blocks are combined into a
larger range.
$ rwpmapcat --map=assigned-slash-8s.pmap | grep 128
128.0.0.0/6|Administered by ARIN|
Working with IPsets
Assume you have a binary IPset file, my-ips.set, that has the contents
shown here, and you want to find the list of unique assignments from
the assigned-slash-8s.pmap file.
$ rwsetcat --cidr-blocks=1 my-ips.set
9.9.9.0/24
13.13.13.0/24
15.15.15.0/24
16.16.16.0/24
17.17.17.0/24
18.18.18.0/24
Since the blocks in the assigned-slash-8s.pmap file are /8, use the
rwsettool(1) command to mask the IPs in the IPset to the unique /8 that
contains each of the IPs.
$ rwsettool --mask=8 my-ips.set \
| rwpmaplookup --map-file=assigned-slash-8s.pmap
key| value|
9.0.0.0| IBM|
13.0.0.0| Xerox Corporation|
15.0.0.0| Hewlett-Packard Company|
16.0.0.0|Digital Equipment Corporation|
17.0.0.0| Apple Computer Inc.|
18.0.0.0| MIT|
Protocol/port prefix maps
Assume the service.pmap prefix map file maps protocol/port pairs to the
name of the service running on the named port.
$ rwpmapcat service.pmap
startPair| endPair| label|
0/0| 0/65535| unknown|
1/0| 1/65535| ICMP|
2/0| 5/65535| unknown|
6/0| 6/21| TCP|
6/22| 6/22| TCP/SSH|
...
17/0| 17/52| UDP|
17/53| 17/53| UDP/DNS|
...
To query this prefix map, the input must contain two numbers separated
by a slash.
$ rwpmaplookup --map-file=service.pmap --no-files 6/80
key| value|
6/80| TCP/HTTP|
Specifying "block", "start-block", and "end-block" in the --fields
switch also works for Protocol/port prefix map files. The "block"
column contains the same information as the "start-block" and
"end-block" columns separated by a single space.
$ rwpmaplookup --map-file=service.pmap --no-files \
--fields=key,value,start,end,block \
6/80 6/6000 17/0 17/53 128/128
key| value|start-blo|end-block| block|
6/80| TCP/HTTP| 6/80| 6/80| 6/80 6/80|
6/6000| TCP| 6/4096| 6/6143| 6/4096 6/6143|
17/0| UDP| 17/0| 17/31| 17/0 17/31|
17/53| UDP/DNS| 17/53| 17/53| 17/53 17/53|
200/200|Unassigned| 192/0|223/65535| 192/0 223/65535|
Using the pmapfilter(3) plug-in to rwcut(1), you can print the label
for the source port and destination port in the SiLK Flow file data.rw.
$ rwcut --pmap-file=service.pmap --num-rec=5 \
--fields=proto,sport,src-service,dport,dst-service data.rw
pro|sPort|src-service|dPort|dst-service|
17|29617| UDP| 53| UDP/DNS|
17| 53| UDP/DNS|29617| UDP|
6|29618| TCP| 22| TCP/SSH|
6| 22| TCP/SSH|29618| TCP|
1| 0| ICMP| 771| ICMP|
The pmapfilter plug-in does not provide a way to print the values based
on the application field. You can get that information by having rwcut
print the protocol and application separated by a slash, and pipe the
result into rwpmaplookup.
$ rwcut --fields=proto,application --num-rec=5 \
--delimited=/ --no-title \
| rwpmaplookup --map-file=service.pmap
key| value|
17/53| UDP/DNS|
17/53| UDP/DNS|
6/22| TCP/SSH|
6/22| TCP/SSH|
1/0| ICMP|
ENVIRONMENT
SILK_IP_FORMAT
This environment variable is used as the value for --ip-format when
that switch is not provided. Since SiLK 3.11.0.
SILK_PAGER
When set to a non-empty string, rwpmaplookup automatically invokes
this program to display its output a screen at a time unless the
--no-files switch is given. If this variable is set to an empty
string, rwpmaplookup does not automatically page its output.
PAGER
When set and SILK_PAGER is not set, rwpmaplookup automatically
invokes this program to display its output a screen at a time.
SILK_COUNTRY_CODES
This environment variable allows the user to specify the country
code mapping file to use when the --country-codes switch is
specified without an argument. The variable's value may be a
complete path or a file relative to SILK_PATH. See the "FILES"
section for standard locations of this file.
SILK_ADDRESS_TYPES
This environment variable allows the user to specify the address
type mapping file to use when the --address-types switch is
specified without an argument. The variable's value may be a
complete path or a file relative to the SILK_PATH. See the "FILES"
section for standard locations of this file.
SILK_CLOBBER
The SiLK tools normally refuse to overwrite existing files.
Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_PATH
This environment variable gives the root of the install tree. When
searching for configuration files, rwpmaplookup may use this
environment variable. See the "FILES" section for details.
FILES
${SILK_COUNTRY_CODES}
${SILK_PATH}/share/silk/country_codes.pmap
${SILK_PATH}/share/country_codes.pmap
/usr/local/share/silk/country_codes.pmap
/usr/local/share/country_codes.pmap
Possible locations for the country codes mapping file when the
--country-codes switch is specified without an argument.
${SILK_ADDRESS_TYPES}
${SILK_PATH}/share/silk/address_types.pmap
${SILK_PATH}/share/address_types.pmap
/usr/local/share/silk/address_types.pmap
/usr/local/share/address_types.pmap
Possible locations for the address types mapping file when the
--address-types switch is specified without an argument.
NOTESrwpmaplookup was added in SiLK 3.0.
rwpmaplookup duplicates the functionality of rwip2cc(1). rwip2cc is
deprecated, and it will be removed in the SiLK 4.0 release. Examples
of using rwpmaplookup in place of rwip2cc are provided in the latter's
manual page.
SEE ALSOrwpmapbuild(1), rwpmapcat(1), ccfilter(3), addrtype(3), pmapfilter(3),
rwgeoip2ccmap(1), rwcut(1), rwset(1), rwsetbuild(1), rwsetcat(1),
rwsettool(1), silk(7)SiLK 3.11.0.1 2016-02-19 rwpmaplookup(1)
