rwbagcat(1) SiLK Tool Suite rwbagcat(1)NAMErwbagcat - Output a binary Bag as text.
SYNOPSISrwbagcat [ --network-structure[=STRUCTURE] | --bin-ips[=SCALE] ]
[--print-statistics[=OUTFILE]]
[--minkey=VALUE] [--maxkey=VALUE] [--mask-set=PATH]
[--mincounter=VALUE] [--maxcounter=VALUE] [--zero-counts]
[--output-path=OUTPUTFILE]
[--key-format=FORMAT] [--integer-keys] [--zero-pad-ips]
[--no-columns] [--column-separator=C]
[--no-final-delimiter] [{--delimited | --delimited=C}]
[--pager=PAGER_PROG] [BAGFILE...]
rwbagcat--help
rwbagcat--version
DESCRIPTIONrwbagcat reads a binary Bag as created by rwbag(1) or rwbagbuild(1),
converts it to text, and outputs it to the standard output or the
specified file. It can also print various statistics and summary
information about the Bag.
rwbagcat reads the BAGFILEs specified on the command line; if no
BAGFILE arguments are given, rwbagcat attempts to read the Bag from the
standard input. BAGFILE may also explicitly be the keyword "stdin" or
a hyphen ("-") to allow rwbagcat to combine files and piped input. If
any input does not contain a Bag, rwbagcat prints an error to the
standard error and exits abnormally.
When multiple BAGFILEs are specified, each is handled individually; to
process the combination of the BAGFILEs, invoke rwbagcat on the output
from rwbagtool(1).
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
--network-structure
--network-structure=STRUCTURE
For each numeric value in STRUCTURE, group the IPs in the Bag into
a netblock of that size and print the number of hosts, the sum of
the counters, and, optionally, print the number of smaller,
occupied netblocks that each larger netblock contains. When
STRUCTURE begins with "v6:", the IPs in the Bag are treated as IPv6
addresses, and any IPv4 addresses are mapped into the ::ffff:0:0/96
netblock. Otherwise, the IPs are treated as IPv4 addresses, and
any IPv6 address outside the ::ffff:0:0/96 netblock is ignored.
Aside from the initial "v6:" (or "v4:", for consistency), STRUCTURE
has one of following forms:
1. NETBLOCK_LIST/SUMMARY_LIST. Group IPs into the sizes specified
in either NETBLOCK_LIST or SUMMARY_LIST. rwbagcat prints a row
for each occupied netblock specified in NETBLOCK_LIST, where
the row lists the base IP of the netblock, the sum of the
counters for that netblock, the number of hosts, and the number
of smaller, occupied netblocks having a size that appears in
either NETBLOCK_LIST or SUMMARY_LIST. (The values in
SUMMARY_LIST are only summarized; they are not printed.)
2. NETBLOCK_LIST/. Similar to the first form, except all occupied
netblocks are printed, and there are no netblocks that are only
summarized.
3. NETBLOCK_LISTS. When the character "S" appears anywhere in the
NETBLOCK_LIST, rwbagcat provides a default value for the
SUMMARY_LIST. That default is 8,16,24,27 for IPv4, and 48,64
for IPv6.
4. NETBLOCK_LIST. When neither "S" nor "/" appear in STRUCTURE,
the output does not include the number of smaller, occupied
netblocks.
5. Empty. When STRUCTURE is empty or only contains "v6:" or
"v4:", the NETBLOCK_LIST prints a single row for the total
network (the /0 netblock) giving the number of hosts, the sum
of the counters, and the number of smaller, occupied netblocks
using the same default list specified in form 3.
NETBLOCK_LIST and SUMMARY_LIST contain a comma separated list of
numbers between 0 (the total network) and the size for an
individual host (32 for IPv4 or 128 for IPv6). The characters "T"
and "H" may be used as aliases for 0 and the host netblock,
respectively. In addition, when parsing the lists as IPv4
netblocks, the characters "A", "B", "C", and "X" are supported as
aliases for 8, 16, 24, and 27, respectively. A comma is not
required between adjacent letters. The --network-structure switch
disables printing of the IPs in the Bag file; specify the "H"
argument to the switch to print each individual IP address and its
counter.
--bin-ips
--bin-ips=SCALE
Invert the bag and count the total number of unique IP addresses
for a given value of the volume bin. For example, turn a Bag
{sip:flow} into {flow:count(sip)}. SCALE is a string containing
the value "linear", "binary", or "decimal".
· The default behavior is "linear": Each distinct counter gets
its own bin. Any counter in the input Bag file that is larger
than the maximum possible key will be attributed to the maximum
key; to prevent this, specify "--maxcounter=4294967295".
· "binary" creates a bag of {log2(flow):count(sip)}. Bin "n"
contains counts in the range "[ 2^n, 2^(n+1) )".
· "decimal" creates one hundred bins for each counter in the
range [1,100), and one hundred bins for each counter in the
range [100,1000), each counter in the range [1000,10000), etc.
Counters are logarithmically distributed among the bins.
--print-statistics
--print-statistics=OUTFILE
Print out breakdown of the network hosts seen, and print out
general statistics about the keys and counters.
· count of unique keys
· sum of all the counters
· minimum key
· maximum key
· minimum counter
· maximum counter
· mean of counters
· variance of counters
· standard deviation of counters
· skew of counters
· kurtosis of counters
· count of nodes allocated
· total bytes allocated for nodes
· count of leaves allocated
· total bytes allocated for leaves
· density of the data
OUTFILE is a filename, named pipe, the values "stdout" or "-" to
print to the standard output, or the value "stderr" to print to the
standard error. Defaults to printing to the standard output.
--minkey=VALUE
Output records whose key value is at least VALUE. VALUE may be an
IP address or an integer in the range 0 to 4294967295 inclusive.
The default is to print all records with a non-zero counter.
--maxkey=VALUE
Output records whose key value is not more than VALUE. VALUE may
be an IP address or an integer in the range 0 to 4294967295
inclusive. The default is to print all records with a non-zero
counter.
--mask-set=PATH
Output records whose key appears in the binary IPset read from the
file PATH. (To build an IPset, use rwset(1) or rwsetbuild(1).)
When used with --minkey and/or --maxkey, output records whose key
is in the IPset and is also within when the specified range.
--mincounter=VALUE
Output records whose counter value is at least VALUE. VALUE is an
integer in the range 1 to 18446744073709551615. The default is to
print all records with a non-zero counter; use --zero-counts to
show records whose counter is 0.
--maxcounter=VALUE
Output records whose counter value is not more than VALUE. VALUE
is an integer in the range 1 to 18446744073709551615, with the
default being the maximum counter value.
--zero-counts
Print keys whose counter is zero. Normally, keys with a counter of
zero are suppressed since all keys have a default counter of zero.
In order to use this flag, either --mask-set or both --minkey and
--maxkey must be specified. When this switch is specified, any
counter limit explicitly set by the --maxcounter switch will still
be applied.
--output-path=OUTPUTFILE
Redirect output of the --network-structure or --bin-ips options to
OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the values
"stdout" or "-" to print to the standard output.
--key-format=FORMAT
Specify the format to use when printing the keys. When this switch
is not specified, a Bag whose keys are known not to be IP addresses
are printed as decimal numbers, and the keys for all other Bags are
printed as IP addresses in the canonical format. The FORMAT is one
of:
canonical
Print keys as IP addresses in the canonical format: dotted quad
for IPv4 (127.0.0.1) and hexadectet for IPv6 ("2001:db8::1").
Note that IPv6 addresses in ::ffff:0:0/96 and some IPv6
addresses in ::/96 will be printed as a mixture of IPv6 and
IPv4.
zero-padded
Print keys as IP addresses in their canonical form, but add
zeros to the output so it fully fills the width of column. The
addresses 127.0.0.1 and "2001:db8::1" are printed as
127.000.000.001 and "2001:0db8:0000:0000:0000:0000:0000:0001",
respectively.
decimal
Print keys as integers in decimal format. The addresses
127.0.0.1 and "2001:db8::1" are printed as 2130706433 and
42540766411282592856903984951653826561, respectively.
hexadecimal
Print keys as integers in hexadecimal format. The addresses
127.0.0.1 and "2001:db8::1" are printed as "7f000001" and
"20010db8000000000000000000000001", respectively.
force-ipv6
Print all keys as IP addresses in the canonical form for IPv6
without using any IPv4 notation. Any integer key or IPv4
address is mapped into the ::ffff:0:0/96 netblock. The
addresses 127.0.0.1 and "2001:db8::1" are printed as
"::ffff:7f00:1" and "2001:db8::1", respectively.
--integer-keys
This switch is equivalent to --key-format=decimal, it is deprecated
as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.
--zero-pad-ips
This switch is equivalent to --key-format=zero-padded, it is
deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0
release.
--no-columns
Disable fixed-width columnar output.
--column-separator=C
Use specified character between columns and after the final column.
When this switch is not specified, the default of '|' is used.
--no-final-delimiter
Do not print the column separator after the final column. Normally
a delimiter is printed. When the network summary is requested
(--network-structure=S), the separator is always printed before the
summary column and never that column.
--delimited
--delimited=C
Run as if --no-columns --no-final-delimiter --column-sep=C had been
specified. That is, disable fixed-width columnar output; if
character C is provided, it is used as the delimiter between
columns instead of the default '|'.
--pager=PAGER_PROG
When output is to a terminal, invoke the program PAGER_PROG to view
the output one screen full at a time. This switch overrides the
SILK_PAGER environment variable, which in turn overrides the PAGER
variable. If the value of the pager is determined to be the empty
string, no paging will be performed and all output will be printed
to the terminal.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
To print the bag:
$ rwbagcat mybag.bag
172.23.1.1| 5|
172.23.1.2| 231|
172.23.1.3| 9|
172.23.1.4| 19|
192.168.0.100| 1|
192.168.0.101| 1|
192.168.0.160| 15|
192.168.20.161| 1|
192.168.20.162| 5|
192.168.20.163| 5|
To print it with full network:
$ rwbagcat --network-structure=TABCHX mybag.bag
172.23.1.1 | 5|
172.23.1.2 | 231|
172.23.1.3 | 9|
172.23.1.4 | 19|
172.23.1.0/27 | 264|
172.23.1.0/24 | 264|
172.23.0.0/16 | 264|
172.0.0.0/8 | 264|
192.168.0.100 | 1|
192.168.0.101 | 1|
192.168.0.96/27 | 2|
192.168.0.160 | 15|
192.168.0.160/27 | 15|
192.168.0.0/24 | 17|
192.168.20.161 | 1|
192.168.20.162 | 5|
192.168.20.163 | 5|
192.168.20.160/27 | 11|
192.168.20.0/24 | 11|
192.168.0.0/16 | 28|
192.0.0.0/8 | 28|
TOTAL | 292|
Or an abbreviated network structure by class A and C only, including
summary information:
$ rwbagcat --network-structure=ACS mybag.bag
172.23.1.0/24 | 264| 4 hosts in 1 /27
172.0.0.0/8 | 264| 4 hosts in 1 /16, 1 /24, and 1 /27
192.168.0.0/24 | 17| 3 hosts in 2 /27s
192.168.20.0/24 | 11| 3 hosts in 1 /27
192.0.0.0/8 | 28| 6 hosts in 1 /16, 2 /24s, and 3 /27s
To bin by number of unique IP addresses by volume:
$ rwbagcat--bin-ips mybag.bag
1| 3|
5| 3|
9| 1|
15| 1|
19| 1|
231| 1|
This means there were 3 source hosts in the bag that had a single flow;
3 hosts that had 5 flows; and one host each that had 9, 15, 19, and 231
flows.
For a log2 breakdown of the counts:
$ rwbagcat --bin-ips=binary mybag.bag
2^0 to 2^1-1| 3|
2^2 to 2^3-1| 3|
2^3 to 2^4-1| 2|
2^4 to 2^5-1| 1|
2^7 to 2^8-1| 1|
Statistics:
$ rwbagcat--stats mybag.bag
Statistics
keys: 10
sum of counters: 292
minimum key: 172.23.1.1
maximum key: 192.168.20.163
minimum count: 1
maximum count: 231
mean: 29.2
variance: 5064
standard deviation: 71.16
skew: 2.246
kurtosis: 8.1
$ rwbagcat--tree-stats mybag.bag
nodes allocated: 5 (10240 bytes)
leaves allocated: 4 (1024 bytes)
keys inserted: 10 (10 unique)
counter density: 7.81%
ENVIRONMENT
SILK_CLOBBER
The SiLK tools normally refuse to overwrite existing files.
Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_PAGER
When set to a non-empty string, rwbagcat automatically invokes this
program to display its output a screen at a time. If set to an
empty string, rwbagcat does not automatically page its output.
PAGER
When set and SILK_PAGER is not set, rwbagcat automatically invokes
this program to display its output a screen at a time.
SEE ALSOrwbag(1), rwbagbuild(1), rwbagtool(1), rwset(1), rwsetbuild(1), silk(7)SiLK 3.11.0.1 2016-02-19 rwbagcat(1)
