YAFMETA2PCAP(1) Yet Another Flowmeter YAFMETA2PCAP(1)NAMEyafMeta2Pcap - YAF PCAP Metadata File Parser and PCAP Creator
SYNOPSISyafMeta2Pcap [--pcap-meta-file META_FILE_NAME]
[--pcap PCAP_FILE_NAME]
[--caplist CAP_FILE_NAME]
[--metalist META_FILE_NAME]
[--out OUTPUT_SPECIFIER]
[--hash FLOW_KEY_HASH]
[--time FLOW_START_MILLISECONDS]
[--etime FLOW_END_MILLISECONDS]
[--packets NUM_PACKETS]
[--yaf-program PATH_TO_YAF]
[--verbose]
DESCRIPTION
yaf has several PCAP export options available. yaf will create a pipe-
delimited file that contains information to assemble a PCAP file for
any particular flow. yafMeta2Pcap parses the metadata file created by
yaf and creates a PCAP file for the given flow identified by the flow
key hash and start time.
yafMeta2Pcap has the ability to read IPFIX from getFlowKeyHash on stdin
to avoid typing the flow key hash and start time on the command line.
yafMeta2Pcap will accept the first 2 records from getFlowKeyHash. If
the first 2 records have the same start time, yafMeta2Pcap will sum the
values for packetTotalCount to determine the total number of packets to
search for (to account for uniflows). yafMeta2Pcap will only search
for the first hash and start time it receives. See below for examples
for using IPFIX input.
OPTIONS
The following options are required for yafMeta2Pcap to create the PCAP
file.
--pcap-meta-file META_FILE_NAME
META_FILE_NAME is the path to the pcap-meta-file as created by yaf.
META_FILE_NAME is a file glob pattern, which must be escaped or
quoted to prevent the shell expansion if using wildcards. Files
that match this pattern will be processed by yafMeta2Pcap.
Alternatively, you can use --metalist to provide a text file with
an ordered list of pcap-meta-files created by yaf. Alternatively,
you can set the YAF_PCAP_META_FILE environment variable to the full
path to the pcap-meta-file created by yaf to avoid including it to
the command line.
--pcap PCAP_FILE_NAME
PCAP_FILE_NAME is the path to the PCAP file that was given to yaf
to create the pcap-meta-file. PCAP_FILE_NAME is a file glob
pattern, which must be escaped or quoted to prevent shell expansion
if using wildcards. The option is only required if full path is
not available in the PCAP metadata file. yafMeta2Pcap will open
subsequent PCAPs only when the third column in the PCAP-meta-file
changes. For example, if the third column is 0, yafMeta2Pcap will
open the first PCAP given to --pcap. Alternatively, you can use
--caplist to give a text file with an ordered list of PCAP files.
--caplist CAP_FILE_NAME
Instead of listing multiple PCAP files, use CAP_FILE_NAME as an
ordered newline-delimited list of pathnames to PCAP dumpfiles as
read by yaf. Alternatively, you can set the YAF_CAPLIST
environment variable to the full path of the CAP_FILE_NAME to avoid
adding it to the command line.
--metalist META_FILE_NAME
Instead of listing multiple PCAP metadata files, use META_FILE_NAME
as an ordered newline-delimited list of pathnames to PCAP metadata
files as created by yaf. Alternatively, you can set the
YAF_PCAP_META_LIST environment variable to the META_FILE_NAME to
avoid adding it to the command line.
--out OUTPUT_SPECIFIER
The full path of the PCAP file that will be created containing just
the packets from the given flow. If this option is not present,
yafMeta2Pcap will simply write the filename(s) of the PCAP(s) that
contained the flow with the given FLOW_KEY_HASH. yafMeta2Pcap will
overwrite any existing file at OUTPUT_SPECIFIER.
--hash FLOW_KEY_HASH
The flow key hash of the flow as created by getFlowKeyHash. Use
getFlowKeyHash with the flow's 5-tuple and vlan to calculate the
flow key hash. If this option is not present, yafMeta2Pcap assumes
IPFIX will be present on stdin.
--time FLOW_START_MILLISECONDS
The milliseconds since epoch time of the start time of the flow.
getFlowKeyHash can calculate the milliseconds since epoch given the
date and time. Not required, but suggested. If
FLOW_START_MILLISECONDS is not provided, all packets matching the
FLOW_KEY_HASH will be given, which may include multiple flows.
--etime FLOW_END_MILLISECONDS
The milliseconds since epoch time of the end time of the flow.
This value is exported by getFlowKeyHash if present and accepted by
yafMeta2Pcap. Not required, but suggested. Including this value
will likely speed up search times. This argument overrides any end
time information collected via IPFIX from getFlowKeyHash/
--packets NUM_PACKETS
NUM_PACKETS is the total number of packets for the bi-flow. This
optional argument may speed up processing because it tells
yafMeta2Pcap when it has found the entire flow. This can also be
used if you only want the first NUM_PACKETS of the flow instead of
the entire flow. This argument overrides any packet count
information collected via IPFIX from getFlowKeyHash.
--yaf-program PATH_TO_YAF
PATH_TO_YAF is the location of the yaf program to use to create the
PCAP file if OUTPUT_SPECIFIER is present and the pcap-meta-file has
the format:
hash | stime | path_to_pcap
If not specified, yafMeta2Pcap assumes there is program yaf on your
$PATH.
--verbose
If present, print debug information to stdout.
Examples
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Lines have been wrapped for improved readability, and the back slash
("\") is used to indicate a wrapped line.
The following example shows how to run yaf, getFlowKeyHash, and
yafMeta2Pcap to create a PCAP for a given flow.
$ yaf --in /data/some_large_file.pcap --out /data/ipfix.yaf \
--pcap-meta-file /tmp/pcap-info
$ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
--dport 80 --protocol 6 --vlan 3 --date 2011-03-21 \
--time 16:15:54.256
sIP| dIP|sPort|dPort|pro| vlan| hash| ms
10.10.25.11| 10.10.26.12| 1586| 80| 6| 3| 100819735| 1300724154256
FILE PATH: 256/100819735-2011321161554_0.pcap
$ yafMeta2Pcap--pcap-meta-file /tmp/pcap-info20150107141533_00000.meta \
--pcap /data/some_large_file.pcap \
--out /data/my-new-pcap.pcap \
--hash 100819735 --time 1300724154
The following example shows how to run getFlowKeyHash directly to
yafMeta2Pcap and use environment variables.
$ export YAF_PCAP_META_FILE=/tmp/pcap-info20150107141533_00000.meta
$ export YAF_CAPLIST=/tmp/yafcaplist.txt
$ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
--dport 80 --protocol 6 --vlan 3 --date 2011-03-21 \
--time 16:15:54.256 -I | yafMeta2Pcap-o /data/my-new-pcap -n 10
The following example demonstrates how to use SiLK tools with
getFlowKeyHash and yafMeta2Pcap. This assumes the appropriate
environment variables are set.
$ rwfilter --application=80 --pass-dest=stdout --max-pass-records=2 file.rw | \
rwsilk2ipfix | getFlowKeyHash -I | yafMeta2Pcap-o /data/mynewpcap.pcap
The following example shows how to configure yaf to create rolling
PCAP files and use yafMeta2Pcap to create a PCAP for a specific flow.
$ yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
--pcap-meta-file /tmp/pcap-info --pcap /data/pcap \
--live pcap --index-pcap
$ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
--dport 80 --protocol 6 --vlan 3 --date 2011-03-21 \
--time 16:15:54.256
sIP| dIP|sPort|dPort|pro| vlan| hash| ms
10.10.25.11| 10.10.26.12| 1586| 80| 6| 3| 100819735| 1300724154256
FILE PATH: 256/100819735-2011321161554_0.pcap
$ yafMeta2Pcap--pcap-meta-file /tmp/pcap-info20150107141533_00000.meta \
--out /data/my-new-pcap.pcap \
--hash 100819735 --time 1300724154
The following example shows how to run yaf creating rolling PCAP files
and use yafMeta2Pcap to create a PCAP file for a particular flow.
$ yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
--pcap-meta-file /tmp/pcap-info --pcap /data/pcap \
--live pcap
$ yafMeta2Pcap --pcap-meta-file="/tmp/pcap-info*"\
-o /data/new.pcap \
--hash 100819735 --time 1300724154
The following example shows how to run yaf over a list of pcap files
and use yafMeta2Pcap to create a PCAP for a specific flow
$ yaf --in /tmp/caplist.txt --out /tmp/yaffile.yaf \
--pcap-meta-file /tmp/pcap-info --caplist
$ yafMeta2Pcap--caplist /tmp/caplist.txt -o flow.pcap \
--hash 100819735 --time 1300724154 \
--pcap-meta-file "/tmp/pcap-info*" -v
Found 1 packets that match criteria.
Known Issues
If the pcap-meta-file was generated in a way that it has the form:
hash | stime | path_to_pcap
and an OUTPUT_SPECIFIER is provided to --out, yafMeta2Pcap creates a
temporary file with the PCAP file names that contain the flow it is
searching for. This temporary file is placed in /tmp by default, but
the location can be configured through the use of the TMPDIR
environment variable. The temporary file is deleted upon termination
of yafMeta2Pcap.
Bug reports may be sent directly to the Network Situational Awareness
team at <netsa-help@cert.org>.
AUTHORS
Emily Sarneso and the CERT/CC Engineering Team,
<http://www.cert.org/netsa>.
SEE ALSOyaf(1), getFlowKeyHash(1)2.8.0 19-Feb-2016 YAFMETA2PCAP(1)
