innfeed.conf(5)innfeed.conf(5)NAMEinnfeed.conf - configuration file for innfeed
DESCRIPTION
This man page describes the configuration file for version 1.0 of inn‐
feed. This format has changed dramatically since version 0.9.3.
The file innfeed.conf is used to control the innfeed(1) program. It is
a fairly free-format file that consists of three types of entries:
key/value, peer and group. Comments are from the hash character ``#''
to the end of the line.
Key/value entries are a keyword and a value separated by a colon (which
can itself be surrounded by whitespace). For example:
max-connections: 10
A legal key starts with a letter and contains only letters, digits, and
``_'', ``-''.
There are 5 different type of values: integers, floating-point numbers,
characters, booleans, and strings. Integer and floating point numbers
are as to be expected except that exponents in floating point numbers
are not supported. A boolean value is either ``true'' or ``false''
(case is not significant). A character value is a single-quoted charac‐
ter as defined by the C-language. A string value is any other sequence
of characters. If the string needs to contain whitespace, then it must
be quoted with double quotes, and uses the same format for embedding
non-printing characters as normal C-language string.
Peer entries look like:
peer <name> {
# body ...
}
The word ``peer'' is required. The ``<name>'' is the same as the site
name in INN's newsfeeds file. The body of a peer entry contains some
number (possibly zero) of key/value entries.
Group entries look like:
group <name> {
# body
}
The word ``group'' is required. The ``<name>'' is any string valid as a
key. The body of a group entry contains any number of the three types
of entries. So key/value pairs can be defined inside a group, and peers
can be nested inside a group, and other groups can be nested inside a
group.
Key/value entries that are defined outside of all peer and group
entries are said to be at ``global scope''. There are global key/value
entries that apply to the process as a whole (for example the location
of the backlog file directory), and there are global key/value entries
that act as defaults for peers. When innfeed looks for a specific value
in a peer entry (for example, the maximum number of connections to set
up), if the value is not defined in the peer entry, then the enclosing
groups are examined for the entry (starting at the closest enclosing
group). If there are no enclosing groups, or the enclosing groups don't
define the key/value, then the value at global scope is used.
A small example could be:
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. ``uunet'' is the name used by innd in
# the newsfeeds file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # override global value.
}
# A group of two peers who can handle more connections
# than normal
group fast-sites {
max-connections: 15
# Another peer. The ``max-connections'' value from the
# ``fast-sites'' group scope is used. The ``ip-name'' value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # he can really cook.
}
}
Given the above configuration file, the defined peers would have the
following values for the ``max-connections'' key.
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
Innfeed ignores key/value pairs it is not interested in. Some config
file values can be set via a command line option, in which case that
setting overrides the settings in the file.
Config files can be included in other config files via the syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example config file, see the supplied innfeed.conf.
GLOBAL VALUES
The following listing show all the keys that apply to the process as
whole. These are not required (compiled-in defaults are used where
needed).
news-spool
This key requires a pathname value. It specifies where the top
of the article spool is. This corresponds to the ``-a'' command-
line option.
input-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) that should be read in fun‐
nel-file mode. This corresponds to giving a filename as an argu‐
ment on the command-line (i.e. its presence also implies that
funnel-file mode should be used).
pid-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) where the pid of the innfeed
process should be stored. This corresponds to the ``-p'' com‐
mand-line option.
debug-level
This key defines the debug level for the process. A non-zero
number generates a lot of messages to stderr, or to the config-
defined ``log-file''. This corresponds to the ``-d'' command-
line option.
use-mmap
This key requires a boolean value. It specifies whether mmaping
should be used if innfeed has been built with mmap support. If
article data on disk is not in NNTP-ready format (CR/LF at the
end of each line), then after mmaping the article is read into
memory and fixed up, so mmaping has no positive effect (and pos‐
sibly some negative effect depending on your system), and so in
such a case this value should be false. This corresponds to the
``-M'' command-line option.
log-file
This key requires a pathname value. It specifies where any log‐
ging messages that couldn't be sent via syslog(3) should go
(such as those generated when a positive value for ``debug-
value'', is used). This corresponds to the ``-l'' command-line
option. A relative pathname is relative to the ``backlog-direc‐
tory'' value.
log-time-format
This key requires a format string suitable for strftime(3). It
is used for messages sent via syslog(3) and to the status-file.
Default value is "%a %b %d %H:%M:%S %Y".
backlog-directory
This key requires a pathname value. It specifies where the cur‐
rent innfeed process should store backlog files. This corre‐
sponds to the ``-b'' command-line option.
backlog-highwater
This key requires a positive integer value. It specifies how
many articles should be kept on the backlog file queue before
starting to write new entries to disk.
backlog-ckpt-period
This key requires a positive integer value. It specifies how
many seconds between checkpoints of the input backlog file. Too
small a number will mean frequent disk accesses, too large a
number will mean after a crash innfeed will re-offer more
already-processed articles than necessary.
backlog-newfile-period
This key requires a positive integer value. It specifies how
many seconds before each checks for externally generated backlog
files that are to be picked up and processed.
backlog-rotate-period
This key requires a positive integer value. It specifies how
many seconds elapse before innfeed checks for a manually created
backlog file and moves the output backlog file to the input
backlog file.
dns-retry
This key requires a positive integer value. It defines the num‐
ber of seconds between attempts to re-lookup host information
that previous failed to be resolved.
dns-expire
This key requires a positive integer value. It defines the num‐
ber of seconds between refreshes of name to address DNS transla‐
tion. This is so long-running processes don't get stuck with
stale data, should peer ip addresses change.
close-period
This key requires a positive integer value. It is the maximum
number of seconds a connection should be kept open. Some NNTP
servers don't deal well with connections being held open for
long periods.
gen-html
This key requires a boolean value. It specifies whether the sta‐
tus-file should be HTML-ified.
status-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) where the periodic status of
the innfeed process should be stored. This corresponds to the
``-S'' command-line option.
connection-stats
This key requires a boolean value. If the value is true, then
whenever the transmission statistics for a peer are logged, then
each active connection logs its own statistics. This corresponds
to the ``-z'' command-line option.
host-queue-highwater
This key requires a positive integer value. It defines how many
articles will be held internally for a peer before new arrivals
cause article information to be spooled to the backlog file.
stats-period
This key requires a positive integer value. It defines how many
seconds innfeed waits between generating statistics on transfer
rates.
stats-reset
This key requires a positive integer value. It defines how many
seconds innfeed waits before resetting all internal transfer
counters back to zero (after logging one final time). This is so
a innfeed-process running more than a day will generate
``final'' stats that will be picked up by logfile processing
scripts.
initial-reconnect-time
This key requires a positive integer value. It defines how many
seconds to first wait before retrying to reconnect after a con‐
nection failure. If the next attempt fails too, then the recon‐
nect time is approximately doubled until the connection suc‐
ceeds, or max-reconnection-time is reached.
max-reconnect-time
This key requires an integer value. It defines the maximum num‐
ber of seconds to wait between attempt to reconnect to a peer.
The initial value for reconnection attempts is defined by ini‐
tial-reconnect-time, and it is doubled after each failure, up to
this value.
stdio-fdmax
This key requires a non-negative integer value. If the value is
greater than zero, then whenever a network socket file descrip‐
tor is created and it has a value less than this, the file
descriptor will be dup'ed to bring the value up greater than
this. This is to leave lower numbered file descriptors free for
stdio. Certain systems, Sun's in particular, require this. SunOS
4.1.x usually requires a value of 128 and Solaris requires a
value of 256. The default if this is not specified, is 0.
GLOBAL PEER DEFAULTS
All the key/value pairs mentioned in this section must be specified at
global scope. They may also be specified inside a group or peer defini‐
tion. Note that when peers are added dynamically (i.e. when innfeed
receives an article for an unspecified peer), it will add the peer site
using the parameters specified at global scope.
article-timeout
This key requires a non-negative integer value. If no articles
need to be sent to the peer for this many seconds, then the peer
is considered idle and all its active connections are torn down.
response-timeout
This key requires a non-negative integer value. It defines the
maximum amount of time to wait for a response from the peer
after issuing a command.
initial-connections
This key requires a non-negative integer value. It defines the
number of connections to be opened immediately when setting up a
peer binding. A value of 0 means no connections will be created
until an article needs to be sent.
max-connections
This key requires positive integer value. It defines the maximum
number of connections to run in parallel to the peer. A value of
zero specifies an unlimited number of maximum connections. In
general use of an unlimited number of maximum connections is not
recommended. Do not ever set max-connections to zero with
dynamic-method 0 set, as this will saturate peer hosts with con‐
nections. [ Note that in previous versions of innfeed, a value
of 1 had a special meaning. This is no longer the case, 1 means
a maximum of 1 connection ].
dynamic-method
This key requires an integer value between 0 and 3. It controls
how connections (up to max-connections) are opened, up to the
maximum specified by max-connections. In general (and specifi‐
cally, with dynamic-method 0), a new connection is opened when
the current number of connections is below max-connections, and
an article is to be sent while no current connections are idle.
Without further restraint (i.e. using dynamic-method 0), in
practice this means that max-connections connections are estab‐
lished while articles are being sent. Use of other dynamic-
method settings imposes a further limit on the amount of connec‐
tions opened below that specified by max-connections. This limit
is calculated in different ways, depending of the value of
dynamic-method. Users should note that adding additional con‐
nections is not always productive - just because opening twice
as many connections results in a small percentage increase of
articles accepted by the remote peer, this may be at consider‐
able resource cost both locally and at the remote site, whereas
the remote site might well have received the extra articles sent
from another peer a fraction of a second later. Opening large
numbers of connections is considered antisocial. The meanings
of the various settings are:
0 no method
Increase of connections up to max-connections is unre‐
strained.
1 maximize articles per second
Connections are increased (up to max-connections) and
decreased so as to maximize the number of articles per
second sent, while using the fewest connections to do
this.
2 set target queue length
Connections are increased (up to max-connections) and
decreased so as to keep the queue of articles to be sent
within the bounds set by dynamic-backlog-low and dynamic-
backlog-high, while using the minimum resources possible.
As the queue will tend to fill if the site is not keeping
up, this method ensures that the maximum number of arti‐
cles are offered to the peer while using the minimum num‐
ber of connections to achieve this.
3 combination
This method uses a combination of methods 1 and 2 above.
For sites accepting a large percentage of articles,
method 2 will be used to ensure these sites are offered
as complete a feed as possible. For sites accepting a
small percentage of articles, method 1 is used, to mini‐
mize remote resource usage. For intermediate sites, an
appropriate combination is used.
dynamic-backlog-low
This key requires an integer value between 0 and 100. It repre‐
sents (as a percentage) the low water mark for the host queue.
If the host queue falls below this level while using dynamic-
method 2 or 3, and if 2 or more connections are open, innfeed
will attempt to drop connections to the host. An IIR filter is
applied to the value to prevent connection flap (see dynamic-
filter). A value of 25.0 is recommended. This value must be
smaller than dynamic-backlog-high.
dynamic-backlog-high
This key requries an integer value between 0 and 100. It repre‐
sents (as a percentage) the high water mark for the host queue.
If the host queue rises above this level while using dynamic-
method 2 or 3, and if less than max-connections are open to the
host, innfeed will attempt to open further connections to the
host. An IIR filter is applied to the value to prevent connec‐
tion flap (see dynamic-filter). A value of 50.0 is recommended.
This value must be larger than dynamic-backlog-low.
dynamic-backlog-filter
This key requires a floating-point value between 0 and 1. It
represents the filter coefficient used by the IIR filter used to
implement dynamic-method 2 and 3. The recommended value of this
filter is 0.7, giving a time constant of 1/(1-0.7) articles.
Higher values will result in slower response to queue fullness
changes, lower values in faster response.
max-queue-size
This key requires a positive integer value. It defines the maxi‐
mum number of articles to process at one time when using stream‐
ing to transmit to a peer. Larger numbers mean more memory con‐
sumed as articles usually get pulled into memory (see the
description of use-mmap).
streaming
This key requires a boolean value. It defines whether streaming
commands are used to transmit articles to the peers.
no-check-high
This key requires a floating-point number which must be in the
range [0.0, 100.0]. When running transmitting with the streaming
commands, innfeed attempts an optimization called ``no-CHECK''
mode. This involves not asking the peer if it wants the article,
but just sending it. This optimization occurs when the percent‐
age of the articles the peer has accepted gets larger than this
number. If this value is set to 100.0, then this effectively
turns off no-CHECK mode, as the percentage can never get above
100.0. If this value is too small, then the number of articles
the peer rejects will get bigger (and your bandwidth will be
wasted). A value of 95.0 usually works pretty well. NOTE: In
innfeed 0.9.3 and earlier this value was in the range [0.0,
9.0].
no-check-low:
This key requires a floating-point number which must be in the
range [0.0, 100.0), and it must be smaller that the value for
no-check-high. When running in no-CHECK mode, as described
above, if the percentage of articles the remote accepts drops
below this number, then the no-CHECK optimization is turned off
until the percentage gets above the no-check-high value again.
If there is small difference between this and the no-check-high
value (less than about 5.0), then innfeed may frequently go in
and out of no-CHECK mode. If the difference is too big, then it
will make it harder to get out of no-CHECK mode when necessary
(wasting bandwidth). Keeping this to between 5.0 and 10.0 less
than no-check-high usually works pretty well.
no-check-filter
This is a floating point value representing the time constant,
in articles, over which the CHECK / no-CHECK calculations are
done. The recommended value is 50.0 which will implement an IIR
filter of time constant 50. This roughly equates to making a
decision about the mode over the previous 50 articles. A higher
number will result in a slower response to changing percentages
of articles accepted; a lower number will result in a faster
response.
bindaddress
This key requires a string value. It specifies which outgoing
IPv4 address innfeed should bind the local end of its connection
to. Must be an IPv4 address in dotted-quad format
(nnn.nnn.nnn.nnn), "any", or "none". If not set or set to
"any", innfeed defaults to letting the kernel choose this
address. If set to "none", innfeed will not use IPv4 for outgo‐
ing connections to peers in this scope (i.e. it forces IPv6).
If not set in innfeed.conf, innfeed defaults to the value of
sourceaddress from inn.conf(5) (which by default is unset).
bindaddress6
This key requires a string value. It behaves like bindaddress
except for outgoing IPv6 connections. Must be in numeric IPv6
format (note that a value containing colons must be enclosed in
double quotes), "any", or "none". If set to "none", innfeed
will not use IPv6 for outgoing connections to peers in this
scope. If not set in innfeed.conf, innfeed defaults to the
value of sourceaddress6 from inn.conf(5) (which by default is
unset).
port-number
This key requires a positive integer value. It defines the
tcp/ip port number to use when connecting to the remote.
force-ipv4
This key requires a boolean value. By default it is set to
false. Setting it to true is the same as setting "bindaddress6:
none" and removing "bindaddress: none" if it was set.
drop-deferred
This key requires a boolean value. By default it is set to
false. When set to true, and a peer replies with code 431 or 436
(try again later) just drop the article and don't try to re-send
it. This is useful for some peers that keep on deferring arti‐
cles for a long time to prevent innfeed from trying to offer the
same article over and over again.
min-queue-connection
This key requires a boolean value. By default it is set to
false. When set to true, innfeed will attempt to use a connec‐
tion with the least queue size (or the first empty connection).
If this key is set to true, it is recommended that dynamic-
method be set to 0. This allows for article propagation with the
least delay.
no-backlog
This key requires a boolean value. It specifies whether spooling
should be enabled (false, the default) or disabled (true). Note
that when no-backlog is set, articles reported as "spooled" are
actually silently discarded.
backlog-limit
This key requires a non-negative integer value. If the number is
0 then backlog files are allowed to grown without bound when the
peer is unable to keep up with the article flow. If this number
is greater than 0 then it specifies the size (in bytes) the
backlog file should get truncated to when the backlog file
reaches a certain limit. The limit depends on whether backlog-
factor or backlog-limit-highwater is used.
backlog-factor
This key requires a floating point value, which must be larger
than 1.0. It is used in conjunction with the peer key backlog-
limit. If backlog-limit has a value greater than zero, then when
the backlog file gets larger than the value backlog-limit *
backlog-factor, then the backlog file will be truncated to the
size backlog-limit. For example if backlog-limit has a value of
1000000, and backlog-factor has a value of 2.0, then when the
backlogfile gets to be larger than 2000000 bytes in size, it
will be truncated to 1000000 bytes. The front portion of the
file is removed, and the trimming happens on line boundaries, so
the final size may be a bit less than this number. If backlog-
limit-highwater is defined too, then backlog-factor takes prece‐
dence.
backlog-limit-highwater
This key requires a positive integer value that must be larger
than the value for backlog-limit. If the size of the backlog
file gets larger than this value (in bytes), then the backlog
file will be shrunk down to the size of backlog-limit. If both
backlog-factor and backlog-limit-highwater are defined, then the
value of backlog-factor is used.
backlog-feed-first
This key requires a boolean value. By default it is set to
false. When set to true, the backlog is fed before new files.
This is intended to enforce in-order delivery, so setting this
to true when initial-connections or max-connections is more than
1 is inconsistent.
username
This key requires a string value. If the value is defined, then
innfeed tries to authenticate by ``AUTHINFO USER'' and this
value used for user name. password must also be defined, if
this key is defined.
password
This key requires a string value. The value is the password
used for ``AUTHINFO PASS''. username must also be defined, if
this key is defined.
deliver
This key is used with imapfeed to authenticate to a remote host.
It is optional. There are several parameters that must be
included with deliver:
deliver-authname
The authname is who you want to authenticate as.
deliver-password
This is the appropriate password for authname.
deliver-username
The username is who you want to "act" as, that is, who is
actually going to be using the server.
deliver-realm
In this case, the "realm" is the realm in which the spec‐
ified authname is valid. Currently this is only needed
by the DIGEST-MD5 SASL mechanism.
deliver-rcpt-to
A printf-style format string for creating the envelope
recipient address. The pattern MUST include a single
string specifier which will be replaced with the newgroup
(e.g "bb+%s"). The default is "+%s".
deliver-to-header
An optional printf-style format string for creating a To:
header to be prepended to the article. The pattern MUST
include a single string specifier which will be replaced
with the newgroup (e.g "post+%s@domain"). If not speci‐
fied, the To: header will not be prepended.
PEER VALUES
As previously explained, the peer definitions can contain redefinitions
of any of the key/value pairs described in the GLOBAL PEER DEFAULTS
section above. There is one key/value pair that is specific to a peer
definition.
ip-name
This key requires a word value. The word is the host's FQDN, or
the dotted quad ip-address. If this value is not specified then
the name of the peer is taken to also be its ip-name. See the
entry for data.ramona.vix.com in the example below.
RELOADING
If innfeed gets a SIGHUP signal, then it will reread the config file.
All values at global scope except for ``backlog-directory'' can be
changed (although note that ``bindaddress'' and ``bindaddress6''
changes will only affect new connections). Any new peers are added and
any missing peers have their connections closed.
EXAMPLE
Below is the sample innfeed.conf file.
#
# innfeed.conf file. See the comment block at the
# end for a fuller description.
#
##
## Global values. Not specific to any peer. These
## are optional, but if used will override the
## compiled in values. Command-line options used
## will override these values.
##
pid-file: innfeed.pid
debug-level: 0
use-mmap: false
log-file: innfeed.log
stdio-fdmax: 0
backlog-directory: innfeed
backlog-rotate-period: 60
backlog-ckpt-period: 30
backlog-newfile-period: 600
dns-retry: 900
dns-expire: 86400
close-period: 3600
gen-html: false
status-file: innfeed.status
connection-stats: false
host-queue-highwater: 200
stats-period: 600
stats-reset: 43200
max-reconnect-time: 3600
initial-reconnect-time: 30
##
## Defaults for all peers. These must all exist at
## global scope. Any of them can be redefined
## inside a peer or group definition.
##
article-timeout: 600
response-timeout: 300
initial-connections: 1
max-connections: 5
max-queue-size: 25
streaming: true
no-check-high: 95.0
no-check-low: 90.0
no-check-filter: 50.0
port-number: 119
backlog-limit: 0
backlog-factor: 1.10
backlog-limit-highwater:0
dynamic-method: 3
dynamic-backlog-filter: 0.7
dynamic-backlog-low: 25.0
dynamic-backlog-high: 50.0
no-backlog: false
##
## Peers.
##
peer decwrl {
ip-name: news1.pa.dec.com
}
peer uunet {
ip-name: news.uunet.uu.net
max-connections: 10
}
peer data.ramona.vix.com {
# ip-name defaults to data.ramona.vix.com
streaming: false
}
peer bb.home.vix.com {
ip-name: 192.5.5.33
}
# Blank lines are ignored. Everything after a '#'
# is ignored too.
#
# Format is:
# key : value
#
# See innfeed.conf(5) for a description of
# necessary & useful keys. Unknown keys and their
# values are ignored.
#
# Values may be a integer, floating-point, c-style
# single-quoted characters, boolean, and strings.
#
# If a string value contains whitespace, or
# embedded quotes, or the comment character
# (``#''), then the whole string must be quoted
# with double quotes. Inside the quotes, you may
# use the standard c-escape sequence
# (\t,\n,\r,\f,\v,\",\').
#
# Examples:
# eg-string: "New\tConfig\tfile\n"
# eg-long-string: "A long string that goes
# over multiple lines. The
# newline is kept in the
# string except when quoted
# with a backslash \
# as here."
# eg-simple-string: A-no-quote-string
# eg-integer: 10
# eg-boolean: true
# eg-char: 'a'
# eg-ctrl-g: '\007'
HISTORY
Written by James Brister <brister@vix.com> for InterNetNews. This is
revision 8243, dated 2008-12-21.
SEE ALSOinnfeed(1), newsfeeds(5)innfeed.conf(5)
