Configuration¶
fedmsg requires some configuration before it will work properly.
The configuration values used at runtime are determined by checking in the following order:
Built-in defaults
All Python files in the /etc/fedmsg.d/ directory
All Python files in the ~/.fedmsg.d/ directory
All Python files in the current working directory’s fedmsg.d/ directory
Command line arguments
Within each directory, files are loaded in lexicographical order.
For example, if a config value does not appear in either the config file or on the command line, then the built-in default is used. If a value appears in both the config file and as a command line argument, then the command line value is used.
You can print the runtime configuration to the terminal by using the
fedmsg-config
command implemented by
fedmsg.commands.config.config()
.
General configuration¶
active¶
A boolean that, if True
, will cause the publishing socket to connect to the
relay_inbound socket, rather than binding its own socket.
topic_prefix¶
A string prefixed to the topics of all outgoing messages.
The default value is org.fedoraproject
.
environment¶
A string that must be one of ['prod', 'stg', 'dev']
. It signifies
the environment in which this fedmsg process is running and can be used
to weakly separate different logical buses running in the same
infrastructure. It is used by fedmsg.publish()
when it is
constructing a fully-qualified topic.
status_directory¶
A string that is the absolute path to a directory where consumers can save the status of their last processed message. In conjunction with datagrepper_url, allows for automatic retrieval of backlog on daemon startup.
datagrepper_url¶
A URL to an instance of the datagrepper web service, such as https://apps.fedoraproject.org/datagrepper/raw. Can be used in conjunction with status_directory to allow for automatic retrieval of backlog on daemon startup.
skip_last_message¶
Upon restart skip the message that we worked on previously. This is an attempt to have the processing of the queue resume at the expense of droppin a message and possibly not awarding a badge.
endpoints¶
dict
- A mapping of “service keys” to “zeromq endpoints”; the
heart of fedmsg.
endpoints
is “a list of possible addresses from which fedmsg can
send messages.” Thus, “subscribing to the bus” means subscribing to
every address listed in this dictionary.
endpoints
is also an index where a fedmsg process can look up
what port it should bind to to begin emitting messages.
When fedmsg.init()
is invoked, a “name” is determined. It is
either passed explicitly, or guessed from the call stack. The name is
combined with the hostname of the process and used as a lookup key in
the endpoints
dict.
When sending, fedmsg will attempt to bind to each of the addresses
listed under its service key until it can succeed in acquiring the port.
There needs to be as many endpoints listed as there will be
processes * threads
trying to publish messages for a given
service key.
For example, the following config provides for four WSGI processes on bodhi on the machine app01 to send fedmsg messages.
>>> config = dict(
... endpoints={
... "bodhi.app01": [
... "tcp://app01.phx2.fedoraproject.org:3000",
... "tcp://app01.phx2.fedoraproject.org:3001",
... "tcp://app01.phx2.fedoraproject.org:3002",
... "tcp://app01.phx2.fedoraproject.org:3003",
... ],
... },
... )
If apache is configured to start up five WSGI processes, the fifth
one will produce tracebacks complaining with
IOError("Couldn't find an available endpoint.")
.
If apache is configured to start up four WSGI processes, but with two threads each, four of those threads will raise exceptions with the same complaints.
A process subscribing to the fedmsg bus will connect a zeromq SUB
socket to every endpoint listed in the endpoints
dict. Using
the above config, it would connect to the four ports on
app01.phx2.fedoraproject.org.
Note
This is possibly the most complicated and hardest to understand part of fedmsg. It is the black sheep of the design. All of the simplicity enjoyed by the python API is achieved at cost of offloading the complexity here.
Some work could be done to clarify the language used for “name” and
“service key”. It is not always consistent in fedmsg.core
.
srv_endpoints¶
list
- A list of domain names for which to query SRV records
to get the associated endpoints.
When using fedmsg.config.load_config(), the DNS lookup is done and the resulting endpoints are added to config[‘endpoint’][$DOMAINNAME]
For example, the following would query the endpoints for foo.example.com.
>>> config = dict(
... srv_endpoints=[foo.example.com]
...)
replay_endpoints¶
dict
- A mapping of service keys, the same as for endpoints
to replay endpoints, each key having only one. The replay endpoints are
special ZMQ endpoints using a specific protocol to allow the client to
request a playback of messages in case some have been dropped, for
instance due to network failures.
If the service has a replay endpoint specified, fedmsg will automatically try to detect such failures and properly query the endpoint to get the playback if needed.
relay_inbound¶
str
- A special zeromq endpoint where the inbound, passive zmq SUB
sockets for instances of fedmsg-relay
are listening.
Commands like fedmsg-logger
actively connect here and publish their
messages.
See Commands for more information.
relay_outbound¶
str
- A list of special zeromq endpoints where the outbound
sockets for instances of fedmsg-relay
should bind.
fedmsg.consumers.gateway.port¶
int
- A port number for the special outbound zeromq PUB socket
posted by fedmsg.commands.gateway.gateway()
. The
fedmsg-gateway
command is described in more detail in
Commands.
ZeroMQ¶
The following settings are ZeroMQ configuration options.
high_water_mark¶
int
- An option to zeromq that specifies a hard limit on the maximum
number of outstanding messages to be queued in memory before reaching an
exceptional state.
For our pub/sub zeromq sockets, the exceptional state means dropping messages. See the upstream documentation for ZMQ_HWM and ZMQ_PUB.
A value of 0
means “no limit” and is the
recommended value for fedmsg. It is referenced when initializing
sockets in fedmsg.init()
.
io_threads¶
int
- An option that specifies the size of a zeromq thread pool to
handle I/O operations. See the upstream documentation for zmq_init.
This value is referenced when initializing the zeromq context in
fedmsg.init()
.
post_init_sleep¶
float
- A number of seconds to sleep after initializing and before
sending any messages. Setting this to a value greater than zero is
required so that zeromq doesn’t drop messages that we ask it to send
before the pub socket is finished initializing.
Experimentation needs to be done to determine and sufficiently small and
safe value for this number. 1
is definitely safe, but annoyingly
large.
zmq_enabled¶
bool
- A value that must be true. It is present solely
for compatibility/interoperability with moksha.
zmq_reconnect_ivl¶
int
- Number of miliseconds that zeromq will wait to reconnect
until it gets a connection if an endpoint is unavailable. This is in
miliseconds. See upstream zmq options for more information.
zmq_reconnect_ivl_max¶
int
- Max delay that you can reconfigure to reduce reconnect storm
spam. This is in miliseconds. See upstream zmq options for more information.
zmq_strict¶
bool
- When false, allow splats (‘*’) in topic names when
subscribing. When true, disallow splats and accept only strict matches
of topic names.
This is an argument to moksha and arose there to help abstract away differences between the “topics” of zeromq and the “routing_keys” of AMQP.
zmq_tcp_keepalive¶
int
- Interpreted as a boolean. If non-zero, then keepalive options
will be set.
See upstream zmq options and general overview.
zmq_tcp_keepalive_cnt¶
int
- Number of keepalive packets to send before considering the
connection dead.
See upstream zmq options and general overview.
zmq_tcp_keepalive_idle¶
int
- Number of seconds to wait after last data packet before
sending the first keepalive packet.
See upstream zmq options and general overview.
zmq_tcp_keepalive_intvl¶
int
- Number of seconds to wait inbetween sending subsequent
keepalive packets.
See upstream zmq options and general overview.
IRC¶
irc¶
list
- A list of ircbot configuration dicts. This is the primary
way of configuring the fedmsg-irc
bot implemented in
fedmsg.commands.ircbot.ircbot()
.
Each dict contains a number of possible options. Take the following .. rubric:: Example
>>> config = dict(
... irc=[
... dict(
... network='irc.freenode.net',
... port=6667,
... nickname='fedmsg-dev',
... channel='fedora-fedmsg',
... timeout=120,
...
... make_pretty=True,
... make_terse=True,
... make_short=True,
...
... filters=dict(
... topic=['koji'],
... body=['ralph'],
... ),
... ),
... ],
... )
Here, one bot is configured. It is to connect to the freenode network
on port 6667. The bot’s name will be fedmsg-dev
and it will
join the #fedora-fedmsg
channel.
make_pretty
specifies that colors should be used, if possible.
make_terse
specifies that the “natural language” representations
produced by fedmsg.meta
should be echoed into the channel instead
of raw or dumb representations.
make_short
specifies that any url associated with the message
should be shortened with a link shortening service. If True, the
https://da.gd/ service will be used. You can alternatively specify a
callable to use your own custom url shortener, like this:
>>> def make_short(url):
... service_url = ('http://api.bitly.com/v3/shorten?login=YOURLOGIN&'
... 'apiKey=YOURAPIKEY&longUrl={url}&format=txt')
... return requests.get(service_url.format(url=url).text.strip()
The filters
dict is not very smart. In the above case, any message
that has ‘koji’ anywhere in the topic or ‘ralph’ anywhere in the JSON
body will be discarded and not echoed into #fedora-fedmsg
. This is
an area that could use some improvement.
irc_color_lookup¶
A dictionary mapping module names to MIRC irc color names. For example:
>>> irc_color_lookup = {
... "fas": "light blue",
... "bodhi": "green",
... "git": "red",
... "tagger": "brown",
... "wiki": "purple",
... "logger": "orange",
... "pkgdb": "teal",
... "buildsys": "yellow",
... "planet": "light green",
... }
irc_method¶
the name of the method used to publish the messages on IRC.
Valid values are msg
and notice
.
The default is notice
.
STOMP Configuration¶
When using STOMP, you need to set zmq_enabled to False
.
Additionally, if you’re using STOMP with TLS (recommended), you do not need
fedmsg’s cryptographic signatures to validate messages so you can turn those
off by setting validate_signatures to False
stomp_uri¶
A string of comma-separated brokers. For example:
stomp_uri='broker01.example.com:61612,broker02.example.com:61612'
There is no default for this setting.
stomp_heartbeat¶
The STOMP heartbeat interval, in milliseconds.
There is no default for this setting.
stomp_user¶
The username to use with STOMP when authenticating with the broker.
There is no default for this setting.
stomp_pass¶
The password to use with STOMP when authenticating with the broker.
There is no default for this setting.
stomp_ssl_crt¶
The PEM-encoded x509 client certificate to use when authenticating with the broker.
There is no default for this setting.
stomp_ssl_key¶
The PEM-encoded private key for the stomp_ssl_crt.
There is no default for this setting.
stomp_queue¶
If set, this will cause the Moksha hub to only listen to the specified queue for all fedmsg consumers. If it is not specified, the Moksha hub will listen to all topics declared by all fedmsg consumers.
There is no default for this setting.