Tips N Tricks

Tips And Tricks

---

1.  Easy debugging

If you wish to debug specific daemon, there is a simple way to accomplish this, avoiding logging to a file. One need to login to a daemon-to-debug e.g. bgpd and issue set of commands, like below, and debugging messages will be put on stdout. No more runaway log files, and similar. (for more info visit this custom essay Example session:

kaermorhen:~# telnet localhost bgpd
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.

Hello, this is Quagga (version 0.99.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification

Password:
kaermorhen.1|bgpd# terminal monitor
kaermorhen.1|bgpd# debug bgp
BGP debugging is on
kaermorhen.1|bgpd# BGP: a.b.c.131 rcv message type 4, length (excl. header) 0
BGP: x.y.z.121 send message type 4, length (incl. header) 19
BGP: z.x.c.100 rcv message type 4, length (excl. header) 0
BGP: q.w.e.2 send message type 4, length (incl. header) 19
BGP: q.w.e.2 rcv message type 4, length (excl. header) 0
BGP: a.b.c.131 send message type 4, length (incl. header) 19
BGP: z.x.c.100 send message type 4, length (incl. header) 19
BGP: z.y.z.121 send message type 4, length (incl. header) 19

kaermorhen.1|bgpd# no debug bgp
BGP debugging is off
kaermorhen.1|bgpd# no terminal monitor
kaermorhen.1|bgpd#

2.  Useful command line switches (Linux):

zebra: (quoting manual)

-s, --nl-bufsize netlink-buffer-size
Set netlink receive buffer size. There are cases where zebra daemon can't handle flood of netlink messages from kernel. If you ever see "recvmsg overrun" messages in zebra log, you are in trouble. Please see essay for more info

Solution is to increase receive buffer of netlink socket. Note that kernel doesn't allow to increase it over maximum value defined in /proc/sys/net/core/rmem_max. If you want to do it, you have to increase maximum before starting zebra.

Note that this affects Linux only.

On large ospfd networks with many areas containing a lot of short prefixes, it would be usefull to set up above entries like this:

net.core.rmem_max=2097152
net.core.rmem_default=2097152

and start zebra with the following parameter:

-s 2097152

This solves problem with 'recvmsd overrun' seen in logs, and problems with prefixes ocassionally disapearing across and/or inside areas.

Set up this parameters, when running an OSPF area with large number of routes.

3.  OSPF tips

To see what routes we're actually originating, one must issue the following CLI command:

router# show ip ospf database self-orig

This will show all routes we announce to other neighbor(s)/areas.

To see all neighbors in a specific area, one must issue the following CLI command:

router# show ip ospf database

and seek for a part entitled "Router Link States (area a.b.c.d)".

To see directly connected neighbors in a specific area, do this:

router# show ip ospf neighbors

4.  Providing backtrace (bt) for dummies ;)

The description below is based on Debian, if you run anyhthing else, check where your binaries are, and provide correct paths.

When you hit a bug in Quagga, that makes one of your daemons segfaulting, a backtrace should be provided for developers to easily found, what is wrong, and where to seek for it. Task is simple. If hitting a segfault, that is reproducible, one should run the segfaulting daemon under gbd control.

host# gdb /usr/lib/quagga/ospfd
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

gdb) run

Starting program: /usr/lib/quagga/ospfd

If you want to attach to a running process, perform the following:

gdb) /usr/lib/quagga/bgpd <pidof bgpd>
gdb) cont

then, you need to reproduce activities that caused segfault telnetting into daemon, e.g.

 
router# what made bgpd segfaults I repeat

(daemon segfaults, now we get backtrace)

gdb) bt

(spits out backtrace)

Example backtrace can be found in e.g. bug #267
ATTENTION: for a good backtrace, quagga's binaries must not be stripped! E.g. Debian strips binaries by default, so to produce a good bt, one must rebuild package with dh_strip option removed from debian/rules script. For more info how to rebuild Debian packages, refer to their homepage.

5.  Installing and making Quagga Suite run on Debian (and derivatives)

First of all, you have to install Quagga:

# aptitude install quagga

Determine which routing protocol you need to use and edit /etc/quagga/daemons.

Example below enables ospfd and bgpd. Do not forget to enable zebra daemon too. Attention: if you do not enable zebra, none of routes exchanged using routing protocols will be installed in routing table.

# cat /etc/quagga/daemons
zebra=yes
bgpd=yes
ospfd=yes
ospf6d=no
ripd=no
ripngd=no
isisd=no
#

next step is creating config files for every daemon.

 
# touch /etc/quagga/{bgpd.conf,ospfd.conf,zebra.conf}

take care of permissions:

chown quagga.quaggavty /etc/quagga/*.conf

and do not forget about setting up password for daemons:

 
# echo 'password YourPassHere' > /etc/quagga/bgpd.conf
# echo 'password YourPassHere' > /etc/quagga/ospfd.conf
# echo 'password YourPassHere' > /etc/quagga/zebra.conf

anything else may be edited by telnetting to appropiate port, like this:

$ telnet localhost ospfd - to configure ospf daemon,
$ telnet localhost zebra - to configure zebra daemon,
$ telnet localhost bgpd - to configure bgp daemon

When asked about password, use the one you've chosen in previous step.

If you seek for a tool to configure all daemons without switching telnet sessions, use vtysh. Create minimal config file for vtysh:

# echo 'username root nopassword' > /etc/quagga/vtysh.conf

Now, you can use vtysh as user root without password, like this:

rtr:~# vtysh

Hello, this is Quagga (version 0.99.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

rtr|vtysh#

After entering vtysh if you see blank screen with highlighted (END) in the left-down corner, press 'q' to exit pager, try setting "VTYSH_PAGER=more" in your .profile or site-wide /etc/environment, and try running vtysh once again. On recent Debian (Etch and >) seems that /etc/environment is ignored . A workaround is to add "export VTYSH_PAGER=more" in /etc/bash.bashrc.

If you want each daemon listening not only on loopback (which is wise and enabled by default in Debian), you have to edit /etc/quagga/debian.conf and remove -A option, or change it to any IP address you want daemons to listen on.

6.  Return Path filters

If you have setup multihomed BGP peerings and if your traffic keep blocked on your router when all your peerings are up check that your return path filters are turned off ...

By default Linux block traffic that go out from an interface and come back from another.

To solve the issue under debian (and probably other distributions) add these lines in your sysctl.conf an call sysctl -p to refresh

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0

7.  Using gdb to debug Quagga on Debian (Lenny/Sid)

When attempting to use gdb on any daemon except ospfd on Debian, gdb refuse to set any breakpoints in the executable.

The reason for this is: All Quagga daemons except ospfd are compiled and linked to be PIE (Position Independent Executable) as default, which is currently (January 2009) not supported by gdb on Debian.

The immediate solution is to do a "make distclean && autoreconf -i && ./configure.... --disable-pie" and rebuild Quagga all the way.

Breakpoints can now be used throughout. To verify the file type of the daemon, which must be EXEC on Debian:

E.g.: readelf -h /usr/local/sbin/zebra |grep Type

Using the standard Makefile DYN will be returned.