• Part 1 – running tftp server non root (xinetd)
• Part 2 – the tftp client requires firewalld changes as well (this blog post)
• Part 3 – replacing xinetd with systemd

firewalld change on TFTP clients

The rest of this blog post will elaborate on what happens if you don’t do this.

The quick bit is: if you want to run the TFTP client on, say, RHEL7, you need to enable a service in firewalld on the client.

Obviously, I’m assuming you’ve got firewalld turned on, otherwise you wouldn’t be here.

/bin/firewall-cmd --permanent --zone public --add-service tftp-client

Puppet code, using crayfishx/firewalld, and the firewalld-cmd fix for this follows.

  firewalld_service {'enable tftp client in firewalld':
    ensure  => 'present',
    zone    => 'public',
    service => 'tftp-client',
  }

Quick test:

tftp -4 -v 192.168.1.223 -c get foo/bar

the files transfer, but they’re empty?!

24 hours earlier.

Once I’d got the TFTP server running cleanly, unhelpful stuff was happening.  I tried various things, in summary:

client:

$ tftp -4 -v 192.168.1.223 -c get foo
Connected to 192.168.1.223 (192.168.1.223), port 69
getting from 192.168.1.223:foo to foo [netascii]
@����
$ ls -al foo
-rw-rw-r--. 1 ben ben 0 Mar 12 17:06 foo

server:

in.tftpd[2235]: RRQ from 192.168.1.245 filename foo

Yeah, complete with random noise from the client. Nice.

The file’s empty.  There’s no error anywhere, and there’s nothing else logged.  (Spoiler: if it was all working correctly, more would be logged on the server.)

• I turned off the server’s firewalld.
• I set selinux permissive on the server, and the following is clean.

ausearch -m AVC,USER_AVC -ts recent

From Part 1: the tftp daemon stays running.

That makes it much easier to trace, which at this point became necessary.  Ran out of ideas.  I’m not a syscall guru by any means, but it all started with truss on Solaris, and I still fall back to it sometimes.

I was expecting a problem reading the file, in which case the syscalls will show this.

I’ve annotated the end of it here; the PID is that root owned TFTP process. (see Part 1.)

# strace -fp <PID>

   # server IP
[pid  2249] bind(0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.1.223")}, 16) = 0
   # client IP
[pid  2249] connect(0, {sa_family=AF_INET, sin_port=htons(53450), sin_addr=inet_addr("192.168.1.245")}, 16) = 0
[pid  2249] setsockopt(0, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
   # open requested fgile
[pid  2249] open("foo", O_RDONLY|O_LARGEFILE) = 1
[pid  2249] fstat64(1, {st_mode=S_IFREG|0444, st_size=29, ...}) = 0
[pid  2249] fcntl64(1, F_SETLK64, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
[pid  2249] fcntl64(1, F_GETFL)         = 0x20000 (flags O_RDONLY|O_LARGEFILE)
[pid  2249] gettimeofday({1552408345, 873122}, NULL) = 0
   # syslog 'Mar 12 16:32:25 in.tftpd[2249]: RRQ from 192.168.1.245 filename foo'
[pid  2249] send(3, "<29>Mar 12 16:32:25 in.tftpd[224"..., 72, MSG_NOSIGNAL) = 72
[pid  2249] fstat64(1, {st_mode=S_IFREG|0444, st_size=29, ...}) = 0
[pid  2249] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f88000
   # the file foo contains the output of 'date'
[pid  2249] read(1, "Tue Mar 12 16:26:48 UTC 2019n", 4096) = 29
[pid  2249] read(1, "", 4096)           = 0
   # send the contents of the file ..
[pid  2249] send(0, "31Tue Mar 12 16:26:48 UTC 2019"..., 33, 0) = 33
[pid  2249] poll([{fd=0, events=POLLIN}], 1, 1000) = 1 ([{fd=0, revents=POLLERR}])
   #
   # here's the key error - E(rror) HOST UNREACH(able)
   #
[pid  2249] recv(0, 0x9690b0, 516, 0)   = -1 EHOSTUNREACH (No route to host)
   # also a bit weird; I think this is error logging
   # perhaps file handles are used to control the volume on debugging.
[pid  2249] write(2, "fatal: ", 7)      = -1 EBADF (Bad file descriptor)
[pid  2249] write(2, "recvfrom_flags_with_timeout: rec"..., 54) = -1 EBADF (Bad file descriptor)
[pid  2249] fstat64(1, {st_mode=S_IFREG|0444, st_size=29, ...}) = 0
[pid  2249] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f87000
[pid  2249] write(1, "n", 1)           = -1 EBADF (Bad file descriptor)
[pid  2249] exit_group(1)               = ?
[pid  2249] +++ exited with 1 +++

Not a problem reading the file.

It reads the file, tries to pass the data to the client, loses the client.

A network issue.

firewalld on the client

Process of elimination.

I don’t know tftp that well, but I do know that FTP traffic is a bit odd and involves two sessions, and firewalls have to cope with that.

Here we have a hand off between daemons on the server side, perhaps there’s some oddness with ports, and the *client* firewall has to keep track of it. Otherwise, you’ve got incoming data from an unexpected source.

Of course, you don’t usually run the TFTP client – that’s a Cisco device or appliance doing a backup, or it’s your PXE client.  It just works, right?

So client side:

$ sudo systemctl stop firewalld 
[sudo] password for ben: 
$ sudo systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-03-12 20:19:32 GMT; 6s ago
     Docs: man:firewalld(1)
$ tftp -4 -v 192.168.1.223 -c get foo
Connected to 192.168.1.223 (192.168.1.223), port 69
getting from 192.168.1.223:foo to foo [netascii]
Received 29 bytes in 0.1 seconds [1854 bit/s]

#
# server side
#

in.tftpd[2724]: RRQ from 192.168.1.245 filename foo
in.tftpd[2724]: Client 192.168.1.245 finished foo

#
# client side
#

$ sudo systemctl start firewalld 
$ tftp -4 -v 192.168.1.223 -c get foo
Connected to 192.168.1.223 (192.168.1.223), port 69
getting from 192.168.1.223:foo to foo [netascii]
Received 29 bytes in 0.1 seconds [2203 bit/s]

Insert scratchy screetching noise here.

Just ignore that last bit. Firewalld on the client I was testing with got screwed up. That shouldn’t have worked.

firewalld[28663]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table mangle 
--delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 
--jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.

• I rebooted and the problem came back.
• Post reboot, stopping firewalld fixed it again, starting it broke it again.