[BIGPATCH] Numerous fixes and improvements

[guest]

Hi all!

It's this time of year again when I have too little time and too much to do. So I'd better wrap up the things I've been doing in very small chunks for the past many weeks. Here it is.

BIG FAT WARNING: This patch changes the protocol to be incompatible with all previous versions. If you use this patch (which you want, trust me), you need to update ALL iodine servers and clients you're using.

END OF BIG FAT WARNING.

I really don't know why I've been doing this, since I don't expect to have any use for iodine in the forseeable future. But I guess I just can't stand thinking of solutions to problems and then not implementing them.

This was all started by a very small e-mail conversation I had with "The" Lucas Nussbaum. Hi Lucas ;-) That got me thinking a bit. And later checking out iodine's progress more closely, seeing that you bumped the protocol version incompatibly anyway, it got me thinking even more. And when I got drawn into it again, I couldn't possibly stop myself from fixing and improving random things I encountered here and there.

So with this I humbly submit another huge patch for your review. Against svn head this time, to ease the pain.

To apply:

download lazy2.patch.gz from this ticket
svn co -r 956 http://svn.kryo.se/iodine iodine
cd iodine
gunzip < ../lazy2.patch.gz | patch -p1
make

The result works on _all_ DNS relays you might encounter.

Well... actually... that's left as an exercise for the reader. There are many hundreds of public DNS servers out there, listed in several places. Try them all. And report which DNS servers do _not_ work, so that things can be fixed. I claim there won't be any. Prove me wrong.

The long list of changes in this patch:

• Add CMC to upstream data packets, to perfectly distinguish real

retries and DNS-generated duplicates. Gives much better "hammering" resistance, and much cleaner logic. This is fully incompatible with the previous protocol.

• Add very space-efficient query memory to handle duplicates that arrive

long after the small full-query+answer cache has forgotten them. Much improvement in moderate-data-speed situations with multiple parallel DNS relays.

• Answer both the original query and the latest duplicate. This all

makes opendns.com perfectly usable with -I25. Yay!

• Detect SERVFAIL problems and switch to -I1 if needed (fixes #81).
  
• Add SRV request support.
  
• Return multi-hostname for SRV and MX, bandwidth now close to NULL/TXT.

Broke out some hostname en/decoding chunks to separate functions to ease this.

• Completely rewritten base32/64 en/decoding to encode maximally within

any given destination buffer space, doing away completely with that blksize nonsense.

• Add base64u and base128 codecs for upstream and downstream.
  
• (Aside: Full-binary hostnames would probably work too in many places

according to RFC2181 sec 11 and RFC1123 sec 6.1.3.5 (end), but this would require eliminating the non-binary-safe un/dotify and associated stuff. Note that RFC1035 sec 2.3.1 is only _preferred_, sec 2.3.3 and 3.1 speak of full-binary.)

• Update and extend upstream codec autodetection.
  
• Autodetect downstream codec. Introduces new 'Y' request.
  
• Autodetect DNS request type (also fixes #84).
  
• Autodetect EDNS0 support (I found four separate relays that give

FORMERR on EDNS0). Thanks to Lucas Nussbaum for bringing this up.

• Generalized handshake reply handling to reduce code duplication; only

accept replies that we want, and don't get distracted by older or wrong stuff (fixes #74 better).

• Fix 'O' Option handshake (down-codec, lazymode) with patch from #83

(so no need to apply #83 separately).

• Accept only fragsize checks that have perfectly correct 107-rotating

data, since we don't support old servers anyway.

• Add option to reduce upstream hostname length to make some br0ken relays

workable.

• Set iodined mtu default to 1130 to give ~200% speedup in many cases, since

the vast majority of sensible DNS relays give just over 1150 fragsize for NULL.

• Fix client-always-times-out in raw UDP mode.
  
• Applied my documentation skills to revamp the first part (example

section) of the README to be better understandable and more real-world oriented.

• Dropped the example section from the manpage because it doesn't really

belong there and would be rather massive text duplication that has already proven to be very hard to keep synchronized.

• (Question: Do Windows users have any possibility to read the manpage?)
  
• Split iodine's growing -h option listing into two distinct groups.
  
• Fix forwarding between users, also between DNS and raw-UDP.
  
• Fix topdomain stripping in several places to not detect topdomain in

the middle of a request, compare case-insensitive, and require a dot before it.

• Answer A type requests for ns.topdomain (required to recursively

resolve an NS type lookup) and www.topdomain (to pretend usefulness).

The unified diff is again quite unreadable in places. Apply and "diff -c" to have a better look.

Iodine maintainers, after patching you still need to do two minor things before checking in:

• Adjust indents where indicated (grep "indent" *.c); I didn't do that yet

to show that nothing was changed there.

• Rename patched protocol doc to proto_00000502.txt and remove the

"NOTE" line at the top. Possibly get original proto_00000501.txt from previous version or drop it completely, since no official version was ever released with it.

As before, this stuff is licensed under whatever license iodine has or will ever have, as long as it's DFSG-compliant.

That's it for today (and for quite some time, probably).

Best regards, and Season's Greetings,

Anne Bezemer
J.A.Bezemer X opensourcepartners Y nl | tr XY @.

[guest]

lazy2.patch.gz

[guest]

Just like with #75 I had to make a few changes to get this to compile with mingw32.

--- windows.h.orig	2009-12-22 23:24:53.000000000 -0500
+++ windows.h	2009-12-22 23:19:41.000000000 -0500
@@ -32,6 +32,9 @@
 #ifndef DNS_TYPE_TXT
 # define DNS_TYPE_TXT 16
 #endif
+#ifndef T_SRV
+# define T_SRV 33
+#endif
 
 #define T_A DNS_TYPE_A
 #define T_NS DNS_TYPE_NS

I also had to define 'sleep':

--- client.c.orig	2009-12-22 23:26:04.000000000 -0500
+++ client.c	2009-12-22 23:19:57.000000000 -0500
@@ -30,6 +30,7 @@
 #ifdef WINDOWS32
 #include "windows.h"
 #include <winsock2.h>
+#define sleep(seconds) Sleep((seconds)*1000)
 #else
 #include <arpa/nameser.h>
 #ifdef DARWIN

It crosscompiles, but is untested.

[guest]

To compile on OSX I needed to patch the src/Makefile because of sed:

sed: 1: "s/\(base64\)/\1u/ig ; s ...": bad flag in substitute command: 'i'

--- Makefile.orig	2009-12-22 23:03:19.000000000 -0500
+++ Makefile	2009-12-22 23:54:44.000000000 -0500
@@ -34,11 +34,11 @@
 base64u.c: base64.c
 	@echo Making $@
 	@echo '/* No use in editing, produced by Makefile! */' > $@
-	@sed -e 's/\(base64\)/\1u/ig ; s/0123456789+/0123456789_/' < $< >> $@
+	@sed -e 's/\([Bb][Aa][Ss][Ee]64\)/\1u/g ; s/0123456789+/0123456789_/' < $< >> $@
 base64u.h: base64.h
 	@echo Making $@
 	@echo '/* No use in editing, produced by Makefile! */' > $@
-	@sed -e 's/\(base64\)/\1u/ig ; s/0123456789+/0123456789_/' < $< >> $@
+	@sed -e 's/\([Bb][Aa][Ss][Ee]64\)/\1u/g ; s/0123456789+/0123456789_/' < $< >> $@
 
 clean:
 	@echo "Cleaning src/"

Also compiled but untested.

[guest]

This patch seems to be working fine for me:

Opened dns0
Opened UDP socket
Sending DNS queries for X.XXX.XX to 172.16.2.5
Autodetecting DNS query type (use -T to override).
Using DNS type NULL queries
Version ok, both using protocol v 0x00000502. You are user #0
Setting IP of dns0 to 10.100.1.2
Setting MTU of dns0 to 1130
Server tunnel IP is 10.100.1.1
Testing raw UDP data to the server (skip with -r)
Server is at A.B.C.D, trying raw login: ....failed
Using EDNS0 extension
Switching upstream to codec Base64
Server switched upstream to codec Base64
No alternative downstream codec available, using default (Raw)
Switching to lazy mode for low-latency
Server switched to lazy mode
Autoprobing max downstream fragment size... (skip with -m fragsize)
768 ok.. ...1152 not ok.. 960 ok.. ...1056 not ok.. ...1008 not ok.. 984 ok.. ...996 not ok.. will use 984-2=982
Setting downstream fragment size to max 982...
Connection setup complete, transmitting data.
Detaching from terminal...

tlund (on ircnet)

[yarrick]

Looks good, applied in [957]. Will apply the other fixes as well.