Subject: Re: [tram] I-D Action: draft-ietf-tram-turnbis-07.txt

From: Oleg Moskalenko <[email-masked]>
Date: Mon, 12 Oct 2015 10:11:41 -0700

Regarding ICMP packets routing.

There is one problem in this proposal: in the way how it is worded
currently, it is impossible to implement on any commonly used and
known OS, without superuser privileges.

Receiving ICMP packets requires one of two options:

1) dedicated listening ICMP socket. It works on many OSes, and it
provides accurate information, but it requires superuser privileges.
2) It can be partially implemented on Linux only, without superuser
privileges, by setting IP_RECVERR socket option on a UDP socket.

I see two problems in the second approach:

1) It is limited to a single platform (Linux);
2) It does not provide full information. If say we are getting IPv4
"port unreachable" ICMP error (type 3, code 3) the ICMP packet
contains everything that is needed - destination IP address and
destination port number. But the control message that is delivered to
the Linux socket, contains only the IP address. We also can read the
original UDP packet payload (without packet header), but it does not
have any useful (for us) information. In all my tests with Linux
kernels 3.x and 4.x, the delivered port number is 0 (zero) in 100% of
all tests. This is either a Linux kernel bug, or a feature - but it
does not matter - in the current state of things, the correct handling
of the ICMP messages cannot be implemented without superuser
privileges.

One option would be dropping port number requirement from ICMP STUN
attribute, but I guess that makes it much less useful.

And I am not sure that requiring superuser privileges is a good
feature, from the security point of view.

Thanks
Oleg

On Tue, Sep 22, 2015 at 5:19 AM, Simon Perreault <[email-masked]> wrote:
> Le 2015-09-22 00:28, Tirumaleswar Reddy (tireddy) a écrit :
>> This revision addresses comments from Simon, removes SendError, and uses Data indication for ICMP attribute.
>
> Changes look good! Thanks!
>
> Simon
>
> _______________________________________________
> tram mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tram

---

From: "Pal Martinsen (palmarti)" <[email-masked]>
Date: Tue, 13 Oct 2015 07:36:05 +0000

> On 12 Oct 2015, at 19:11, Oleg Moskalenko <[email-masked]> wrote:
> 

I think this is doable. See https://tools.ietf.org/html/draft-martinsen-tram-stuntrace-01#appendix-A

> provides accurate information, but it requires superuser privileges.

On OS-X and IOS (And probably *BSD, have not tested) it is possible to crate a separate socket that listens for ICMP. This does not require admin privileges. 

icmpSocket=socket(config.remoteAddr.ss_family, SOCK_DGRAM, IPPROTO_ICMP);

Or with IPPROTO_ICMPV6 for IPv6.

> privileges, by setting IP_RECVERR socket option on a UDP socket.
I think it can be fully implemented. As you say by using IP_RECVERR.
First tell Linux you want to receive it:
setsockopt (config.sockfd, SOL_IP, IP_RECVERR, &val, sizeof (val)) < 0)

And then later in your poll() you check for POLLERR

> of the ICMP messages cannot be implemented without superuser

ICMP will give you at least 8 bytes of the offending packet, so the complete IP and UDP header should be there.

The information should be available in cmsg.

for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != NULL; 
       cmsg = CMSG_NXTHDR(&msg,cmsg)) {
    if (cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_RECVERR){
        struct sock_extended_err *ee;
        ee = (struct sock_extended_err *) CMSG_DATA(cmsg);
                          
        if (ee->ee_origin == SO_EE_ORIGIN_ICMP) {
            //Have mercy.. Must fix this
            struct sockaddr_in *addr = &((struct sockaddr_in*)SO_EE_OFFENDER(ee))->sin_addr;
                                
             config->icmp_handler(config, 
                                                 (struct sockaddr*)addr,
                                                  ee->ee_type);
            }
      }
 }

> feature, from the security point of view.
Complete src code for a client that runs on OS-X, IOS, Android and Linux can be found at:
https://github.com/palerikm/ICMPTest

Code is messy, sorry about that.

I am worried about any performance impact this might have on a server. Have not tested that.

.-.
Pål-Erik

---

From: Oleg Moskalenko <[email-masked]>
Date: Tue, 13 Oct 2015 12:09:54 -0700

> I think this is doable. See https://tools.ietf.org/html/draft-martinsen-tram-stuntrace-01#appendix-A

That does not address my concerns. Please see below.

> Or with IPPROTO_ICMPV6 for IPv6.

I think nobody consider iOS as a platform for TURN server, right ?

This works on Mac OS X. It does not work on BSD. BSD requires SOCK_RAW
(it does not allow SOCK_DGRAM there) and SOCK_RAW requires superuser
privileges.

>              config->icmp_handler(config,

Have you ever tried this approach yourself ? I mean, have you tried to
get full ICMP error information with that approach ? I did. I tried it
with Linux kernels 2.x, 3.x and 4.x. All give me the same result, so I
consider that as a stable Linux functionality:

The total size of cmsg for ICMP packet is 48 bytes. 16 bytes are taken
by cmsg header. 16 bytes are taken by "struct sock_extended_err" error
information structure. The remaining 16 bytes are taken by struct
sockaddr_in *addr that, remarkably, contains only the IP address of
the offending party - but it DOES NOT CONTAIN THE PORT NUMBER. This is
the whole point of my concern. I am sure about my findings because I
tried it on many different Linuxes with different kernel versions.

Another available piece of information is the received message itself
on IP_RECVERR-enabled-socket. But according to the man pages, and
according to my own tests, the delivered message is the payload of the
original UDP packet; it is NOT the payload of the ICMP package. So we
do not have the original UDP header information, and we cannot deduce
the original destination port number from there.

With the SOCK_RAW, we do have the full ICMP payload, but the SOCK_RAW
requires superuser privileges.

> https://github.com/palerikm/ICMPTest

That code will not work under unprivileged user in Linux or BSD.

That is the point of my concern. TURN-bis is calling for a feature
that can be implemented (for unprivileged user) only in one single
modern common available OS - Mac OS X. That is a wonderful, powerful
OS that is a very good solid platform for a TURN Server. But the
reality of the modern world is that the major portion of the TURN
server market is taken by Linux (and some portion by BSD). Both Linux
and BSD require superuser privileges for TURN server to implement that
feature.

This is just my concern. But I am pretty much sure that
security-conscious people may object that feature.

Thanks
Oleg

---

From: "Pal Martinsen (palmarti)" <[email-masked]>
Date: Tue, 13 Oct 2015 20:42:54 +0000

On 13 Oct 2015, at 21:09, Oleg Moskalenko <[email-masked]<mailto:[email-masked]>> wrote:

On Tue, Oct 13, 2015 at 12:36 AM, Pal Martinsen (palmarti)
<[email-masked]<mailto:[email-masked]>> wrote:

On 12 Oct 2015, at 19:11, Oleg Moskalenko <[email-masked]<mailto:[email-masked]>> wrote:

Well, if it is the only platform that can implement the RFC it might get a boost.. :-)

That is a shame. I was hoping it was a BSD feature.

They 8 bytes are at least on the wire, at least according to RFC1122

Thanks for pointing that out. I was able to get the sockaddr and assumed the rest of the information was there.

*sigh*
So the kernel stores the entire UDP payload so it can be reported back to the user? The TURN server can keep a list or hash of outgoing packets and port numbers, but that would be cumbersome and to much overhead for this.

Another idea.. If that UDP payload is a STUN packet being sent over the TURN relay as part of the initial ICE connectivity checks. It can be useful for the ICE agent to know that that particular STUN message encountered a ICMP error at the other side of the TURN server. Knowing what STUN transactions caused an ICMP response would be nice.

They claim the internet would break if it was not so..

Unless you are happy with just the IP addr. And that was all I tested.

Ok. I have learnt a lot now. Thanks.

I now do share your concern.

.-.
Pål-Erik

Thanks
Oleg

.-.
Pål-Erik

Thanks
Oleg

On Tue, Sep 22, 2015 at 5:19 AM, Simon Perreault <[email-masked]<mailto:[email-masked]>> wrote:
Le 2015-09-22 00:28, Tirumaleswar Reddy (tireddy) a écrit :
This revision addresses comments from Simon, removes SendError, and uses Data indication for ICMP attribute.

Simon

_______________________________________________
tram mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tram

_______________________________________________
tram mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tram

---

From: Simon Perreault <[email-masked]>
Date: Tue, 13 Oct 2015 16:50:20 -0400

Can't we simply make it optional to set the port in the XOR-PEER-ADDRESS
attribute? That is, set it to zero if you don't have access to it. Am I
missing something?

Simon

Le 2015-10-13 16:42, Pal Martinsen (palmarti) a écrit :
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>> I think this is doable.
>>> See https://tools.ietf.org/html/draft-martinsen-tram-stuntrace-01#appendix-A
>>
>>> On OS-X and IOS (And probably *BSD, have not tested) it is possible
>>> to crate a separate socket that listens for ICMP. This does not
>>> require admin privileges.
>>>
>>> ICMP will give you at least 8 bytes of the offending packet, so the
>>> complete IP and UDP header should be there.
>>
>>>            struct sockaddr_in *addr = &((struct
>>> sockaddr_in*)SO_EE_OFFENDER(ee))->sin_addr;
>>>
> So the kernel stores the entire UDP payload so it can be reported back
> to the user? The TURN server can keep a list or hash of outgoing packets
> and port numbers, but that would be cumbersome and to much overhead for
> this. 
> 
> Another idea.. If that UDP payload is a STUN packet being sent over the
> TURN relay as part of the initial ICE connectivity checks. It can be
> useful for the ICE agent to know that that particular STUN message
> encountered a ICMP error at the other side of the TURN server. Knowing
> what STUN transactions caused an ICMP response would be nice.
> 
>>
>>>> On Tue, Sep 22, 2015 at 5:19 AM, Simon Perreault
>>>> <[email-masked] <mailto:[email-masked]>> wrote:
>>>>>> This revision addresses comments from Simon, removes SendError,
>>>>>> and uses Data indication for ICMP attribute.
>>>>>
>>>>> [email-masked] <mailto:[email-masked]>
>>>> [email-masked] <mailto:[email-masked]>

---

From: Oleg Moskalenko <[email-masked]>
Date: Tue, 13 Oct 2015 14:15:13 -0700

On Tue, Oct 13, 2015 at 1:50 PM, Simon Perreault <[email-masked]> wrote:
> attribute? That is, set it to zero if you don't have access to it.

Yes, that would be my suggestion.

See also below:

> Am I missing something?

it does not need any boost - it is on every iPhone and iPad and
iWatch, already :)

>> That is a shame. I was hoping it was a BSD feature.

Mac OS X is "a BSD" but it has lots and lots of upgrades and
improvements in many areas over the original "BSD". Modern BSD and
modern Mac OS X are different in many senses.

Thanks
Oleg

---

From: "Tirumaleswar Reddy (tireddy)" <[email-masked]>
Date: Wed, 14 Oct 2015 08:22:04 +0000

> -----Original Message-----
> From: Oleg Moskalenko [mailto:[email-masked]]
> Sent: Wednesday, October 14, 2015 2:45 AM
> To: Simon Perreault
> Cc: Pal Martinsen (palmarti); [email-masked]; Tirumaleswar Reddy (tireddy)
> Subject: Re: [tram] I-D Action: draft-ietf-tram-turnbis-07.txt
> 
> > Can't we simply make it optional to set the port in the
> > XOR-PEER-ADDRESS attribute? That is, set it to zero if you don't have access
> to it.
> 
Hi Oleg,

Thanks for the analysis on various OS, I will update draft.

-Tiru
> >>>> See
> >>>> https://tools.ietf.org/html/draft-martinsen-tram-stuntrace-01#appen
> >>>> dix-A
> >>>
> it does not need any boost - it is on every iPhone and iPad and iWatch,
> already :)
> 
> 
> >>
> >>>
> >>> This works on Mac OS X. It does not work on BSD. BSD requires
> >>> SOCK_RAW (it does not allow SOCK_DGRAM there) and SOCK_RAW
> requires
> >>> superuser privileges.
> >>>
> Mac OS X is "a BSD" but it has lots and lots of upgrades and improvements in
> many areas over the original "BSD". Modern BSD and modern Mac OS X are
> different in many senses.
> 
> Thanks
> Oleg
> 
> >>
> >>>>
> >>>>
> >>>>> 2) It can be partially implemented on Linux only, without
> >>>>> superuser privileges, by setting IP_RECVERR socket option on a UDP
> socket.
> >>>>>
> >>>>> 2) It does not provide full information. If say we are getting
> >>>>> IPv4 "port unreachable" ICMP error (type 3, code 3) the ICMP
> >>>>> packet contains everything that is needed - destination IP address
> >>>>> and destination port number. But the control message that is
> >>>>> delivered to the Linux socket, contains only the IP address. We
> >>>>> also can read the original UDP packet payload (without packet
> >>>>> header), but it does not have any useful (for us) information. In
> >>>>> all my tests with Linux kernels 3.x and 4.x, the delivered port
> >>>>> number is 0 (zero) in 100% of all tests. This is either a Linux
> >>>>> kernel bug, or a feature - but it does not matter - in the current
> >>>>> state of things, the correct handling of the ICMP messages cannot
> >>>>> be implemented without superuser privileges.
> >>>>>
> >>>>
> >>> Have you ever tried this approach yourself ? I mean, have you tried
> >>> to get full ICMP error information with that approach ? I did. I
> >>> tried it with Linux kernels 2.x, 3.x and 4.x. All give me the same
> >>> result, so I consider that as a stable Linux functionality:
> >>>
> >>> The total size of cmsg for ICMP packet is 48 bytes. 16 bytes are
> >>> taken by cmsg header. 16 bytes are taken by "struct
> >>> sock_extended_err" error information structure. The remaining 16
> >>> bytes are taken by struct sockaddr_in *addr that, remarkably,
> >>> contains only the IP address of the offending party - but it DOES
> >>> NOT CONTAIN THE PORT NUMBER. This is the whole point of my
> concern.
> >>> I am sure about my findings because I tried it on many different Linuxes
> with different kernel versions.
> >>>
> >>
> >> Thanks for pointing that out. I was able to get the sockaddr and
> >> assumed the rest of the information was there.
> >>
> >>> Another available piece of information is the received message
> >>> itself on IP_RECVERR-enabled-socket. But according to the man pages,
> >>> and according to my own tests, the delivered message is the payload
> >>> of the original UDP packet; it is NOT the payload of the ICMP
> >>> package. So we do not have the original UDP header information, and
> >>> we cannot deduce the original destination port number from there.
> >>
> >> *sigh*
> >> So the kernel stores the entire UDP payload so it can be reported
> >> back to the user? The TURN server can keep a list or hash of outgoing
> >> packets and port numbers, but that would be cumbersome and to much
> >> overhead for this.
> >>
> >> Another idea.. If that UDP payload is a STUN packet being sent over
> >> the TURN relay as part of the initial ICE connectivity checks. It can
> >> be useful for the ICE agent to know that that particular STUN message
> >> encountered a ICMP error at the other side of the TURN server.
> >> Knowing what STUN transactions caused an ICMP response would be nice.
> >>
> >>>
> >>> With the SOCK_RAW, we do have the full ICMP payload, but the
> >>> SOCK_RAW requires superuser privileges.
> >>>
> >>>>> One option would be dropping port number requirement from ICMP
> >>>>> STUN attribute, but I guess that makes it much less useful.
> >>>>>
> >>> server market is taken by Linux (and some portion by BSD). Both
> >>> Linux and BSD require superuser privileges for TURN server to
> >>> implement that feature.
> >>>