Subject: [ietf-smtp] Characteristics of Isolated (or mostly-isolated) industrial IP Networks

From: Keith Moore <[email-masked]>
Date: Sat, 4 Jan 2020 02:26:53 -0500

This is an attempt to summarize my observations about (mostly-)isolated networks, and also about some dubious assumptions that I've seen some equipment developers make about security requirements on such networks.

Most of these observations and assumptions aren't at all specific to email, but several of them might inform email recommendations.

Sorry for material duplicated from other posts; I thought it useful to try to collect everything in one place.

1. Scope

I'm only considering IP networks that are disconnected or mostly disconnected, and devices on such networks which use some IP protocols, including IETF application layer protocols.   While these networks may have no external IP connectivity at all, they may indirectly interact with hosts on private IP networks and/or the public Internet, through proxies or application-level gateways.

What I've seen is mostly (broadly speaking) "industrial" networks, used in laboratories, manufacturing, energy production, and energy delivery.   Some of these networks are used to monitor and occasionally control production equipment in factories, some used to monitor and/or control handling of hazardous materials, some used in critical infrastructure, some where explosive materials are present with consequent risks to human lives.   I've also worked with production networks In non-industrial settings that have similar characteristics to networks described here.

Even equipment used only for "monitoring" may still present risks if the equipment is compromised, if the monitoring is employed in a control feedback loop, and/or because compromised monitoring equipment can be used to create diversions from attacks made elsewhere.

What I've seen is a small sample.   I can't claim that it's representative, but I've seen the same patterns in so many different contexts that I suspect they're very common.   And I understand why many of these things are the way they are.   They are generally not easy problems to fix.    I expect wide variation       given that every industry has different regulations, certification requirements, and standard practices including safety practices.   It would be worthwhile to do a comprehensive survey across a variety of industries.

My biggest concerns for these networks are: 

(a) security.   I'm not only concerned about the potential for such networks and devices hosted on such networks to be attacked via external network connections; I'm also concerned about the potential for such networks and devices to be attacked locally, e.g. by attacks over WiFi, by unauthorized devices plugged into local Ethernet, compromise of authorized devices via physical access, and compromise of authorized devices on the isolated network by devices that also connect to other networks.

(b) design choices and recommendations in Internet protocol specifications that don't take into account the particular characteristics of these networks.

2. The Networks

- Common layer 2 media are Ethernet, 802.11, and "EtherNet/IP" (perhaps the most confusing name for any protocol on Earth).    There are a few other variants of Ethernet media but adapted for industrial use, for instance, using a different kind of media access control than collision detection.   There are "industrial wireless" networks that mimic Ethernet but over radio.    I've seen some interest in using IP over mesh networks, but haven't personally seen this deployed.

- Generally they are running IPv4 using RFC1918 address space.   I have seen a tiny amount of interest in IPv6.   I suspect most operators of such networks will prefer IPv4 out of familiarity and because IPv4 addresses are easier to transcribe.   Given that the networks are usually isolated at the IP layer anyway, use of IPv4 with RFC 1918 addresses doesn't present much of a problem.

- Static IP address assignment is common.   DHCP might not be available.

- Often there is no external IP connectivity, though some of the same equipment used on such isolated networks will also be used on other networks that do have external IP connectivity.   (I mean this in several ways: one is that the same design and implementation of a device may be expected to function well on both an isolated network and a more conventional network; another is that the same device may be connected to conventional and otherwise-isolated networks at the same time via multiple network interfaces; another is that it's common for portable devices such as phones and laptops to migrate from one environment to another and connect to such isolated networks when needed (even if this violates safety or security regulations).   Sometimes there is intermittent IP connectivity, say, used for firmware update.

- DNS may not be supported or even wanted.   I've been told that reliance on DNS is a risk in a manufacturing environment that provides significant potential for costly disruption with no benefit.   There may be no name lookup mechanism at all, not even host files.   (I don't know how much mDNS is used, but suspect that the same considerations apply; in my limited experience it wasn't reliable enough to depend on.)   I have seen a few devices that supported NetBios lookup so that PCs could find them by name.

- There may be some interaction with external networks via proxies or application-level gateways.   (sometimes these are called "firewalls" even if they never forward IP packets.)

- Zero threat analysis and handwaving security arguments seem to be the norm.    I should not give specific examples.

- Generally, these networks are not managed or operated by networking professionals.   Sometimes there are defensible reasons for that.   Sometimes the network is just a few hosts on WiFi or Ethernet, some of them are in very remote areas,  some are very disruption-sensitive.  

3. Devices typically used on such networks

- Most interaction on such networks is between "appliance" devices that primarily aren't designed to interact with humans.   They may have no human input or output devices at all (e.g. no keyboard, no display) or support only very limited interactions with humans (a few buttons, a numeric keypad, maybe a tiny display or touchscreen).    That's why they don't need DNS; the devices generally communicate mostly with other devices that are pre-configured and there might not even be a way to type letters in.    There's some pressure to keep configuration settings to a bare minimum, not only because of input difficulties but also because of customer support issues.

- I've seen many such devices with configuration interfaces and status monitoring based on HTTP, ssh, bare tcp/telnet (sometimes with no authentication at all), or occasionally rs232.

- I've seen several such devices that were configurable to send email reports or notices of exceptional conditions.

- It seems to be common to bring a laptop into such an environment as needed to configure devices, without dedicating a computer for that purpose.  Of course PCs still need firmware updates which are most easily accomplished by connecting them to the public internet.

- I've seen wide variation in the ways that such devices can have their firmware updated, if such capability even exists.   Sometimes you can insert a USB stick, sometimes you replace the device's SD card, sometimes you can push a firmware update to the device from a web browser.   Not only can it usually not be automated, an update can be a cumbersome process and each device is different.

- It's increasingly common to have such devices be based on Linux or similar open source platforms (much as we've seen for internet appliances used in other environments), but without much thought given to physical security, secure firmware update, supply chain issues, back doors, removable media, JTAG probes, etc.   In my experience it's often trivially easy to alter such a device's functionality, often with no physical modification needed.

- I've seen far too many devices for which every serial number had the same root or "back door" password.  Even if the device doesn't provide a shell login from a network interface, there may still be a serial "console" interface.

- The problem of configuring and updating such devices, coupled with the problem of managing passwords in such an environment (these passwords may rarely need to be used by humans), is such that devices may be (re)configurable or updated with no credentials at all - just using the app provided by the vendor.

4. Implications

These networks remind me of IP networks from the 1980s.    For instance, password sniffing is often trivial because either no encryption is used, or the encryption is TLS but with self-signed certificates.   (There's an increasing bias against use of self-signed certificates in applications, which makes sense on the public Internet and most private IP networks.   But the same browsers are used on these isolated networks as are used on the public Internet.)

TLS, as normally used, doesn't work terribly well in such environments because they don't use host names and there's no good way to rotate keys or update trusted certificates. 

Even HTTP digest authentication doesn't work very well because the protocol specification doesn't really support all the features needed (how do you logoff in a way that works for all browsers?  how do you timeout a login session?) and the user interface is so poor. 

There's a widespread assumption that connected devices need on-demand firmware updates  in order to minimize the threat from zero-day exploits, but firmware update is especially difficult in these environments.

Since DNS is not generally available and devices are generally not assigned DNS names, and it can be difficult even to explicitly configure some of them,  it's not reasonable to expect these devices to use DNS names, say, in email.

---

From: "John Levine" <[email-masked]>
Date: 4 Jan 2020 21:18:39 -0500

In article <[email-masked]> you write:
>-=-=-=-=-=-
>This is an attempt to summarize my observations about (mostly-)isolated networks, and also about some dubious assumptions
>that I've seen some equipment developers make about security requirements on such networks.

Thanks, this is very helpful.

It looks like, insofar as we're thinking about mail, a reasonable
design is to put a reasonably capable submission server on a network
(e.g., Raspberry Pi running linux) and point the IPs of the IoT mail
senders at it.  We could give some more thought about what the
submission server could reasonably do to avoid relaying hostile
messages.

R's,
John

---

From: Keith Moore <[email-masked]>
Date: Sat, 4 Jan 2020 23:03:02 -0500

On 1/4/20 9:18 PM, John Levine wrote:

> In article<[email-masked]>  you write:
>> -=-=-=-=-=-
Thanks.
> submission server could reasonably do to avoid relaying hostile

At first glance it might seem like good sense to use a Raspberry Pi or 
similar devices in some such scenarios. Though it's trickier that it 
might seem at first.

Any device that uses an SD card as its primary storage medium is likely 
to be flaky.   At a minimum, it really needs a good power supply and 
ideally a UPS to minimize the risk that the SD card will be trashed.  (I 
think newer Pis can boot from USB devices, so the SD card might not be 
an insurmountable problem.)

In some environments there are other issues associated with such 
devices, e.g. environmental (temperature, humidity, etc.).   And a Pi is 
not approved for use in an environment where explosive gases are 
present, etc.

(It's probably possible to build an device which is basically a sealed 
IP-rated enclosure containing a Pi with a 24v or lower voltage power 
supply, and pay a testing lab to certify that such a device meets 
relevant standards.   Someone might even have done that already, but 
such certification costs many thousands of dollars, so the certified 
devices won't be cheap.   A wall wart power supply would never be 
acceptable.)

And to the person in charge of such a facility, a Raspberry Pi doesn't 
look like industrial equipment.   It might look more like a security 
threat, which it coincidentally is.   They're extremely easy to hack via 
multiple paths, and it's difficult to protect them from being hacked.   
They were, after all, basically designed to be easy to hack, not to 
protect their code and data from attack.

-------

Rather than try to define what such a site's hardware configuration 
should be, I suspect ietf-smtp would do better to define the behavior of 
a submission service that is designed to accept inbound email from 
devices in such an environment and forward such email to a "smarthost" 
(or whatever you want to call it) that can get it to the appropriate 
destinations.

Such a service might be provided by hardware located within that 
environment, or external to that environment via a 
NAT/proxy/firewall/whatever.   Different enterprises will require 
different hardware solutions.   But the "IIoT submission service" may 
still need to be able to operate in the same environment as the devices 
it serves, which  may still mean no DNS, etc.    Or maybe two service 
profiles are needed - one for on-site and one for upstream?

Keith

---

From: Dave Crocker <[email-masked]>
Date: Sat, 4 Jan 2020 21:58:26 -0800

On 1/4/2020 8:03 PM, Keith Moore wrote:
> I suspect ietf-smtp would do better to define the behavior of a 
> submission service that is designed to accept inbound email from devices 
> in such an environment 

The clients in such an environment appear to simply be MUAs.  The 
submission server appears simply to be an MSA.

I gather what makes them distinctive is methods of identification and 
authentication -- notably reliance on network address rather than domain 
names -- rather than anything involving email semantics or basic 
protocol details (beyond ID & Auth)?

What else is distinctive?

and forward such email to a "smarthost"

What are the differences from a classic originating MTA?

 From your note, the reliance on IP Address and boundary security rather 
than encryption appear to be the primary points of distinction.

In any event, I suspect it would be helpful to create an Informational 
RFC that discusses design, administration and operations issues that are 
noteworthy in common IoT environments.  Sounds like it doesn't need to 
be lengthy of complicated, but formulating a discrete set of statements 
and getting community consensus on it could facilitate detailed wg 
decision-making, here and elsewhere.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Keith Moore <[email-masked]>
Date: Sun, 5 Jan 2020 02:38:02 -0500

On 1/5/20 12:58 AM, Dave Crocker wrote:

>> submission service that is designed to accept inbound email from 
>> devices in such an environment 
>

The clients might have MTA-like functions like queuing and retry, but 
aren't likely to bounce mail.   From the outside they look a lot like MUAs.

> authentication -- notably reliance on network address rather than 
> domain names -- rather than anything involving email semantics or 
> basic protocol details (beyond ID & Auth)?

I should probably preface this by stating my belief that having 
different kinds of Internet for different kinds of devices is overall a 
Bad Idea.   I don't want to see the Internet split up into lots of 
special kinds of networks each with their own protocols.   (I feel the 
same way about homenet, and IoT - as opposed to IIoT - devices.)

IMO, to the extent possible, an IIoT device should be capable of, and 
configurable to, submit mail via an ordinary MSA, authenticating in the 
same way that an MUA would authenticate to an ordinary MSA, etc.   Being 
able to talk to an IIoT MSA should be an extra feature for use by 
devices that operate on more constrained networks - without public IP 
connectivity, without DNS, etc.

But the IIoT MSA might still be operating in an environment without DNS, 
so it can't be expected to do all of the sanity checking and fixup that 
an MSA on the public Internet would do. The IIoT MSA might need to be 
configured to forward to a "real" MSA downstream for such sanity 
checking and fixup.

Something I didn't mention earlier is that such networks may not have a 
reliable source of time, and a lot of these devices' clocks will drift, 
their batteries will fail and not be replaced, and so on.   So an IIoT 
MSA might not be able to put an accurate Date field or accurate 
timestamps in Received fields which is something else that might better 
be done downstream.   (Better to not add a Date header than to add one 
that's not reliable. )

I think an IIoT MSA should support some specific authentication methods 
(most of them probably already existing) other than having to rely on 
source IP address, even if it can still be configured to trust based on 
source IP address alone.

It might need some rate limiting provisions to remedy vulnerabilities 
caused by lack of real authentication.

I'd also like to see if there's a way to use TLS to let the connection 
be secured by a certificate on the client end, using a certificate 
issued by the device manufacturer.   For SMTP, what I have in mind is a 
SLTTRATS SMTP extension that acts like STARTTLS except that the the 
client and server roles on the TLS connection are flipped - the SMTP 
client is the TLS server and vice versa. Then the IIoT devices don't 
need a trusted CA list, the IIoT MSAs that they talk to do - but the 
MSAs are closer to the public network so the update might be easier.   A 
properly designed IIoT device could safeguard its private key from 
exfiltration enough that key rotation might be unnecessary - or maybe 
enough key pairs and certificates could be generated at manufacture to 
last for the lifetime of the device.    This is admittedly more 
ambitious than other ideas, and should probably be investigated separately.

It might be useful to have a standard for mail-generating appliances to 
uniquely identify themselves - maybe a convention for local parts 
containing globally-unique tokens - so they don't need explicit 
per-device configuration in order to have distinct originator addresses.

One service I've found to be useful in such networks is "deliver this 
mail if it can be sent immediately, otherwise don't bother".    This is 
used when implementing alarms/alerts that generate repeated messages 
while an exceptional condition persists.   The first such message might 
have normal delivery semantics, while subsequent messages for the same 
condition (until it's cleared) don't clutter up an outgoing queue, say, 
if there's a network outage.   DELIVERBY seems fairly well suited for 
that, especially when combined with an SMTP client implementation that, 
for messages so marked, simply drops the message (doesn't bother to 
queue it) if it can't be relayed immediately.

> What are the differences from a classic originating MTA?

To me most of this stuff seems pretty similar to a typical mid-1980s IP 
lab network, say a local Ethernet with locally-chosen IP addresses, no 
external IP connectivity, no DNS, maybe hosts files, maybe a smarthost 
on the LAN that could forward mail via UUCP.   I don't think any of us 
who were running mail systems back then, would find much of this novel.

But I also think that we're going to see a lot more such systems and 
networks in the 21st century than we ever saw in the 1980s, and the 
scaling issues and security threats will be greater than they were then.

> From your note, the reliance on IP Address and boundary security 
> rather than encryption appear to be the primary points of distinction.
>
> RFC that discusses design, administration and operations issues that 
> are noteworthy in common IoT environments. Sounds like it doesn't need 
> to be lengthy of complicated, but formulating a discrete set of 
> statements and getting community consensus on it could facilitate 
> detailed wg decision-making, here and elsewhere. 

Yeah, I'm thinking along similar lines.  Thanks for the feedback.

Keith

---

From: "John R Levine" <[email-masked]>
Date: 5 Jan 2020 10:59:49 -0500

> similar devices in some such scenarios. Though it's trickier that it might 
> seem at first.

I totally take your point about equipment needing to be suitable for an 
industrial environment.  My point, perhaps not well expressed, is that a 
pi has all the computing power needed for a mail gateway.

FWIW, there are a lot of vendors selling industrial raspberry pi's. 
Here's one of them:

https://revolution.kunbus.com/revolution-pi-series/

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "John Levine" <[email-masked]>
Date: 5 Jan 2020 12:19:29 -0500

>What else is distinctive?

I'd think that typically you know in advance a lot about what mail
each device should send.  For example, there is a printer on my desk
that sends alerts about paper jams and the like.  It only sends mail
to one preconfigured address, and the rate at which it sends mail is
limited by the mechanical parts it's reporting on.

If it sent mail to any other address, or if the message rate was much above
one a minute, we'd know something was screwed up with its mail.

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 5 Jan 2020 15:14:35 -0500

On Sun, Jan 05, 2020 at 02:38:02AM -0500, Keith Moore wrote:

> But the IIoT MSA might still be operating in an environment without
> DNS, so it can't be expected to do all of the sanity checking and
> fixup that an MSA on the public Internet would do.

The usual-suspect MSAs (at least Postfix and Exim) operate just fine in
isolated non-DNS networks.  An MSA only needs DNS when:

    * Delivering to someone else's domain on the public internet, via
      the MX records for that domain.
    * Attempting to do DANE SMTP.

neither of these apply on isolated non-DNS networks.

> issued by the device manufacturer.
>
> For SMTP, what I have in mind is a SLTTRATS SMTP extension that acts
> like STARTTLS except that the the client and server roles on the TLS
> connection are flipped - the SMTP client is the TLS server and vice
> versa.

That's possible now, using STARTTLS.  No need for a new ESMTP extension.
The client ignores the server certificate, while the server, requests a
client certificate and uses it to authenticate the client (by
certificate or public key fingerprint).

> Then the IIoT devices don't need a trusted CA list, the IIoT MSAs that
> they talk to do - but the MSAs are closer to the public network so the
> update might be easier.

They don't need a CA bundle if they are not going to authenticate the
server.  Since they can't be assumed to have a reliable clock, certs
work poorly in any case, so they could at best be (on site) configured
with a list of public key fingerprints of trusted MSAs.  In most cases
this is likely overkill.  A rogue MSA can just reject connections
blocking mail delivery, and no amount of authentication will stop that.

-- 
    Viktor.

---

From: Hector Santos <[email-masked]>
Date: Mon, 06 Jan 2020 09:13:27 -0500

> aren't likely to bounce mail.   From the outside they look a lot like

Hi Keith,

You certainly touched base and highlighted the fact there are both 
legacy and current forms of small footprint and specialized SMTP 
transport clients.

I have a number of these.  One is a proprietary server/client that 
works over a virtual dialup/socket system. Its for our GUI frontend 
"wcNavigator" app that includes a Mail Reader/Writer component (MUA). 
  The connection and thus, its MTA is always authenticated so the 
backend hosting SMTP would be an MSA per se.  The backend is a GUI 
server that contains a smtp server for the gui smtp clients that can 
be a mail client or a game client sending notifications.

The mail client can send reply, forwarded, new messages via an API or 
the backend hosting SMTP server.  When the SMTP method is enabled, one 
mail client uses HELO with no field data and the game client uses HELO 
with the user's name (No verification necessary).  They also use other 
proprietary SMTP commands and attributes that the server recognizes 
such as:

HELO <host.user.info.name>
MAIL CONF:<conf number> [FORWARD:<conf number>;<message id>]
RCPT CC:<receiver-path>
ATCH NAME:<filename> SIZE:<filesize>

But this is within a closed proprietary boundary environment.  The GUI 
MUA/MTA is not going to connect to a general SMTP server.

In my opinion, RFC5321bis should be about the public service port 
where we have unsolicited, anonymous senders and we are basically 
trying to clarify what can be commonly done to strengthen SMTP 
compliancy or maybe perhaps relax it.

Thanks

-- 
HLS

---

From: Dave Crocker <[email-masked]>
Date: Mon, 6 Jan 2020 06:28:11 -0800

>> What else is distinctive?
...
> one a minute, we'd know something was screwed up with its mail.

This sounds like a variant of blocking outbound port 25.

d/

---

From: "John R Levine" <[email-masked]>
Date: 6 Jan 2020 09:37:14 -0500

> This sounds like a variant of blocking outbound port 25.

It's exactly forcing all the mail through the submission host so it can do 
outbound filtering.  That's been common on consumer mail systems for 
decades.

Regards,