Subject: Re: [OPSEC] Fwd: Last Call for Comments:draft-oreirdan-mody-bot-remediation-06

From: Jason Livingood <[email-masked]>
Date: Wed, 21 Apr 2010 11:00:29 -0400

I¹m just getting around to doing a 08 document update now that IETF-77 has
concluded.  (Joel¹s feedback has been incorporated into 08 as well.)

Thanks for this feedback, now incorporated into
draft-oreirdan-mody-bot-remediation-08 (
http://www.ietf.org/id/draft-oreirdan-mody-bot-remediation-08.txt).

Specific comments/replies inline below if you are so inclined.

Regards
Jason

On 2/15/10 1:16 PM, "Smith, Donald" <[email-masked]> wrote:

>> >    This document contains recommendations on how Internet Service
>> >    Providers can manage the effects of computers used by their
> * subscribers, which have been infected with malicious bots, via
> 
> donald: A bot is a tool. It can't really be malicious nor benign any more that
> a gun, or screwdriver is malicious however I understand as this is widely
> used, incorrectly, within the industry. But malicious implies intent to do
> harm.
> As long as that is addressed in the malicious bot/bot definition it's probably
> acceptable to use it in this way.
> 
> [JL] Quite true.  I have tried to ensure when using bots in the doc that I am
> referring to malicious bots, rather than other bots which may have positive
> uses.  I try to define this at length in Terminology, Section 2.1.  (I
> received similar feedback from Robb Topolski in an earlier draft.)
> 
>> >    various remediation techniques.  Internet users with infected
>> >    computers are exposed to risks such as loss of personal data, as well
>> >    as increased susceptibility to online fraud and/or phishing.  Such
>> >    computers can also become an inadvertent participant in or component
>> >    of an online crime network, spam network, and/or phishing network, as
>> >    well as be used as a part of a distributed denial of service attack.
> 
> donald: The risks to users of infected systems and others are included/defined
> in many places within this document.
> It should probably be defined clearly in one place and referenced in all other
> places.
> 
> [JL] Fair point.  I¹m going to call that out in my open issues list and
> address in a future version if I don¹t have the chance to get it into -08.
> 
>> > 2.1.  Malicious Bots, or Bots
>> >
>> >    A malicious "bot" (derived from the word "robot", hereafter simply
> 
> donald: Malicious bot implies intent to do harm, while these are really bots
> that can be used to perform potentially harmful tasks for the bot master.
> Perhaps potentially harmful bot?
> 
> [JL] I¹ve added a bunch of new text to 2.1 to address this.  Please let me
> know if there¹s anything that could be improved upon.
> 
>> >    referred to as a "bot") refers to a program that is surreptitiously
> 
> donald: Some times they are installed on purpose but the user doesn't
> understand the full functionality of the software installed or the amount of
> remote control the application provides to the remote controller (bot master)
> 
> [JL] Ack, and addressed in 2.1 of the doc.
> 
>> >    It is important to note that there are 'good', or benign bots.  Such
>> >    benign bots are often found in such environments such as gaming and
>> >    Internet Relay Chat (IRC) [RFC1459], where a continual, interactive
>> >    presence can be a requirement for participating in the games,
>> >    interacting with a computing resource, or other purposes.  Since such
>> >    benign bots are performing useful, lawful, and non-disruptive
>> >    functions, there is no reason for a provider to monitor for their
>> >    presence and/or alert users to their presence.
> 
> donald: Again Benign implies intent. Perhaps harmless instead?
> 
> [JL] I¹ve retained benign since it seems to be used a great deal but added
> harmless as well: ³Thus, while there may be benign, or harmless bots, for the
> purposes of this document...²
> 
>> >    include: identity theft, spam, distributed denial of service (DDoS)
>> >    attacks, key-logging, fraudulent DNS (pharming), proxy services, fast
> * flux hosting and click fraud.
> 
> donald: This is a pretty good risk identification. You might want to add spim,
> spit, email address harvesting, hosting of highly questionable or illegal
> content (BulletProof hosting), various MITM attacks (banking ssl ala Zeus) and
> web site scraping.
> 
> [JL] Added  good suggestions.
> 
>> >
>> >    Infection vectors include un-patched operating systems, software
>> >    vulnerabilities, weak/non-existent passwords, malicious websites, un-
>> >    patched browsers, malware, vulnerable helper applications and social
>> >        engineering techniques to gain access to the user's computer.
> 
> donald: The use of vulnerable and un-patched seems slightly redundant.
> An un-patched piece of software is likely to be vulnerable.
> I think your trying to cover the concept of un-patched vs zero day (no patch
> publicly available) exploitation.
> 
> [JL] Correct. I modified to ³... software vulnerabilities (which include
> so-called zero-day vulnerabilities where no patch yet exists)²
> 
>> >         Initially, bots used IRC to communicate but were easy to shutdown
>> if
> 
> donald: Initally SOME bots used IRC, Q, TFN, Trinoo, etc... all had non-IRC
> based control.
> 
> [JL] Modified as suggested
> 
>> >    Increasingly, other household systems and devices contain embedded
>> >    computers which are connected to or can connect to the public
>> >    Internet and/or private IP networks.  However, these devices may not
>> >    be under interactive control of the Internet user, such as may be the
>> >    case with various smart home and smart grid devices.
> 
> donald: So do you intend for your computer definition to include smart ip
> enabled  devices or not. It's hard to tell from this definitation.
> I would limit the definition to computers that humans use as general purpose
> computers.
> 
> [JL] I did intend to refer to these sorts of devices.  As I have generalized
> ³computers² to now be ³hosts² I¹ve made some modifications to make this
> hopefully more clear in the next version.  But I expressly want to include
> both typical PC-type devices as well as, say, Internet-connected picture
> frames.
> 
>> >    can sometimes cause their computer to be infected with malware, which
>> >    may include a bot or cause a bot to install itself, via inadvertently
>> >    accessing a specific website, downloading a specific file, or other
> 
> donald: a website, downloading a file ...
> 
> [JL] Change made
> 
>> >    Alternatively, Internet-connected computers may become infected with
> donald: Not really alternatively, since you mention vulnerabilities above but
> I think your trying to say without user interaction (scan and spolit) right?
> 
> [JL] Correct.  Changed ³Alternatively² to ³In other cases²
> 
>> >    DNS Fast Flux is typically used in conjunction with proxies which
>> >    then route the web requests to the real host which serves the data
>> >    being sought.  The proxies are normally hosted on compromised user
>> >    computers.  The effect of this is to make the detection of the real
>> >    host for a piece of content much more difficult and to ensure that
>> >    the site remains up for as long as possible.
> donald: The backend/hidden site
> 
> [JL] Change made
> 
>> >    Threats in 2009 and Beyond].  Bots are frequently used as part of
>> >    concerted Distributed Denial of Service (DDoS) attacks for criminal,
> donald: coordinated
> 
> [JL] Change made
> 
>> >    operating in their networks.  Furthermore, ISPs may also be in a
>> >    unique position to be able to communicate to Internet users which are
> donald: unique position to be able to notify their customers of potential
> infection by
> bots or other infections.
> 
> [JL] Change made
> 
>> >    From an end user perspective, knowing that their computer has been
> donald: From an end user perspective, being notified that they may have an
> infected computer on their network is important information.
> 
> [JL] change made
> 
>> >    infected with one or more bots is very important information.  Once
>> >    they know this, they can take steps to remove the bot, protect
>> >    themselves in the future, and resolve any problems which may stem
>> >    from the bot infection.  Given that bots can drain local computing
> donald: Given that bots can consume vast amounts of local computing and
> network resources ...
> 
> [JL] change made
> 
>> >    As a result, the intent of this document is to provide a guide to
> donald: The intent of this document is to provide guidence to ISPs...
> 
> [JL] change made
> 
>> >    generally, as well as on individual Internet users.  Efforts by ISPs
>> >    and other organizations could therefore, over time, reduce the pool
> donald: and other organizations can over time reduce the pool of ...
> 
> [JL] change made
> 
>> >    The potential mitigation of bots is accomplished through a process of
>> >    detection, notification to Internet users, and remediation of that
>> >    bot with a variety of tools, as described later in this document.
> donald: bot infections with a variety ...
> 
> [JL] change made
> 
>> >    ability of average users.  Attempts at bot removal may frequently be
>> >    unsuccessful, or only partially successful, and may leave a user's
> donald: unsuccessful or only partially successful leaving the system ...
> 
> [JL] change made
> 
>> >    system in an unstable and unsatisfactory state or even in a state
>> >    where it is still infected.  Attempts at bot removal can also result
>> >    in side effects ranging from a loss of data or other files, all the
>> >    way through partial or complete loss of system usability.
> donald: Attempts at bot removal can result in side effects from a loss of data
> to partial or complete loss of system usablility.
> 
> [JL] change made
> 
>> >    In general, the only way a user can be sure they have removed some of
>> >    today's increasingly sophisticated malware is by 'nuking-and-paving'
>> >    the system: reformatting the drive, reinstalling the operating system
>> >    and applications (including all patches) from scratch, and then
>> >    restoring user files from a clean backup.  However the introduction
> donald: restoring user files from a known clean backup...
> 
> [JL] chg made
> 
>> >    of BIOS-based malware may mean that, in some cases, this will not be
> donald: of persistent memory based malware may meant that, in some cases, this
> may not be ...
> 
> [JL] chg made
> 
>> >    enough and may prove to be more than any end user can be reasonably
>> >    expected to resolve [Persistent BIOS Infection].  Experienced users
> donald: Persistent memory infections.
> 
> [JL] chg made 
> 
>> >    would have to re-flash BIOS in their machines in order to remove
>> >    BIOS-based malware.  However, in some cases, not even 'nuking-and-
> donald: would have to re-flash or re-image persistent memory sections on their
> machines in order ...
> 
> [JL] chg made
> 
>> >    Devices with embedded operating systems, such as video gaming
>> >    consoles and smart home appliances, will most likely be beyond a
>> >    user's capability to remediate by themselves, and will typically
>> >    require the aid of vendor-specific advice, updates and tools.  Care
>> >    must be taken when imparting remediation advice to Internet users
>> >    given the increasingly wide array of computing devices that can be,
>> >    or could be, infected by bots in the future.
> donald: Many such devices have a "reset to factory defaults" switch which will
> be the main method to clear such devices if infected.
> 
> [JL] Good suggestion.  Added ³However, in some cases, such devices will have a
> function or switch to enable the user to reset that device to a factory
> default configuration, which may in some cases enable the user to remediate
> the infection.²
>> >
>> >
>> > 5.  Detection of Bots
>> >
>> >    An ISP must first identify that an Internet user, in this case a user
>> >    that is assumed to be their customer or otherwise connected to the
>> >    ISP's network, is determined to be infected, or likely to have been
>> >    infected with a bot.  The ISP must attempt to detect the presence of
>> >    bots using methods, processes, and tools which maintain the privacy
>> >    of the personally identifiable information (PII) of their customers.
>> >    The ISP also must not block legitimate traffic in the course of bot
>> >    detection, and must instead employ detection methods, tools, and
>> >    processes which seek to be non-disruptive, as well as being
>> >    transparent to Internet users and end-user applications.
> donald: You can't be totally transparent nor completely non-disruptive.
> Nothing can be observed without affecting what your observing.
> Perhaps rewrite with the concept of MINUMIZING such effects?
> 
> [JL] I see what you mean but it¹s tough to get that across.  I think there¹s
> enough wiggle room insofar as I have  said ³seek to be non-disruptive² which
> means it is really simply a goal.
> 
>> >
>> >    Detection methods, tools, and processes may include things such as
> donald: may include analysis of specific ...
> 
> [JL] change made
>  
>> >    or other Internet users, as well as a wide variety of other
>> >    possibilities.  It is likely that a combination of multiple bot
> donald: In order to validate any bot net detection source a combination of
> multiple bot
> detection sources has proven to be an effective approach ...
> 
> [JL] made a change to address your concern
> 
>> >    detection data points will prove to be an effective approach in order
>> >    to corroborate information of varying dependability or consistency,
>> >    as well as to avoid or minimize the possibility of false positive
>> >    identification of computers.  Detection should also, where possible
>> >    and feasible, attempt to classify a bot in order to confirm that it
> donald: to classify the specific bot infection type in order to ...
> 
> [JL] chg made
> 
>> >    can be stopped.  This may mean that an ISP may need to balance the
>> >    desire or need to definitively classify and/or confirm a bot, which
>> >    may take an extended period of time, with the ability to predict the
>> >    strong likelihood of a bot in a very short period of time.  This
>> >    'definitive-vs-likely' challenge is difficult and, when in doubt,
>> >    ISPs should probably err on the side of caution by communicating when
> donald: I agree erring on the side of caution BUT that should be don't
> communicate with your customers unless your fairly certain they are infected
> (False positive rate of 5% or less)! Notifying customers just on "likely" (ie
> greater then 50%) will cause customers loose faith in such notification.
> 
> [JL] Good point  added text to address this
>  
>> >    a likely bot infection has taken place.  This also means that
>> >    Internet users may benefit from the deployment of client-based
>> >    software protections or other software tools, which can enable rapid
>> >    performance of heuristically-based detection bot activity, such as
> donald: There is a risk here in that this teaches users to install software on
> their computer that may not be effective given todays botnet + root kit hiding
> abilities. Thus a false sense of security plus you've trained them to install
> software outside of normal OS maintenance. Would this even be included if it
> weren't the approach you choose?
> 
> [JL] It¹s not a statement that we should only rely on host-based protections,
> merely an argument for layered defenses.  I don¹t want to see things like I
> used to see in enterprises, where folks felt that since a firewall existed in
> the network that hosts could go largely unprotected. In this case, just
> because an ISP may have detection and perhaps protection measures in place, I
> don¹t want to obviate users of the responsibility to run some local software
> to protect themselves (something > nothing).  :-)
> 
>> >    the detection of a bot as it starts to communicate a bot net and
> donald: communicate to a bot net ...
> 
> [JL] chg made.  
> 
>> >    execute some type of command.  Any bot detection systems should also
> donald: execute commands.
> 
> [JL] chg made
>> >    be capable of learning and adapting, either via manual intervention
>> >    or automatically, in order to cope with a rapidly evolving threat.
> donald: If done learning is done automatically you have just solved the turing
> test.
> Computers don't learn.
> 
> [JL] I removed Œlearning¹
>> >
>> >    As noted above, detection methods, tools, and processes should ensure
>> >    that privacy of customers' PII is maintained.  While bot detection
>> >    methods, tools, and processes are similar to spam and virus defenses
>> >    deployed by the ISP for the benefit of their customers (and may be
>> >    directly related to those defenses), attempts to detect bots should
>> >    take into account the need of an ISP to take care to ensure any PII
>> >    collected or incidentally detected is properly protected.  The
>> >    definition of PII varies from one jurisdiction to the next so proper
>> >    care must be taken to ensure that any actions taken comply with
>> >    legislation and good practice in the jurisdiction in which the PII is
>> >    gathered.  Finally, depending upon the geographic region within which
> donald: While this is true in many cases the customers PII is already at risk
> from criminal elements that want to exploit it. So being overly concerned
> about the privacy of PII (such as hiding PII from internal analysis teams)
> should be recommended against.
> 
> [JL] That¹s why I say ³should² and not ³must² since as you point out there may
> be cases where this is not possible.  Also, what legally constitutes PII is
> typically fairly small.  I added this text into the section though: ³This is
> important, as just as spam defenses may involve scanning the content of email
> messages which may contain PII, then so to may bot defenses similarly come
> into incidental contact with PII.²
> 
>  a.  Where legally permissible or otherwise an industry accepted
>> >        practice in a particular market region, an ISP may in some manner
>> >        "scan" their IP space in order to detect un-patched or otherwise
>> >        vulnerable hosts.  This may provide the ISP with the opportunity
>> >        to easily identify Internet users who appear to already be or are
>> >        at great risk of being infected with a bot.  ISP's should note
>> >        that some types of port scanning may leave network services in a
>> >        hung state or render them unusable due to common frailties, and
>> >        that many modern firewall and host-based intrusion detection
>> >        implementations may alert the Internet user to the scan.  As a
>> >        result the scan may be interpreted as a malicious attack against
>> >        the computer.  Vulnerability scanning has a higher probability of
>> >        leaving accessible network services and applications in a damaged
>> >        state and will often result in a higher probability of detection
>> >        by the Internet user and subsequent interpretation as a targeted
>> >        attack.  Depending upon the vulnerability being scanned, some
>> >        automated methods of vulnerability checking may result in data
>> >        being altered or created afresh on the Internet users computer
>> >        which be a problem in many legal environments.
> donald: I would strongly recommend against this as a common practice. You
> provided most of the risks that make this a dangerous course.
> 
> [JL] Good point.  If we delete it, we¹ll just be asked later to add it back I
> fear.  So I have added this to the end: ³Thus, while we note that this is one
> technique which may be utilized, it is unlikely to be particularly effective
> and it has problematic side effects, which leads the authors to recommend
> against the use of this particular method.²
> 
>> >        monitoring to identify network anomalies that may be indicative
>> >        of botnet attacks or bot communications.  For example, an ISP may
>> >        be able to identify compromised hosts by identifying traffic
>> >        destined to IP addresses associated with the command and control
> donald: destined to an IP address and control port associated with the ...
> Most of the time the host associated with c and c has other legit traffic so
> having the control port helps a LOT when running potentially infected bot net
> reports.
> 
> [JL] good point  added some text to address this
> 
>> >        of botnets.  In addition, bots can be identified when a remote
>> >        host is under a DDoS attack because computers participating in
>> >        the attack will likely be infected by a bot, frequently as
> donald: I would caution against FPs due to spoofing of source IP addresses.
> With netflow you can check which router and interface the attack traffic is
> coming in from as method to reduce FPs.
> 
> [JL] added: ³...(though ISPs should beware of source IP address spoofing
> techniques to avoid or confuse detection).²
> 
>> >    e.  User complaints: Because hosts infected by bots are frequently
>> >        used to send spam or participate in DDoS attacks, the ISP
>> >        servicing those hosts will normally receive complaints about the
>> >        malicious network traffic.  Those complaints may be sent to
>> >        RFC2142-specified [RFC2142] role accounts, such as abuse@ or
>> >        postmaster@ or to abuse or security addresses specified by the
> donald: postmaster is for email delivery issues and we shouldn't be
> encouraging people to use it for abuse unless no other contact info is
> available.
> NOC or security should be encouraged instead if abuse@ doesn't work.
> 
> [JL] made a change to address this
> 
>> >        site as part of its WHOIS (or other) contact data.
>> >
>> >    f.  ISPs may also discover likely bot infected hosts located at other
>> >        sites; when legally permissible or otherwise an industry accepted
> donald: located on other ISPs networks; when legally ...
> You mention industry accepted practice here as an or legally permissible but I
> think you mean and not or. It has to be legal or you shouldn't do it period.
> If it is an industry accepted practice shouldn't really come in to it. The
> first of us that started share such reports to other ISPs we were the first to
> do so. It wasn't the industry accepted practice at that time.
> 
> [JL] Good catch  made a change to address that
> 
>> > 6.  Notification to Internet Users
>> >
>> >    Once an ISP has detected a bot, or the strong likelihood of a bot,
>> >    steps should be undertaken to inform the Internet user that they may
>> >    have a bot-related problem.  Depending upon a range of factors, from
>> >    the technical capabilities of the ISP, to the technical attributes of
>> >    their network, financial considerations, available server resources,
>> >    available organizational resources, the number of likely infected
>> >    computers detected at any given time, and the severity of any
>> >    possible threats, among many other things, an ISP should decide the
>> >    most appropriate method or methods for providing notification to one
> 
> donald: Ideally the customer should be notified via as many methods as
> possible at nearly the same time if possible. While most ISPs could do more
> then one notification method most of the "bad guys" won't be able to spoof
> many different
> notification types within a small amount of time.
> I don't know anyone that has implemented this but it seems like a really good
> recommendation.
> 
> [JL] Good suggestion, and added text to that effect.
>  
>> >    or more of their customers or Internet users.  Such notification
>> >    methods may include one or more of the following, as well as other
>> >    possible methods not described below
>> >
>> >    It is important to note that none of these methods are guaranteed to
>> >    be successful, and that each has its own set of limitations.  In
> donald: 100% successful,
> 
> [JL] Ack, added.
> 
>> >    addition, in some cases, an ISP may determine that a combination of
>> >    two or more methods is most appropriate.  Finally, notification is
>> >    also considered time sensitive; if the user does not receive or view
>> >    the notification or a timely basis, then a particular bot could
>> >    launch an attack, exploit the user, or cause other harm.  If
>> >    possible, an ISP should establish a preferred means of communication
>> >    when the subscriber first signs up for service.  As a part of the
>> >    notification process, ISPs should maintain a record of the allocation
>> >    of IP addresses to subscribers for such a period as allows any bot
> donald: commonly used bot detection
> 
> [JL] ack, added
> 
>> >    detection technology to be accurately able to link an infected IP
> donald: to be able to accurately link an infected ...
> 
> [JL] change made
> 
>> >    address to a subscriber.  This record should only be maintained for a
>> >    period of time which is necessary, in order to maintain the
>> >    protection of the privacy of an individual subscriber.
>> >
>> >    One important factor to bear in mind is that notification to end
>> >    users needs to be defended against spoofing by third parties.  This
>> >    must be done to protect against the possibility of notifications
>> >    being spoofed and used by bots to deliver additional malware.
> donald: must be done to protect as well as reasonably possible against the
> possibility of legitimate notifications being spoofed and used by persons with
> malicious intent to perform additional malicious acts against the victims.
> 
> They may not be bots and they may not use this to "infect" with a bot in many
> cases "they" will be an AD that sells fake AV/scareware.
> 
> [JL] Good suggestion  change made
> 
>> >
>> > 6.1.  Email Notification
>> >
>> >    This is probably the most common form of notification used by ISPs.
> donald: Is it? Do we have any surveys or stats on this? I think your correct
> but it would be good to have some kind of backup.
> 
> [JL] changed to ³common² since I have no data to backup that it¹s the most
> common.
> 
>> >    delete the message or otherwise file it in an email folder that the
>> >    user may not check on a regular and/or timely basis.  Bot masters
>> >    have also been known to impersonate the ISP or trusted sender and
>> >    send fradulent emails to the users.  This technique of social
> donald: fraudulent
> 
> [JL] corrected
> 
>> >    engineering, generally referred to as 'phishing', often leads to new
> 
> donald: Phishing is generally defined as attempting to gain user credentials
> or other sensitive information by spoofing a trusted source.
> What your describing is a method used by some viruses such as beagle to spread
> by convincing customers to install it thinking it was a fix to some malware or
> other issue. This is commonly refered to as social engineering as you state
> but not phishing.
> 
> [JL] That may have been an artifact of an earlier draft, but in any case I
> dropping the phishing part.
> 
>> >    as such not welcome it, or not accept the call at all.  Finally, even
>> >    if a representative of the ISP is able to connect with and speak to a
>> >    user, that user is very likely to lack the necessary technical
>> >    expertise to understand or be able to effectively deal with the
>> >    threat.
> donald: Isn't this last statement true for ALL notifications methods.
> I agree it is true, but it probably needs to be a caveat for all notification
> methods. You mention it else where but perhaps a special paragraph at the
> beginning that states this is true for all notification types.
> 
> [JL] I¹ll take that into consideration  added to my open issues list for the
> next version.
> 
>> >
>> > 6.3.  Postal Mail Notification
>> >
>> >    This form of notification is probably the least popular and effective
>> >    means of communication, due to both preparation time, delivery time,
>> >    the cost of printing and paper, and the cost of postage.
> 
> donald: For ISPs that are already billing their customers there wouldn't need
> to be additional cost of postage, printing or paper it could be incorporated
> into the bill its self. The biggest risk here is delivery time and the fact
> that some customers don't review their bills (similar to the email issue
> identified above).
> 
> [JL] In most cases, it¹s not realistic to get that sort of info into a billing
> system.  Thus, a mailer is likely to be separate.
>> >
>> > 6.4.  Walled Garden Notification
>> >
>> >    Placing a user in a walled garden is another approach that ISPs may
>> >    take to notify users.  A walled garden refers to an environment that
>> >    controls the information and services that a subscriber is allowed to
>> >    utilize and what network access permissions are granted.  This is an
> donald: That is really the definition of a quarantine walled garden.
> Walled garden's don't have to quarantine the user (much) to notify the
> customer.
> They can be fairly leaky and still notify the customer of their infection.
> 
> [JL] I¹d like to ponder that more, so have added to our open issues list.
> 
>> >    not always the case that a user is actively using a computer that
>> >    uses a web browser or which has a web browser actively running on it.
> donald: not always the case that a user is using a application that used the
> ports being redirected to the walled garden.
> 
> This is more of an application issue then the "computer".
> 
> [JL] Was modified to Œhost.¹  Also added the reference to other ports.
> 
>> >    In one example, a user could be playing a game online, via the use of
>> >    a dedicated, Internet-connected game console.  In another example,
>> >    the user may not be using a computer with a web browser when they are
>> >    placed in the walled garden and may instead be in the course of a
>> >    telephone conversation, or may be expecting to receive a call, using
>> >    a Voice Over IP (VoIP) device of some type.  As a result, the ISP may
>> >    feel the need to maintain a potentially lengthy white list of domains
>> >    which are not subject to the typical restrictions of a walled garden,
>> >    which could well prove to be an onerous task, from an operational
>> >    perspective.
> donald: Or design a leaky walled garden that allows most traffic without
> interference only redirecting the traffic needed to mitigate or notify.
> 
> [JL] Seems related to the question raised before re a quarantine WG.  I¹ll
> keep it on the open issues list.
> 
>> >        This option is suggested when the purpose of the
> donald: primary purpose of the
> 
> [JL] change made
> 
>> >    Once the user acknowledges the notification, then the user decides to
> donald: the notification, decides to remediate and exit the walled garden or
> exit the walled garden without remediating the issue.
> 
> [JL] made a chg to address this
> 
>> >    either remediate and then exit the walled garden, or exit the walled
>> >    garden without addressing the issue.  Another approach may be to
>> >    enforce a stricter policy and require the user to clean the computer
>> >    prior to permitting the user to exit the walled garden, though this
>> >    may not be technically feasible depending upon the type of bot,
>> >    obfuscation techniques employed by a bot, and/or a range of other
>> >    factors.  Thus, the ISP may also need to support tools to scan the
>> >    infected computer and determine whether it is still infected or rely
>> >    on user judgment that the bot has been disabled or removed.  One
> 
> donald: Scanning is risky and should be identified as such.
> Port scanning or vulnerability scanning is dangerous as you identified else
> where.
> Anti-virus scanning is also risky if the ISP initiates it as that means you
> have taken some level of control (and responsibility) for that infected
> machine.
> A better method would be to use SNORT or similar tool to passively watch for
> packets indicating an continued infection.
> Another good approach is just to keep track of whom was put into the walled
> garden for which infection and if they keep showing up as infected via the
> trusted third party reports then you can assume they are unable/unwilling to
> remediate and take what ever additional actions you deem appropriate.
> 
> [JL] Added a clarifying statement ³(in the style of a virus scan, rather than
> a port scan)²
> 
>> >    There are several drawbacks with this communications method.  There
>> >    is a high probability that subscriber may interpret the communication
>> >    to be spam, and as such ignore it.  Also, not every user uses IM
>> >    and/or the user may not provide their IM identity to the ISP so some
>> >    alternative means have to be used.  Even in those cases where a user
>> >    does have an IM address, they may not be signed onto that IM system
>> >    when the notification is attempted.  There maybe also be a privacy
>> >    concern on the part of users, when such an IM notification must be
>> >    transmitted over a third-party network and/or IM service.  As such,
>> >    should this method be used, the notification should be discreet and
>> >    not include any PII in the notification itself.
> 
> donald: Isn't this true for email, sms and possibly other methods.
> Again if so it could become part of the opening statement rather then tied to 
> just one notification type.
> 
> [JL] Good suggestion to ponder   I will drop that into my open issues for a 
> future rev.
> 
> 
>> > Livingood, et al.        Expires August 16, 2010               [Page 15]
>> >
>> > Internet-Draft     Remediation of Bots in ISP Networks     February 2010
>> >
>> >
>> >    typically pay on a per-message basis to send SMS notifications.  An
> 
> donald: No in most cases the ISP would use the customers SMS providers SMTP to 
> SMS gateway and wouldn't encore additional costs for an sms msg.
> 
> [JL] I just generalized to the following: ³In both cases an ISP may incur fees 
> related to SMS notifications, depending upon the method used to send the 
> notifications²
> 
>> >    additional downside is that SMS messages sent to a user may result in
>> >    a charge to the user by their wireless provider, depending upon the
>> >    plat to which they subscribe.  Another minor disadvantage is that it
> 
> donald: did you mean platform or feature set here?
> 
> [JL] I meant Œplan¹ and corrected it.
> 
>> >    is possible to notify the wrong user if the intended user changes
>> >    their mobile number but forgets to update it with the ISP.
> 
> donald: Again this applies to more then just sms, certainly to email accounts 
> and probably other notification methods.
> 
> [JL] True, but given that most users are metered in some manner for their SMS 
> messages, this is more pronounced of a risk.  An SMS could cost a user a few 
> cents.
> 
>> > 6.9.  Considerations for Notification to Network Locations Using a
>> >       Shared IP Address
>> >
>> >    Delivering a notification to a location that Internet access routed
>> >    through one or more shared public IP addresses may be of low value
>> >    since it may be quite difficult to differentiate between users when
>> >    providing a notification.  For example, on a business network of 500
>> >    users, all sharing one public IP address, it may be sub-optimal to
>> >    provide a notification to all 500 users if you only need one specific
>> >    user to be notified and take action.  As a result, such networks may
> 
> donald: In cases like this to assist the business network you describe 
> providing a date/timestamp, source port of the infected system, and malicious 
> sites it was visiting often helps them find the infected system.
> 
> [JL] Added: However, should an ISP implement some form of notification to such 
> networks, it may be better to simply send notifications to a designated 
> network administrator at the site. In such a case the local network 
> administrator may like to receive additional information in such a 
> notification, such as a date and timestamp, the source port of the infected 
> system, and malicious sites that may have been visited.
> 
>> >          notice regular disk activity even when you are not doing
>> >          anything.  Ignoring this problem is not really an option since
> donald: Ignoring this problem is risky to you and your personal information ..
> 
> [JL]  change made
> 
>> >    As an additional consideration, it may be useful to create a process
>> >    by which users could choose, at their option and with their express
>> >    consent, to share data regarding their bot infection with their ISP
>> >    and/or another authorized third party.  Such third parties may
>> >    include governmental entities that aggregate threat data, such as the
>> >    Internet Crime Complaint Center referred to earlier in this document,
>> >    to academic institutions, and/or security researchers.  While in many
>> >    cases the information shared with the user's ISP or designated third
>> >    parties will only be used for aggregated statistical analysis, it is
>> >    also possible that certain research needs may be best met with more
>> >    detailed data.  Thus, any such data sharing from a user to the ISP or
>> >    authorized third party may contain some type of personally
>> >    identifiable information, either by design or inadvertently.  As a
>> >    result, any such data sharing must be enabled on an opt-in based,
>> >    where users review and approve of the data being shared and the
>> >    parties with which it is to be shared.
> 
> donald: Unless the ISP is already required to provide said information due 
> local laws or judicial requirements (subpoena)...
> 
> [JL] change made
> 
>> >    In addition, regarding notifications, as described in the Section 6,
>> >    care should be taken to assure users that notifications have been
>> >    provided by a trustworthy site and/or party, so that the notification
>> >    is more difficult for phishers to mimic, or that the user has some
> 
> donald: I think you are talking social engineers not phishers.
> 
> [JL] changed to include both.
> 
> Thanks again for this highly detailed feedback!!
> 
>

---

From: "Smith, Donald" <[email-masked]>
Date: Wed, 21 Apr 2010 10:39:06 -0600

Thanks Jason, a few minor comments below.
Mostly I agreed with your changes.
Danny there is one comment aimed at you wrt the SP survey you do every year.

(coffee != sleep) & (!coffee == sleep)
[email-masked] gcia

> -----Original Message-----
> From: Jason Livingood [mailto:[email-masked]]
> Sent: Wednesday, April 21, 2010 9:00 AM
> To: Smith, Donald; Joel Jaeggli; Danny McPherson
> Cc: Nirmal Mody; opsec wg mailing list; Mike O'Reirdan
> Subject: Re: [OPSEC] Fwd: Last Call for
> Comments:draft-oreirdan-mody-bot-remediation-06
>
> I'm just getting around to doing a -08 document update now
> that IETF-77 has concluded.  (Joel's feedback has been
> incorporated into -08 as well.)
>
>       donald: A bot is a tool. It can't really be malicious
> nor benign any more that a gun, or screwdriver is malicious
> however I understand as this is widely used, incorrectly,
> within the industry. But malicious implies intent to do harm.
>       As long as that is addressed in the malicious bot/bot
> definition it's probably acceptable to use it in this way.
>
>       [JL] Quite true.  I have tried to ensure when using
> bots in the doc that I am referring to malicious bots, rather
> than other bots which may have positive uses.  I try to
> define this at length in Terminology, Section 2.1.  (I
>       donald: The risks to users of infected systems and
> others are included/defined in many places within this document.
>       It should probably be defined clearly in one place and
> referenced in all other places.
>
>       [JL] Fair point.  I'm going to call that out in my open
> issues list and address in a future version if I don't have
> the chance to get it into -08.
>
>       donald: Malicious bot implies intent to do harm, while
> these are really bots that can be used to perform potentially
> harmful tasks for the bot master. Perhaps potentially harmful bot?
>
>       [JL] I've added a bunch of new text to 2.1 to address
> this.  Please let me know if there's anything that could be
> improved upon.
>
>       donald: Some times they are installed on purpose but
> the user doesn't understand the full functionality of the
> software installed or the amount of remote control the
> application provides to the remote controller (bot master)
>
>       [JL] I've retained benign since it seems to be used a
> great deal but added harmless as well: "Thus, while there may
> be benign, or harmless bots, for the purposes of this document..."
>
>       donald: This is a pretty good risk identification. You
> might want to add spim, spit, email address harvesting,
> hosting of highly questionable or illegal content
> (BulletProof hosting), various MITM attacks (banking ssl ala
> Zeus) and web site scraping.
>
>       [JL] Added - good suggestions.
>
>       >
>       I think your trying to cover the concept of un-patched
> vs zero day (no patch publicly available) exploitation.
>
>       [JL] Correct. I modified to "... software
> vulnerabilities (which include so-called zero-day
> vulnerabilities where no patch yet exists)"
>
>       >         Initially, bots used IRC to communicate but
> were easy to shutdown if
>
>       donald: Initally SOME bots used IRC, Q, TFN, Trinoo,
> etc... all had non-IRC based control.
>
>       donald: So do you intend for your computer definition
> to include smart ip enabled  devices or not. It's hard to
> tell from this definitation.
>       I would limit the definition to computers that humans
> use as general purpose computers.
>
>       [JL] I did intend to refer to these sorts of devices.
> As I have generalized "computers" to now be "hosts" I've made
> some modifications to make this hopefully more clear in the
> next version.  But I expressly want to include both typical
> PC-type devices as well as, say, Internet-connected picture frames.
>
>       donald: Not really alternatively, since you mention
> vulnerabilities above but I think your trying to say without
> user interaction (scan and spolit) right?
>
>       [JL] Correct.  Changed "Alternatively" to "In other cases"
>
>       donald: unique position to be able to notify their
> customers of potential infection by
>       donald: From an end user perspective, being notified
> that they may have an infected computer on their network is
> important information.
>
>       [JL] change made
>
>       donald: Given that bots can consume vast amounts of
> local computing and network resources ...
>
>       [JL] change made
>
>       donald: Attempts at bot removal can result in side
> effects from a loss of data to partial or complete loss of
> system usablility.
>
>       [JL] change made
>
>       donald: of persistent memory based malware may meant
> that, in some cases, this may not be ...
>
>       [JL] chg made
>
>       donald: would have to re-flash or re-image persistent
> memory sections on their machines in order ...
>
>       [JL] chg made
>
>       donald: Many such devices have a "reset to factory
> defaults" switch which will be the main method to clear such
> devices if infected.
>
>       [JL] Good suggestion.  Added "However, in some cases,
> such devices will have a function or switch to enable the
> user to reset that device to a factory default configuration,
> which may in some cases enable the user to remediate the infection."
>       >
>       >
>       > 5.  Detection of Bots
>       >
>       [JL] I see what you mean but it's tough to get that
> across.  I think there's enough wiggle room insofar as I have
>  said "seek to be non-disruptive" which means it is really
> simply a goal.
>
>       >
>       donald: In order to validate any bot net detection
> source a combination of multiple bot
>       donald: I agree erring on the side of caution BUT that
> should be don't communicate with your customers unless your
> fairly certain they are infected (False positive rate of 5%
> or less)! Notifying customers just on "likely" (ie greater
> then 50%) will cause customers loose faith in such notification.
>
>       [JL] Good point - added text to address this
>
>       donald: There is a risk here in that this teaches users
> to install software on their computer that may not be
> effective given todays botnet + root kit hiding abilities.
> Thus a false sense of security plus you've trained them to
> install software outside of normal OS maintenance. Would this
> even be included if it weren't the approach you choose?
>
>       [JL] It's not a statement that we should only rely on
> host-based protections, merely an argument for layered
> defenses.  I don't want to see things like I used to see in
> enterprises, where folks felt that since a firewall existed
> in the network that hosts could go largely unprotected. In
> this case, just because an ISP may have detection and perhaps
> protection measures in place, I don't want to obviate users
> of the responsibility to run some local software to protect
> themselves (something > nothing).  :-)

Agreed and in some cases I realize providing a link to a tool to find/remove the malware is pretty much a requirement if you want the customer to be able to clean up.
So there is a bit of an conundrum here as bad guys have used fake notifications to socially engineer vicitims to install malware. That is the primary method used for the class of malware refered to as scareware/fakeAV.

>       donald: If done learning is done automatically you have
> just solved the turing test.
>       Computers don't learn.
>
>       [JL] I removed 'learning'
>       >
>       donald: While this is true in many cases the customers
> PII is already at risk from criminal elements that want to
> exploit it. So being overly concerned about the privacy of
> PII (such as hiding PII from internal analysis teams) should
> be recommended against.
>
>       [JL] That's why I say "should" and not "must" since as
> you point out there may be cases where this is not possible.
> Also, what legally constitutes PII is typically fairly small.
>  I added this text into the section though: "This is
> important, as just as spam defenses may involve scanning the
> content of email messages which may contain PII, then so to
> may bot defenses similarly come into incidental contact with PII."
Good thanks.

>       donald: I would strongly recommend against this as a
> common practice. You provided most of the risks that make
> this a dangerous course.
>
>       [JL] Good point.  If we delete it, we'll just be asked
> later to add it back I fear.  So I have added this to the
> end: "Thus, while we note that this is one technique which
> may be utilized, it is unlikely to be particularly effective
> and it has problematic side effects, which leads the authors
> to recommend against the use of this particular method."
Agreed!

I missed something in b.
" Good examples of this include SNDS from Microsoft, XBL and PBL
  from Spamhaus and the DSHIELD AS tool from the SANS Institute."

That is a list of good examples but there are a lot of other good 3rd party infected client notification resources. Arbor via atlas, ShadowServer, Cymru, SIE, Support Intelligence and many others.
It might be worth having a seperate section near the end that
has a list of 3rd parties that are willing to share this kind of information with ISPs and provide contact info such as was done for conficker.
http://www.confickerworkinggroup.org/wiki/pmwiki.php/SP/ServiceProviders

That list is NOT complete in any sense of the word.
Providing contact information may help other ISPs get a more complete set of third party reports.
I am a bit concerned that some of the 3rd party contact information could change later and be outdated but I believe most of them will use a role based email account or a link.

>       Most of the time the host associated with c and c has
> other legit traffic so having the control port helps a LOT
> when running potentially infected bot net reports.
>
>       [JL] good point - added some text to address this
>
>       With netflow you can check which router and interface
> the attack traffic is coming in from as method to reduce FPs.
>
>       [JL] added: "...(though ISPs should beware of source IP
> address spoofing techniques to avoid or confuse detection)."
>
>       donald: postmaster is for email delivery issues and we
> shouldn't be encouraging people to use it for abuse unless no
> other contact info is available.
>       You mention industry accepted practice here as an or
> legally permissible but I think you mean and not or. It has
> to be legal or you shouldn't do it period. If it is an
> industry accepted practice shouldn't really come in to it.
> The first of us that started share such reports to other ISPs
> we were the first to do so. It wasn't the industry accepted
> practice at that time.
>
>       [JL] Good catch - made a change to address that
>
>       donald: Ideally the customer should be notified via as
> many methods as possible at nearly the same time if possible.
> While most ISPs could do more then one notification method
> most of the "bad guys" won't be able to spoof many different
>       I don't know anyone that has implemented this but it
> seems like a really good recommendation.
>
>       donald: must be done to protect as well as reasonably
> possible against the possibility of legitimate notifications
> being spoofed and used by persons with malicious intent to
> perform additional malicious acts against the victims.
>
>       They may not be bots and they may not use this to
> "infect" with a bot in many cases "they" will be an AD that
> sells fake AV/scareware.
>
>       [JL] Good suggestion - change made
>
>       >
>       > 6.1.  Email Notification
>       >
>       donald: Is it? Do we have any surveys or stats on this?
> I think your correct but it would be good to have some kind of backup.
>
>       [JL] changed to "common" since I have no data to backup
> that it's the most common.

Good enough. I would still like to see a survey at some point maybe we could get Danny at Arbor to add a set of customer notification questions to their SP survey for 2011.

>       donald: Phishing is generally defined as attempting to
> gain user credentials or other sensitive information by
> spoofing a trusted source.
>       What your describing is a method used by some viruses
> such as beagle to spread by convincing customers to install
> it thinking it was a fix to some malware or other issue. This
> is commonly refered to as social engineering as you state but
> not phishing.
>
>       [JL] That may have been an artifact of an earlier
> draft, but in any case I dropping the phishing part.
>
>       I agree it is true, but it probably needs to be a
> caveat for all notification methods. You mention it else
> where but perhaps a special paragraph at the beginning that
> states this is true for all notification types.
>
>       [JL] I'll take that into consideration - added to my
> open issues list for the next version.
>
>       >
>       donald: For ISPs that are already billing their
> customers there wouldn't need to be additional cost of
> postage, printing or paper it could be incorporated into the
> bill its self. The biggest risk here is delivery time and the
> fact that some customers don't review their bills (similar to
> the email issue identified above).
>
>       [JL] In most cases, it's not realistic to get that sort
> of info into a billing system.  Thus, a mailer is likely to
> be separate.
Ok, I haven't tried to use billing for notification so you could be correct.
It doesn't sound HARD to me but no real background doing so ...

>       Walled garden's don't have to quarantine the user
> (much) to notify the customer.
>       [JL] I'd like to ponder that more, so have added to our
> open issues list.
Sounds good we can talk about the qwest walled garden seperately if you like to better understand our "leaky" approach:)

>       donald: not always the case that a user is using a
> application that used the ports being redirected to the walled garden.
>
>       [JL] Was modified to 'host.'  Also added the reference
> to other ports.
>
>       donald: Or design a leaky walled garden that allows
> most traffic without interference only redirecting the
> traffic needed to mitigate or notify.
>
>       [JL] Seems related to the question raised before re a
> quarantine WG.  I'll keep it on the open issues list.
>
>       donald: the notification, decides to remediate and exit
> the walled garden or exit the walled garden without
> remediating the issue.
>
>       Port scanning or vulnerability scanning is dangerous as
> you identified else where.
>       Anti-virus scanning is also risky if the ISP initiates
> it as that means you have taken some level of control (and
> responsibility) for that infected machine.
>       A better method would be to use SNORT or similar tool
> to passively watch for packets indicating an continued infection.
>       Another good approach is just to keep track of whom was
> put into the walled garden for which infection and if they
> keep showing up as infected via the trusted third party
> reports then you can assume they are unable/unwilling to
>       [JL] Added a clarifying statement "(in the style of a
> virus scan, rather than a port scan)"
>
>       Again if so it could become part of the opening
> statement rather then tied to just one notification type.
>
>       [JL] Good suggestion to ponder  - I will drop that into
> my open issues for a future rev.
>
>
>       donald: No in most cases the ISP would use the
> customers SMS providers SMTP to SMS gateway and wouldn't
> encore additional costs for an sms msg.
>
>       [JL] I just generalized to the following: "In both
> cases an ISP may incur fees related to SMS notifications,
> depending upon the method used to send the notifications"
Ok

>       [JL] I meant 'plan' and corrected it.
>
>       donald: Again this applies to more then just sms,
> certainly to email accounts and probably other notification methods.
>
>       [JL] True, but given that most users are metered in
> some manner for their SMS messages, this is more pronounced
> of a risk.  An SMS could cost a user a few cents.

Ok but the fact that an email account could be retired and reused is still a risk.
I am sure some of the other notification methods could be directed to the wrong party due to account
reuse also so we may need to add some generic statement to that effect.

I agree sms could cost an invalid end user so we can put some emphisis on that.

>       donald: In cases like this to assist the business
> network you describe providing a date/timestamp, source port
> of the infected system, and malicious sites it was visiting
> often helps them find the infected system.
>
>       [JL] Added: However, should an ISP implement some form
> of notification to such networks, it may be better to simply
> send notifications to a designated network administrator at
> the site.
Most ISPs have a "network contact" for their enterprise customers.
That is whom would normally be notified.

That happens in the broadband world via the primary email account
that is used for any type of notification.

> In such a case the local network administrator may
> like to receive additional information in such a
> notification, such as a date and timestamp, the source port
> of the infected system, and malicious sites that may have
> been visited.
malicious sites and ports that may have been visited.

When notifying enterprises they can often find an infected system behind a NAT/PAT system
if they have the date/timestamp, malicious site (ip or url) and port if they have logs.

>       donald: Unless the ISP is already required to provide
> said information due local laws or judicial requirements (subpoena)...
>
>       [JL] change made
>

This communication is the property of Qwest and may contain confidential or
privileged information. Unauthorized use of this communication is strictly
prohibited and may be unlawful.  If you have received this communication
in error, please immediately notify the sender by reply e-mail and destroy
all copies of the communication and any attachments.

---

From: Jason Livingood <[email-masked]>
Date: Thu, 22 Apr 2010 11:11:09 -0400

Inline.

> That is a list of good examples but there are a lot of other good 3rd party
> infected client notification resources. Arbor via atlas, ShadowServer, Cymru,
> SIE, Support Intelligence and many others.
> has a list of 3rd parties that are willing to share this kind of information
> with ISPs and provide contact info such as was done for conficker.
> I am a bit concerned that some of the 3rd party contact information could
> change later and be outdated but I believe most of them will use a role based
> email account or a link.

[JL] I have added this to my issues tracker for -09.

> if they have the date/timestamp, malicious site (ip or url) and port if they

[JL] Change made.

Thanks again,
Jason

---

From: Danny McPherson <[email-masked]>
Date: Thu, 22 Apr 2010 15:06:50 -0600

> Good enough. I would still like to see a survey at some point maybe we could get Danny at Arbor to add a set of customer notification questions to their SP survey for 2011.

Good idea Don.  We'd be happy to add some questions related to
this in attempts to quantify and qualify.  Year before last we 
had a few but not directly along these lines.  If any of you would 
like to have something added to the survey to better support work 
here, we most welcome it.  

I've cc'd a couple colleagues (Mike Hollyman, Craig Labovitz, and
Roland Dobbins) that are charged with compiling the new questionnaire 
this year, please drop them an email and we'll get it in there.

Thanks!

-danny