Subject: GSS-APIv3 sketch

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 12:11:40 -0600

As I mentioned yesterday, I started with a goal of writing a new, model
mechglue based on GSS-APIv2u1, with some extensions, and with the
"PGSSAPI" idea fully incorporated.  For those who don't remember, the
PGSSAPI proposal was to add a [non-security] context handle to all GSS
functions, and which I'd refined to "turn the minor_context argument
into that context handle" proposal.

Well, I realized at one point that there were a number of other problems
with the GSS-APIv2u1 that I wanted to solve, and that my solutions were
less than ideal.  For example: memory management, which involved lots of
interning of things in my scheme.  That led me to consider a close-to-
wipe-the-slate approach -- the slate can't be wiped entirely clean, both
because the effort to update existing applications of the GSS-API to the
new-thing must be minimal, and because protocol aspects of the GSS-API
cannot change.

Here, then, is my proposal for a GSS-APIv3 (from memory, and not
necessarily complete -- it's still a work in progress).  My approach to
implementation is described below.

 - Problems to be solved:

    - memory management
    - extensibility
    - credentials limitations
    - application/instance-specific configuration
    - error reporting
    - QoP issues
    - C-Bindings ABI issues (primarily: remove ABI non-determinism
      caused by certain aspects of the v2u1 API that are left open to
      implementors)

 - At the abstract level the API changes very little:

    - minor_status indeed becomes a "call context handle"

       - functions are added to deal with that (more on this below)
       - this provides for better error reporting
       - this provides for application-specific mechglue configuration
       - for portability: a function to get from a call context handle
	 what used to be a discrete minor status number (providers need
	 not support this)

    - must distinguish between input and output buffers (more on this
      below)

    - sundry extensions

       - including a credential store handle and related functions (more
	 on this below), mechglue/provider configuration control, some
	 amount of reflection, ...

       - see below

    - optional output arguments, of functions where an output is not
      essential and can be obtained via inquiry functions, are
      eliminated (e.g., context expiration time is no longer output by
      init/accept_sec_context functions)

    - extensibility: new flags can be added that are critical or
      non-critical (see below)

    - credentials are more set-like objects
       - add_cred() function is revamped, simplified
	  - it only adds elements to an existing credential
	  - the requirement that desired_name be the same for add_cred()
	    calls that add elements to existing creds is gone
       - allows for better control over what creds are used for
	 accept_sec_context()
       - default initiator principal for a credential set can be
	 inquired/set (but this is cred handle-specific -- a credential
	 store's default initiator principal notion is distinct)
       - see below about credential stores

    - All existing extensions will be included

 - C-Bindings:

    - new symbol/macro prefix name, either gss3_ or gs3_ (capitalized,
      for macros)

    - requires C99

    - requires uint64_t

       - all uses of OM_uint32 become uses of uint64_t

       - for flags arguments, half the word contains critical and legacy
	 flags, while the other half contains non-critical flags

    - requires that all opaque types be declared as pointers to
      incomplete structures, and that the opaque types all be distinct
      (for static type safety)

    - memory management:

       - option for simple, single-threaded apps: sec context and
	 per-msg output tokens automatically release when either the
	 next token is output or the sec context is deleted

       - buffer types:

	  - struct gss3_in_buffer_desc
	  - gss3_in_buffer_t
	  - gss3_out_buffer_t

	 The gss3_out_buffer_t type is opaque, but accessor macros are
	 provided (these may expand to function calls or to casts and
	 derefences; accessor functions provided so one can take their
	 addresses and pass them around, if desired).

	 gss3_release_buffer() releases only output buffers.  No more
	 accidentally calling it to release application buffers!
	 (Static type safety ensures this.)

	 (Internally the gss3_out_buffer_t need only track length,
	 value, and provider release function pointer.)

	  - a function to allocate provider-specific input buffers for
	    optimizing context/per-msg input token processing (returned
	    as both, an output buffer, for releasing, and an input
	    buffer, for the app to place tokens into)

	     - think of crypto HW where using a buffer allocated for the
	       purpose can be used to reduce overhead in using the
	       crypto HW (the newest UltraSPARC crypto units support
	       this, for example)

       - OID, OID set, and output buffer set types

	  - no input buffer set type yet
	  - OID type is opaque
	  - all set types are opaque, manipulated only via functions and
	    accessed only via accessor getter macros/functions

	  - str_to_oid() and oid_to_str() revived

	     - OID description functions are provided, outputting symbol
	       name, the type of thing the OID refers to, stringified
	       OID (with and without arcs resolved to names),
	       description

	  - release_oid() revived

	 (Internally these types need only track count/length,
	 value/elements and provider ops vectors.)

    - all opaque _objects_ (creds, cred stores, contexts, names, call
      contexts), but not buffers, OIDs, OID sets, can be duplicated and
      refcounted.

      There's a function per-object type to duplicate, and a function
      per-object type to get an additional reference to an object.  The
      traditional release functions decrement reference counts,
      destroying the object when the last reference is gone.

    - minor_status indeed becomes a call context handle
      (gss3_call_ctx_id_t)

       - there's a default handle, GSS3_C_DEFAULT_CALL_CONTEXT, that can
	 be duplicated to get a call context handle that refers to the
	 default mechglue/provider configuration

       - new call contexts can be allocated, with a given configuration
	 file, or with manual mechglue configuration (add provider, add
	 provider, ...)

	  - and these can be duplicated, of course

	  - memory management functions can be specified, if desired

       - configuration is frozen when one call indicate_mechs() or one
	 of the extended mech inquiry functions

       - display_status() clearly has more information now
	  - internally the mechglue maintains a cache of
	    provider-specific call contexts in each mechglue call
	    context; this allows providers to track more error/status
	    information in a context-specific manner
	  - on next call to anything other than display_status() or
	    release_call_context() the previous error information is
	    cleared
	  - mechglue vs. provider errors better delineated (there's a
	    mechanism OID for the mechglue itself)

    - thread safety

       - v3 API is thread-safe provided that: only one thread is
	 accessing any given object at any given time

	  - for buffers and sets, what matters is destructive calls

	  - security contexts can allow concurrent per-msg token
	    functions if the app requests this (rationale: the mechanism
	    may be able to produce better concurrency than the
	    application could by locking around these functions)

    - well-defined SPI

       - the API is the SPI, _mostly_, but there's a function that each
	 provider must provide by which a mechglue above it can
	 configure the provider

	  - among other things, the mechglue passes a dlsym()-like
	    function and handle to providers that providers can use to
	    re-enter the mechglue (for pseudo-mechanisms, such as SPNEGO
	    and stackable mechanisms); re-entrance must be done this way

       - mechglues can be stacked; this is only interesting for testing
	 purposes, and for accessing providers with a v2u1 interface
	 only (see below)

       - I've several approaches for the delegated credential issue

    - credential stores

       - opaque object
       - anonymous credential stores (think MEMORY ccache type) can be
	 created at any time
       - notion of "current credential store" associated with call
	 contexts
       - a credential store has a notion of "default initiator
	 principal"
       - external credential stores supported for: process, session,
	 user
	  - a reference to such a cred store can be obtained if it
	    exists
	  - credential stores can be set for a process, session, user,
	    but creation of those things is not provided for (i.e., the
	    API is agnostic as to how processes, sessions, or users are
	    created; a login program still has to create such items in
	    an OS-specific manner)
	     - when the input cred store is anonymous it becomes
	       non-anonymous (think taking a MEMORY ccache and copying
	       it to a FILE ccache)
	  - note: no per-thread stores -- not needed because there's
	    always a call context and call contexts have a "current
	    credential store" notion
       - optional cred store argument for credential acquisition and
	 credential listing functions (not really needed, but makes API
	 easier to use, I think)
       - _of course_, there's a store_cred() function

    - QoP issues fixed

       - QoP integer argument for per-msg token functions replaced with
	 optional policy names
       - able to inquire what QoP _policies_ a credential supports
       - able to set what QoP _policies_ a credential can/must not use
       - able to inquire what QoP _policies_ a sec context supports
       - able to set what QoP _policies_ a sec context can/must not use,
	 and what QoP policy is the default for per-msg token functions
       - policies are named with well-known names with well-known
	 semantics or locally-controllable semantics
	  - Fixed semantics
	     - FIPS-140-2		(fixed semantics)
	     - NIST something or other	(fixed semantics)
	  - Fixed relative semantics, but with locally-definable exact
	    semantics
	     - strong	(strong algs only)
	     - fast	(fast algs only; may or may not be strong)
	     - interop	(required-to-implement algs included)
	     - weak	(weak algs included)
	     - notweak	(same as interop but with weak algs excluded;
			 may not be as strong as 'strong' policy)

	    I.e., sysadmin can say "'strong' means AES enctypes for krb5
	    mech, ...".

	  - locally-definable policy names (and semantics)?  locally-
	    definable policy names complicate the API...

    - Sundry extensions
       - I don't remember them all, but...
       - a per-msg token normalized-to-zero-start sequence number, for
	 mechanisms/sec contexts that can support it (the rationale:
	 apps like RPCSEC_GSS have had to build their own -- what a
	 waste!)

Implementation approach:

 - There's a file format for describing GSS-API functions:

    - function name, acronym (for internal purposes)
    - function attributes
       - versions of the API where the function is present
       - mechglue core (code can't be generated) or not core (code can
	 be generated)
       - stackable mech core or not core
       - whether it deals with call context allocation/release
       - whether providers include this function (whether to generate
	 function pointer typedef)
    - function arguments and per-argument attributes
       - type
       - input/output (optional, as often this can be inferred)
       - whether present in v2u1 and/or v3 APIs
       - ...
    - function description
    - header comments

 - A compiler reads the above and generates:

    - header file contents:
       - C function prototypes
       - file C function pointer typedefs (named <function_name>_fct)

    - code:
       - mechglue core function stubs
       - mechglue non-core functions, including code
	  - there are several patterns, inferred from function
	    name/attrs and argument list
       - stackable mechanism core function stubs
       - "null" stackable mechanism non-core functions, including code
	  - to be used as basis for constructing a variety of stackable
	    mechanisms
	  - there are several patterns, inferred from function
	    name/attrs and argument list
       - ssh-agent-like shell interface to GSS-API (to make it possible
	 to write most tests as shell scripts!)

    - compatibility shim code:
       - v2u1->v3 provider shim (so you can use existing v2u1 mechglues
	 and providers as providers for a v3 mechglue)
       - v3->v2u1 shim (so you can use a v3 mechglue implementation in
	 v2u1 applications)
	  - this uses interning to solve memory management issues (i.e.,
	    output buffers are interned)

    - documentation:
       - xml2rfc stubs (including generic abstract and C-bindings
	 argument descriptions + function description supplied in input
	 file); these can be accessed from a larger xml2rfc document as
	 XML entities
       - POD (or whatever) manpage stubs (including generic abstract and
	 C-bindings argument descriptions + function description
	 supplied in input file, SEE ALSOs, ...)

Currently my compiler is written as a ksh script.  Ideally the input
file should be XML and the compiler would be mostly a collection of XSLT
transformations of the input, but I've no clue as to how to write XSLT
transforms, so for me ksh it is for now.

Also, since I switched to a v3 design mid-stream, my compiler is not
entirely updated, and the mechglue core is incomplete and busted.

For me this is a weekends-only project, which means it's progressing
_slowly_.  I intend to ask our legal department for permission to open
source this with a permissive license, as I want to have a reference
implementation to speed standardization and adoption, but I make no
guarantees.  There's no IPR here that I know of (I've not applied for
any patents covering any aspect of the above; I have described this
project to others before; I believe most of this obvious enough).

Note too that the above approach can easily be re-purposed to generate a
SASL API.  I've considered having a SASL-as-GSS API, but there are too
many minor impedance mismatches for me to be comfortable with that
approach.  (The same problem applies to TLS, IMO, though a reasonable
subset of TLS/DTLS can be made available as a "non-standard" GSS
mechanism, in the GGF way.)

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 12:42:44 -0600

On Wed, Nov 11, 2009 at 10:40:50AM -0800, Love Hörnquist Åstrand wrote:
> 
> I have so many comments that I don't know where to start.
> 
> First out: any work that doesn't include async is dead in the water.

Oh, I forgot to mention that.  It does include async.

> Its not mentioned how this is gss3 objects relates to gss2 object.

They are distinct.  The compat shims take care of that.

> Well-defined SPI will force everything to reimplemnent GSS-API
> semantics in all layer, this sucks and duplicates code.

No, see the compat shims.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 12:47:27 -0600

> Oh, I forgot to mention that.  It does include async.

I should describe it:

 - The simplest way to do async is:

    - add a new return code, S_ASYNC_CONTINUE_NEEDED
    - add a callback, to be set on a call context handle, that will be
      called (on a different thread) when the suspended function is
      ready to make progress

   This requires threads.  I don't mind that at all.  Any alternative
   based on poll(2)/select(2)/libevent/kevent/event ports/whatever is
   going to either be OS-specific or will require a third party event
   library (e.g., libevent), neither of which seems more appropriate
   than just requiring threads (note: not POSIX threads, specifically,
   nor any other specific thread implementation).

The same approach of associating callbacks with the call context works
for prompter callbacks for dealing with initial credential acquisition.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 12:55:29 -0600

On Wed, Nov 11, 2009 at 10:57:30AM -0800, Love Hörnquist Åstrand wrote:
> 
> 11 nov 2009 kl. 10:42 skrev Nicolas Williams:
> 
> If the api requires calling sane function serveral times with the same arguments, the API is broken.
> 
> That doesn't answer my question, making the SPI look like the API is the problem.

Actually, I have options to set the SPI symbol prefix to anything you
like, so as long as you don't have additional SPI arguments (as we do in
OpenSolaris) you're OK.  But ALSO, I did answer your question: providers
that don't support he new SPI can be accessed via v2u1 mechglues used
through the v3 mechglue (that's what the note about shims was about).

---

From: Love Hörnquist Åstrand <[email-masked]>
Date: Wed, 11 Nov 2009 10:40:50 -0800

Well-defined SPI will force everything to reimplemnent GSS-API semantics in all layer, this sucks and duplicates code.

Love

11 nov 2009 kl. 10:11 skrev Nicolas Williams:

> _______________________________________________
> Kitten mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/kitten

---

From: Love Hörnquist Åstrand <[email-masked]>
Date: Wed, 11 Nov 2009 10:57:30 -0800

Love

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 13:06:30 -0600

> through the v3 mechglue (that's what the note about shims was about).

I should add too that this is how I plan to test the mechglue, since I
don't intend to port any existing providers to this SPI:

 - configure a v3 call context to use a v2u1 mechglue as a provider
 - add stackable mechanism providers (to excercise the v3 SPI)
 - test

Also, my plan is to mix OpenSolaris and MIT mechglues to access
different mechanism providers indirectly:

 - configure a v3 call context to use the OpenSolaris v2u1 mechglue as a
   provide mech_dh
 - add the MIT v2u1 mechglue (compiled to have the same ABI as the
   OpenSolaris one) as a provider to provide mech_krb5
 - add stackable mechanism providers (to excercise the v3 SPI)
 - test

Nico
--

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 11 Nov 2009 14:41:27 -0500

> through the v3 mechglue (that's what the note about shims was about).

I think you're missing Love's point.  Regardless of what the symbol names 
are, if the API and SPI are the same, then both the mechglue and each 
mechanism have to implement all of the semantics of that interface.  That 
was sort of necessary with GSSAPIv2, because the mechglue concept was 
introduced after the original API design, which envisioned selecting a 
mechanism at link time.  Now that we understand how silly that is, there's 
no reason to presume that anyone will link to a GSSAPIv3 mechanism provider 
directly, and so the API and SPI need not be the same.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 14:14:01 -0600

> directly, and so the API and SPI need not be the same.

I didn't say the SPI is exactly the same as the API.  I said "mostly".

The reason for the two to be mostly the same is simple: I can't think of
why I'd want the SPI to be radically different than the API, that is, I
haven't found an implementation reason for wanting the SPI to be
radically different than the API.

In fact, having them be at least very similar helps make the code
generation approach possible.  For example, suppose I created a single
SPI function interface to all per-message token functions -- I sure
could, but then the code generator would have to learn all the specific
details of how to map per-msg token API functions onto that one SPI
function, thus negating the whole point of a code generator approach.

Let's look at possible SPI departures from the API:

 - symbol prefixes -- easy to handle
 - extra arguments, as in the OpenSolaris SPI (which is not open)
    - I don't see a need for such extra arguments, but as long as the
      pattern is regular, it can be easily handled
 - instead of using dlopen()/dlsym() you call a function that returns a
   struct full of function pointers -- easy to handle
    - I also forgot to mention that the mechglue configuration interface
      allows you do implement statically-linked providers if you want

 - radical departures from the API, such as merging API functions at the
   SPI layer -- NOT easy to handle

 - gratuitous departures, such as different argument ordering -- why
   bother??

 - other departures??

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 14:34:28 -0600

On Wed, Nov 11, 2009 at 12:11:40PM -0600, Nicolas Williams wrote:
> 	  - among other things, [...]

To be clear, the SPI is not exactly the API, and my description of the
differences between the two was not intended to be complete ("among
other things, ...").

Besides the differences I pointed out in the original and follow-ups,
it's reasonable to have some slight semantic differences, such as: a
provider not providing async support, with the mechglue invoking the
provider on a helper thread so as to layer async support above it
(Jeff's suggestion), or a provider not providing per-msg token
concurrency but the mechglue providing it on the provider's behalf.

There may be other differences, but the point stands: the SPI will be
_mostly_ the same as the API.  The rationale is as given in one of those
follow-ups: the more similar the two, the easier it is to generate
mechglue code, plus, there's no obvious reason to want the two
interfaces to diverge in more than minor ways.

But if someone has any reasons to want the two interfaces to diverge
radically, I'd love to hear them.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 15:45:54 -0600

I did forget to mention a variety of other things, one of them is:

 - The code generator will also generate marshall/unmarshall code for
   encoding function calls and results for proxying over
   sockets/doors/...

   This is necessary for the agent-like shell interface to the GSS-API.

   It's also useful for things like kgssapi/gssd interfaces.

   It's also useful for doing some privilege separation kinds of things.
   In particular it's useful for a  stackable mechanism I've in mind
   that authenticates _two_ initiator principals to a single target
   acceptor principal.

   It's also useful for GSS-API credentials forwarding ala ssh-agent
   forwarding.

Yes, I should look at my notes and produce the full list of things I've
already got code or comments for.  I left them at home :(

Nico
--

---

From: Andrew Bartlett <[email-masked]>
Date: Thu, 12 Nov 2009 09:59:26 +1100

> for prompter callbacks for dealing with initial credential acquisition.

I'm hoping that Volker or Metze (CC'ed async experts on the Samba Team)
can fill out some more detail, but I'll start by nothing that anything
that requires threads, or causes the library to start threads is a
non-starter for Samba.  

Creating threads behind the applications' back is a really bad idea. 

We really need a proper state machine, with integration available into
our event library (or any other the event library the caller
provides).  

As to portability, if no event lib is provided by the caller, then just
block like GSSAPI does at the moment.  

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Cisco Inc.

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 00:08:50 +0100

> block like GSSAPI does at the moment.  

Just to confirm: Samba and probably other applications
really need an async gensec without threads. I've very
recently seen a talk by a SUN engineer talking about doing
threads behind an app's back. This really, really nasty
stuff.

For a possible model how to do the async API, I would
suggest looking at the avahi libraries:

http://avahi.org/download/doxygen/struct_avahi_poll.html

Writing an async state machine is definitely a non-trivial
task, but it is possible given the right infrastructure. In
Samba, it has taken us several iterations to come up with
something that makes state machines relatively easy to do.
Unfortunately it heavily depends on our hierarchical memory
allocator called talloc. Without such a thing, memory
management can very quickly become a real nightmare.

Volker

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 17:00:45 -0600

> Creating threads behind the applications' back is a really bad idea. 

In the Solaris unified process model this can just happen.  That is,
libraries are free to create worker threads.

> our event library (or any other the event library the caller

That's much more work, and much less portable, and much more OS-/event-
API-specific.  I'm speaking here from a standardization point of view.
(In fact, using libevent would result in reasonably portable interfaces,
certainly across all Unix, Linux, and BSD type operating systems.  But
that's beside the point that I'm making.)

I think implementations could certainly have such interfaces, but I
think we'd not standardize them here.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 17:04:42 -0600

> threads behind an app's back. This really, really nasty

I'd like to hear more about this.  Can you please describe the problem
with libraries creating worker threads?  You can use private e-mail if
you wish.  References to bug reports would be great.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 11 Nov 2009 17:20:07 -0600

> think we'd not standardize them here.

On second thought, I think we could do non-threaded async APIs as
separate RFCs, one for each event interface.  We might end up with only
one: libevent.

One sketch:

 - app calls set_async_XYZ() on a call context handle to register
   XYZ-specific event loops
 - init/accept_sec_context() return S_ASYNC_CONTINUE_NEEDED
 - app calls init/accept_sec_context() again when a related event is
   posted

Another:

 - app calls set_async_XYZ() on a call context handle to indicate the
   desire to do XYZ-specific async calls
 - init/accept_sec_context() return S_ASYNC_CONTINUE_NEEDED
 - app calls get_async_XYZ() on the call context handle to get
   XYZ-specific event descriptions that the app then handles in an
   XYZ-specific way
 - app calls init/accept_sec_context() again when a related event is
   posted

Nico
--

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 08:20:18 +0100

> you wish.  References to bug reports would be great.

The most nasty implications come from initialization and
from fork. If you have several libraries doing threads, it
is easy for unsuspecting applications creating races. The
worst one is fork: You have to shut down all threads before
the fork and potentially re-start them after it, because you
need to drop all locks/mutexes/etc during the fork. If
needed, I can try to contact the author and get his slides

Volker

---

From: "Stefan (metze) Metzmacher" <[email-masked]>
Date: Thu, 12 Nov 2009 13:03:49 +0100

>  - app calls init/accept_sec_context() again when a related event is

I think such models could nicely work from the callers view.
But init/accept_sec_context() should only return A_ASYNC_CONTINUE_NEEDED
once! And the caller gets only notified when the function has completed,
so that the caller can get the result.

The main remaining problem has to do with the mech implementations, they
need an api to register different types of events.

Think about this:

A system has a libgssv3.so with mechglue code and a few
mech implementations by different vendors.

Then some application wants to use libevent with the
libgssv3.so.

On the same box samba wants to use it's tevent library
with libgssv3.so.

And a third application wants to use the glib event code
with libgssv3.so.

The mech implementations should work in all 3 cases
without knowledge about the specific event system.

metze

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 11:04:03 -0600

> needed, I can try to contact the author and get his slides

I'd love to see that, please :)

OpenSolaris' fork() == fork1(), and it has forkx(2) for libraries.
forkx(2) allows library-forked child procs to not interfere with the
application running in the parent, e.g., by not triggering SIGCHLD and
not showing up in plain wait(3C) calls.  And libraries in OpenSolaris
that create worker threads take care to use pthread_atfork() and
atexit() handlers to do fork/exit cleanup.

Of course, pthread_atfork() and atexit() are standard functions...
Libraries on Linux should use them too.

If there are issues in OpenSolaris (apologies to the list) about this, I
really want to hear about them, because we need to make sure they are
addressed if it's at all possible.  I also care to hear about issues on
Linux and *BSD though (I do some development on Linux, but mostly I want
to be informed).

Nico
--

---

From: Love Hörnquist Åstrand <[email-masked]>
Date: Thu, 12 Nov 2009 10:52:03 -0800

> I'd love to see that, please :)

http://www.opengroup.org/onlinepubs/009695399/functions/fork.html

"A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] 
  Fork handlers may be established by means of the pthread_atfork()function in order to maintain application invariants across fork() calls."

Does libc regiester all mutexes it uses with pthread_atfork() so malloc will continue to work ? As long as all libraries doesn't you pthread_atfork() you are dead.

Fork-and-not-do-exec is dead concept in multi threaded apps.

Love

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 19:54:56 +0100

> I'd love to see that, please :)

It was a talk at the OpenSolaris developer conference in
Dresden a couple of weeks ago:

http://www.osdevcon.org/2009/program_detail.html#zdenek

> Libraries on Linux should use them too.

Yes, sure. It is all possible and doable. But I am just
scared by its complexity. Race conditions and deadlocks can
stay undetected for very long and if they happen they can
turn out to be very hard to reproduce and fix. But that
might be just me. I know event-based programming has its own
quirks and complexities, but at least you can the programs
under a debugger.

> Linux and *BSD though (I do some development on Linux, but mostly I want

There were no special Solaris issues mentioned. All standard
pthreads stuff. The author (interestingly a SUN employee)
talked about Solaris crypto libraries having issues, but he
mostly used them as examples from his own experience fixing
threaded libs. The problems he mentioned however were pretty
generic.

Let me point out another thing: By using threads and a
purely blocking API, you force everyone into threads. If you
do it async, it is very easy to create blocking wrappers
around the async calls which then can be used in threaded
programs. The other way round is a lot harder.

Volker

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 13:41:33 -0600

On Thu, Nov 12, 2009 at 10:52:03AM -0800, Love Hörnquist Åstrand wrote:
> "A process shall be created with a single thread. If a multi-threaded
> process calls fork(), the new process shall contain a replica of the
> calling thread and its entire address space, possibly including the
> states of mutexes and other resources. Consequently, to avoid errors,
> the child process may only execute async-signal-safe operations until
> such time as one of the exec functions is called. [THR] 

>   Fork handlers may be established by means of the
>   pthread_atfork()function in order to maintain application invariants
>   across fork() calls."
> 
> Does libc regiester all mutexes it uses with pthread_atfork() so
> malloc will continue to work ? As long as all libraries doesn't you
> pthread_atfork() you are dead.

If there are libraries in OpenSolaris that use threads but don't clean
up their synchronization state in fork() children via pthread_atfork(),
then that's _BUG_.  I'll grant that some such libraries may not be part
of core Solaris, that they may be FOSS with upstreams that don't care
about such bugs.  But here we're talking about libraries (GSS-API
mechglue and providers) that are part of core Solaris.

> Fork-and-not-do-exec is dead concept in multi threaded apps.

Fork-and-not-exec is a suspect idea no matter what, and if you'll
fork-and-exec then you should use posix_spawn(3C).

But why is any of this a concern here?  Does Samba fork-and-not-exec?
Are you afraid that there are libraries that would?

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 13:42:51 -0600

> programs. The other way round is a lot harder.

That ship has sailed: Solaris 10 and up has a unified process model, and
it's a threaded process model.

Nico
--

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 21:05:37 +0100

> But why is any of this a concern here?  Does Samba fork-and-not-exec?

Yes.

Volker

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 21:06:01 +0100

> it's a threaded process model.

So GSSAPI is a Solaris only thing?

Volker

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 14:06:12 -0600

> then that's _BUG_.  I'll grant that some such libraries may not be part

Also, to be fair, I'd consider such behavior to be a BUG in Linux, *BSD,
...  pthread_atfork() and atexit() are standard interfaces, and they
exist to avoid the problems that you fear will crop up.  If those
interfaces are not used properly, or not used at all, where they should
be, then that has to be considered a bug.

For a _standard_ GSS-API async extension I'd like the REQUIRED to
implement to be one that encodes the least OS-specific detail.  The
callback API is that extension (it can be implemented on any OS that
provides for either threads or a global event loop -- that's Windows,
Linux, Solaris, *BSD, and probably a great many others).  I don't mind,
and will help design, specify and implement alternatives that are
RECOMMENDED or OPTIONAL to implement; such alternatives will be less
portable than the callback design since details of async I/O event
systems vary greatly (though with libevent we can get most
Unix/Unix-like OSes as well as Windows).

Do you agree?

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 14:10:15 -0600

> So GSSAPI is a Solaris only thing?

How does that follow?  I'm advising you that on Solaris you can expect
core libraries to launch threads, and you can expect them to use
pthread_atfork() and atexit() propoerly.

Note that I'm NOT saying "you can't have a libevent-/whatever-based GSS
async extension".  I'm not being hostile to what you propose nor to you.

I'm saying that I think we must have a callback-based GSS async
extension as _one_ choice, and probably the required-to-implement choice
because that one makes the fewest assumptions about operating systems.

Nico
--

---

From: Volker Lendecke <[email-masked]>
Date: Thu, 12 Nov 2009 21:34:03 +0100

> because that one makes the fewest assumptions about operating systems.

That sounds good! I had understood that you only want the
blocking API, depending on threads.

Thanks :-)

Volker

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 12 Nov 2009 14:33:43 -0600

> blocking API, depending on threads.

I thought so :)

> Thanks :-)

NP.  Thanks for the info.

---

From: Tom Yu <[email-masked]>
Date: Mon, 16 Nov 2009 21:04:35 -0500

>           - struct gss3_in_buffer_desc

Should we be more conforming about namespaces and avoid the
POSIX-reserved "_t" suffix?  (as long as we are trying to atone for
our past sins...)