Subject: [nfsv4] Disk protection problem for pNFS block

From: sfaibish <[email-masked]>
Date: Thu, 21 Oct 2010 13:21:25 -0400

I want to discuss this issue in preparation for the WG discussion on
this topic. In the current pNFS block protocol the client has to
contact the MDS in order to get the signature offset of the devices
used by the block client.

Users of pNFS block in Linux want to be able to identify a LUN being used
to store pNFS data without having to contact the metadata server but the
protocol doesn't define a fixed location; the MDS will send it at mount
time. The location of the signature is not defined in the protocol but
we can make it a configuration parameter in the client that will define
the signature location for each pNFS block server. My prefered solution
would be to enhance the block protocol.

In more details the pNFS FS writes a signature to the LUN, but to find
out the offset of the signature the client first needs to talk with the
MDS to be told the offset that the signature is located when using
GETDEVICEINFO. But this is too late and can only be done after the boot
ends and as a result some boot time applications can destroy the FS. One
possible solution is to "hide" pNFS devices or write protect the devices
that contain pNFS data even if that host doesn't mount the pNFS block  
volume.

The concern was that there is a window of time before the devices can be
protected at mount time but it leaves a window of possible data corruption.
I would also like to include some kind of protection mechanism in the  
protocol
as an extenssion to RFS5663 for example. So Jason and I propose something  
like
this:

1) Clients can have configuration file that specifies the signatures to  
protect.
    The signature should be vendor agnostic: "pNFS block device" but at a  
fixed offset.
2) Clients could come pre-configured with a protection file that recognizes
    "most signatures"
3) pNFS servers could define a function that will return such signatures.
    It will be late, but every time one does a mount, the client could  
retrieve
    the signatures, and then make sure that they are listed in the  
configuration
    file.  This way the system learns what it needs to protect in cases  
where a
    new non-standard pNFS server is introduced.

What do people think about the problem and the solution idea? I will  
collect
all the opinion and include in the discussion in Beijing. Thank you for  
your
consideration

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Unified Storage Division
         EMC²
where information lives

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 21 Oct 2010 14:19:22 -0400

On Thu, Oct 21, 2010 at 01:21:25PM -0400, sfaibish wrote:
> ends and as a result some boot time applications can destroy the FS.

I'm confused: you're assuming such applications may write to every disk
they discover that isn't on some list of disks not to touch?

The pnfs block protocol has to assume *some* level of sanity from
clients.

--b.

> One
> that contain pNFS data even if that host doesn't mount the pNFS
> block volume.
> 
> I would also like to include some kind of protection mechanism in
> the protocol
> as an extenssion to RFS5663 for example. So Jason and I propose
> something like
> this:
> 
> 1) Clients can have configuration file that specifies the signatures
> to protect.
>    The signature should be vendor agnostic: "pNFS block device" but
> at a fixed offset.
>    It will be late, but every time one does a mount, the client
> could retrieve
>    file.  This way the system learns what it needs to protect in
> cases where a
> all the opinion and include in the discussion in Beijing. Thank you
> for your
> consideration
> 
> /Sorin
> 
> 
> -- 
> Best Regards
> 
> Sorin Faibish
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Thu, 21 Oct 2010 20:24:49 +0200

>> to store pNFS data without having to contact the metadata server but the

You mean before sending GETDEVICELIST and GETDEVICEINFO?

>> time. 

The mount time thing is linux client specific actually.
The client can issue GETDEVICELIST and GETDEVICEINFO at any time.

>> The location of the signature is not defined in the protocol but

Hmm, now I'm confused too :-/
http://tools.ietf.org/html/rfc5663
2.2.  GETDEVICELIST and GETDEVICEINFO

2.2.1.  Volume Identification

   /// struct pnfs_block_sig_component4 { /* disk signature component */
   ///     int64_t bsc_sig_offset;        /* byte offset of component
   ///                                       on volume*/
   ///     opaque  bsc_contents<>;        /* contents of this component
   ///                                       of the signature */
   /// };

Benny

---

From: sfaibish <[email-masked]>
Date: Thu, 21 Oct 2010 15:11:33 -0400

> they discover that isn't on some list of disks not to touch?
Yes. We encountered such cases of DB applications that use fixed device
name that put a signature on devices. There is no such thing as list
of devices not to touch. How do you identify them if there is no signature?

> The pnfs block protocol has to assume *some* level of sanity from
Well this was the premise when w5663 was written but I can show you cases
when happens specially with applications using raw SCSI devices FC  
connected.
We had this problem with our private MPFS many times. And this is in the  
boot
sequence and you cannot change that as some applications are started at  
boot
time. I discussed this with Jason and he agrees to this.

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: sfaibish <[email-masked]>
Date: Thu, 21 Oct 2010 15:16:30 -0400

> You mean before sending GETDEVICELIST and GETDEVICEINFO?
In order to issue these you need to have a connection to the MDS and it
is only established at mount time and it cannot be used during the boot.

> The mount time thing is linux client specific actually.
Nope. It is for any OS (Solaris as an example).

> The client can issue GETDEVICELIST and GETDEVICEINFO at any time.
To whom? How does he know the MDS before mount?

>    ///     opaque  bsc_contents<>;        /* contents of this component
The disk signature location is sent to the client in the GETDEVICEINFO
but it only includes signature identifying the device not anything else.
we need something to be specific just ID is not enough to protect. This is
the real problem I want to solve.

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]