Subject: [nfsv4] Inter server-side copy performance

From: "Mora, Jorge" <[email-masked]>
Date: Wed, 12 Apr 2017 14:47:30 +0000

Hello,

The following shows a comparison between inter server-side copy and a traditional copy.
Setup:
    Client: 16 CPUs, 32GB
    SRC server: 4 CPUs, 8GB
    DST server: 4 CPUs, 8GB
    All machines are running 4.10.0-ssc-02242017.1345,
      4.10.0 kernel on top RHEL 7.3 with Olga’s SSC patches 02242017.tgz (asynchronous SSC)
Test:
    The test runs 10 times for each copy size and the average is compared.
    The servers are mounted and un-mounted for each copy (using default mount options)
    Testing copy size: 1KB – 4GB, NOTE: the last copy is 4GB-1 bytes, the maximum copy_file_range() accepts.
Results:
    For a copy size below 16MB, traditional copy runs faster than server-side copy
    For a copy size of 32MB and above, server-side copy is at least 30% faster.
    For a copy size of 128MB and above, server-side copy is about 50% faster.
    For the 1GB and 2GB copy sizes, the performance improvement is only about 30-40%, investigating why this is happening.

*** Inter-SSC performance test
    TEST: Running test 'perf02'

>> Performance degradation:
    INFO: 00:08:50.035165 - perf02 copy with size 1KB
    INFO: 00:09:01.302004 - Server-side COPY: 0.104049992561 seconds
    INFO: 00:09:01.302206 - Traditional COPY: 0.0319610118866 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 1KB file: 225%
    INFO: 00:09:01.302378 - perf02 copy with size 2KB
    INFO: 00:09:12.332144 - Server-side COPY: 0.0923588514328 seconds
    INFO: 00:09:12.332335 - Traditional COPY: 0.033596777916 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 2KB file: 174%
    INFO: 00:09:12.332510 - perf02 copy with size 4KB
    INFO: 00:09:23.320714 - Server-side COPY: 0.100715565681 seconds
    INFO: 00:09:23.320915 - Traditional COPY: 0.0336240530014 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 4KB file: 199%
    INFO: 00:09:23.321086 - perf02 copy with size 8KB
    INFO: 00:09:34.542515 - Server-side COPY: 0.0881641149521 seconds
    INFO: 00:09:34.542742 - Traditional COPY: 0.0336608886719 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 8KB file: 161%
    INFO: 00:09:34.542924 - perf02 copy with size 16KB
    INFO: 00:09:45.439424 - Server-side COPY: 0.0939898014069 seconds
    INFO: 00:09:45.439630 - Traditional COPY: 0.0277063846588 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 16KB file: 239%
    INFO: 00:09:45.439795 - perf02 copy with size 32KB
    INFO: 00:09:56.103480 - Server-side COPY: 0.0798309087753 seconds
    INFO: 00:09:56.103691 - Traditional COPY: 0.0335722208023 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 32KB file: 137%
    INFO: 00:09:56.103848 - perf02 copy with size 64KB
    INFO: 00:10:07.217026 - Server-side COPY: 0.106491327286 seconds
    INFO: 00:10:07.217228 - Traditional COPY: 0.034387588501 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 64KB file: 209%
    INFO: 00:10:07.217400 - perf02 copy with size 128KB
    INFO: 00:10:18.022320 - Server-side COPY: 0.105627346039 seconds
    INFO: 00:10:18.022516 - Traditional COPY: 0.0511318922043 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 128KB file: 106%
    INFO: 00:10:18.022686 - perf02 copy with size 256KB
    INFO: 00:10:29.452353 - Server-side COPY: 0.124787020683 seconds
    INFO: 00:10:29.452538 - Traditional COPY: 0.0552408218384 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 256KB file: 125%
    INFO: 00:10:29.452713 - perf02 copy with size 512KB
    INFO: 00:10:41.357315 - Server-side COPY: 0.110602092743 seconds
    INFO: 00:10:41.357489 - Traditional COPY: 0.0643961668015 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 512KB file: 71%
    INFO: 00:10:41.357656 - perf02 copy with size 1MB
    INFO: 00:10:52.704159 - Server-side COPY: 0.118950200081 seconds
    INFO: 00:10:52.704341 - Traditional COPY: 0.0703304767609 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 1MB file: 69%
    INFO: 00:10:52.704499 - perf02 copy with size 2MB
    INFO: 00:11:05.367265 - Server-side COPY: 0.13633646965 seconds
    INFO: 00:11:05.367451 - Traditional COPY: 0.103643512726 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 2MB file: 31%
    INFO: 00:11:05.367614 - perf02 copy with size 4MB
    INFO: 00:11:18.872004 - Server-side COPY: 0.181165075302 seconds
    INFO: 00:11:18.872204 - Traditional COPY: 0.164586114883 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 4MB file: 10%
    INFO: 00:11:18.872368 - perf02 copy with size 8MB
    INFO: 00:11:35.093071 - Server-side COPY: 0.29020409584 seconds
    INFO: 00:11:35.093297 - Traditional COPY: 0.283315181732 seconds
    FAIL: SSC should outperform traditional copy, performance degradation for a 8MB file: 2%

>> Performance gain:
    INFO: 00:11:35.093446 - perf02 copy with size 16MB
    INFO: 00:11:54.779844 - Server-side COPY: 0.455569577217 seconds
    INFO: 00:11:54.780038 - Traditional COPY: 0.506252598763 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 16MB file: 11%
    INFO: 00:11:54.780185 - perf02 copy with size 32MB
    INFO: 00:12:22.415131 - Server-side COPY: 0.71369125843 seconds
    INFO: 00:12:22.415319 - Traditional COPY: 0.93473637104 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 32MB file: 30%
    INFO: 00:12:22.415466 - perf02 copy with size 64MB
    INFO: 00:13:03.498260 - Server-side COPY: 1.23845338821 seconds
    INFO: 00:13:03.498456 - Traditional COPY: 1.73098192215 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 64MB file: 39%
    INFO: 00:13:03.498637 - perf02 copy with size 128MB
    INFO: 00:14:11.502475 - Server-side COPY: 2.22639911175 seconds
    INFO: 00:14:11.502896 - Traditional COPY: 3.34778087139 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 128MB file: 50%
    INFO: 00:14:11.503102 - perf02 copy with size 256MB
    INFO: 00:16:12.323350 - Server-side COPY: 4.29401872158 seconds
    INFO: 00:16:12.323537 - Traditional COPY: 6.54622249603 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 256MB file: 52%
    INFO: 00:16:12.323703 - perf02 copy with size 512MB
    INFO: 00:19:58.311598 - Server-side COPY: 8.3841770649 seconds
    INFO: 00:19:58.311793 - Traditional COPY: 12.9200412273 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 512MB file: 54%
    INFO: 00:19:58.311935 - perf02 copy with size 1GB
    INFO: 00:26:01.275809 - Server-side COPY: 14.5967838049 seconds
    INFO: 00:26:01.276020 - Traditional COPY: 20.1957453728 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 1GB file: 38%
    INFO: 00:26:01.276175 - perf02 copy with size 2GB
    INFO: 00:35:29.990220 - Server-side COPY: 23.606795764 seconds
    INFO: 00:35:29.990432 - Traditional COPY: 31.4261539459 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 2GB file: 33%
    INFO: 00:35:29.990580 - perf02 copy with size 4GB
    INFO: 00:53:17.198827 - Server-side COPY: 41.2261408091 seconds
    INFO: 00:53:17.199033 - Traditional COPY: 62.9210350513 seconds
    PASS: SSC should outperform traditional copy, performance improvement for a 4GB file: 52%
    TIME: 44m27.164286s

Logfile: /home/mora/logs/nfstest_ssc_20170412000849.log

23 tests (9 passed, 14 failed)

Total time: 44m28.334177s

--Jorge

---

From: [email-masked] (J. Bruce Fields)
Date: Thu, 13 Apr 2017 13:45:15 -0400

> The following shows a comparison between inter server-side copy and a traditional copy.

Thanks for doing this!  Adding the linux-nfs list, I hope that's OK.

>     DST server: 4 CPUs, 8GB

Could you also tell us about the network?  How much bandwidth is
available between the two servers, and between the each server and the
client?  Disk bandwidth might be useful to know too.

>     Testing copy size: 1KB – 4GB, NOTE: the last copy is 4GB-1 bytes, the maximum copy_file_range() accepts.

If the client unmounts the servers, but if the servers keep their
exports mounted then the performance may depend on whether data is
already cached on the source server, since all these copies fit in RAM.

Are you timing just the copy_file_range() call, or do you include a
following sync?

>     For the 1GB and 2GB copy sizes, the performance improvement is only about 30-40%, investigating why this is happening.

Might also be interesting to look at performance when copying a larger
file with multiple copy_file_range() calls.

--b.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mora, Jorge" <[email-masked]>
Date: Fri, 14 Apr 2017 20:09:54 +0000

> client?  Disk bandwidth might be useful to know too.

Client:     pair    (1000baseT/Full)
SRC Server: ricoh   (1000baseT/Full)
DST Server: haddock (1000baseT/Full)

Transfer from source server to client:
[mora@pair test]$ scp ricoh:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file                       100% 8192MB 110.7MB/s   01:14

Transfer from destination server to client:
[mora@pair test]$ scp haddock:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file                       100% 8192MB 112.2MB/s   01:13

Transfer from source to destination server:
[mora@haddock ~]$ scp ricoh:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file                       100% 8192MB 110.7MB/s   01:14

Disk I/O on source server:
[mora@ricoh ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@ricoh ~]$ dd if=/home/exports/nfstest_ssc_source_file of=/dev/null bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 56.8953 s, 151 MB/s
[mora@ricoh ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@ricoh ~]$ dd if=/dev/zero of=/home/exports/test bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 46.9064 s, 183 MB/s

Disk I/O on destination server:
[mora@haddock ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@haddock ~]$ dd if=/home/exports/nfstest_ssc_source_file of=/dev/null bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 57.3356 s, 150 MB/s
[mora@haddock ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@haddock ~]$ dd if=/dev/zero of=/home/exports/test bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 47.3809 s, 181 MB/s

> already cached on the source server, since all these copies fit in RAM.

Good point, I will clear the VM caches using "echo 3 > /proc/sys/vm/drop_caches"
on both servers before each copy.

> Are you timing just the copy_file_range() call, or do you include a

I am timing right before calling copy_file_range() up to doing an fsync() and close() of the destination file.
For the traditional copy is the same, I am timing right before the first read on the source file up to the
fsync() and close() of the destination file.

> file with multiple copy_file_range() calls.

I will do this as well.

--Jorge

---

From: Olga Kornievskaia <[email-masked]>
Date: Fri, 14 Apr 2017 17:22:13 -0400

> fsync() and close() of the destination file.

Why should do we need a sync after copy_file_range(). kernel
copy_file_range() will send the commits for any unstable copies it
received.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 17 Apr 2017 09:36:04 -0400

> copy_file_range() will send the commits for any unstable copies it

Why does it do that?  As far as I can tell it's not required by
documentation for copy_file_range() or COPY.  COPY has a write verifier
and a stable_how argument in the reply.  Skipping the commits would
allow better performance in case a copy requires multiple COPY calls.

But, in any case, if copy_file_range() already committed then it
probably doesn't make a significant difference to the timing whether you
include a following sync and/or close.

--b.

---

From: Olga Kornievskaia <[email-masked]>
Date: Mon, 17 Apr 2017 11:30:06 -0400

> include a following sync and/or close.

Hm. It does make sense. Anna wrote the original code which included
the COMMIT after copy which I haven't thought about.

Anna, any comments?

---

From: Anna Schumaker <[email-masked]>
Date: Mon, 17 Apr 2017 11:57:41 -0400

(Resending because Google Inbox didn't switch to plain text)

> the COMMIT after copy which I haven't thought about.

I think the commit just seemed like a good idea at the time.  I'm okay with changing it if it doesn't make sense.

Anna

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 18 Apr 2017 13:28:39 -0400

On Mon, Apr 17, 2017 at 11:37 AM, Anna Schumaker
<[email-masked]> wrote:
>
>
> On Mon, Apr 17, 2017 at 11:30 AM Olga Kornievskaia <[email-masked]> wrote:
>>
>> >> > I am timing right before calling copy_file_range() up to doing an

Given how the code is written now it looks like it's not possible to
save up commits....

Here's what I can see happening:

nfs42_proc_clone() as well as nfs42_proc_copy() will call
nfs_sync_inode(dst) "to make sure server(s) have the latest data"
prior to initiating the clone/copy. So even if we just queue up (not
send) the commit after the executing nfs42_proc_copy, then next call
into vfs_copy_file_range() will send out that queued up commit.

Is it ok to relax the requirement that requirement, I'm not sure...

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 18 Apr 2017 14:33:33 -0400

> Is it ok to relax the requirement that requirement, I'm not sure...

Well, if the typical case of copy_file_range is just opening a file,
doing a single big copy_file_range(), then closing the file, then this
doesn't matter.

The linux server is currently limiting COPY to 4MB at a time, which will
make the commits more annoying.

Even there the typical case will probably still be an open, followed by
a series of non-overlapping copies, then close, and that shouldn't
require the commits.

--b.