Subject: Re: [dtn-users] DTN2 custody signal handling

From: Vangelis Anifantis <[email-masked]>
Date: Sat, 13 Dec 2014 02:02:05 +0200

Hello David,

Thanks for your response. It actually helps a lot, since it answers my
question regarding the custody signal handling from the administrative
element.

However, on the subject of the Custody Timer, I still have problems on
setting the custody timer default values using the param command (e.g.,
param set custody_timer_max 1000). It seems that I cannot manually
override the default values by using the command "param" (it has been
validated by the debug logs, the forwarding log of each bundle, as well
as in practice by measuring the retransmission delay).

Following your suggestion/implementation, I have changed the timeout
calculation in the CustodyTimer.cc so as to cover my own needs and it
works fine, but it would be convenient for me to be able to change
on-the-fly the min/max values via the param command. May i miss any
configuration setup (or external package) that prevent my dtn
installation (i have not used any special customization) from this kind
of functionality?

I would appreciate any help.

Regards,
Vangelis Anifantis

On 10/22/2014 12:04 AM, Zoller, David A. (MSFC-EO50)[HOSC SERVICES
CONTRACT] wrote:
> Hi Vangelis,
> Since the bundles are displaying as (NOT PENDING) the custody signal was actually processed by the administrative element (AdminRegistration.cc). 
> The initial bundles remain in the all_bundles list because the Custody Timer still has a reference to the bundle - when the timer expires the bundle will be deleted. The Custody Signal remains in the all_bundles and pending list because AdminRegistration does not update the fwdlog to indicate that the bundle was delivered.
>
> I have not noticed a functional issue after changing the custody timer default values using the param command but may not have paid close enough attention. It could just be a logging issue but I can't say for sure. 
>
> While on the subject of the Custody Timer, I will say that I do not like the implementation which sets the timeout to the min plus the percentage and then overrides that with the max if appropriate. We changed ours so that it starts with the percentage and then overrides with the min and max if needed.
>
> Hope this helps a bit,
> DZ
>
> -----Original Message-----
> From: dtn-users [mailto:[email-masked]] On Behalf Of Vangelis Anifantis
> Sent: Friday, October 17, 2014 9:16 AM
> To: [email-masked]
> Subject: [dtn-users] DTN2 custody signal handling
>
> hello,
>
> I have been recently playing around with DTN2 (dtn2.9.0 version) reference implementation to better understand the custody transfer capability (e.g., sending a bundle via dtnsend with activated the option -c).
>
> What i am seeing is that, as expected, each dtn node in the communication chain accepts the custody request, becomes the custodian and sends back a custody signal administrative record to indicate "custody_succeeded set to 1" (with indications regarding custody timer parameters as configured in the current custodian). However, I observe that the custody signal is not handled (neither received) by the administrative element, thus, remaining on the "all_bundles list" and "pending list"; and consequently, the bundle protocol agent does not follow the custody transfer success procedure by deleting the initial bundle (i.e., there is still a reference on the "all_bundles list" 
> regarding the initial bundle with indication (NOT PENDING)). Do i miss any configuration setup regarding the administrative element which is deemed necessary?
>
> Furthermore, even though I have set manually (custody_timer_min, custody_timer_lifetime_pct,  custody_timer_max) at all dtn nodes to differ from the default values, the forwarding log of the initial generated bundle keeps displaying the default values, e.g., e.g.,
>   forwarding log:
>          TRANSMITTED -> link_name [dtn:none] FORWARD at 1412929830.51299 [custody min 1800 pct 25 max 0] Why is this happening?
>
> Thanks in advance for any help.
>
> Regards,
> Vangelis Anifantis
>
> _______________________________________________
> dtn-users mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/dtn-users
>
-- 

OpenPGP public key ID: 
pub  4096R/A97BA940 <https://pgp.mit.edu/pks/lookup?op=get&search=0xDFB631A0A97BA940> 2014-11-15 Evangelos Anifantis <[email-masked]> <https://pgp.mit.edu/pks/lookup?op=vindex&fingerprint=on&search=0xDFB631A0A97BA940>
	 Fingerprint=9759 E043 8873 9FD3 D095  C717 DFB6 31A0 A97B A940

---

From: Vangelis Anifantis <[email-masked]>
Date: Sun, 14 Dec 2014 12:25:36 +0200

Just for the records, I have finally found out what was going on
regarding the param command.

For my testing scenarios I used a static routing scheme by which each
bundle was forwarded to a predetermined link. To create disruptions, I
used to shut down the dtnd daemon of the remote dtn node. In this case,
each generated bundle was directly forwarded to the predetermined link
(as specified in the route entry by the initial configuration file),
where the custody timers (i.e., min, lifetime_pct, max) had values equal
to the ones at the time of the specific route entry creation/definition
(for my case, the route entry was static and created by the initial
dtn.conf, namely, before any param command reconfiguration). Thus, I
always noticed the default custody timer values ([custody min 1800 pct
25 max 0]).

In case that I change the custody timer values (using param), and then I
re-create the route entry, e.g.,
route del dtn://node20.dtn/*
route add dtn://node20.dtn/* link_0
any new custodian bundle will be bound with the new custody timer values
(as set using the param command).

Besides, I have also noticed that a custody timer is not rescheduled.
Specifically, if the custody_timeout fires and the bundle is not
successfully delivered to the next custodian node, the custody_timeout
is not rescheduled so as to try a new retransmission of the bundle in
the future (does this make sense?).

Regards,
Vangelis Anifantis

On 12/13/2014 2:02 AM, Vangelis Anifantis wrote:
> setting the custody timer default values using the param command
> (e.g., param set custody_timer_max 1000). It seems that I cannot
> manually override the default values by using the command "param" (it
> has been validated by the debug logs, the forwarding log of each
> bundle, as well as in practice by measuring the retransmission delay).
>
> installation (i have not used any special customization) from this
> kind of functionality?
>

---

From: "Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Mon, 15 Dec 2014 14:50:06 +0000

Hello Vangelis,
Thanks for digging into and figuring out how the custody timer specs and route specs interrelate. It looks like it would be useful to be able to override custody spec params on the "route add" command and possibly to implement a "route reconfigure" command to change them on the fly.

A custody timer is started for the bundle as part of the BundleDaemon's processing of the BundleTransmitted event which the convergence layer issues. When a custody_timeout fires, the bundle should be re-routed and transmitted resulting in another BundleTransmitted event being issued resulting in starting a new custody timer. If there is not an open connection when the custody_timeout fires, then the bundle should either be queued as deferred on a known link(s) or there may not be a link to add it to at the time. If the known link eventually establishes its connection then the bundle should be transmitted which starts the next custody timer. If a new link is established then all pending bundles are checked to see if they can be routed through the new connection which when transmitted would start a new custody timer.

Basically, the custody timer is only started after it is known that the bundle has been transmitted. There is a special case in the comments in BundleDaemon::handle_bundle_transmitted() if configured to wait for reliable transmission that could result in returning without starting a new custody timer which should issue a warning "XXX/demmer fixme transmitted special case".

Could you be hitting the special case?
Or, can you provide more detail on the custody timer not being rescheduled scenario including which convergence layer?
Regards,
DZ

From: Vangelis Anifantis [mailto:[email-masked]]
Sent: Sunday, December 14, 2014 4:26 AM
To: Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]; [email-masked]
Subject: Re: [dtn-users] DTN2 custody signal handling

Just for the records, I have finally found out what was going on regarding the param command.

For my testing scenarios I used a static routing scheme by which each bundle was forwarded to a predetermined link. To create disruptions, I used to shut down the dtnd daemon of the remote dtn node. In this case, each generated bundle was directly forwarded to the predetermined link (as specified in the route entry by the initial configuration file), where the custody timers (i.e., min, lifetime_pct, max) had values equal to the ones at the time of the specific route entry creation/definition (for my case, the route entry was static and created by the initial dtn.conf, namely, before any param command reconfiguration). Thus, I always noticed the default custody timer values ([custody min 1800 pct 25 max 0]).

In case that I change the custody timer values (using param), and then I re-create the route entry, e.g.,
route del dtn://node20.dtn/*
route add dtn://node20.dtn/* link_0
any new custodian bundle will be bound with the new custody timer values (as set using the param command).

Besides, I have also noticed that a custody timer is not rescheduled. Specifically, if the custody_timeout fires and the bundle is not successfully delivered to the next custodian node, the custody_timeout is not rescheduled so as to try a new retransmission of the bundle in the future (does this make sense?).

Regards,
Vangelis Anifantis

Thanks for your response. It actually helps a lot, since it answers my question regarding the custody signal handling from the administrative element.

However, on the subject of the Custody Timer, I still have problems on setting the custody timer default values using the param command (e.g., param set custody_timer_max 1000). It seems that I cannot manually override the default values by using the command "param" (it has been validated by the debug logs, the forwarding log of each bundle, as well as in practice by measuring the retransmission delay).

Following your suggestion/implementation, I have changed the timeout calculation in the CustodyTimer.cc so as to cover my own needs and it works fine, but it would be convenient for me to be able to change on-the-fly the min/max values via the param command. May i miss any configuration setup (or external package) that prevent my dtn installation (i have not used any special customization) from this kind of functionality?

Regards,
Vangelis Anifantis

On 10/22/2014 12:04 AM, Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT] wrote:

Hi Vangelis,

Hope this helps a bit,

DZ

-----Original Message-----

To: [email-masked]<mailto:[email-masked]>

hello,

  forwarding log:

Regards,

Vangelis Anifantis

dtn-users mailing list

[email-masked]<mailto:[email-masked]>

--

OpenPGP public key ID:

pub  4096R/A97BA940<https://pgp.mit.edu/pks/lookup?op=get&search=0xDFB631A0A97BA940> 2014-11-15 Evangelos Anifantis <[email-masked]><https://pgp.mit.edu/pks/lookup?op=vindex&fingerprint=on&search=0xDFB631A0A97BA940>

---

From: Vangelis Anifantis <[email-masked]>
Date: Sat, 20 Dec 2014 21:14:52 +0200

Hello David,

Thanks again for your response.
The problem with the custody timer not being rescheduled happens with 
the UDPconvergenceLayer. To reproduce the problem with the custody timer 
rescheduling, consider a trivial scenario:
Let assume two dtn nodes (node A and node B) communicating using udp 
convergence layer and via a static routing. By using "link dump" I 
observe the corresponding link, e.g., my_link  ONDEMAND udp 
state=AVAILABLE. Now,  shut down the dtnd in node B and then send a 
bundle with custody transfer enabled from node A to node B. I observe 
that the custody_timer is started. When it fires, node A attempts to 
retransmit the bundle and now i have a debug message
[1419099989.900814 CustodyTimer 0xb4416a60 info] CustodyTimer::timeout
[1419099989.940186 UDPConvergenceLayer::Sender 0xb4404e24 error] 
send_bundle: error sending bundle (wrote -1/110): Connection refused
Then, the bundle remains in the "custody_bundles" list and it is added 
to the list "my_link:queue". Now on, it would be never been 
retransmitted again. Start the dtnd daemon at node B. The bundle remains 
as custodian in node A, but it would not be rescheduled for retransmission.

Sorry, I have not checked if i hit the special case that you mentioned 
in the previous mail. I will check it out and i ll come back soon.

p.s. By using the TCP convergence layer, the problem does not exist.

Regards,
Vangelis

On 12/15/2014 4:50 PM, Zoller, David A. (MSFC-EO50)[HOSC SERVICES 
CONTRACT] wrote:
>
> Hello Vangelis,
>
> Thanks for digging into and figuring out how the custody timer specs 
> and route specs interrelate. It looks like it would be useful to be 
> able to override custody spec params on the “route add” command and 
> possibly to implement a “route reconfigure” command to change them on 
> the fly.
>
> A custody timer is started for the bundle as part of the 
> BundleDaemon’s processing of the BundleTransmitted event which the 
> convergence layer issues. When a custody_timeout fires, the bundle 
> should be re-routed and transmitted resulting in another 
> BundleTransmitted event being issued resulting in starting a new 
> custody timer. If there is not an open connection when the 
> custody_timeout fires, then the bundle should either be queued as 
> deferred on a known link(s) or there may not be a link to add it to at 
> the time. If the known link eventually establishes its connection then 
> the bundle should be transmitted which starts the next custody timer. 
> If a new link is established then all pending bundles are checked to 
> see if they can be routed through the new connection which when 
> transmitted would start a new custody timer.
>
> Basically, the custody timer is only started after it is known that 
> the bundle has been transmitted. There is a special case in the 
> comments in BundleDaemon::handle_bundle_transmitted() if configured to 
> wait for reliable transmission that could result in returning without 
> starting a new custody timer which should issue a warning “XXX/demmer 
> fixme transmitted special case”.
>
> Or, can you provide more detail on the custody timer not being 
> rescheduled scenario including which convergence layer?
>
> Regards,
>
> DZ
>
> *From:*Vangelis Anifantis [mailto:[email-masked]]
> *Sent:* Sunday, December 14, 2014 4:26 AM
> *To:* Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]; 
> [email-masked]
> *Subject:* Re: [dtn-users] DTN2 custody signal handling
>
> used to shut down the dtnd daemon of the remote dtn node. In this 
> case, each generated bundle was directly forwarded to the 
> predetermined link (as specified in the route entry by the initial 
> configuration file), where the custody timers (i.e., min, 
> lifetime_pct, max) had values equal to the ones at the time of the 
> specific route entry creation/definition (for my case, the route entry 
> was static and created by the initial dtn.conf, namely, before any 
> param command reconfiguration). Thus, I always noticed the default 
> custody timer values ([custody min 1800 pct 25 max 0]).
>
>     Thanks for your response. It actually helps a lot, since it
>     answers my question regarding the custody signal handling from the
>     administrative element.
>
>     However, on the subject of the Custody Timer, I still have
>     problems on setting the custody timer default values using the
>     param command (e.g., param set custody_timer_max 1000). It seems
>     that I cannot manually override the default values by using the
>     command "param" (it has been validated by the debug logs, the
>     forwarding log of each bundle, as well as in practice by measuring
>     the retransmission delay).
>
>     Following your suggestion/implementation, I have changed the
>     timeout calculation in the CustodyTimer.cc so as to cover my own
>     needs and it works fine, but it would be convenient for me to be
>     able to change on-the-fly the min/max values via the param
>     command. May i miss any configuration setup (or external package)
>     that prevent my dtn installation (i have not used any special
>     customization) from this kind of functionality?
>
>         To:[email-masked]  <mailto:[email-masked]>
>
>         [email-masked]  <mailto:[email-masked]>
>
>     pub  4096R/A97BA940  <https://pgp.mit.edu/pks/lookup?op=get&search=0xDFB631A0A97BA940>  2014-11-15Evangelos Anifantis <[email-masked]>  <https://pgp.mit.edu/pks/lookup?op=vindex&fingerprint=on&search=0xDFB631A0A97BA940>
>

---

From: "Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Fri, 26 Dec 2014 16:07:22 +0000

Hi Vangelis,
Good catch! No need to check for the special case because you are not hitting it. I experimented with the UDP CL and reproduced the issue just as you outlined. After the connection refused error is generated, the state of the Link remains "OPEN" so there is no attempt to ever re-establish the connection. After bringing up node B again, if you send a new bundle then that bundle will be sent but the original one remains queued at the Link level instead of being sent.

I think that to continue cleanly along the lines of the current implementation would require at a minimum issuing an event to close the contact when an error is detected on the socket write.  However, after reading through the latest specification for the UDP CL (RFC 7122), I do not see any reason to use a connected UDP socket and my opinion is that the UDP CL should be changed to use an unconnected socket and it should always transmit bundles as they are queued. It is fairly easy to change to using an unconnected socket:

In UDPConvergenceLayer::Sender::init()

*         Insert these lines right after the line socket_.set_logfd(false):

socket_.init_socket();

socket_.set_remote_addr(addr);

socket_.set_remote_port(port);

*         Comment out or delete the socket_.connect() code block.

In UDPConvergenceLayer::Sender::send_bundle()

*         Change the socket_.write to use sendto:

int cc = socket_.sendto((char*)buf_, total_len, 0,

                        socket_.remote_addr(), socket_.remote_port());

That is my take on it.
Regards,
DZ

From: Vangelis Anifantis [mailto:[email-masked]]
Sent: Saturday, December 20, 2014 1:15 PM
To: Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]; [email-masked]
Subject: Re: [dtn-users] DTN2 custody signal handling

Hello David,

Thanks again for your response.
The problem with the custody timer not being rescheduled happens with the UDPconvergenceLayer. To reproduce the problem with the custody timer rescheduling, consider a trivial scenario:
Let assume two dtn nodes (node A and node B) communicating using udp convergence layer and via a static routing. By using "link dump" I observe the corresponding link, e.g., my_link  ONDEMAND udp state=AVAILABLE. Now,  shut down the dtnd in node B and then send a bundle with custody transfer enabled from node A to node B. I observe that the custody_timer is started. When it fires, node A attempts to retransmit the bundle and now i have a debug message
[1419099989.900814 CustodyTimer 0xb4416a60 info] CustodyTimer::timeout
[1419099989.940186 UDPConvergenceLayer::Sender 0xb4404e24 error] send_bundle: error sending bundle (wrote -1/110): Connection refused
Then, the bundle remains in the "custody_bundles" list and it is added to the list "my_link:queue". Now on, it would be never been retransmitted again. Start the dtnd daemon at node B. The bundle remains as custodian in node A, but it would not be rescheduled for retransmission.

Sorry, I have not checked if i hit the special case that you mentioned in the previous mail. I will check it out and i ll come back soon.

Regards,
Vangelis
On 12/15/2014 4:50 PM, Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT] wrote:
Hello Vangelis,
Thanks for digging into and figuring out how the custody timer specs and route specs interrelate. It looks like it would be useful to be able to override custody spec params on the "route add" command and possibly to implement a "route reconfigure" command to change them on the fly.

From: Vangelis Anifantis [mailto:[email-masked]]
Sent: Sunday, December 14, 2014 4:26 AM
To: Zoller, David A. (MSFC-EO50)[HOSC SERVICES CONTRACT]; [email-masked]<mailto:[email-masked]>
Subject: Re: [dtn-users] DTN2 custody signal handling

Regards,
Vangelis Anifantis

Regards,
Vangelis Anifantis

Hi Vangelis,

Hope this helps a bit,

DZ

-----Original Message-----

hello,

  forwarding log:

Regards,

Vangelis Anifantis

dtn-users mailing list

--

OpenPGP public key ID:

---

From: Vangelis Anifantis <[email-masked]>
Date: Fri, 26 Dec 2014 19:33:35 +0200

Thanks a lot David for your time on addressing all my inquiries.

Merry Christmas and a Happy New Year,
--Vangelis

On 12/26/2014 6:07 PM, Zoller, David A. (MSFC-EO50)[HOSC SERVICES
CONTRACT] wrote:
>
> Hi Vangelis,
>
> Good catch! No need to check for the special case because you are not
> hitting it. I experimented with the UDP CL and reproduced the issue
> just as you outlined. After the connection refused error is generated,
> the state of the Link remains “OPEN” so there is no attempt to ever
> re-establish the connection. After bringing up node B again, if you
> send a new bundle then that bundle will be sent but the original one
> remains queued at the Link level instead of being sent.
>
>  
>
> I think that to continue cleanly along the lines of the current
> implementation would require at a minimum issuing an event to close
> the contact when an error is detected on the socket write.  However,
> after reading through the latest specification for the UDP CL (RFC
> 7122), I do not see any reason to use a connected UDP socket and my
> opinion is that the UDP CL should be changed to use an unconnected
> socket and it should always transmit bundles as they are queued. It is
> fairly easy to change to using an unconnected socket:
>
>  
>
> ·         Insert these lines right after the line
> socket_.set_logfd(false):
>
> socket_.init_socket();
>
> ·         Comment out or delete the socket_.connect() code block.
>
>  
>
>  
>
> ·         Change the socket_.write to use sendto:
>
> *Sent:* Saturday, December 20, 2014 1:15 PM
> the UDPconvergenceLayer. To reproduce the problem with the custody
> timer rescheduling, consider a trivial scenario:
> retransmitted again. Start the dtnd daemon at node B. The bundle
> remains as custodian in node A, but it would not be rescheduled for
> retransmission.
>
>     Thanks for digging into and figuring out how the custody timer
>     specs and route specs interrelate. It looks like it would be
>     useful to be able to override custody spec params on the “route
>     add” command and possibly to implement a “route reconfigure”
>     command to change them on the fly.
>
>      
>
>     deferred on a known link(s) or there may not be a link to add it
>     to at the time. If the known link eventually establishes its
>     connection then the bundle should be transmitted which starts the
>     next custody timer. If a new link is established then all pending
>     bundles are checked to see if they can be routed through the new
>     connection which when transmitted would start a new custody timer.
>
>      
>
>     Basically, the custody timer is only started after it is known
>     that the bundle has been transmitted. There is a special case in
>     the comments in BundleDaemon::handle_bundle_transmitted() if
>     configured to wait for reliable transmission that could result in
>     returning without starting a new custody timer which should issue
>     a warning “XXX/demmer fixme transmitted special case”.
>
>      
>
>     For my testing scenarios I used a static routing scheme by which
>     each bundle was forwarded to a predetermined link. To create
>     disruptions, I used to shut down the dtnd daemon of the remote dtn
>     node. In this case, each generated bundle was directly forwarded
>     to the predetermined link (as specified in the route entry by the
>     initial configuration file), where the custody timers (i.e., min,
>     specific route entry creation/definition (for my case, the route
>     entry was static and created by the initial dtn.conf, namely,
>     before any param command reconfiguration). Thus, I always noticed
>     the default custody timer values ([custody min 1800 pct 25 max 0]).
>
>     Besides, I have also noticed that a custody timer is not
>     rescheduled. Specifically, if the custody_timeout fires and the
>     bundle is not successfully delivered to the next custodian node,
>     the custody_timeout is not rescheduled so as to try a new
>     retransmission of the bundle in the future (does this make sense?).
>
>     Regards,
>     Vangelis Anifantis
>
>
>         answers my question regarding the custody signal handling from
>         the administrative element.
>
>         param command (e.g., param set custody_timer_max 1000). It
>         seems that I cannot manually override the default values by
>         using the command "param" (it has been validated by the debug
>         logs, the forwarding log of each bundle, as well as in
>         practice by measuring the retransmission delay).
>
>         timeout calculation in the CustodyTimer.cc so as to cover my
>         own needs and it works fine, but it would be convenient for me
>         to be able to change on-the-fly the min/max values via the
>         param command. May i miss any configuration setup (or external
>         package) that prevent my dtn installation (i have not used any
>         special customization) from this kind of functionality?
>
>             To: [email-masked] <mailto:[email-masked]>
>