Subject: [dtn-users] dtncpd causes dtnd crash

From: Tom O'Reilly <[email-masked]>
Date: Wed, 9 May 2012 17:32:51 -0700

dtncpd frequently causes dtnd to crash on my RedHat machine. I'm  
running dtn version 2.8.0.

I start dtnd as follows, logging at debug level:

dtnd -c trex.dtn.conf -d -o dtnd.log -l debug

Then I start dtncpd:

dtncpd incoming/
opening connection to dtn router...
fatal error opening dtn handle: error in ipc communication

At this point dtnd has crashed, and dtnd.log contains the following:

[1336608944.240116 /log debug] adding log reparse signal handler
[1336608944.240997 /timer debug] adding signal handler 0x8328426 for  
signal 1
[1336608944.241245 /log debug] adding log rotate signal handler
[1336608944.241445 /timer debug] adding signal handler 0x832780c for  
signal 10
[1336608944.241670 /dtnd debug] ignoring SIGPIPE
[1336608944.241887 /dtnd notice] random seed is 241854
[1336608944.244618 /dtnd notice] DTN daemon starting up... (pid 8886)
[1336608944.294499 /dtn/tclcmd/debug debug] debug command registering
[1336608944.295102 /dtn/tclcmd/gettimeofday debug] gettimeofday  
command register
ing
[1336608944.295575 /dtn/tclcmd/help debug] help command registering
[1336608944.296358 /dtn/tclcmd/log debug] log command registering
[1336608944.299855 /thread debug] activating thread creation barrier
[1336608944.300293 /dtn/apiserver/notifier debug] created pipe, fds: 7 8
[1336608944.300530 /dtn/apiserver/notifier debug] set_nonblocking:  
nonblocking m
ode set
[1336608944.300752 /dtn/apiserver/notifier debug] set_nonblocking:  
nonblocking m
ode set
[1336608944.301106 /dtn/apiserver debug] APIServer init (evironment  
set addr 127
.0.0.1 port 5010)
[1336608944.301752 /dtn/tclcmd/api debug] api command registering
[1336608944.302159 /command/completion_notifier/notifier debug]  
created pipe, fd
s: 9 10
[1336608944.302399 /command/completion_notifier/notifier debug]  
set_nonblocking:
  nonblocking mode set
[1336608944.302621 /command/completion_notifier/notifier debug]  
set_nonblocking:
  nonblocking mode set
[1336608944.303340 /dtn/tclcmd/bundle debug] bundle command registering
[1336608944.303809 /dtn/tclcmd/interface debug] interface command  
registering
[1336608944.304499 /dtn/tclcmd/link debug] link command registering
[1336608944.306444 /dtn/tclcmd/param debug] param command registering
[1336608944.306930 /dtn/tclcmd/registration debug] registration  
command register
ing
[1336608944.309768 /dtn/tclcmd/route debug] route command registering
[1336608944.310304 /dtn/tclcmd/discovery debug] discovery command  
registering
[1336608944.311984 /dtn/tclcmd/prophet debug] prophet command  
registering
[1336608944.312356 /dtn/tclcmd/shutdown debug] shutdown command  
registering
[1336608944.312704 /dtn/tclcmd/quit debug] quit command registering
[1336608944.315738 /dtn/tclcmd/storage debug] storage command  
registering
[1336608944.316378 /dtn/tclcmd/block debug] block command registering
[1336608944.317253 /dtn/tclcmd/ecla debug] ecla command registering
[1336608944.317527 /dtnd debug] registered dtn commands
[1336608944.319542 /oasys/util/uri/ debug] URI::normalize: normalized  
URI dtn:no
ne
[1336608944.320341 /dtn/bundle/daemon/notifier debug] created pipe,  
fds: 11 12
[1336608944.320584 /dtn/bundle/daemon/notifier debug] set_nonblocking:  
nonblocking
[1336608944.320584 /dtn/bundle/daemon/notifier debug] set_nonblocking:  
nonblocki
ng mode set
[1336608944.320805 /dtn/bundle/daemon/notifier debug] set_nonblocking:  
nonblocki
ng mode set
[1336608944.321231 /dtnd debug] intialized dtn components
[1336608944.321468 /dtn/tclcmd/console debug] console command  
registering
[1336608944.322186 /dtn/tclcmd/test debug] test command registering
[1336608944.322585 /dtnd info] parsing configuration file  
trex.dtn.conf...
[1336608944.322816 /dtn/tclcmd debug] executing command file  
trex.dtn.conf
[1336608944.334233 /oasys/util/uri/ debug] URI::normalize: normalized  
URI dtn://
trex.dtn
[1336608944.335274 /dtn/interface/table info] adding interface tcp0  
(tcp)
[1336608944.335619 /dtn/cl/tcp debug] adding interface tcp0
[1336608944.336048 /dtn/cl/tcp/listener/notifier debug] created pipe,  
fds: 13 14
[1336608944.336287 /dtn/cl/tcp/listener/notifier debug]  
set_nonblocking: nonbloc
king mode set
[1336608944.336512 /dtn/cl/tcp/listener/notifier debug]  
set_nonblocking: nonbloc
king mode set
[1336608944.336936 /dtn/cl/tcp/iface/tcp0 debug] created socket 15
[1336608944.337180 /dtn/cl/tcp/iface/tcp0 debug] IPSocket::configure
[1336608944.337381 /dtn/cl/tcp/iface/tcp0 debug]      
params.reuseaddr_: 1
[1336608944.337586 /dtn/cl/tcp/iface/tcp0 debug]      
params.reuseport_: 0
[1336608944.337789 /dtn/cl/tcp/iface/tcp0 debug]      
params.broadcast_: 0
[1336608944.337993 /dtn/cl/tcp/iface/tcp0 debug]      
params.multicast_: 0
[1336608944.338339 /dtn/cl/tcp/iface/tcp0 debug]      
local_addr_      : 0
[1336608944.338552 /dtn/cl/tcp/iface/tcp0 debug]      
remote_addr_     : -1
[1336608944.338755 /dtn/cl/tcp/iface/tcp0 debug]      
local_port_      : 0
[1336608944.338958 /dtn/cl/tcp/iface/tcp0 debug]      
remote_port_     : 0
[1336608944.339162 /dtn/cl/tcp/iface/tcp0 debug] setting SO_REUSEADDR
[1336608944.339499 /dtn/cl/tcp/iface/tcp0 debug] binding to 0.0.0.0:4556
[1336608944.339897 /dtn/cl/tcp/iface/tcp0 debug] listening
[1336608944.340407 /dtn/cl/tcp/iface/tcp0 debug] state INIT -> LISTENING
[1336608944.340629 /thread debug] delaying start of thread 0x9ddf0ec  
[TCPConverg
enceLayer::Listener] due to barrier
[1336608944.341349 /dtn/link/manta_link debug] set_state UNAVAILABLE - 
 > UNAVAILA
BLE
[1336608944.341595 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 0 -> 1 add Link::create_link:  
return value
[1336608944.342239 /dtn/cl/tcp debug] adding ALWAYSON link 10.250.32.202
[1336608944.342957 /dtn/link/manta_link info] new link manta_link  
[10.250.32.202
  dtn:none ALWAYSON tcp state=UNAVAILABLE]
[1336608944.343210 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 1 -> 2 add !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.343545 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 1 del Link::create_link:  
return value
[1336608944.343888 /dtn/contact/manager debug] adding NEW link  
manta_link
[1336608944.344121 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 1 -> 2 add ContactManager
[1336608944.344378 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 3 add ContactManager
[1336608944.344629 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 2 del ContactManager
[1336608944.344876 /dtn/contact/manager debug] posting LinkCreatedEvent
[1336608944.345087 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 3 add LinkCreatedEvent
[1336608944.345363 /dtn/bundle/daemon debug] posting event (0x9ddf948)  
with type
  LINK_CREATED (at back)
[1336608944.345646 /dtn/bundle/daemon/msgqueue debug] notifier notify
[1336608944.345925 /dtn/bundle/daemon/msgqueue debug] notify count = 1
[1336608944.346189 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 2 del !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.346773 /oasys/util/uri/ debug] URI::normalize: normalized  
URI dtn://
manta.dtn/*
[1336608944.347086 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 3 add  
ContactManager::find_link: ret
urn value
[1336608944.347415 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dtn:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 4 add !! 
DEFAULT REF CONSTRUCTOR!!
[1336608944.347659 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 3 del  
ContactManager::find_link: ret
urn value
[1336608944.348044 /oasys/util/uri/ debug] URI::normalize: normalized  
URI *:*
[1336608944.348364 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 4 add RouteEntry
[1336608944.348656 /dtn/bundle/daemon debug] posting event (0x9dd8ac8)  
with type
  ROUTE_ADD (at back)
[1336608944.348907 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 3 del !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.349462 /oasys/util/uri/ debug] URI::normalize: normalized  
URI dtn://
vesuvio.dtn/*
[1336608944.349775 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 4 add  
ContactManager::find_link: ret
urn value
[1336608944.350053 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.350438 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 4 del  
ContactManager::find_link: ret
urn value
[1336608944.350728 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add RouteEntry
[1336608944.350998 /dtn/bundle/daemon debug] posting event (0x9ddf728)  
with type ROUTE_ADD (at back)
[1336608944.351247 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 4 del !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.351843 /oasys/util/uri/ debug] URI::normalize: normalized  
URI dtn://
shore.dtn/*
[1336608944.352154 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add  
ContactManager::find_link: ret
urn value
[1336608944.352433 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 6 add !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.352669 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 6 -> 5 del  
ContactManager::find_link: ret
urn value
[1336608944.352944 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 6 add RouteEntry
[1336608944.353207 /dtn/bundle/daemon debug] posting event (0x9dd8c08)  
with type
  ROUTE_ADD (at back)
[1336608944.353456 /dtn/link/refs_b debug] refcount manta_link  
[10.250.32.202 dt
n:none ALWAYSON tcp state=UNAVAILABLE] 6 -> 5 del !!DEFAULT REF  
CONSTRUCTOR!!
[1336608944.353798 /dtnd debug] Initializing datastore.
[1336608944.354065 /dtn/storage debug] DurableStore instantiated  
(0x9ddf808)
[1336608944.356052 /dtn/storage info] initializing db name=DTN (not  
shared), dir
=/home/dorado1/mydtn/var/trex/db
[1336608944.374653 /dtn/storage/deadlock_timer debug] rescheduling in  
5000 msecs
[1336608944.374926 /timer debug] scheduling timer 0x9e3fdf8 in 5000 ms  
at 133660
8949:374875
[1336608944.375531 /dtn/storage info] datastore /home/dorado1/mydtn/ 
var/trex/db
was not cleanly shut down
[1336608944.375839 /dtnd debug] directory validated: /home/dorado1/ 
mydtn/var/tre
x/bundles
[1336608944.376172 /dtnd debug] directory access validated: /home/ 
dorado1/mydtn/
var/trex/bundles
[1336608944.377762 /dtn/storage debug] get_table -- notfound database  
globals
[1336608944.378195 /dtn/storage/globals error] error initializing  
global store:
table not found
[1336608944.378421 /dtnd critical] error initializing data store
[1336608944.378641 /dtnd error] error initializing data store,  
exiting...
[1336608944.379403 /log debug] shutting down
[1336608944.385404 /log debug] shutting down

And the trex.dtn.conf configuration file:

set dtn_node "trex"
set dtn_dir $env(HOME)/mydtn

set manta_tcp_adr  "10.250.32.202"
set vesuvio_tcp_adr "10.250.32.206"

###################################################
# DTN configuration.
###################################################
set dtn_eid "dtn://$dtn_node.dtn"
set storage_dir "$dtn_dir/var/$dtn_node"
file mkdir $storage_dir

storage set dbdir "$storage_dir/db"
storage set payloaddir "$storage_dir/bundles"
file mkdir $storage_dir/db
file mkdir $storage_dir/bundles

storage set dbname DTN
route set type static
route local_eid $dtn_eid
console set port 5050
console set addr 127.0.0.1

# Interfaces
interface add tcp0 tcp

# Link and route to Manta
link add manta_link $manta_tcp_adr ALWAYSON tcp keepalive_interval=1  
min_retry_i
nterval=1 max_retry_interval=4
route add dtn://manta.dtn/* manta_link

# Link and route to vesuvio
###link add vesuvio_link $vesuvio_tcp_adr ALWAYSON tcp  
keepalive_interval=1 min_
retry_interval=1 max_retry_interval=4
### route add dtn://vesuvio.dtn/* vesuvio_link

# Route to vesuvio through manta
route add dtn://vesuvio.dtn/* manta_link

# Route to "shore" through manta
route add dtn://shore.dtn/* manta_link

Has anyone seen similar behavior?
I also observe the problem appears to fix itself if I first run dtnd  
with the --init-db flag:

dtnd -c trex.dtn.conf --init-db

Initializing the database each time I run dtnd seems like a rather  
draconian solution though...

Thanks,
Tom

------------------------------------------------
Thomas C. O'Reilly
Monterey Bay Aquarium Research Institute
7700 Sandholdt Road
Moss Landing, CA 95039
831-775-1766
[email-masked]
http://www.mbari.org
"The machine does not isolate man from the great mysteries of nature,
but plunges him more deeply into them."

     - ANTOINE DE SAINT-EXUPERY
       "Wind, Sand, and Stars" (1939)

---

From: Alex McMahon <[email-masked]>
Date: Thu, 10 May 2012 10:57:53 +0100

Hi Tom,

dtncpd is not causing dtnd to crash in this instance - the dtnd has not
started in the first place due an error initializing the global store
which appears to be the result of some corruption as the database was
not cleanly shut down.

The error 'in ipc communication' in this case occurs as there is no dtnd
daemon running. The problem appears to fix itself when you run dtnd with
the --init-db flag as this initializes the database

All the best,

Alex 

On Wed, 2012-05-09 at 17:32 -0700, Tom O'Reilly wrote:
> dtncpd incoming/ opening connection to dtn router... fatal error
> opening dtn handle: error in ipc communication 
> 
> signal 1    [1336608944.241245 /log debug] adding log rotate signal
> handler                  [1336608944.241445 /timer debug] adding
> signal handler 0x832780c for signal 10   [1336608944.241670 /dtnd
> debug] ignoring SIGPIPE
> command register ing
> [1336608944.300293 /dtn/apiserver/notifier debug] created pipe, fds: 7
> 8         [1336608944.300530 /dtn/apiserver/notifier debug]
> set_nonblocking: nonblocking m ode set
> nonblocking m ode set
> set addr 127 .0.0.1 port 5010)
> created pipe, fd s: 9 10
> set_nonblocking:  nonblocking mode set
> set_nonblocking:  nonblocking mode set
> [1336608944.303340 /dtn/tclcmd/bundle debug] bundle command
> registering          [1336608944.303809 /dtn/tclcmd/interface debug]
> interface command registering    [1336608944.304499 /dtn/tclcmd/link
> debug] link command registering
> command register ing
> registering    [1336608944.311984 /dtn/tclcmd/prophet debug] prophet
> command registering        [1336608944.312356 /dtn/tclcmd/shutdown
> debug] shutdown command registering
> registering        [1336608944.316378 /dtn/tclcmd/block debug] block
> command registering            [1336608944.317253 /dtn/tclcmd/ecla
> debug] ecla command registering              [1336608944.317527 /dtnd
> debug] registered dtn commands
> URI dtn:no ne
> fds: 11 12   [1336608944.320584 /dtn/bundle/daemon/notifier debug]
> set_nonblocking: nonblocking
> nonblocki ng mode set
> nonblocki ng mode set
> registering        [1336608944.322186 /dtn/tclcmd/test debug] test
> command registering              [1336608944.322585 /dtnd info]
> parsing configuration file trex.dtn.conf...
> trex.dtn.conf       [1336608944.334233 /oasys/util/uri/ debug]
> URI::normalize: normalized URI dtn:// trex.dtn
> (tcp)        [1336608944.335619 /dtn/cl/tcp debug] adding interface
> tcp0
> fds: 13 14 [1336608944.336287 /dtn/cl/tcp/listener/notifier debug]
> set_nonblocking: nonbloc king mode set
> set_nonblocking: nonbloc king mode set
> params.reuseaddr_: 1        [1336608944.337586 /dtn/cl/tcp/iface/tcp0
> debug]     params.reuseport_: 0
> params.broadcast_: 0        [1336608944.337993 /dtn/cl/tcp/iface/tcp0
> debug]     params.multicast_: 0
> local_addr_      : 0        [1336608944.338552 /dtn/cl/tcp/iface/tcp0
> debug]     remote_addr_     : -1
> local_port_      : 0        [1336608944.338958 /dtn/cl/tcp/iface/tcp0
> debug]     remote_port_     : 0
> [1336608944.339499 /dtn/cl/tcp/iface/tcp0 debug] binding to
> 0.0.0.0:4556         [1336608944.339897 /dtn/cl/tcp/iface/tcp0 debug]
> listening
> [1336608944.340407 /dtn/cl/tcp/iface/tcp0 debug] state INIT ->
> LISTENING         [1336608944.340629 /thread debug] delaying start of
> thread 0x9ddf0ec [TCPConverg enceLayer::Listener] due to barrier
> [1336608944.341349 /dtn/link/manta_link debug] set_state UNAVAILABLE
> -> UNAVAILA BLE
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 0 -> 1 add
> Link::create_link: return value
> [1336608944.342239 /dtn/cl/tcp debug] adding ALWAYSON link
> 10.250.32.202         [1336608944.342957 /dtn/link/manta_link info]
> new link manta_link [10.250.32.202  dtn:none ALWAYSON tcp
> state=UNAVAILABLE]
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 1 -> 2 add !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.343545 /dtn/link/refs_b
> debug] refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 2 -> 1 del Link::create_link: return value
> manta_link        [1336608944.344121 /dtn/link/refs_b debug] refcount
> manta_link [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 1
> -> 2 add ContactManager
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 3 add
> ContactManager                 [1336608944.344629 /dtn/link/refs_b
> debug] refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 3 -> 2 del ContactManager
> [1336608944.344876 /dtn/contact/manager debug] posting
> LinkCreatedEvent          [1336608944.345087 /dtn/link/refs_b debug]
> refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 2 -> 3 add LinkCreatedEvent
> with type  LINK_CREATED (at back)
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 2 del !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.346773 /oasys/util/uri/
> debug] URI::normalize: normalized URI dtn:// manta.dtn/*
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 2 -> 3 add
> ContactManager::find_link: ret urn value
> DEFAULT REF CONSTRUCTOR!!    [1336608944.347659 /dtn/link/refs_b
> debug] refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 4 -> 3 del ContactManager::find_link: ret urn value
> URI *:*    [1336608944.348364 /dtn/link/refs_b debug] refcount
> manta_link [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 3
> -> 4 add RouteEntry
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 3 del !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.349462 /oasys/util/uri/
> debug] URI::normalize: normalized URI dtn:// vesuvio.dtn/*
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 3 -> 4 add
> ContactManager::find_link: ret urn value
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.350438 /dtn/link/refs_b
> debug] refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 5 -> 4 del ContactManager::find_link: ret urn value
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add
> RouteEntry                     [1336608944.350998 /dtn/bundle/daemon
> debug] posting event (0x9ddf728) with type ROUTE_ADD (at back)
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 4 del !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.351843 /oasys/util/uri/
> debug] URI::normalize: normalized URI dtn:// shore.dtn/*
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 4 -> 5 add
> ContactManager::find_link: ret urn value
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 6 add !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.352669 /dtn/link/refs_b
> debug] refcount manta_link [10.250.32.202 dt n:none ALWAYSON tcp
> state=UNAVAILABLE] 6 -> 5 del ContactManager::find_link: ret urn value
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 5 -> 6 add
> RouteEntry                     [1336608944.353207 /dtn/bundle/daemon
> debug] posting event (0x9dd8c08) with type  ROUTE_ADD (at back)
> [10.250.32.202 dt n:none ALWAYSON tcp state=UNAVAILABLE] 6 -> 5 del !!
> DEFAULT REF CONSTRUCTOR!!    [1336608944.353798 /dtnd debug]
> Initializing datastore.
> (0x9ddf808)     [1336608944.356052 /dtn/storage info] initializing db
> name=DTN (not shared), dir =/home/dorado1/mydtn/var/trex/db
> 5000 msecs [1336608944.374926 /timer debug] scheduling timer 0x9e3fdf8
> in 5000 ms at 133660 8949:374875
> [1336608944.375531 /dtn/storage info]
> datastore /home/dorado1/mydtn/var/trex/db  was not cleanly shut down
> [1336608944.375839 /dtnd debug] directory
> validated: /home/dorado1/mydtn/var/tre x/bundles
> [1336608944.376172 /dtnd debug] directory access
> validated: /home/dorado1/mydtn/ var/trex/bundles
> globals    [1336608944.378195 /dtn/storage/globals error] error
> initializing global store:  table not found
> exiting...        [1336608944.379403 /log debug] shutting down

---

From: Tom O'Reilly <[email-masked]>
Date: Thu, 10 May 2012 08:47:03 -0700

Thanks for that info Alex.

In general, when is it "necessary" to reinitialize the database with  
the --init-db flag?
I assume when the network topology changes... how about other  
conditions? I guess it may be safe to initialize things every time  
dtnd starts, but that would wipe out any cached bundles and kind of  
defeat the purpose...

Thanks
Tom

---

From: Elwyn Davies <[email-masked]>
Date: Thu, 10 May 2012 17:52:40 +0100

Hi, Tom,

On Thu, 2012-05-10 at 08:44 -0700, Tom O'Reilly wrote:
> Hi Elwyn,
> 
> 
> 
> On May 10, 2012, at 2:07 AM, Elwyn Davies wrote:
> 
> > 
> > I think you may have misunderstood how to get the DTN2 system going
> > on a
> > node.
> > 
> > The 'problem goes away' when you run dtnd with --init-db because you
> > have to run this once to get the database initialized.  After that
> > you
> > should be able to run the daemon as many times as you like in
> > 'production mode'. Thus you would run:
> > dtnd -c trex.dtn.conf -o dtnd.log -l debug --init-db
> > [This runs and terminates having set up the database - check the log
> > says this happened.]
> > and then:
> > 
> > [which goes off into the background]
> > Check it is running by telneting in to port 5050 on the localhost
> > (or
> > run it undaemonized until you are happy it is working).
> > 
> 
> 
> I have run with --init-db on all of my nodes at least once - probably
> several days before the problem recurs. 
> Before yesterday's problem occurred, I had run dtncpd several times
> before dtnd began its crashing behavior.
> Under what conditions is it necessary to re-initialize dtnd again?
> 
Hmm.
The log file indicated that the globals table was missing when you
started dtnd.  That indicates that the database was either deleted or
corrupted after the previous run.  Also the .ds_clean file that is
created when the database is cleanly shut down at the end of the
previous run was missing.  This indicates that something went wrong at
the end of the previous run or you changed where the database was
stored.

You normally only need to use --init-db at cold start, or after moving
the database to a new directory (and not moving the files), or after
changing the storage mechanism, although it is sometimes necessary
during development when things go badly wrong.  It might also be
necessary if, e.g., you manually deleted some bundle files.

Usually using --tidy is enough to get things going again - this
reinitializes the tables and gets rid of existing data.

v2.8.0 has been somewhat superseded but I haven't seen this problem
despite using DTN2 a great deal over the last 6 months, and leaving
instances running for days at a time.

> > 
> > Check it with dtnping and then use dtncpd.
> > 
> > BTW Quite a lot of the directory creation stuff that you have added
> > in
> > to dtn.conf is unnecessary - --init-db automatically creates the db
> > and
> > bundles directories provided the base dbdir is present and
> > writeable.
> > Its also wise to be explicit about the storage type you are using -
> > you
> > are using the default berkeleydb I believe.
> > 
> > If you haven't done so, (and maybe even so) have a read of the
> > tutorial
> > in the DTN2/doc/manual directory which describes getting started.
> > 
> 
> 
> 
> 
> Thanks for your help Elwyn - I have gone through the tutorial in the
> past; it's okay as a starter, but do you know of a more extensive
> tutorial?

A comment we have heard before (and quite recently).  No, there isn't.
One of us will have to settle down and do some more.

Regards,
Elwyn

> > Regards,
> > Elwyn
> > > 8        
> > > [1336608944.338339 /dtn/cl/tcp/iface/tcp0 debug]     local_addr_
> > > : 0       
> > > [1336608944.338552 /dtn/cl/tcp/iface/tcp0 debug]     remote_addr_
> > >  : -1      
> > > [1336608944.338755 /dtn/cl/tcp/iface/tcp0 debug]     local_port_
> > > : 0       
> > > [1336608944.338958 /dtn/cl/tcp/iface/tcp0 debug]     remote_port_
> > >  : 0       
> > > 0.0.0.0:4556        
> > > LISTENING        
> > > -> UNAVAILA
> > > 10.250.32.202        
> > > LinkCreatedEvent         
> > > [1336608944.345925 /dtn/bundle/daemon/msgqueue debug] notify count
> > > =
> > > 1          
> > > _______________________________________________
> > > dtn-users mailing list
> > > [email-masked]
> > > https://www.irtf.org/mailman/listinfo/dtn-users
> > 
> > 
>

---

From: Tom O'Reilly <[email-masked]>
Date: Fri, 11 May 2012 14:17:53 -0700

Hi Elwyn,

> the end of the previous run or you changed where the database was

What is the clean way to shut down dtnd?
When it's running in the foreground, I use ^C.
When it's running as a daemon, I usually issue 'kill' with the default  
TERM signal.

>
> You normally only need to use --init-db at cold start,

What do you mean by "cold start"?
After rebooting the host computer, do I need to use --init-db?

Thanks,
Tom

> or after moving