Subject: Re: [ippm] [spring] Monitoring metric to detect and locate congestion

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 1 Mar 2020 21:25:01 +0000

Hi Rudiger,

I’m looking at your proposal again after several months.

Thanks for preparing the slides, which help.
I have a question about one of the inferred results
on slide 2, where it says:

Simultaneous packet loss or a change in delay of measurement
paths 1, 2 and 3 indicate loss of connectivity between LSR a and LER i

What are the details leading-up to the conclusion above?
I might be more inclined to agree if the conditions were
“Simultaneous packet loss AND a change in delay of measurement”
possible because the data plane has been re-routed.

Just a thought, more to consider here,
Al

From: ippm [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, February 27, 2020 5:42 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [ippm] [spring] Monitoring metric to detect and locate congestion

Hi Robert,

thanks for your support and comments. I’d be glad if other interested operator representatives indicate their interest on the list...

Regards, Ruediger

Von: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Gesendet: Donnerstag, 27. Februar 2020 10:38
An: Geib, Rüdiger <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; SPRING WG <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Betreff: Re: [spring] Monitoring metric to detect and locate congestion

Hi,

As I mentioned in my first mail I am a big supporter of end to end path measurement - mainly have been focusing on hop by hop timestamping approach.

So my comments were not really to discourage in any way end to end path probing - it is extremely useful. They were just to see your point of view on alternative options just for the specific goal here - congestion detection.

Will be watching how your proposal evolves with interest !

Cheers,
R.

On Thu, Feb 27, 2020 at 9:17 AM <[email-masked]<mailto:[email-masked]>> wrote:
Hi Robert,

regarding scalability, I hope the difference between our positions is just whether it’s a router or a dedicated CPE. I don’t promote deploying 20k PCs (I hope to promote a metric to replace them). I prefer the dedicated CPE, but routers do as well.

The telemetry threshold might be an option too, if congestion is to be detected. In September last year, we had to replace a line card at a production router, whose ingress port hardware was corrupted. There were no drops, just random delay variations, which showed in our performance measurement system. I wonder whether problems like that can be detected by evaluating telemetry.
Further, telemetry needs to work reliable if the router hardware is busy dealing with heavy loads (i.e., telemetry must be sufficiently privileged). External measurements on forwarding layer don’t rely on router internal processing resources.

Regards,

Ruediger

Von: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Gesendet: Mittwoch, 26. Februar 2020 14:19
An: Geib, Rüdiger <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; SPRING WG <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Betreff: Re: [spring] Monitoring metric to detect and locate congestion

Hi,

Two clarifications:

[RG1] the measurements pass the routers on forwarding plane.

Well if I have 20K CEs and would like to measure this end to end that means it better run on a router ... I can not envision installing 20K new PCs just for this. At min such end point it should run on a well designed CPE as an LXC.

[RG1] Up to now, the conditions in Deutsche Telekom’s backbone network require a careful design of router probing interval and counters to be read.

Sure. I actually had in mind telemetry push model where you only get notified to yr monitoring station when applied queue threshold is crossed. Very little mgmt traffic in the network limited to only information which is important.

I know some vendors resist to locally (at LC or local RE) apply filtering to telemetry streaming but this is IMHO just sick approach.

Thx,
R.

On Wed, Feb 26, 2020 at 12:01 PM <[email-masked]<mailto:[email-masked]>> wrote:
Hi Robert,

Thanks, my replies in line marked [RG1]

I have read your draft and presentation with interest as I am a big supporter and in some lab trials  of end to end network path probing.

Few comments, observations, questions:

You are essentially measuring and comparing delay across N paths traversing known network topology (I love "network tomography" name !)

[RG1] it’s telemetry with a constrained set up, but the term doesn’t appear in the draft yet…that can be changed.

------
* First question - this will likely run on RE/RP and in some platforms path between LC and RE/RP is completely deterministic and can take 10s or 100s of ms locally in the router. So right here the proposal to compare anything may not really work - unless the spec mandates that actually timestamping is done in hardware on the receiving LC. Then CPU can process it when it has cycles.

[RG1] the measurements pass the routers on forwarding plane. High-end routers add variable processing latencies. It’s on a level of double or lower triple digit [us] on Deutsche Telekom backbone routers. If a dedicated sender receiver system is used, timestamping may be optimized for the purpose.
------

* Second question is that congestion usually has a very transient character ... You would need to be super lucky to find any congestion in normal network using test probes of any sort. If you have interfaces always congested then just the queue depth time delta may not be visible in end to end measurements.

[RG1] The probing frequency depends on the characteristics of congestion the operator wants to be aware of. Unplanned events may cause changes in measurement delays lasting for minutes or longer (congestion or hardware issues). The duration of a reliably detectable “event” correspond to the measurement packet distance (I don’t intend to replace hello-exchanges or BFD by the metric).
------

* Third - why not simply look at the queue counters at each node ? Queue depth, queue history, min, avg, max on a per interface basis offer tons of information readily available. Why would anyone need to inject loops of probe packets in known network to detect this ? And in black box unknown networks this is not going to work as you would not know the network topology in the first place. Likewise link down/up is already reflected in your syslog via BFD and IGP alarms. I really do not think you need end to end protocol to tell you that.

[RG1] Up to now, the conditions in Deutsche Telekom’s backbone network require a careful design of router probing interval and counters to be read. The proposed metric allows to capture persistent issues impacting forwarding. It points out, where these likely occur. An operator may then have a closer look at an interface/router to analyse what’s going on, using the full arsenal of accessible information and tools. As unusual events happen rarely, it may still be a fair question for which purpose linecard- and central processing cycles of routers are consumed.
-------
+ Thanks for catching the nit below..

Regards, Ruediger

s/nodes L100 and L200 one one/nodes L100 and L200 on one/

:)

Many thx,
R.

On Wed, Feb 26, 2020 at 8:55 AM <[email-masked]<mailto:[email-masked]>> wrote:
Dear IPPM (and SPRING) participants,

I’m solliciting interest in a new network monitoring metric which allows to detect and locate congested interfaces. Important properties are

  *   Same scalability as ICMP ping in the sense one measurement relation required per monitored connection
  *   Adds detection and location of congested interfaces as compared to ICMP ping (otherwise measured metrics are compatible with ICMP ping)
  *   Requires Segment Routing (which means, measurement on forwarding layer, no other interaction with passed routers – in opposite to ICMP ping)
  *   Active measurement (may be deployed using a single sender&receiver or separate sender and receiver, Segment Routing allows for both options)

I’d be happy to present the draft in Vancouver.. If there’s community interest. Please read and comment.

You’ll find slides at

https://datatracker.ietf.org/meeting/105/materials/slides-105-ippm-14-draft-geib-ippm-connectivity-monitoring-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_meeting_105_materials_slides-2D105-2Dippm-2D14-2Ddraft-2Dgeib-2Dippm-2Dconnectivity-2Dmonitoring-2D00&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=c38PpFTaJ0AWTj9q3yN4NtYUD_ieugCWMnRqjd3ZmPQ&s=3zzlAlab399Bw3a6xX8yWYKNiIgul-FxWiQX3l0n758&e=>

Draft url:

https://datatracker.ietf.org/doc/draft-geib-ippm-connectivity-monitoring/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dgeib-2Dippm-2Dconnectivity-2Dmonitoring_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=c38PpFTaJ0AWTj9q3yN4NtYUD_ieugCWMnRqjd3ZmPQ&s=Zu1SLrrHSC0XeySk7ZSZxZ5CnEjdnlo1TdDq8R07ZJE&e=>

Regards,

Ruediger
_______________________________________________
spring mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spring<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_spring&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=c38PpFTaJ0AWTj9q3yN4NtYUD_ieugCWMnRqjd3ZmPQ&s=J2AhOl6DRqfg3lk90anUJEvOQygGdIMAMyfJwdZtrVU&e=>

---

From: <[email-masked]>
Date: Mon, 2 Mar 2020 08:34:55 +0000

Hi Al,

the draft explains the “or” in section “2.  A brief segment routing connectivity monitoring framework” (the last bullet point list at the end of the section).

Segment Routing allows to concatenate Tranport Labels, which carry routing information following SPF. A loss in connectivity is followed by a re-route and the delay changes (might be a rather small difference in a DC).

Segment Routing also allows to insert so called Adjacency Segments, which may be interpreted as a static label (or SRv6 Segment) pointing to a local router interface. If such an interface looses connectivity, all traffic will be dropped.

The behavior depends on the method preferred to set up the measurement path. The slides don’t explain that, but the draft does (to accommodate a usually densely populated IPPM agenda, I preferred a brief presentation with a correct, but not complete extract from the draft).

Regards,

Ruediger

Von: MORTON, ALFRED C (AL) <[email-masked]>
Gesendet: Sonntag, 1. März 2020 22:25
An: Geib, Rüdiger <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Betreff: RE: [spring] Monitoring metric to detect and locate congestion

Hi Rudiger,

From: ippm [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Thursday, February 27, 2020 5:42 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] [spring] Monitoring metric to detect and locate congestion

Hi Robert,

Regards, Ruediger

Hi,

Cheers,
R.

Regards,

Ruediger

Hi,

Two clarifications:

Thx,
R.

------

Regards, Ruediger

:)

Many thx,
R.

You’ll find slides at

Draft url:

Regards,

Ruediger

---

From: <[email-masked]>
Date: Mon, 2 Mar 2020 15:53:24 +0000

Hi Haoyu,

thanks for your remarks. Let me pick up your numbering

  1.  IOAM information could be added by a passed router, if there's interest. The draft doesn't exclude that. But that's not in focus either. I didn't make up my mind, whether and which IOAM information may add value to such measurements. But that doesn't preclude this discussion. I'd prefer any related information in a separate document (and happily add a reference, should there be one).
  2.  The path set up chosen carries a design and it is optimized by the goal one monitored path/interface, one measurement path. It allows active monitoring of connectivity, congestion location (queue depth) and delay at forwarding layer. The draft assumes a single outage only. I didn't try to seize the same information with less measurements (I'm happy I found the solution shown in the draft).
I agree that automated set up of large scale network monitoring measurements is an important and related topic. I prefer to look at it as another building block. I think, other optimized measurement path set ups may be used to reach other monitoring goals (e.g., just connectivity, just congestion location or just delays).

I don't want to overload this draft, even if contents are related. I'd be glad if it is a useful part of a bigger picture "highly automated network monitoring".

Regards,

Ruediger

Von: Haoyu Song <[email-masked]>
Gesendet: Donnerstag, 27. Februar 2020 20:44
An: Geib, Rüdiger <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]
Betreff: RE: Monitoring metric to detect and locate congestion

Hi Ruediger,

I like the general idea that using pre-determined paths in SR to collect performance metrics. I think this approach provides some unique benefits compared with the other approaches. It is also coincident with some of related research work I'm doing.
Here are some thoughts I have.

  1.  I think IOAM could be used as the standard approach for such probing packets. It can collect the performance metrics mentioned in the draft and does more.
  2.  An interesting problem raised by the draft but not fully addressed is the method to plan the optimal paths. There is a work called INT-PATH (https://ieeexplore.ieee.org/document/8737529) which applies Eulerian Path algorithm to find the minimum set of paths with network-wide coverage. However, the problem here seems different: you need path coverage redundancy. My question is: do we really need the redundancy to achieve the measurement goal? If so, what's the best planning algorithm should be? In a real and large scale network, we have constraint on where the probing device(s) can be placed, and we usually want to monitoring the entire network, so an efficient algorithm is necessary.

Best regards,
Haoyu

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Tuesday, February 25, 2020 11:55 PM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [ippm] Monitoring metric to detect and locate congestion

I'm solliciting interest in a new network monitoring metric which allows to detect and locate congested interfaces. Important properties are

  *   Same scalability as ICMP ping in the sense one measurement relation required per monitored connection
  *   Adds detection and location of congested interfaces as compared to ICMP ping (otherwise measured metrics are compatible with ICMP ping)
  *   Requires Segment Routing (which means, measurement on forwarding layer, no other interaction with passed routers - in opposite to ICMP ping)
  *   Active measurement (may be deployed using a single sender&receiver or separate sender and receiver, Segment Routing allows for both options)

I'd be happy to present the draft in Vancouver.. If there's community interest. Please read and comment.

You'll find slides at

https://datatracker.ietf.org/meeting/105/materials/slides-105-ippm-14-draft-geib-ippm-connectivity-monitoring-00<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F105%2Fmaterials%2Fslides-105-ippm-14-draft-geib-ippm-connectivity-monitoring-00&data=02%7C01%7Chaoyu.song%40futurewei.com%7C4f7ef087db7046424e2508d7ba915817%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637183005756536633&sdata=98D7jZDubbhdB3ext94tjEElItEBVyDUld6cQtND6O4%3D&reserved=0>

Draft url:

https://datatracker.ietf.org/doc/draft-geib-ippm-connectivity-monitoring/<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-geib-ippm-connectivity-monitoring%2F&data=02%7C01%7Chaoyu.song%40futurewei.com%7C4f7ef087db7046424e2508d7ba915817%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637183005756546590&sdata=sRAxvsAvs2nHOFme1%2FVZV%2FcFgvZ5AIFtFe5jInmfy7Q%3D&reserved=0>

Regards,

Ruediger

---

From: <[email-masked]>
Date: Mon, 2 Mar 2020 16:02:23 +0000

Hi Tianran,

using node based information like added by IOAM is another valid option. The method I propose is to stay at forwarding layer and work without node support. That way forwarding issues are detected, even if the control plane isn’t aware of it. Also that happens.

Active measurements and IOAM can be combined, and that might be useful.

Regards,

Ruediger

Von: spring <[email-masked]> Im Auftrag von Tianran Zhou
Gesendet: Freitag, 28. Februar 2020 02:37
An: Haoyu Song <[email-masked]>; Robert Raszuk <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Betreff: Re: [spring] Monitoring metric to detect and locate congestion

I take the second point. But it also gives short coming, the probe cannot take many data in one packet.
The first point I do not understand.
You said “combination ingress+egress queuing information on each transit node”. I suppose you mean all the queues information in this node. This in my opinion can only be collected in control plane.

Best,
Tianran

From: Haoyu Song [mailto:[email-masked]]
Sent: Friday, February 28, 2020 9:23 AM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [spring] Monitoring metric to detect and locate congestion

Two key differences. (1) you want to do this purely in data plane fast path. (2) you don’t want to keep a streaming session from each node for scalability reason
The path probing approach can meet both requirements.

Haoyu

From: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 27, 2020 5:14 PM
To: Haoyu Song <[email-masked]<mailto:[email-masked]>>; Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [spring] Monitoring metric to detect and locate congestion

This is a good point. In this way, the probe is generically used to collect node data.
At each node, the probe will be send to the slow path to get the node data. Data could be carried in the payload which has more space.
Then, what’s the difference to existing way that Robert mentioned? Use streaming telemetry to export from each node.
As far as I know the state of art, many device support streaming telemetry can export in a period of 1-10 seconds. For customized data, the period could even be reduced to 10-100ms.

Best,
Tianran

From: ippm [mailto:[email-masked]] On Behalf Of Haoyu Song
Sent: Friday, February 28, 2020 4:44 AM
To: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] [spring] Monitoring metric to detect and locate congestion

Can the combination ingress+egress queuing information on each transit node be collected by simply visit the node? If so, one or more probing paths that can cover all the nodes are sufficient.

Best,
Haoyu

From: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 27, 2020 12:18 PM
To: Haoyu Song <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [spring] Monitoring metric to detect and locate congestion

The point is not to assure all egress ports are inspected.

The point of any really useful end to end path probing is to find out if combination of ingress+egress queuing on each transit node my packet may traverse are not congested.

Looking at Eulerian path algorithm I do not see such guarantees.

Best,
R.

PS. Path probing is A1 to B1. It is not A1 to B1, B2 .. Bn. where An are the ingress ports to the network and Bn are the egress ports.

On Thu, Feb 27, 2020 at 9:14 PM Haoyu Song <[email-masked]<mailto:[email-masked]>> wrote:
Hi Robert,

The Eulerian path algorithm guarantees to visit all the edges of a graph. In the SR context, we can consider the sub-path between two segments an edge.

Haoyu

From: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 27, 2020 11:50 AM
To: Haoyu Song <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [spring] Monitoring metric to detect and locate congestion

Hi Haoyu,

> which applies Eulerian Path algorithm to find the minimum set of paths with network-wide coverage.

In practice networks use ECMP. ECMP decision may happen at each hop. Your end to end flows get spread over all ECMP paths. So limiting number of probed paths is inaccurate to the fundamental objective of the exercise.

That is infact main challenge with any end to end path probing today ... if you do not cover all possible paths your packets may take between ingress and egress you just do not get full picture of the network.

Thx a lot,
R.

On Thu, Feb 27, 2020 at 8:44 PM Haoyu Song <[email-masked]<mailto:[email-masked]>> wrote:
Hi Ruediger,

I like the general idea that using pre-determined paths in SR to collect performance metrics. I think this approach provides some unique benefits compared with the other approaches. It is also coincident with some of related research work I’m doing.
Here are some thoughts I have.

  1.  I think IOAM could be used as the standard approach for such probing packets. It can collect the performance metrics mentioned in the draft and does more.
  2.  An interesting problem raised by the draft but not fully addressed is the method to plan the optimal paths. There is a work called INT-PATH (https://ieeexplore.ieee.org/document/8737529<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fieeexplore.ieee.org%2Fdocument%2F8737529&data=02%7C01%7Chaoyu.song%40futurewei.com%7C03811f6f80554f7dad6408d7bbeb74f0%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637184492271070940&sdata=CMFuDR26gKPUjOYu3vcVlWNoTP%2FA%2FGiCIEnxERVBZmk%3D&reserved=0>) which applies Eulerian Path algorithm to find the minimum set of paths with network-wide coverage. However, the problem here seems different: you need path coverage redundancy. My question is: do we really need the redundancy to achieve the measurement goal? If so, what’s the best planning algorithm should be? In a real and large scale network, we have constraint on where the probing device(s) can be placed, and we usually want to monitoring the entire network, so an efficient algorithm is necessary.

Best regards,
Haoyu

I’d be happy to present the draft in Vancouver... If there’s community interest. Please read and comment.

You’ll find slides at

https://datatracker.ietf.org/meeting/105/materials/slides-105-ippm-14-draft-geib-ippm-connectivity-monitoring-00<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F105%2Fmaterials%2Fslides-105-ippm-14-draft-geib-ippm-connectivity-monitoring-00&data=02%7C01%7Chaoyu.song%40futurewei.com%7C03811f6f80554f7dad6408d7bbeb74f0%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637184492271070940&sdata=tL%2FtbMJ0Ei2e%2FUFZyBB2GFPEhj8qzCuWXo5upM9GNsE%3D&reserved=0>

Draft url:

https://datatracker.ietf.org/doc/draft-geib-ippm-connectivity-monitoring/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-geib-ippm-connectivity-monitoring%2F&data=02%7C01%7Chaoyu.song%40futurewei.com%7C03811f6f80554f7dad6408d7bbeb74f0%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637184492271080924&sdata=qteR3ORsHz6mnbguKfMMQwcls9kaSi8NhAh9hFaH7b8%3D&reserved=0>

Regards,

Ruediger
_______________________________________________
spring mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spring<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fspring&data=02%7C01%7Chaoyu.song%40futurewei.com%7C03811f6f80554f7dad6408d7bbeb74f0%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637184492271080924&sdata=cUScWh29Li%2BFBJHIYBm6fcylGx6zrv%2F4RtEM6b5YMvs%3D&reserved=0>