Subject: MIC calculation discrepancy?

From: Michael Milner <[email-masked]>
Date: Wed, 27 Aug 2003 08:23:45 -0700 (PDT)

Hello, I am hoping someone can help me... we are
testing some in-house software to do AS2
communications, and we have had problems connecting to
Walmart. We can send signed and encrypted messages
just fine and we receive a proper MDN; however we are
not able to verify the Message Integrity Check
returned in Walmart's MDN. In particular if we sign,
encrypt, and send the same test message repeatedly we
get a different value for the MIC each time!

I am a bit confused as to how this could be, as RFC
3335 and the AS2 spec indicate that the MIC should be
calculated over the original EDI data and MIME header.
Moreover we have not had this issue connecting with
other servers. Is this a known issue with Wal-Mart's
EDI solution? Or am I doing something wrong?

Thank you!

Best regards,

Michael Milner

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

---

From: Tim McCarthy <[email-masked]>
Date: Wed, 27 Aug 2003 13:00:17 -0400

Hi Phil, it's been a while. Escaped the interops? I've seen the same thing
that you have with regards to MD5 but every once in a while I see MDN's with
bad MIC's returned for customers using SHA. If Michael is seeing different
MICs for every retransmission then it's probably his software but if it's
every once in a while I believe it could very well be iSoft. I'm still
trying to get data from customers to nail this down but once they
re-transmit and get a good MDN back they move on.

Tim
 
-----Original Message-----
From: Phil Arcuri [mailto:[email-masked]] 
Sent: Wednesday, August 27, 2003 12:06 PM
To: Michael Milner; [email-masked]
Subject: RE: MIC calculation discrepancy?

Michael,

We have many customers who trade with Walmart, and might be
able to help.  Walmart's AS2 vendor chose not to support MD5,
and we have seen very peculiar behaviour when messages signed
with MD5 are sent to Walmart.  Are you using MD5 in your test 
messages?  If not, someone in our Support group who regularly
handles issues with Walmart/AS2 communications can work with
you (at least for a little while...).  Can you provide a
copy of the test message sent to Walmart?

- Phil

Phil Arcuri, D.Phil.
Director, Desktop Applications Development
bTrade, Inc.
2324 Gateway Drive
Irving, TX 75063
972-580-2916 

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Michael Milner
Sent: Wednesday, August 27, 2003 10:24 AM
To: [email-masked]
Subject: MIC calculation discrepancy?

Thank you!

Best regards,

Michael Milner

---

From: "Phil Arcuri" <[email-masked]>
Date: Wed, 27 Aug 2003 16:02:10 -0500

No problem.

I think you will find that interoperability may be
somewhat elusive without going thru the UCC/DGI Interop tests
- that is, to be interoperable with a wide range of vendors.
Although we have a spec for AS2, each vendor has their own
little peculiarities - some won't handle folded headers, for example.
I think it likely that some products will not correctly handle some
case which they have never encountered before due to a previously 
undiscovered bug or two.  Each AS2 Interop test round brings new bugs
to light in existing previously certified products because new vendors
join the mix and do something a little differently.
Finding the lowest common denominator rather than following the
various RFCs and drafts is what it boils down to.
If you only want to interoperate with Wal-Mart, then you will
only have to learn their peculiarities, but since you will not have the
weight of the Interop Test forum to help, it may be a bit painful.  

The vendor of the product that Wal-Mart uses has passed the DGI AS2 
interop tests, and unless you were unlucky and Wal-Mart happened
to have just upgraded their software to a new and buggy release (which 
seems unlikely), then it can be argued that the problem is probably with
your software.  That is, if you change your software you can probably 
get Wal-Mart's software to send you back good MDNs.

To your attached messages:

(1) The encrypted message looks OK in general.  Some comments:
- Some vendors include 'To:', 'From:', 'Reply-To:' headers in the top-most
  message header (they are not required but be prepared for them).
- AS2-Version 1.0 implies that you will not support compression using
  ZLIB and an SMIME compressed-data CMS object (this is the only
  standard compression for AS2 to date).
- Some vendors leave out the Subject header.
- Many vendors do not base64 encode their AS2 messages, since this adds
  significantly to the overall message size.
- I assume your message-ID value was changed to hide it, and that you use
  a globally unique message-Id generation scheme.  You can't generate
  two messages with the same message-ID.  If you did you would cause
  all kinds of problems for those receiving systems which assume it
  will be a unique value (and have it as a primary key)

(2) The signed message (which was subsequently encrypted for Wal-Mart)
is probably the problem with respect to the Received-Content-MIC.  
While you may argue that you have followed the various RFCs and draft
RFCs, until you argue your case in a Interop Test setting, the Wal-Mart
vendor is unlikely to change their product at your request even if you
can prove it is their fault.  To change their product would invalidate
their interop certification.  You should be able to get it to work by
changing your end (I know, but nobody said it was fair...).
A couple of things about your signed message:
- The content-type header is folded - some vendors may not handle this.
  Certainly, some vendors do NOT handle this at the top message level because
  of a bug in IIS that Microsoft has been very slow to fix.
- Your choice of boundary text is not a good one - if your data or signature
  happened to contain the same text, the message syntax would be corrupt.
- You seem to have an extra CRLF after the header before the first boundary.
  Not sure if this is legal or not, and it doesn't matter.  This may be causing
  your problem - try using just two CRLF's - one to end the last header line and
  one signify the end of the headers.
- Some vendors actually require that the edi data be valid edi data, as wrong 
  as that sounds -- AS2 is about transport, right?.  Since your payload claims to 
  be X12 data, you might try including a simple syntactically correct X12
  payload instead of your "TEST". 
- As mentioned above - many vendors do not filter their data - you have filtered
  your signature - because it adds unnecessary bytes.  Probably no big deal on
  signatures since they are always small.

That's all I see at a casual glance.
While it seems very peculiar that the Received-Content-MIC has a different value
in every MDN you receive for the same payload message sent to them, and you may 
very well have uncovered a problem in their software, you may need to change
your code to work around it.

Of course, I may be missing something obvious, too.
Others in this list may see something more - anyone?

- Phil

Phil Arcuri, D.Phil.

-----Original Message-----
From: Michael Milner [mailto:[email-masked]]
Sent: Wednesday, August 27, 2003 2:01 PM
To: Phil Arcuri; [email-masked]
Subject: RE: MIC calculation discrepancy?

Thanks for your help.

I'm attaching wmrequest.bin which is the signed,
encrypted request to Walmart. I'm also attaching
testcase.bin which is the signed data that was
encrypted with Walmart's certificate.

The MIC returned is always different -- even though I
am encrypting the same data each time. (I suppose
Walmart is somehow calculating the MIC on the
encrypted data?!)

p.s. Am using SHA1.

Thanks.

Michael

---

From: Tim McCarthy <[email-masked]>
Date: Wed, 27 Aug 2003 17:52:01 -0400

I thought Wal-Mart had stated that they would only test with certified
products?  

-----Original Message-----
From: Phil Arcuri [mailto:[email-masked]] 
Sent: Wednesday, August 27, 2003 5:02 PM
To: Michael Milner; [email-masked]
Subject: RE: MIC calculation discrepancy?

No problem.

(2) The signed message (which was subsequently encrypted for Wal-Mart)
is probably the problem with respect to the Received-Content-MIC.  
While you may argue that you have followed the various RFCs and draft
RFCs, until you argue your case in a Interop Test setting, the Wal-Mart
vendor is unlikely to change their product at your request even if you
can prove it is their fault.  To change their product would invalidate
their interop certification.  You should be able to get it to work by
changing your end (I know, but nobody said it was fair...).
A couple of things about your signed message:
- The content-type header is folded - some vendors may not handle this.
  Certainly, some vendors do NOT handle this at the top message level
because
  of a bug in IIS that Microsoft has been very slow to fix.
- Your choice of boundary text is not a good one - if your data or signature
  happened to contain the same text, the message syntax would be corrupt.
- You seem to have an extra CRLF after the header before the first boundary.
  Not sure if this is legal or not, and it doesn't matter.  This may be
causing
  your problem - try using just two CRLF's - one to end the last header line
and
  one signify the end of the headers.
- Some vendors actually require that the edi data be valid edi data, as
wrong 
  as that sounds -- AS2 is about transport, right?.  Since your payload
claims to 
  be X12 data, you might try including a simple syntactically correct X12
  payload instead of your "TEST". 
- As mentioned above - many vendors do not filter their data - you have
filtered
  your signature - because it adds unnecessary bytes.  Probably no big deal
on
  signatures since they are always small.

That's all I see at a casual glance.
While it seems very peculiar that the Received-Content-MIC has a different
value
in every MDN you receive for the same payload message sent to them, and you
may 
very well have uncovered a problem in their software, you may need to change
your code to work around it.

- Phil

Phil Arcuri, D.Phil.

-----Original Message-----
From: Michael Milner [mailto:[email-masked]]
Sent: Wednesday, August 27, 2003 2:01 PM
To: Phil Arcuri; [email-masked]
Subject: RE: MIC calculation discrepancy?

Thanks for your help.

p.s. Am using SHA1.

Thanks.

Michael