From miclai@microsoft.com Thu Jan 16 20:04:22 2014 Return-Path: X-Original-To: http-devops@ietfa.amsl.com Delivered-To: http-devops@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C24241ADF23 for ; Thu, 16 Jan 2014 20:04:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jyaN5xjnCZTx for ; Thu, 16 Jan 2014 20:04:20 -0800 (PST) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0182.outbound.protection.outlook.com [207.46.163.182]) by ietfa.amsl.com (Postfix) with ESMTP id B7F4D1ADF12 for ; Thu, 16 Jan 2014 20:04:19 -0800 (PST) Received: from BL2PR03MB164.namprd03.prod.outlook.com (10.255.230.148) by BL2PR03MB370.namprd03.prod.outlook.com (10.141.89.13) with Microsoft SMTP Server (TLS) id 15.0.847.13; Fri, 17 Jan 2014 04:04:06 +0000 Received: from BL2PR03MB164.namprd03.prod.outlook.com ([169.254.10.174]) by BL2PR03MB164.namprd03.prod.outlook.com ([169.254.10.174]) with mapi id 15.00.0847.008; Fri, 17 Jan 2014 04:04:05 +0000 From: Michelle Lai To: "'http-devops@ietf.org'" Thread-Topic: HTTP/2 test suite is now available! Thread-Index: Ac8TOJjpyP3ypoPuRO6eFkiQjActJA== Date: Fri, 17 Jan 2014 04:04:05 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.147.53] x-o365ent-eop-header: Message processed by - O365_ENT: Allow from ranges (Engineering ONLY) x-forefront-prvs: 0094E3478A x-forefront-antispam-report: SFV:NSPM; SFS:(10009001)(679001)(689001)(779001)(164054003)(53754006)(199002)(189002)(15975445006)(81686001)(33646001)(81816001)(79102001)(56776001)(80976001)(46102001)(77982001)(59766001)(66066001)(49866001)(47736001)(80022001)(47976001)(50986001)(63696002)(4396001)(65816001)(54356001)(54316002)(76482001)(19580395003)(51856001)(83322001)(53806001)(76176001)(76796001)(76786001)(76576001)(77096001)(19300405004)(85852003)(83072002)(92566001)(93136001)(74706001)(31966008)(47446002)(74502001)(74662001)(16236675002)(15202345003)(90146001)(93516002)(56816005)(2656002)(81542001)(87936001)(81342001)(74316001)(74366001)(85306002)(87266001)(69226001)(74876001)(24736002)(491001); DIR:OUT; SFP:1101; SCL:1; SRVR:BL2PR03MB370; H:BL2PR03MB164.namprd03.prod.outlook.com; CLIP:131.107.147.53; FPR:; RD:InfoNoRecords; A:1; MX:1; LANG:en; Content-Type: multipart/alternative; boundary="_000_d167bbd5f75f4a97882c126cab095cddBL2PR03MB164namprd03pro_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com Cc: "Parashuram Narasimhan \(MS OPEN TECH\)" , "Brian Raymor \(MS OPEN TECH\)" , "Elancheziyan, Anbu" , Rob Trace Subject: [http-devops] HTTP/2 test suite is now available! X-BeenThere: http-devops@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion about operation, deployment and testing of HTTP" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 04:04:23 -0000 --_000_d167bbd5f75f4a97882c126cab095cddBL2PR03MB164namprd03pro_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Everyone, I'd like to announce that HTTP/2 test suite is now available at https://git= hub.com/http2/http2-test/ The test suite is based on the principles of HTTP/2 Protocol test draft: ht= tp://www.ietf.org/id/draft-trace-httpbis-http2-test-00.txt to: =B7 focus on HTTP/2 protocol compliance testing =B7 contain client and server test cases =B7 use platform-independent programming language - tests are writt= en in JavaScript =B7 provide cross-platform support - use NodeJS, grunt, mocha, and = G=C3bor's node-http2 module as the 09 implementation (will be updated as th= e draft moves forward) Usage and proposed test scenarios are listed in readme and wiki at GitHub. Currently we have few contributors: Akamai, Microsoft Open Tech, and G=C3bo= r and are seeking more contributors to: =B7 Implement more test cases =B7 Improve framework robustness =B7 Make framework as part of your build process, tweak it so that = it works Thanks, Michelle --_000_d167bbd5f75f4a97882c126cab095cddBL2PR03MB164namprd03pro_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi Everyone,

I’d like to a= nnounce that HTTP/2 test suite is now available at https://github.com/http2/http2-test/

 

The test suite is b= ased on the principles of HTTP/2 Protocol test draft: http://www.ietf.org/id/draft-trace-httpbis-http= 2-test-00.txt to:

=B7         focus on HT= TP/2 protocol compliance testing

=B7         contain cli= ent and server test cases

=B7         use platfor= m-independent programming language - tests are written in JavaScript

=B7         provide cro= ss-platform support - use NodeJS, grunt, mocha, and G=C3bor’s node-ht= tp2 module as the 09 implementation (will be updated as the draft moves for= ward)

 

Usage and proposed = test scenarios are listed in readme and wiki at GitHub.

 

Currently we have f= ew contributors: Akamai, Microsoft Open Tech, and G=C3bor and are seeking m= ore contributors to:

=B7         Implement m= ore test cases

=B7         Improve fra= mework robustness

=B7         Make framew= ork as part of your build process, tweak it so that it works 

 

Thanks,<= /span>

Michelle=

--_000_d167bbd5f75f4a97882c126cab095cddBL2PR03MB164namprd03pro_-- From tatsuhiro.t@gmail.com Sat Jan 18 04:52:52 2014 Return-Path: X-Original-To: http-devops@ietfa.amsl.com Delivered-To: http-devops@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2EE4B1A1F5F for ; Sat, 18 Jan 2014 04:52:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ta5EocsXQ1L4 for ; Sat, 18 Jan 2014 04:52:50 -0800 (PST) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by ietfa.amsl.com (Postfix) with ESMTP id 4A2091A1F1A for ; Sat, 18 Jan 2014 04:52:50 -0800 (PST) Received: by mail-ie0-f181.google.com with SMTP id tq11so4768209ieb.12 for ; Sat, 18 Jan 2014 04:52:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=8SY7Rl4iA6UC5jiVu9KfpQuoqJidW6+wCOvmWYxI54s=; b=ZOOfTywa4m57VtziKQQNPQC+0ouKESWZMbXXqZhM8XUTqhCKk0vYGP3oHaDFOIzVcG LJYmolvgNGGxFQnVWiLrjA6vXbnchYKZ4G+U4f9GeVZ2rmOydMh7HlSnJyPRU5yjeXLI ffpduIYbSpKJrAQT3mVyrsZfAcPGLrAGTdZZZLo5+FoEP9K3SNr7ECcl7Fk04UKvQ7jw JRUF1cOFlIqnwXFHqiypC6bA2DhxRvRdxmkR2b81SQ3PQFgSbEmB+3A7GUIW447fxlHJ YX6iNfpxdnlmr+34jDPBCEAAO3h/JQXTmlZKV9JWxrX/vC5qXQgRMz/j0n8wD+auMkhV uKGA== X-Received: by 10.42.133.199 with SMTP id i7mr489293ict.64.1390049557418; Sat, 18 Jan 2014 04:52:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.64.24.137 with HTTP; Sat, 18 Jan 2014 04:52:17 -0800 (PST) In-Reply-To: References: From: Tatsuhiro Tsujikawa Date: Sat, 18 Jan 2014 21:52:17 +0900 Message-ID: To: Michelle Lai Content-Type: multipart/alternative; boundary=90e6ba6e858617195d04f03e223b Cc: "http-devops@ietf.org" , "Brian Raymor \(MS OPEN TECH\)" , "Elancheziyan, Anbu" , "Parashuram Narasimhan \(MS OPEN TECH\)" , Rob Trace Subject: Re: [http-devops] HTTP/2 test suite is now available! X-BeenThere: http-devops@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion about operation, deployment and testing of HTTP" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jan 2014 12:52:52 -0000 --90e6ba6e858617195d04f03e223b Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, Thank you for this effort, testing platform is a great step forward for HTTP/2.0 development. I'd like to test nghttp2 client/server with this test suite, but I could not find the document how to hook the test to the ordinary command line tool. Could you tell me how to do it? Best regards, Tatsuhiro Tsujikawa On Fri, Jan 17, 2014 at 1:04 PM, Michelle Lai wrote: > Hi Everyone, > > I=92d like to announce that HTTP/2 test suite is now available at > https://github.com/http2/http2-test/ > > > > The test suite is based on the principles of HTTP/2 Protocol test draft: > http://www.ietf.org/id/draft-trace-httpbis-http2-test-00.txt to: > > =B7 focus on HTTP/2 protocol compliance testing > > =B7 contain client and server test cases > > =B7 use platform-independent programming language - tests are > written in JavaScript > > =B7 provide cross-platform support - use NodeJS, grunt, mocha, an= d > G=C3bor=92s node-http2 module as the 09 implementation (will be updated a= s the > draft moves forward) > > > > Usage and proposed test scenarios are listed in readme and wiki at GitHub= . > > > > Currently we have few contributors: Akamai, Microsoft Open Tech, and G=C3= bor > and are seeking more contributors to: > > =B7 Implement more test cases > > =B7 Improve framework robustness > > =B7 Make framework as part of your build process, tweak it so tha= t > it works > > > > Thanks, > > Michelle > > _______________________________________________ > http-devops mailing list > http-devops@ietf.org > https://www.ietf.org/mailman/listinfo/http-devops > > --90e6ba6e858617195d04f03e223b Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi,

Thank you for this effort, testing = platform is a great step forward for HTTP/2.0 development.

I'd like to test nghttp2 client/server with this test suite, b= ut I could not find the document how to hook the test to the ordinary comma= nd line tool.
Could you tell me how to do it?

Best regards,=
Tatsuhiro Tsujikawa

On Fri, Jan 17, 2014 at 1:04 PM, Michelle Lai <= miclai@microsoft.com> wrote:

Hi Everyone,

I=92d like to annou= nce that HTTP/2 test suite is now available at https://github.com/http2/http2-test/

=A0

The test suite is b= ased on the principles of HTTP/2 Protocol test draft: http://www.ietf.org/id/draft-= trace-httpbis-http2-test-00.txt to:

=B7=A0=A0=A0=A0=A0=A0=A0=A0 focus on HTTP/= 2 protocol compliance testing

=B7=A0=A0=A0=A0=A0=A0=A0=A0 contain client= and server test cases

=B7=A0=A0=A0=A0=A0=A0=A0=A0 use platform-i= ndependent programming language - tests are written in JavaScript=

=B7=A0=A0=A0=A0=A0=A0=A0=A0 provide cross-= platform support - use NodeJS, grunt, mocha, and G=C3bor=92s node-http2 mod= ule as the 09 implementation (will be updated as the draft moves forward)

=A0

Usage and proposed = test scenarios are listed in readme and wiki at GitHub.

=A0

Currently we have f= ew contributors: Akamai, Microsoft Open Tech, and G=C3bor and are seeking m= ore contributors to:

=B7=A0=A0=A0=A0=A0=A0=A0=A0 Implement more= test cases

=B7=A0=A0=A0=A0=A0=A0=A0=A0 Improve framew= ork robustness

=B7=A0=A0=A0=A0=A0=A0=A0=A0 Make framework= as part of your build process, tweak it so that it works=A0<= /span>

=A0

Thanks,

Michelle<= /u>


_______________________________________________
http-devops mailing list
http-devops@ietf.org
https://www.ietf.org/mailman/listinfo/http-devops


--90e6ba6e858617195d04f03e223b-- From tatsuhiro.t@gmail.com Fri Jan 31 23:40:12 2014 Return-Path: X-Original-To: http-devops@ietfa.amsl.com Delivered-To: http-devops@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE5201A0536 for ; Fri, 31 Jan 2014 23:40:12 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.693 X-Spam-Level: X-Spam-Status: No, score=-0.693 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001, TRACKER_ID=1.306] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7NCKEN49E37 for ; Fri, 31 Jan 2014 23:40:07 -0800 (PST) Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 2925F1A0534 for ; Fri, 31 Jan 2014 23:40:07 -0800 (PST) Received: by mail-oa0-f43.google.com with SMTP id h16so6265682oag.16 for ; Fri, 31 Jan 2014 23:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=KfnMQIpQD6bLnopu9RCHHFM5tpIxFmPxgc0vP5IrvDw=; b=JoYp/9iRjLwr758JLTWYUApW6EBHQ0snpIhp6Oh3IY+oo890vuFbYxy2UFuSmQZQSB K0ZvINMhvMWduaWnj1PYQJJrmuGP4vmyy1r69uEbnbQF2WwrwGIa/tZ0lLv0K8xmC7Ky jriub+1/kYdjA6fJJO3Mh+buM7ckVg658/qsO3UrSjkESVb82zLegQn6p00JJ1WoUJpO Pd1L7HfPFUeeO+ty1TF2hs5EznLJ/yBjO6K60XUf0tjoKcsgs+/27trqfyAe2ySd+7EO fNcMa5PkcysTldskwgeeKjJ0lFt5yWDZdT1O6E3GD5Wtr2gljM7WG+tbboKUmmKY7Cqg nAAg== X-Received: by 10.60.165.72 with SMTP id yw8mr18419oeb.71.1391240403101; Fri, 31 Jan 2014 23:40:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.172.80 with HTTP; Fri, 31 Jan 2014 23:39:42 -0800 (PST) In-Reply-To: References: From: Tatsuhiro Tsujikawa Date: Sat, 1 Feb 2014 16:39:42 +0900 Message-ID: To: "Parashuram Narasimhan (MS OPEN TECH)" Content-Type: multipart/alternative; boundary=047d7b41904b062e8204f15366b5 Cc: "Brian Raymor \(MS OPEN TECH\)" , "Jeff Mendoza \(MS OPEN TECH\)" , "http-devops@ietf.org" , "Elancheziyan, Anbu" , Michelle Lai , Rob Trace Subject: Re: [http-devops] HTTP/2 test suite is now available! X-BeenThere: http-devops@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion about operation, deployment and testing of HTTP" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Feb 2014 07:40:13 -0000 --047d7b41904b062e8204f15366b5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Sat, Feb 1, 2014 at 7:31 AM, Parashuram Narasimhan (MS OPEN TECH) < panarasi@microsoft.com> wrote: > Hi, > > > > I wanted to ping you again to see if you were able to get the test cases > running for nghttp. > > We were also able to download your code and ran some initial test cases t= o > ensure that we are doing it right on our end. The left column is the > filenames and should give you an idea of how the http2 payload was sent t= o > your server. > > > With HTTP2_BROWSER environment variable, I can run the test against nghttp client. Thank you. > We did find a few bugs in our implementation, thanks to your code and > have fixed the tests. Many of the tests passed. The ones labelled > INCORRECT TEST were issues with the test framework and we have fixed most > of the issues. We also found some issues other issues that may be in > nghttp2. It would be great if you could take a look at it and confirm tha= t > the fails are not caused due to the test framework. > > > > It would be great if you can continue running your implementation against > the test framework and let us know if you encounter issues with the test > framework itself. > > > - invalid-level-goaway-test I fixed this sometime ago in nghttp2 - *-when-closed-test I'm thinking they are really necessary. The spec says it is RST_STREAM situation, but responding each frame with RST_STREAM is just wasting the processing and memory (e.g., many DATA frames comes from the remote and responding each with RST_STREAM). So "smart" client may remember when RST_STREAM is issued to certain stream, but what gains if we do that using extra memory. As a good compromise, I think it is best to ignore these frames. - *-when-idle-test I thought that these are the same categories of *-when-closed-test. So I did not implement the check for this idle test. It is not generally possible to detect that which stream ID is currently idle without remembering all stream ID used so far, especially those which are smaller than the last used/received stream ID. But it is possible to detect idle stream which has larger stream ID than the currently used. The current test case uses stream ID 2, which is just the case. I have pending local commits to detect this situation to pass the tests. - push-promise-when-reserved-remote-test With the latest nghttp2 and http2-test, this test passes. - priority-when-reserved-remote What does WRONG TEST mean? I don't know why but the following tests fail on my PC: - huge-index-test Error: done() invoked with non-Error: Not appropriate error code: COMPRESSION_ERROR - invalid-data-1-test Error: timeout of 10000ms exceeded - invalid-data-2-test Error: timeout of 10000ms exceeded - oversized-ping-test Error: done() invoked with non-Error: Not appropriate error code: FRAME_SIZE_ERROR Best regards, Tatsuhiro Tsujikawa > > *Multiplexing* > > *NGHTTP2* > > invalid-level-data-test > > passed > > invalid-level-goaway-test > > FAIL - nghttp2 did not respond any error when receiving GOAWAY with > specified StreamId - 6.8. GOAWAY > > invalid-level-headers-test > > passed > > invalid-level-ping-test > > passed > > invalid-level-priority-test > > passed > > invalid-level-push-promise-test > > passed > > invalid-level-rst-stream-test > > passed > > invalid-level-settings-test > > passed > > *Framing* > > oversized-ping-test > > passed > > *Push* > > invalid-promised-id-1-test > > passed > > invalid-promised-id-2-test > > passed > > invalid-promised-id-3-test > > passed > > *Compression* > > huge-index-test > > passed > > invalid-data-1-test > > passed > > invalid-data-2-test > > passed > > *Stream* > > data-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > priority-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > push-promise-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > reservation-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > rst-stream-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > window-update-when-closed-test > > INCORRECT TEST -waiting for connection error on closed stream - 5.1 > (closed state) > > data-when-idle-test > > FAIL - nghttp2 did not fail (did not send GOAWAY) when sending data on > idle stream - 5.1 - last paragraph > > priority-when-idle-test > > FAIL - nghttp2 did not fail (did not send PRIORITY) when sending data on > idle stream - 5.1 - last paragraph > > push-promise-when-idle-test > > FAIL - RST_STREAM (stream error) instead of GOAWAY (connection error) whe= n > sending PUSH_PROMISE on stream in IDLE state - 5.1 - last paragraph > > rst-stream-when-idle-test > > FAIL - nghttp2 did not fail when sending RST_STREAM on stream in IDLE > state - 6.4 last paragraph > > window-update-when-idle-test > > FAIL - nghttp2 did not fail when sending WINDOW_UPDATE on stream in IDLE > state - 5.1 - last paragraph > > data-when-reserved-remote-test > > passed > > priority-when-reserved-remote > > WRONG TEST - An endpoint MAY send a PRIORITY frame in this state to > reprioritize the reserved stream. - 5.1 (reserved-remote) > > push-promise-when-reserved-remote-test > > FAIL - RST_STREAM instead of GOAWAY - Receiving any other type of fram= e > other than HEADERS or RST_STREAM MUST be treated as a connection error > (Section 5.4.1) of type PROTOCOL_ERROR. - 5.1 (reserved-remote) > > reservation-when-reserved-remote-test > > passed > > window-update-when-reserved-remote-test > > passed > > > > > > *From:* Parashuram Narasimhan (MS OPEN TECH) > *Sent:* Saturday, January 18, 2014 9:13 AM > *To:* 'Tatsuhiro Tsujikawa'; Michelle Lai > *Cc:* http-devops@ietf.org; Brian Raymor (MS OPEN TECH); Elancheziyan, > Anbu; Rob Trace > *Subject:* RE: [http-devops] HTTP/2 test suite is now available! > > > > Hi, > > > > You can see the instructions to run the client test cases here - > https://github.com/http2/http2-test/blob/master/README.md#testing-the-cli= ent. > We do not have any server test cases and are working on adding more tests > for both client and server. Note that the client is expected to talk HTTP= 2 > directly without upgrade. > > > > *From:* Tatsuhiro Tsujikawa [mailto:tatsuhiro.t@gmail.com] > > *Sent:* Saturday, January 18, 2014 4:52 AM > *To:* Michelle Lai > *Cc:* http-devops@ietf.org; Parashuram Narasimhan (MS OPEN TECH); Brian > Raymor (MS OPEN TECH); Elancheziyan, Anbu; Rob Trace > *Subject:* Re: [http-devops] HTTP/2 test suite is now available! > > > > Hi, > > > > Thank you for this effort, testing platform is a great step forward for > HTTP/2.0 development. > > > > I'd like to test nghttp2 client/server with this test suite, but I could > not find the document how to hook the test to the ordinary command line > tool. > > Could you tell me how to do it? > > > > Best regards, > > Tatsuhiro Tsujikawa > > > > On Fri, Jan 17, 2014 at 1:04 PM, Michelle Lai > wrote: > > Hi Everyone, > > I'd like to announce that HTTP/2 test suite is now available at > https://github.com/http2/http2-test/ > > > > The test suite is based on the principles of HTTP/2 Protocol test draft: > http://www.ietf.org/id/draft-trace-httpbis-http2-test-00.txt to: > > =B7 focus on HTTP/2 protocol compliance testing > > =B7 contain client and server test cases > > =B7 use platform-independent programming language - tests are > written in JavaScript > > =B7 provide cross-platform support - use NodeJS, grunt, mocha, an= d > G=C3bor's node-http2 module as the 09 implementation (will be updated as = the > draft moves forward) > > > > Usage and proposed test scenarios are listed in readme and wiki at GitHub= . > > > > Currently we have few contributors: Akamai, Microsoft Open Tech, and G=C3= bor > and are seeking more contributors to: > > =B7 Implement more test cases > > =B7 Improve framework robustness > > =B7 Make framework as part of your build process, tweak it so tha= t > it works > > > > Thanks, > > Michelle > > > _______________________________________________ > http-devops mailing list > http-devops@ietf.org > https://www.ietf.org/mailman/listinfo/http-devops > > > --047d7b41904b062e8204f15366b5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Sat, Feb 1, 2014 at 7:31 AM, Parashuram Narasimhan (MS OPEN TECH= ) <panarasi@microsoft.com> wrote:

Hi,

 

I wanted to ping you again to see if you wer= e able to get the test cases running for nghttp.

We were also able to download your code and = ran some initial test cases to ensure that we are doing it right on our end= . The left column is the filenames and should give you an idea of how the http2 payload was sent to your server. =

 


With HTTP2_BROWSER environment variable, I can run= the test against nghttp client. Thank you.
 

We did find a few bugs in our implementation= , thanks to your code and have fixed the tests. Many of the tests passed. The = ones labelled INCORRECT TEST were issues with the test framework and we hav= e fixed most of the issues. We also found some issues other issues that may= be in nghttp2. It would be great if you could take a look at it and confirm that the fails are not caused due to the test fra= mework.

 

It would be great if you can continue runnin= g your implementation against the test framework and let us know if you enc= ounter issues with the test framework itself.

 


- invalid-level-goaway-test

I fixed this sometime ago in nghttp2

- *-when-closed-test

I'= m thinking they are really necessary. The spec says it is RST_STREAM situat= ion, but responding each frame
with RST_STREAM is just wasting th= e processing and memory (e.g., many DATA frames comes from the remote and r= esponding each with
RST_STREAM). So "smart" client may remember when RST_STREAM = is issued to certain stream, but what gains if we do that using extra
=
memory. As a good compromise, I think it is best to ignore these frame= s.

- *-when-idle-test

I thought that these are the s= ame categories of *-when-closed-test. So I did not implement the check for = this idle test.
It is no= t generally possible to detect that which stream ID is currently idle witho= ut remembering all stream ID used so far,
especially those which are sm= aller than the last used/received stream ID.
But it i= s possible to detect idle stream which has larger stream ID than the curren= tly used. The current test case uses
stream ID 2, which is just the cas= e. I have pending local commits to detect this situation to pass the tests.=


- push-promise-when-reserved-remote-test
=
With the = latest nghttp2 and http2-test, this test passes.

=
- priority-when-reserved-remote
What does WRONG TEST mean?


I don't know why but the following tests fail on my PC:

- huge-index-test
  Error: done() invoked with non-Err= or: Not appropriate error code: COMPRESSION_ERROR

=
- invalid-data-1-test
  Error: tim= eout of 10000ms exceeded

- invalid-data-2-test
=
&nbs= p; Error: timeout of 10000ms exceeded

- oversized-ping-test
<= div>   Er= ror: done() invoked with non-Error: Not appropriate error code: FRAME_SIZE_= ERROR
 
Best regards,
<= div>
Tatsuhiro Tsujikawa

 

Multiplexing

NGHTTP2

invalid-level-data-test

passed

invalid-level-goaway-test

FAIL - nghttp2 did not respond any error when receiving GOAWAY wi= th specified StreamId - 6.8.  GOAWAY

invalid-level-headers-test

passed

invalid-level-ping-test

passed

invalid-level-priority-test

passed

invalid-level-push-promise-test

passed

invalid-level-rst-stream-test

passed

invalid-level-settings-test

passed

Framing

oversized-ping-test

passed

Push

invalid-promised-id-1-test

passed

invalid-promised-id-2-test

passed

invalid-promised-id-3-test

passed

Compression

huge-index-test

passed

invalid-data-1-test

passed

invalid-data-2-test

passed

Stream

data-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

priority-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

push-promise-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

reservation-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

rst-stream-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

window-update-when-closed-test

INCORRECT TEST  -waiting for connection error on closed stre= am - 5.1 (closed state)

data-when-idle-test

FAIL - nghttp2 did not fail (did not send GOAWAY) when sending da= ta on idle stream  - 5.1 - last paragraph

priority-when-idle-test

FAIL - nghttp2 did not fail (did not send PRIORITY) when sending = data on idle stream  - 5.1 - last paragraph

push-promise-when-idle-test

FAIL - RST_STREAM (stream error) instead of GOAWAY (connection er= ror) when sending PUSH_PROMISE on stream in IDLE state - 5.1 - last paragraph

rst-stream-when-idle-test

FAIL - nghttp2 did not fail when sending RST_STREAM on stream in = IDLE state - 6.4 last paragraph

window-update-when-idle-test

FAIL - nghttp2 did not fail when sending WINDOW_UPDATE on stream = in IDLE state - 5.1 - last paragraph

data-when-reserved-remote-test

passed

priority-when-reserved-remote

WRONG TEST -      An endpoint MAY send a= PRIORITY frame in this state to reprioritize the reserved stream. - 5.1 (reserved-remote)

push-promise-when-reserved-remote-test

FAIL - RST_STREAM instead of GOAWAY -    Receiving= any other type of frame other than HEADERS or RST_STREAM MUST be treated a= s a connection error (Section 5.4.1) of type      PROTOCOL_ERROR. - 5.1 (reserved-= remote)

reservation-when-reserved-remote-test

passed

window-update-when-reserved-remote-test

passed

 

 

From: Parashuram Narasimhan (MS OPEN TECH)
Sent: Saturday, January 18, 2014 9:13 AM
To: 'Tatsuhiro Tsujikawa'; Michelle Lai
Cc: http-d= evops@ietf.org; Brian Raymor (MS OPEN TECH); Elancheziyan, Anbu; Rob Tr= ace
Subject: RE: [http-devops] HTTP/2 test suite is now available!

 

Hi,

 

You can see the instructions to run the clie= nt test cases here - https://github.com/http2/http2-test/blob/master/README.md#testing-the-clien= t. We do not have any server test cases and are working on adding more = tests for both client and server. Note that the client is expected to talk = HTTP2 directly without upgrade.

 

From: Tatsuhiro Tsujikawa [mailto:tatsuhiro.t@gmail.com]
Sent: Saturday, January 18, 2014 4:52 AM
To: Michelle Lai
Cc: http-d= evops@ietf.org; Parashuram Narasimhan (MS OPEN TECH); Brian Raymor (MS = OPEN TECH); Elancheziyan, Anbu; Rob Trace
Subject: Re: [http-devops] HTTP/2 test suite is now available!

 

Hi,

 

Thank you for this effort, testing platform is a gre= at step forward for HTTP/2.0 development.

 

I'd like to test nghttp2 client/server with this= test suite, but I could not find the document how to hook the test to the = ordinary command line tool.

Could you tell me how to do it?

 

Best regards,

Tatsuhiro Tsujikawa

 

On Fri, Jan 17, 2014 at 1:04 PM, Michelle Lai <miclai@microsoft.co= m> wrote:

Hi Everyone,<= /u>

I’d like to announce that HTTP/2 test suite is= now available at https:/= /github.com/http2/http2-test/

 

The test suite is based on the principles of HTTP/2 = Protocol test draft: http://www.ietf.org/id/draft-trace-httpbis-http2-test-00.t= xt to:

=B7         focus on HTTP/2 protocol compliance testing

=B7         contain client and server test cases

=B7         use platform-independent programming language - tests are written in= JavaScript

=B7         provide cross-platform support - use NodeJS, grunt, mocha, and G=C3b= or’s node-http2 module as the 09 implementation (will be updated as t= he draft moves forward)

 

Usage and proposed test scenarios are listed in read= me and wiki at GitHub.

 

Currently we have few contributors: Akamai, Microsof= t Open Tech, and G=C3bor and are seeking more contributors to:

=B7         Implement more test cases

=B7         Improve framework robustness

=B7         Make framework as part of your build process, tweak it so that it wo= rks 

 

Thanks,

Michelle


_______________________________________________
http-devops mailing list
http-devops@ietf.= org
https://www.ietf.org/mailman/listinfo/http-devops

 


--047d7b41904b062e8204f15366b5--