From nobody Tue Feb 2 04:18:46 2021 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2D8D03A1A3C for ; Tue, 2 Feb 2021 04:18:45 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1qljDM7Z5bV2 for ; Tue, 2 Feb 2021 04:18:43 -0800 (PST) Received: from ssdrsserver2.hosting.co.uk (ssdrsserver2.hosting.co.uk [185.185.84.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EC9A23A1A3B for ; Tue, 2 Feb 2021 04:18:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:MIME-Version:Date:Message-ID: Subject:From:Cc:To:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=3qNLH4qjfFAAFIu4AO1+ruI2u2ojIbOOxyRQ6ZobTYY=; b=Xa5gRR1KcoQbM2F2buPh7fET3m fNbaOm5GwIZ6yMrU5EriMtArmua4w2eXbEHNEtdZSqNvVgSCJDk1hj0AIvRpEfJjvBXOH9j3vskbw jHHYw47DFbn1gm7cN0+34wMTcBC96I621ArRsYvVXFD2oS96PZX6imxO6a/HOOAibz3RvBWBOXt9G QVnLpBdkkikGXv75rnuscAbEEUfC1rdp0PyVm3Z+1qgt/nCv7TepGy4qCIRasO0IL0nJV88olk8CF zBoWCA5s4km7shIkQDRxkVg22ZfigH+h7lz1i4my2nxQgRBOVrv0yPpZgp+ZgKH7E1DpdFTQq8qCw kHF2ZCag==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:45874 helo=[192.168.1.5]) by ssdrsserver2.hosting.co.uk with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1l6udh-0000Wp-Hc; Tue, 02 Feb 2021 12:18:37 +0000 To: "Tilmans, Olivier (Nokia - BE/Antwerp)" Cc: TCP Prague List From: Bob Briscoe Message-ID: <1ddb42c7-596b-031c-6215-84e675d08b7d@bobbriscoe.net> Date: Tue, 2 Feb 2021 12:18:36 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------ABDF2AAAF8722BCCC1BCBB0C" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hosting.co.uk X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hosting.co.uk: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hosting.co.uk: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: [tcpPrague] Pacing IW X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2021 12:18:45 -0000 This is a multi-part message in MIME format. --------------ABDF2AAAF8722BCCC1BCBB0C Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Olivier, I notice you've pushed code to TCP Prague, to optionally pace out IW over the estimated rtt (srtt). https://github.com/L4STeam/linux/commit/aea3ff36380bec82804eba8e3f99e5f14342978c Thanks for this. Some thoughts: srtt can tend to be rather wildly wrong at the start of a connection. I'd say a high estimate will be more concerning than a low, by this reasoning: - If the initial srtt estimate is too low, there will be a queuing delay spike, but it will never be worse than just sending IW at line rate. - If srtt is too high, a short flow would take longer to complete because the first ACK will return before the IW has finished sending. I recall Mark Handley presenting data showing longer RTT outliers at the start of connections (cause was unknown, but could have been a CDN getting its act together to serve the connection from an optimal location). Sorry, can't find the paper - it was perhaps 5-10 years ago. Whatever, in general, an over-high initial estimate is surely more likely than over-low, given the initial handshake might be delayed more than other packets, but it's unlikely to find some worm-hole that gets it back home faster than the speed of light. So, possible solutions (either or both): 1. In tcp_prague.c, multiply the rate returned from prague_update_pacing_rate() by N(where N=2, say). So IW is at least paced over the first half of the RTT on average. Then, if the initial srtt estimate is longer than reality, the error has to be 2x before IW will be delayed more than a whole round. 2. Introduce a third sysctl option (tcp_pace_iw = 2) that only paces IW if srtt was initialized using the destination cache? Altho this is in TCP Prague now, I guess eventually it could be taken up by any transport with any CC. Cheers Bob PS. Defensive coding nit: In tcp_output.c, I noticed you've kept the condition     if ( ... || tp->data_segs_out >= 10 ) Surely this shouldn't have been a hard-coded '10' in the first place? -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------ABDF2AAAF8722BCCC1BCBB0C Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Olivier,

I notice you've pushed code to TCP Prague, to optionally pace out IW over the estimated rtt (srtt).
https://github.com/L4STeam/linux/commit/aea3ff36380bec82804eba8e3f99e5f14342978c
Thanks for this. Some thoughts:

srtt can tend to be rather wildly wrong at the start of a connection. I'd say a high estimate will be more concerning than a low, by this reasoning:

- If the initial srtt estimate is too low, there will be a queuing delay spike, but it will never be worse than just sending IW at line rate.
- If srtt is too high, a short flow would take longer to complete because the first ACK will return before the IW has finished sending.

I recall Mark Handley presenting data showing longer RTT outliers at the start of connections (cause was unknown, but could have been a CDN getting its act together to serve the connection from an optimal location). Sorry, can't find the paper - it was perhaps 5-10 years ago. Whatever, in general, an over-high initial estimate is surely more likely than over-low, given the initial handshake might be delayed more than other packets, but it's unlikely to find some worm-hole that gets it back home faster than the speed of light.

So, possible solutions (either or both):

1. In tcp_prague.c, multiply the rate returned from prague_update_pacing_rate() by N (where N=2, say). So IW is at least paced over the first half of the RTT on average. Then, if the initial srtt estimate is longer than reality, the error has to be 2x before IW will be delayed more than a whole round.
2. Introduce a third sysctl option (tcp_pace_iw = 2) that only paces IW if srtt was initialized using the destination cache?


Altho this is in TCP Prague now, I guess eventually it could be taken up by any transport with any CC.

Cheers


Bob

PS. Defensive coding nit: In tcp_output.c, I noticed you've kept the condition
    if ( ... || tp->data_segs_out >= 10 )
Surely this shouldn't have been a hard-coded '10' in the first place?

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------ABDF2AAAF8722BCCC1BCBB0C-- From nobody Tue Feb 2 05:07:55 2021 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 559D13A1AA3 for ; Tue, 2 Feb 2021 05:07:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.151 X-Spam-Level: X-Spam-Status: No, score=-2.151 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.25, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=nokia.onmicrosoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hGaYtB92KmBJ for ; Tue, 2 Feb 2021 05:07:53 -0800 (PST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60101.outbound.protection.outlook.com [40.107.6.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D676D3A1AA2 for ; Tue, 2 Feb 2021 05:07:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UNNUy7RIh09iLhjB3KrM+kzjhFKeUB5MfFJqN7r9YAZwTk8xTv/GDDOw9aY51BQRomxZAIHfDAnRUMPdCadiKdBDtUArQj8sGibcMQ1Oh55r7dhazZclNqxX1/9ZTE32kHNsH3RBKvvL7QiFhJOo3Nb4bRq2KMoTbKrnx/QkUD2bMxv3EukELQ263dsRGFXm7MWAVhOLKU59Vm6sXdDFXqcLof18w1qqs9H0hU79CJi7aR6uQF7Rn1mJNZ9zLbtl12PtlRaf8iUVECel/cuaYiKd3PAgduTbuC9vlJKso8S4fXcqbjwj96N9iJFjrz+1w23b4MLNJGceotwyiEF/Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LC41VkjuQGBLdCQiAgAaomZKpvKjphq5yzeMc5/kzjw=; b=X9BVdtmseKN7At0P98X3eAf+MT9JzVQOPaGZevFjI0//TRqYzJtF9W1/jR+lFjHA+llf9Opnk2xy2vOP/6+or+nxphppGHN1gIK/ydUKz7LMARu7zhZb2pw1NzbulLJ8FHS7LVbO+QreJDy4ilLPF+j7bVcD6uUbk0uCaoFiChAvMvozrDEap2k5KPV848bfDBvPUmaiGAYGDhDRYbwJme/EZQVPpMuSZ+xUcja0M2NOjvVOzwig91Nk8xgcmoEAAkUr8W0bq2gxmMN/N/8oZQOdyIrBTzArokYsz90ujdFb5pnok/yxQtxj6lpqvTW8HGHdZmXa8+27CY2HSC0G4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nokia-bell-labs.com; dmarc=pass action=none header.from=nokia-bell-labs.com; dkim=pass header.d=nokia-bell-labs.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nokia.onmicrosoft.com; s=selector1-nokia-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LC41VkjuQGBLdCQiAgAaomZKpvKjphq5yzeMc5/kzjw=; b=AclvojvycmTc9bObwflMuf4ErxWazeznz5pdXbKQD4oB9aQkRliXzrAOLTNaQ+3fO4p+Kg8oCN5pWM46++X48gniPpyLWPCtZU7Z76R8PIQyw1gwz6hdFefxzPDqxctpNEN6G2IrMSuzJbugaF1ML1nWDZnW0ucNV745xUylHgY= Received: from (2603:10a6:20b:24c::5) by AM9PR07MB7282.eurprd07.prod.outlook.com (2603:10a6:20b:2c6::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.8; Tue, 2 Feb 2021 13:07:48 +0000 Received: from AM8PR07MB7521.eurprd07.prod.outlook.com ([fe80::95c9:a55b:5f52:1260]) by AM8PR07MB7521.eurprd07.prod.outlook.com ([fe80::95c9:a55b:5f52:1260%7]) with mapi id 15.20.3825.013; Tue, 2 Feb 2021 13:07:48 +0000 From: "Tilmans, Olivier (Nokia - BE/Antwerp)" To: Bob Briscoe CC: TCP Prague List Thread-Topic: Pacing IW Thread-Index: AQHW+V2JFweiH7ZHME6bivIt0mTC/apE0Vvw Date: Tue, 2 Feb 2021 13:07:48 +0000 Message-ID: References: <1ddb42c7-596b-031c-6215-84e675d08b7d@bobbriscoe.net> In-Reply-To: <1ddb42c7-596b-031c-6215-84e675d08b7d@bobbriscoe.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: bobbriscoe.net; dkim=none (message not signed) header.d=none;bobbriscoe.net; dmarc=none action=none header.from=nokia-bell-labs.com; x-originating-ip: [2a02:a03f:636b:f600:dcd3:e0c3:2103:a4a4] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 4f3a5866-cfd5-44a3-28cc-08d8c77b8a46 x-ms-traffictypediagnostic: AM9PR07MB7282: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: euPb1/QuTny+wx8fx4AuoxoNviQvKAiiADBHxX44BM7lNNTs55/kguseN2AQ1sbcsNy0MbxNSU492x0TOqWdIzIOgV+b2pG0mCdshGPEer7YdKRmA53C7gSZgpFjXx/bTxZK/qk4cwuzyudLOaowTJF0iWaciZ5k0KQzwb2q1ChPBhjpJj3G14xSjaNs9a23XZHp/tmHYiKTOc+oZpGHCHDAbfnw0fdJEfBgM+khHUlXJJ3yB7oIh7/deVPfDfQyDFWyhIWm5QEMAZI6m9e3Uk5EXRHLbQUWglO66svDrhc38/67PybukN0N0bpaAVD7Ji5nSlLms5L9hnoabPpNMmME2MuS85nUYj1O6a4Yaff4erJqykkQmqtU358R/p0L3fFJ+j6NWrIuf3SIvZMGV3ggUuEr5YDwL15EIvm1sXZ6PbgKbRfoSqhSzpDW0Nz90XkOJrFeWIxE5IlVnOdrYOy+ofqkbQ/C0pQI0q+pbP6NmGJt0lxHkpFuP5zknsB0KwuphMt/uj/6WbHBp+LIWiN8kF6fTT83STIMSukXSJDYVCRbEvCWXXwoDiuyHss8m24fT9skr3514Eyi1tqAbeg80t86PGw/+JH/yaWvRpiPDcxYyeDZfbIiYY12WXLgAHFkEZ+UiamX4qhaEvo8r69KMcIJz10fM7m/bQgO0BIdxG5UxopeEJnxKfyxEO1i x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM8PR07MB7521.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(376002)(396003)(136003)(39860400002)(8676002)(316002)(55016002)(4326008)(8936002)(7116003)(2906002)(86362001)(6916009)(186003)(76116006)(966005)(7696005)(64756008)(52536014)(71200400001)(5660300002)(33656002)(66446008)(6506007)(83380400001)(9686003)(3480700007)(478600001)(66476007)(66556008)(66946007)(336755003)(18886065003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?utf-8?B?U3J5c2hqT25sOWpnK0cza2t5OWxaZFBVNktibjYvTXBHRDJiYXJqSnpqK1Ru?= =?utf-8?B?aVpKTXVWYUtrQWRRLzI2VU5jNXF0eWxVUjI5M21uS3dQak9OQ3dhaFVValpu?= =?utf-8?B?NmJWRkE0bWlic2Vrd1A1ZGkzVW1nTmJBU09VTXZ4SExxb2Nqbnd0c2VOTzBk?= =?utf-8?B?a05jMm8yMXZOZ3lVNEJyTi9KQ0tQTnhOTk1zd2U2WGpoT3NROFE5YWtPc29L?= =?utf-8?B?dGZpMCs1MmUxMDJ0MlNvWTAzNzBGVnpUOG12NTNuTzNieG5lRzhCQkxmSi8x?= =?utf-8?B?NzV6eVBSekdLNFZHTmRKTW5vYjJCWVVBM3NMRmJlNWN6eHg0eWZ5QURkVUdJ?= =?utf-8?B?SDNVM2pBdUZIM0lxVzBRbWpVaTNmQXVndkFMc3FodEE2cEN1MVAzY3dnb1Jl?= =?utf-8?B?OGJkdmVzZWFQbzF5QnM0WTNJaXkrTXlDUWxNV08vMWloZUpwUTFVVGFLM0Jy?= =?utf-8?B?dEdrQ21BVWJ3VkJzSzB5V1o1ZnhaN1JKUWwvUXFISmhnOHRicHhSSy9kOGN5?= =?utf-8?B?eng0UE9OT3VyU0wwOUo1ejN3ZTBnY043c0h2V2xuZGFuc3ZDVE93SHJUdjNU?= =?utf-8?B?RTVhdnd3c1U4T1dGc3RycEEwWE5XY1k5RjdLa2pwcU9sd0xhL1VOOFM0ZGcx?= =?utf-8?B?WkpTWkZuSUJla0RqT0FRRzhFT3lvMXRVQTVQZHg1K3ZTNjk1bjlFZ3BSc3FZ?= =?utf-8?B?T3o1VjNrU0d6TXk5aEg1c2d2dEV3UWRrOEFEeWs4dk9MYTU3Y2tzTmQxY0Nt?= =?utf-8?B?ZU9yV2RGKzRYYWNsZEZkWit3VmJrUTI5Z01keVlSVm9RbVdVUmh2WEtsbzgy?= =?utf-8?B?QzFvbXRZZEk4cFowejFyNnN4aE1QTWdEdU52cGhkNXBiVmFxV2JlWlFlOU1l?= =?utf-8?B?OTB4eGJTay9VRHUwZnZkQjF4aGxmb25SbnVaOGtEVkRxVHNVNXJFTHdwUXh4?= =?utf-8?B?aFVDeUlaNVA2c2lCd2x3b3VjQzN1cHREUmVPenZMVzdvcUFnS0xqcTc0alhi?= =?utf-8?B?cGhtb0MzYlNQTkZ5cVRUZlJhWVRDWjh6aEkyMkN3TDFWYkZ4cE1rS1MvTUJC?= =?utf-8?B?K0JBUG1vdTlBTXBhZVRiZTBjb3lyUmE3eWFxVEtXVGRaWldXRW11UkdKN0dx?= =?utf-8?B?WDVMbTNhTlJyWnJ4QmR6dVhoVjFKWmQ1ZHVwNUppK3dqaTljS0g0WFVrdzd6?= =?utf-8?B?NzhjZm1yYW9XRVdoMVhOdzRVbXBsc2NQd3BPTDZpQzRVTS9pWDJ6MGFONWYw?= =?utf-8?B?NVE0Y0RVeGNQcDU1NFQxMElBdU82SFdsdlNCbENuc2tOY3Y5a3VVVk11cThK?= =?utf-8?B?K3hxeUZndTdvYWgzd0FVbWMrekNpQXBZSnNFVGtieVFrdndCL1c4NElzbmpm?= =?utf-8?B?MktvMm54YUYyOWhCTEFUUytlTE8zTzB4cG1TejBURkcyeWFyMnZqcnl0c0VJ?= =?utf-8?B?RTBpN1FQVDdESjJITkd4TDhRYnMwRjU4YU9MWVBoWHFYbllKNXptUG14ZTA1?= =?utf-8?Q?S2UjrZY/LT3hsbR30E1YWpSo8Eb?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: nokia-bell-labs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM8PR07MB7521.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f3a5866-cfd5-44a3-28cc-08d8c77b8a46 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Feb 2021 13:07:48.6508 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: CUpYUR8INXEwSPhPBdJ68k0UmiRHPPpcnm6nIAOi0BdszRLn6itZ31kp1a7di13dYBpPpKdtndEzc8ZmO2mMJyAOTZYIgtkDdjWtM9VomBmSKnf6fT8QdKvJfghW2oif X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR07MB7282 Archived-At: Subject: Re: [tcpPrague] Pacing IW X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2021 13:07:54 -0000 SGkgQm9iLA0KDQogPiBJIG5vdGljZSB5b3UndmUgcHVzaGVkIGNvZGUgdG8gVENQIFByYWd1ZSwg dG8gb3B0aW9uYWxseSBwYWNlIG91dCBJVyBvdmVyDQogPiB0aGUgZXN0aW1hdGVkIHJ0dCAoc3J0 dCkuDQogPiBodHRwczovL2dpdGh1Yi5jb20vTDRTVGVhbS9saW51eC9jb21taXQvYWVhM2ZmMzYz ODBiZWM4MjgwNGViYThlM2Y5OWU1ZjE0DQogPiAzNDI5NzhjDQoNCk5vdGUgdGhhdCB0aGlzIGlz IGRpc2FibGVkIGJ5IGRlZmF1bHQ6IHRoaXMgYXBwcm9hY2ggaXMgcXVpdGUgZnJhZ2lsZSBhcw0K eW91IHJpZ2h0ZnVsbHkgcG9pbnQgbGF0ZXIgb247IGl0IGVuYWJsZXMgcXVpY2sgZXhwZXJpbWVu dGF0aW9uLiBJJ2QgZXhwZWN0DQpzb21lb25lIHdpbGxpbmcgdG8gdXNlIHRoaXMgaW4gcHJvZHVj dGlvbiB0byBoYXZlIGEgZmluZXItZ3JhaW5lZCBhcHByb2FjaCwNCmUuZy4sIHVzaW5nIHRoZSBk ZXN0aW5hdGlvbiBjYWNoZSB0byBnZXQgcmVjZW50IHNydHQvY3duZCB2YWx1ZXMuDQoNCiA+IFsu Li5dDQogPiAtIElmIHRoZSBpbml0aWFsIHNydHQgZXN0aW1hdGUgaXMgdG9vIGxvdywgdGhlcmUg d2lsbCBiZSBhIHF1ZXVpbmcgZGVsYXkNCiA+IHNwaWtlLCBidXQgaXQgd2lsbCBuZXZlciBiZSB3 b3JzZSB0aGFuIGp1c3Qgc2VuZGluZyBJVyBhdCBsaW5lIHJhdGUuDQoNCkFncmVlZC0tSSB3b3Vs ZCBleHBlY3QgdGhpcyB0byBoYXBwZW4sIGUuZy4sIHdpdGggUEVQUyB0ZXJtaW5hdGluZyBhDQpj b25uZWN0aW9uLg0KDQogPiAtIElmIHNydHQgaXMgdG9vIGhpZ2gsIGEgc2hvcnQgZmxvdyB3b3Vs ZCB0YWtlIGxvbmdlciB0byBjb21wbGV0ZSBiZWNhdXNlDQogPiB0aGUgZmlyc3QgQUNLIHdpbGwg cmV0dXJuIGJlZm9yZSB0aGUgSVcgaGFzIGZpbmlzaGVkIHNlbmRpbmcuDQoNCkFncmVlZC4gTW9y ZSBnZW5lcmFsbHksIHRoYXQgdHJhZGVvZmYgaXMgZXh0cmVtZWx5IGFwcGxpY2F0aW9uIGRlcGVu ZGVudCwNCnNvIGFwcGxpY2F0aW9ucyBzZW5zaXRpdmUgdG8gdGhpcyBleHRyYSBkZWxheSB3b3Vs ZCBlaXRoZXIgbm90IHVzZSB0aGlzIG9yDQpjb250cm9sIHRoZSBwYWNpbmcgcmF0ZSB1c2VkIHRo ZW1zZWx2ZXMgYnkgb3ZlcndyaXRpbmcgdGhlIHNvY2tldCBtYXggcGFjaW5nDQpyYXRlLg0KDQog PiBbLi4uXQ0KID4gU28sIHBvc3NpYmxlIHNvbHV0aW9ucyAoZWl0aGVyIG9yIGJvdGgpOg0KID4g DQogPiAxLiBJbiB0Y3BfcHJhZ3VlLmMsIG11bHRpcGx5IHRoZSByYXRlIHJldHVybmVkIGZyb20N CiA+IHByYWd1ZV91cGRhdGVfcGFjaW5nX3JhdGUoKSBieSBOICh3aGVyZSBOPTIsIHNheSkuIFNv IElXIGlzIGF0IGxlYXN0DQogPiBwYWNlZCBvdmVyIHRoZSBmaXJzdCBoYWxmIG9mIHRoZSBSVFQg b24gYXZlcmFnZS4gVGhlbiwgaWYgdGhlIGluaXRpYWwNCiA+IHNydHQgZXN0aW1hdGUgaXMgbG9u Z2VyIHRoYW4gcmVhbGl0eSwgdGhlIGVycm9yIGhhcyB0byBiZSAyeCBiZWZvcmUgSVcNCiA+IHdp bGwgYmUgZGVsYXllZCBtb3JlIHRoYW4gYSB3aG9sZSByb3VuZC4NCg0KVGhpcyBpcyBhbHJlYWR5 IHRoZSBjYXNlLCBnaXZlbiB0aGUgY29ubmVjdGlvbiBpcyBzZWVuIGFzIGluIHNsb3dzdGFydCAo DQpzc3RocmVzaD1pbmZpbml0eSksIHRoZSBwYWNpbmcgcmF0ZSB3aWxsIGJlICgyICogSVcvc3J0 dCkNCmh0dHBzOi8vZ2l0aHViLmNvbS9MNFNUZWFtL2xpbnV4L2Jsb2IvdGVzdGluZy9uZXQvaXB2 NC90Y3BfcHJhZ3VlLmMjTDMwNg0KDQogPiAyLiBJbnRyb2R1Y2UgYSB0aGlyZCBzeXNjdGwgb3B0 aW9uICh0Y3BfcGFjZV9pdyA9IDIpIHRoYXQgb25seSBwYWNlcyBJVw0KID4gaWYgc3J0dCB3YXMg aW5pdGlhbGl6ZWQgdXNpbmcgdGhlIGRlc3RpbmF0aW9uIGNhY2hlPw0KIA0KSSdkIGhhcHBpbHkg bWVyZ2UgYW55IGFkZGl0aW9uIG9yIGJldHRlciB3YXlzIHRvIGRvIHRoaXMuDQoNCiA+IEFsdGhv IHRoaXMgaXMgaW4gVENQIFByYWd1ZSBub3csIEkgZ3Vlc3MgZXZlbnR1YWxseSBpdCBjb3VsZCBi ZSB0YWtlbiB1cA0KID4gYnkgYW55IHRyYW5zcG9ydCB3aXRoIGFueSBDQy4NCg0KSW5kZWVkLCB0 aGlzIHdhcyB0aGUgcXVpY2tlc3Qgd2F5IHRvIGdldCBpdCB0byB3b3JrLiBJdCBjYW4gZGVmaW5p dGVseSBiZQ0KYWx0ZXJlZCB0byB1c2UgdGhlIGRlc3RpbmF0aW9uIGNhY2hlLCBhbmQvb3IgdG8g dXNlIHNrX21heF9wYWNpbmdfcmF0ZSBpbiBhDQpzbWFydGVyIHdheSAoZS5nLiwhPSBpbmZpbml0 eSBpbmRpY2F0ZXMgdGhlIGFwcGxpY2F0aW9uIGV4cGxpY2l0bHkNCnJlcXVlc3RlZCBpdCBiZWZv cmUgcXVldWluZyBkYXRhIGZvciBpdHMgSVcsIHdoaWNoIGNvdWxkL3Nob3VsZCBiZSBwcmVmZXJy ZWQNCnRvIGFueSBnZW5lcmljIGluLWtlcm5lbCBjb21wdXRhdGlvbi9hc3N1bXB0aW9uKSwgYW5k L29yIHRvIGJlIENDQQ0KaW5kZXBlbmRlbnQuDQoNCg0KQmVzdCwNCk9saXZpZXINCg0KID4gDQog PiBDaGVlcnMNCiA+IA0KID4gDQogPiBCb2INCiA+IA0KID4gUFMuIERlZmVuc2l2ZSBjb2Rpbmcg bml0OiBJbiB0Y3Bfb3V0cHV0LmMsIEkgbm90aWNlZCB5b3UndmUga2VwdCB0aGUNCiA+IGNvbmRp dGlvbg0KID4gICAgIGlmICggLi4uIHx8IHRwLT5kYXRhX3NlZ3Nfb3V0ID49IDEwICkNCiA+IFN1 cmVseSB0aGlzIHNob3VsZG4ndCBoYXZlIGJlZW4gYSBoYXJkLWNvZGVkICcxMCcgaW4gdGhlIGZp cnN0IHBsYWNlPw0KID4gDQogPiANCiA+IC0tDQogPiBfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQogPiBCb2IgQnJpc2NvZSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBodHRwOi8vYm9iYnJpc2NvZS5uZXQvDQo= From nobody Fri Feb 5 02:48:39 2021 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A32843A0C22 for ; Fri, 5 Feb 2021 02:48:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.351 X-Spam-Level: X-Spam-Status: No, score=-2.351 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.25, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KVUIjTtzaR7d for ; Fri, 5 Feb 2021 02:48:35 -0800 (PST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60083.outbound.protection.outlook.com [40.107.6.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 51CA63A0BBA for ; Fri, 5 Feb 2021 02:48:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SG54YndVGDp50VWxieEovqJpgmEkxtmGUL5Ca5akgMSQPSR/DNqKSbtWfVE0QAnWiuxJK/PeuqwOdbCYWJhX48uF2TmtZ0hS8y7QI0AYPuZOytZ9VNJMV3B+xD96bYA5rJvnjLdgODewvfl0bE34ndmPsxxMnidgiDsAIZZJp42GdDvrR8xLtigLN0stO1QR89O+dbqV12q2sXEFiGHCW8YA8DgzVqzxbTLpZhJXu7/UvUKNOZT4jLVNvPblS8rBNeHUpHf4FLoOieiOK2VnwCcVkIvab83LKbKsKNjPSys83m4Mu1DuWx/P/mP+s/s5mECc2aTC7XPQxIlr9CFUTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D++sTDNSRPPejv/tf8bKX0D0jlivU3D5tWTQjrF/g7s=; b=cH6X5ph7BLagNcjGEZBoYEe4kWHfS6xCPhdPVMuRzDqWkWV002qN5prrjnHHk5kBzUckM3iJ4RfWCHME+KFfVDMDIzU4CetXY5yHmFDEQ7GMkNgPKbaxwY11EkuCCObfNJ0cIDyc7egcJ3HG47wU+eiQ2yQ0ny6g5DR8sCnwGf/AskF3CNWG944gxTt32c0KLcpudFG32mQnMIaRNRcMYrKkMa7pL3cejaaIL1SMQ9lcBQxRC4HF+oSQ3QS9xobDbBsxUZ+an3wSeoHrC90bZsDZpQJQYms/r3JEvfTz3efRBSnMvgbvj5py8MAa0ljoldciHvRhRo588hxpiMDaMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ericsson.com; dmarc=pass action=none header.from=ericsson.com; dkim=pass header.d=ericsson.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D++sTDNSRPPejv/tf8bKX0D0jlivU3D5tWTQjrF/g7s=; b=Dj3PWisIowu30orGCCNmzpNE3UAQq2Ham9RyCEfG8brk8VILKAlJx4wTokV5DXOmvuk6nKXZM9j9BYHCoWJLm0TP3M28LStjTymZ95wZectpXrTzGny8KF+lYM4qattmGodRSxXEMciz0KQNQE5aLm8xhhxSI6dths9uSf9F+eo= Received: from (2603:10a6:3:6c::8) by HE1PR0702MB3579.eurprd07.prod.outlook.com (2603:10a6:7:7e::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.12; Fri, 5 Feb 2021 10:48:31 +0000 Received: from HE1PR0701MB2299.eurprd07.prod.outlook.com ([fe80::494d:6160:61fd:5b1]) by HE1PR0701MB2299.eurprd07.prod.outlook.com ([fe80::494d:6160:61fd:5b1%9]) with mapi id 15.20.3825.020; Fri, 5 Feb 2021 10:48:31 +0000 From: Ingemar Johansson S To: "Tilmans, Olivier (Nokia - BE/Antwerp)" , Bob Briscoe CC: TCP Prague List , Ingemar Johansson S Thread-Topic: [tcpPrague] Pacing IW Thread-Index: AQHW+Z4LM/xHQu3tVkq5ODsEOVqBUqpJYn1g Date: Fri, 5 Feb 2021 10:48:30 +0000 Message-ID: References: <1ddb42c7-596b-031c-6215-84e675d08b7d@bobbriscoe.net> In-Reply-To: Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: nokia-bell-labs.com; dkim=none (message not signed) header.d=none;nokia-bell-labs.com; dmarc=none action=none header.from=ericsson.com; x-originating-ip: [83.227.122.88] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 133a7d3c-eeea-4a63-7d6e-08d8c9c393ea x-ms-traffictypediagnostic: HE1PR0702MB3579: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: aPKvquRAPYVYpsO6OBnaLAEr7ZH5QFEhWHDFuP6XvvXibcnCmPWDoSL+qn/CoBjOLjYRG2ZIAvmGjrYB6e7Daf4eY5SPJe20E8ZgxQ9VS21AuaOEFPZMFQmHbfjacS036DfwhvtaQ9BG7YN3gZYgXQZCvFQ4Xg2uAp8/vsIYviOdMUCdEbs80HfZaGEShv4CajAb4bYt12U5jPiiI07IUJGBytD2t4GoQ36SV/fXJt3IbI8CaO/gllh1DuPHIOHU0e8DXpTbbepU1kWw72QFlE37/+Vd+261jFUsniCHwHJJbDIn69JoYQ7z6iWQN6R6GA7OhcKB2vwJRxhxjntySrk3zXuI6t3xpEg58RmNJNCBnkD3x7v1SF1TT+Nlb8A8+/pXPc8i7m90iYcbVPsnsxvjEGuUu9G5TWQuH7e353H1BUQ1eoDIoBDKzd0IqARg3xUyQw4v3loo9rG6d0qELOReQx7PAnsqdX2Towb75ceN2pLHzXR49aUUz94tiKiOkyISUFEB2k+rYcx7I2ZxxRQZDGc48JhHw5a9utV2RjgDWs4/2nSbFrIa8V9eOEJvTbJdF8UtartdXqCeJ9XoXDesIGTuwffD6qAhrY9Z9oaCh1R9qpYjGTQUNDj+RD6Vz6ogh+B/GjCqVhJ38YLIGPN8DibTfYLFt+MCHVR8Csh6D6RHwDok086ILLAp6r+6 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:HE1PR0701MB2299.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(54906003)(110136005)(2906002)(53546011)(6506007)(966005)(9686003)(5660300002)(55016002)(107886003)(86362001)(33656002)(66556008)(64756008)(4326008)(66616009)(66476007)(83380400001)(66946007)(8676002)(186003)(66446008)(52536014)(26005)(71200400001)(7696005)(8936002)(99936003)(498600001)(76116006)(336755003)(18886065003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?utf-8?B?N0p6MHYzbE1ZRTNlK1lXaU9ldHFZOXpaUXNaTVNBSFIwWTVaK2tId0FCQUZr?= =?utf-8?B?M3NtSjI0WStuRDVXREFuVWxyOW5EUVNZSVI4aHNGd0JzRXViVmlTRkVsWVVD?= =?utf-8?B?ZE1RSUdkbVFTOGg4Y2twTzQybGFWUkhRUHZORXJneTEyaXlKbU9XK2tVQ2VR?= =?utf-8?B?Q1dSanFCeDNoNTg2YSs0V1B5eCtaZHIvRDhST1JWYnpJUE53bmIzeTZaUVht?= =?utf-8?B?TEpBcVlzdmRmYXlndSs1UkxYejkvalNhS3d6TjhsYzQwZE1uRmNFNXdLbi9P?= =?utf-8?B?aE1oQnNkTGY5SFIydnFMbzd6enBIb0l4dWw1V2lOSitYS1NzeUhGUk1lZGZa?= =?utf-8?B?aHpKazRQSTF4d1lwNVdRek0xaUpzY1VyUW02WmUxQ1lZR3pMU1FtSGJOai9V?= =?utf-8?B?VUJINDRsRlZ0Tyt2Q1RsR0FDMllTczMxNVBtajdHOEJIT0YrUlpPeThDWlgr?= =?utf-8?B?K1JOeTBWU2laMnRFajJsckNWOWYrejRhNGJIT0paRWh1T0F4TjhJU1QxUi9r?= =?utf-8?B?dGI4SHNpSlRaRUNtcllyUmxXRTZWTWxyeHF6WDAwVzg4OC9RbUVjRlF6ZEVw?= =?utf-8?B?M1RHOHJoMUUwcXdmS2tCNkN2aVBzWWxmeFc2RXNzaWVDZElQODU5ajh2MjRI?= =?utf-8?B?dHF5cHRvUmRlL3EySjVnZDdXZ0NHRUVGUjJlaWhNU1RYZENZMEE4K1pRQU5m?= =?utf-8?B?Y3JwTmQ1TWlKeERpcXMrVE82MHFXRER4VENzUStORkk1S2wzejBxYVpkZkxj?= =?utf-8?B?UjJjZ1BtRW5hcEJrZXY2OUt1ZXNDWk5NL2UwUFhaeGkraEQxc1c1Z1ZZczZQ?= =?utf-8?B?UUhxTmlHZW14U1dIVTRWdjIrUmJMTC9yS3RyaDNqbUE2TnNhSDNqZmhyUGpq?= =?utf-8?B?c2lyV1VzRnJCNm1Kc2N6OUgvTnMrTU5yZmRYY1IwdUN0K2Y2SnM0ZUhQcGFX?= =?utf-8?B?SEpueGRROG5kZHZGRHVyK243SUxMeEVtck5JZytRS3RIZk5adU45bXdiTHFE?= =?utf-8?B?UEFnbC9xK3Y4c3dZaVlSN2M3L1BxOWJJWmdBa21rb2doRFJQV1hjb3A4ZVpU?= =?utf-8?B?SUdWelNZczUzVDIxek13b2tFVG0wektySjFZNGt2VVdXQURxV2Z3aDNuMjhx?= =?utf-8?B?VEp4RVhSdGNZT2ZBdkhob3JETVpxdUt6QnlTWTRYcUsvVXhLVU5yRlM4anlZ?= =?utf-8?B?RVNwNC9TTjdZbitWWXFnSXRlTzBpY2k4UWtJQzc0R3duN1JSeURoYTlWOVQ3?= =?utf-8?B?a1g5c3pQTnZnSURKR0FwOUk1UWdBQ2dhN3RRQXVKd051WUZ5aDFqSDQzVmVP?= =?utf-8?B?VTFCQ0orTWxYdTYwRURMdkd1MTN5cFRRSVJnL1FBSU5CYjVpMmZuN3FXcFVM?= =?utf-8?B?aXQrMXAyazM0NDNabkdwNVh5NXQwbmV3RmVzOXZ0Vm9LK1AyWlpidW15QWl4?= =?utf-8?B?bTJSTTFZcUplOEtoR0xrUjVUa05ydDNoUE1IRWlhbEE0Vk1DeWZMMzNGMStT?= =?utf-8?B?eWk2TDBZMlNGRStaWG1kdldjREszbXA0Rnh0Y3l5K2V3TS96Mm5UVm8wRHdo?= =?utf-8?B?YUJoVGlxQkdSdzNzYWhHMTNCd0laN2JkaXc0SDQvSFB3SlFxd1NTSElOMWxi?= =?utf-8?B?Q3NJdldJTE82akd0MTVhZHlqNGoyM0xpREVpL1ZYQXYySEsreWtmQ1BMVUhS?= =?utf-8?B?YUI0bWsrMUFKWFhDZ2N3ckhLVWFZSzVDbFpmeFpkdjZqc0xITG9tVGQrblpK?= =?utf-8?Q?IXUaIpgfHY+1QwY8QvgDTX/6Ry0J4Kn8Fuz5BFy?= Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_021A_01D6FBB4.D25C55C0" MIME-Version: 1.0 X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: HE1PR0701MB2299.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 133a7d3c-eeea-4a63-7d6e-08d8c9c393ea X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Feb 2021 10:48:30.9069 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ud9Duw2wWyuVWe2B1TSMnmxuElOdsjS5/Ox8Iv1MKLiP0DPn11cSA6Q+bxbWEQ3M/NQoVAnIxzOgEtHWkNuFMkJkU+zRUiYQyZoA4EO8hxiOaj9AvyKnOhGE2AGKE0UW X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0702MB3579 Archived-At: Subject: Re: [tcpPrague] Pacing IW X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 10:48:38 -0000 ------=_NextPart_000_021A_01D6FBB4.D25C55C0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit On the pacing matters based on (initial RTT). It is true that an initial RTT based on e.g. 3WHS can give values that are higher than the actual values when for instance cellular access is used. There are a few reasons to this : 1) DRX (Discontinous reception) : A battery saving feature that makes the cell phone (a.k.a UE) sleep periodically. This feature is noticeable e.g. when ping measurements are done 2) UL scheduling : To transmit in uplink, an UE must first send a scheduling request (SR) to the base station, the base station then transmits a grant a short while after. The grant may however be too small, in which case the UE must piggy back a buffer status report (BSR) to the transmitted data. All this gives the potential that the RTT may become higher than the true value /Ingemar > -----Original Message----- > From: Tilmans, Olivier (Nokia - BE/Antwerp) labs.com> > Sent: den 2 februari 2021 14:08 > To: Bob Briscoe > Cc: TCP Prague List > Subject: Re: [tcpPrague] Pacing IW > > Hi Bob, > > > I notice you've pushed code to TCP Prague, to optionally pace out IW over > > the estimated rtt (srtt). > > > https://github.com/L4STeam/linux/commit/aea3ff36380bec82804eba8e3f99 > e5f14 > > 342978c > > Note that this is disabled by default: this approach is quite fragile as you > rightfully point later on; it enables quick experimentation. I'd expect > someone willing to use this in production to have a finer-grained approach, > e.g., using the destination cache to get recent srtt/cwnd values. > > > [...] > > - If the initial srtt estimate is too low, there will be a queuing delay > > spike, > but it will never be worse than just sending IW at line rate. > > Agreed--I would expect this to happen, e.g., with PEPS terminating a > connection. > > > - If srtt is too high, a short flow would take longer to complete because > > > the first ACK will return before the IW has finished sending. > > Agreed. More generally, that tradeoff is extremely application dependent, > so applications sensitive to this extra delay would either not use this or > control the pacing rate used themselves by overwriting the socket max > pacing rate. > > > [...] > > So, possible solutions (either or both): > > > > 1. In tcp_prague.c, multiply the rate returned from > > prague_update_pacing_rate() by N (where N=2, say). So IW is at least > > paced over the first half of the RTT on average. Then, if the initial > > srtt > estimate is longer than reality, the error has to be 2x before IW > will be > delayed more than a whole round. > > This is already the case, given the connection is seen as in slowstart ( > ssthresh=infinity), the pacing rate will be (2 * IW/srtt) > https://github.com/L4STeam/linux/blob/testing/net/ipv4/tcp_prague.c#L30 > 6 > > > 2. Introduce a third sysctl option (tcp_pace_iw = 2) that only paces IW > > if > srtt was initialized using the destination cache? > > I'd happily merge any addition or better ways to do this. > > > Altho this is in TCP Prague now, I guess eventually it could be taken up > > > by any transport with any CC. > > Indeed, this was the quickest way to get it to work. It can definitely be > altered to use the destination cache, and/or to use sk_max_pacing_rate in a > smarter way (e.g.,!= infinity indicates the application explicitly requested > it > before queuing data for its IW, which could/should be preferred to any > generic in-kernel computation/assumption), and/or to be CCA independent. > > > Best, > Olivier > > > > > Cheers > > > > > > Bob > > > > PS. Defensive coding nit: In tcp_output.c, I noticed you've kept the > > condition > > if ( ... || tp->data_segs_out >= 10 ) > > Surely this shouldn't have been a hard-coded '10' in the first place? > > > > > > -- > > > __________________________________________________________ > ______ > > Bob Briscoe http://bobbriscoe.net/ ------=_NextPart_000_021A_01D6FBB4.D25C55C0 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIVaTCCAyAw ggIIoAMCAQICAR0wDQYJKoZIhvcNAQEFBQAwOTELMAkGA1UEBhMCRkkxDzANBgNVBAoTBlNvbmVy YTEZMBcGA1UEAxMQU29uZXJhIENsYXNzMiBDQTAeFw0wMTA0MDYwNzI5NDBaFw0yMTA0MDYwNzI5 NDBaMDkxCzAJBgNVBAYTAkZJMQ8wDQYDVQQKEwZTb25lcmExGTAXBgNVBAMTEFNvbmVyYSBDbGFz czIgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQF0o1ncrwDZbHRPoWN/xIvb1/ gC01O+FvqGepvwMcTYxvMkfVQWikEwTBNQyahEP8XB3/ibPoFxjNkV/7iePqv05dfBsm03V57eaE 41flrSnE9Doo56V7hDZps/1edr2jLZnTkE4jKH0YY/FUOyaddluXQrL/rvBO7N05lU6DBn/nSUDI xQGyVFpmHT38+ek8Cp6BuHDwAYvkI1R8yK74kB4AlnLUVM9hI7zq+50CldG2uXE6aQg/D7ThQseI 9T+YqKe6HOBxce9YV4FQelxrdEYOgwOYw46obvJ2Mm4ng8Jz89wY6LST6nVEawRgIHFXh53zvqCQ Iz2KJOHaIdvDAgMBAAGjMzAxMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0OBAoECEqgqliE0148MAsG A1UdDwQEAwIBBjANBgkqhkiG9w0BAQUFAAOCAQEAWs6H+RZyFVdLHdmb56ImMOyTZ9/WLdI0r/c4 pc6rFrmrL3w1y6zQD7RMK/yA72uMkV82dvfbsxsZ6vSyEf1hcUS/KLM6Hb+zQ+ifv9wxCHGwnY3W NEcykMZlJPegSnwEc485bxeMcrW9S8h6+HuDwyhOnAnqZz+yZwQbwxTa+OdJJJHQHWr6YTnva+ch dQYH2BK0ISBwQnGB2jyaNr6mWw1qbJofkXv5+e9Cuk5OnswMjZTc2UWcXuxCUGOu9F3EsRLcyjuo Lp0UWgV1t+zXY+K6NbYECJHo2p2c9ma1GKwKplQmNDPSG8HUfxo6jguqMm7b/E8ln9kyx5ZacKzf TDCCBX0wggRloAMCAQICEQCH7S4aKCZKxRmqOuu5DaLLMA0GCSqGSIb3DQEBCwUAMDkxCzAJBgNV BAYTAkZJMQ8wDQYDVQQKEwZTb25lcmExGTAXBgNVBAMTEFNvbmVyYSBDbGFzczIgQ0EwHhcNMTQx MjA1MDgxOTE1WhcNMjEwNDA1MTAyOTAwWjA3MRQwEgYDVQQKDAtUZWxpYVNvbmVyYTEfMB0GA1UE AwwWVGVsaWFTb25lcmEgUm9vdCBDQSB2MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIB AMK+6yfwIaPzaSZVfp3FVRaRXP3vIb9TgHot0pGMYzHw7CTww6XScnwQbfQ3t+XmfHnqjLWCi65I tqwA3GV17CpNX8GH9SBlK4GoRz6JI5UwFpB/6FcHSOcZrr9FZ7E3GwYq/t75rH2D+1665I+XZ75L jo1kB1c4VWk0Nj0TSO9P4tNmHqTPGrdeNjPUtAa9GAH9d4RQAEX1jF3oI7x+/jXh7VB7qTCNGdMJ jmhnXb88lxhTuylixcpecsHHltTbLaC0H2kD7OriUPEMPPCs81Mt8Bz17Ww5OXOAFshSsCPN4D7c 3TxHoLs1iuKYaIu+5b9y7tL6pe0S7fyYGKkmdtwoSxAgHNN/Fnct7W+A90m7UwW7XWjH1Mh1Fj+J Wov3F0fUTPHSiXk+TT2YqGHeOh7S+F4D4MHJHIzTjU3TlTazN19jY5szFPAtJmtTfImMMsJu7D0h ADnJoWjiUIMusDor8zagrC/kb2HCUQk5PotTubtn2txTuXZZNp1D5SDgPTJghSJRt8czu90VL6R4 pgd7gUY2BIbdeTXHlSw7sKMXNeVzH7RcWe/a6hBle3rQf5+ztCo3O3CLm1u5K7fsslESl1MpWtTw EhDcTwK7EpIvYtQ/aUN8Ddb8WHUBiJ1YFkveupD/RwGJBmr2X7KQarMCpgKIv7NHfirZ1fpoeDVN AgMBAAGjggGAMIIBfDBOBggrBgEFBQcBAQRCMEAwPgYIKwYBBQUHMAKGMmh0dHA6Ly9jYS50cnVz dC50ZWxpYXNvbmVyYS5jb20vc29uZXJhY2xhc3MyY2EuY2VyMA8GA1UdEwEB/wQFMAMBAf8wGQYD VR0gBBIwEDAOBgwrBgEEAYIPAgMBAQIwDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBTwj1k4ALP1 j5qWDNXr+nuqF+gTEjCBuQYDVR0fBIGxMIGuMG+gbaBrhmlsZGFwOi8vY3JsLTEudHJ1c3QudGVs aWFzb25lcmEuY29tL2NuPVNvbmVyYSUyMENsYXNzMiUyMENBLG89U29uZXJhLGM9Rkk/Y2VydGlm aWNhdGVyZXZvY2F0aW9ubGlzdDtiaW5hcnkwO6A5oDeGNWh0dHA6Ly9jcmwtMi50cnVzdC50ZWxp YXNvbmVyYS5jb20vc29uZXJhY2xhc3MyY2EuY3JsMBMGA1UdIwQMMAqACEqgqliE0148MA0GCSqG SIb3DQEBCwUAA4IBAQAQ1elFTM6fGkQ/aRKdkUZicO3Cb9uzBJOpOtFctw+1El0/17lsjoVvJkZB D3KnUobnrriFdAa+7FAN55KLmZeB/3Y2bG0bB4toSyaVHjOQnQY9M0dv8U852w0Q7GwchKfebLUI bh9TMt2hI3Xc6j4knFTBUo7C1WAfO51K4bn1irmX6/Ej2VTgiOFsvOAny28W6enFSEQpSHw60VhN fSttSqTOxyrRR/7kW7Y8yb/3DZDZ/dH6ZCfx/y+BNIv2NuSd85M9HXUzplXXohti4Ql/qeaMn6by Ius6XlMWZZfkdVRvTuk2PkeC7UmAJ2+/DUWOPpawaytMXVfF4Hvxk34NMIIF+jCCA+KgAwIBAgIP AXWsTwOGurO54gJ+lN9MMA0GCSqGSIb3DQEBCwUAMEcxCzAJBgNVBAYTAlNFMREwDwYDVQQKDAhF cmljc3NvbjElMCMGA1UEAwwcRXJpY3Nzb24gTkwgSW5kaXZpZHVhbCBDQSB2MzAeFw0yMDExMDkw OTIwNTlaFw0yMzExMTAwOTIwNTlaMGIxETAPBgNVBAoMCEVyaWNzc29uMRwwGgYDVQQDDBNJbmdl bWFyIEpvaGFuc3NvbiBTMS8wLQYJKoZIhvcNAQkBFiBpbmdlbWFyLnMuam9oYW5zc29uQGVyaWNz c29uLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKM8NLGONT21wL/w2E/7OdQi eLiQdF9r2oH7aowAVCWcHrdaIg+RidSZj9MVEjymz2GNC+/NuJRpssy9ueaT3SeJYqw7rBBEDeIs bwqR4dB3HnbH4nJEJ+/RWt/M6qvkGWRIaKXWulA2Gpi4ogD8w7qcp+g2SOXod7Zk4ieqOvi3Bfqf a5TG8BLWxb8iOo/mtf6zRE8XDpe522ZcPbl3cl0JH9Ec57wS7Q4wtsTxqJKoFBbFrTwxV1M+MZq2 j4qYlJYy0R9dsXPOlFYN+iDoDNi0lHLLLg+eiSgFXf3mQ4wUTuZpc6WJ4lpkOooRjFcR7lTaIsCF 4viQbxnEe2ihL4sCAwEAAaOCAcYwggHCMB8GA1UdIwQYMBaAFBx7GZ6XnHasID3Y3OORauPbLaZT MB0GA1UdDgQWBBQWsmSmqV2RO4zberN0nQ1XJ2q3fjAOBgNVHQ8BAf8EBAMCBaAwVQYDVR0gBE4w TDBKBgwrBgEEAYIPAgMBARIwOjA4BggrBgEFBQcCARYsaHR0cHM6Ly9yZXBvc2l0b3J5LnRydXN0 LnRlbGlhc29uZXJhLmNvbS9DUFMwKwYDVR0RBCQwIoEgaW5nZW1hci5zLmpvaGFuc3NvbkBlcmlj c3Nvbi5jb20wSAYDVR0fBEEwPzA9oDugOYY3aHR0cDovL2NybC50cnVzdC50ZWxpYS5jb20vZXJp Y3Nzb25ubGluZGl2aWR1YWxjYXYzLmNybDAdBgNVHSUEFjAUBggrBgEFBQcDBAYIKwYBBQUHAwIw gYIGCCsGAQUFBwEBBHYwdDAoBggrBgEFBQcwAYYcaHR0cDovL29jc3AyLnRydXN0LnRlbGlhLmNv bTBIBggrBgEFBQcwAoY8aHR0cDovL2NhLnRydXN0LnRlbGlhc29uZXJhLmNvbS9lcmljc3Nvbm5s aW5kaXZpZHVhbGNhdjMuY2VyMA0GCSqGSIb3DQEBCwUAA4ICAQCkqYizUblUVAO2Hd8qacFV0dUO cJ4M/ZOMfrm/6KByKPObAVjz2j8eQ9O6oiaE/Wwhqv5XAVeE+F1dxhajzbuav+NjRHqMfOGLYsgM BLZkz/F9yHVNAo6YUtfABs2maFgOrfezn31tGupRCLb12H2MjGcMTHGNEqyXUTysrjMIyz87h5l5 5Q7eXelpaXYGKY+2W0JXSzNwek6jhipnXcjaLp+4cA5Oj4RwWxORl7HTCOStsbXIv91Hl/05+8qR XIAESI1emcrd9QjJ0r1b9MSlGMB4ORfMXfRutKn6/+m5vnyeq9JT4qGSJWZi3PN1Z4D6QMY8xDs+ 6tebCfPYzXzvNYvxPbyTxbNPi1FNszXbQpo4y3LvP7wuQJat7wUmD2CmxIv9Qu7OKkmKzT6sndY0 4VRJdQb2S+q+onf/g+WE4JPNS2dQ9oTPu6JwdYaWCAE80D2Sz7NSGncJkywh/e78EhyTfJGrm7o8 dpkuT6+IxWV/kiIVQchx9vTOZGvWhN4XcFaB+EuKzgNHlyDSmYP3E9UQ5MtSemsaUgRI6xk2adt4 tf2qN/CF24K93BIM6YPj+eKB7Y662IGwOye/OIL1p5XprdQV4uKfazQ0ZiawYErTK/qiCdQG0+n1 ORDMeVSUbznRwL5sklhsHeNGrJd/G7pU0UpZtDan4Or+8FERdTCCBsIwggSqoAMCAQICEFO4foPh nJkok7CbSRzsuOswDQYJKoZIhvcNAQELBQAwNzEUMBIGA1UECgwLVGVsaWFTb25lcmExHzAdBgNV BAMMFlRlbGlhU29uZXJhIFJvb3QgQ0EgdjEwHhcNMTUxMDI3MTIxNjQ2WhcNMjUxMDI3MTIxNjQ2 WjBHMQswCQYDVQQGEwJTRTERMA8GA1UECgwIRXJpY3Nzb24xJTAjBgNVBAMMHEVyaWNzc29uIE5M IEluZGl2aWR1YWwgQ0EgdjMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDs8t8AALhQ 8qe72FS3xpP348GqO9TDRjS0s85eQ7Y0LTLZdmSz2cl+lYqs0zfSTm+7meisbhkqUXkL7fFzoe4i IZCh/VuYUaW407CZlDCXes4n4TqTSuoklN6uOPhY7EC9ZVbXILlLhRummTdDdxhVW4Leo0awEhfL f98MvWxzwCHzMj8m6YOmNjx+f9TcJE3qaA0piuvSxlfpVdiCulPTlmsmV2RSBSAwqBshZYRcQBID fqmdvkaoP9EzNKAh7yjthC0hpgHZyZMIs0eNo4v2PUmE0rhu+Zs0nujnwhljPA2/8b8v9tGixD1z btT7zoM2Ot1menJpFp4zJVSfdKVgtoWqg5t2H/E0XY1LwJez89W07nscEocyBmpC+zJAmKxKhzEW qIyP1UrZaEIFu+hO+s0Nm8sOUMa4TlG4rAUikc5U5TmUIGBRQGxulYhfAzqSYf8oLUMLky1DOa9e Ru3sp0FdQDEzQlnF/h1L4AK1MOkX1vS+fLgOvBo5LRU1fLPUZQ7FKrDXC6nl2ldvEtljHWstGBmq v25aEvAA+yrrplCh/kYvSBjvZibz9Obbwx4yqS77/NHN1iyZyVP2s52B2BLdvo4yhzk6nRk8S/8z HaUUkBUrrvijPDaGK5FNVSaioGvkC7IKioITKffYLtT9XuirKrHlh3VzkazG46pAVwIDAQABo4IB uDCCAbQwgYoGCCsGAQUFBwEBBH4wfDAtBggrBgEFBQcwAYYhaHR0cDovL29jc3AudHJ1c3QudGVs aWFzb25lcmEuY29tMEsGCCsGAQUFBzAChj9odHRwOi8vcmVwb3NpdG9yeS50cnVzdC50ZWxpYXNv bmVyYS5jb20vdGVsaWFzb25lcmFyb290Y2F2MS5jZXIwEgYDVR0TAQH/BAgwBgEB/wIBADBVBgNV HSAETjBMMEoGDCsGAQQBgg8CAwEBAjA6MDgGCCsGAQUFBwIBFixodHRwczovL3JlcG9zaXRvcnku dHJ1c3QudGVsaWFzb25lcmEuY29tL0NQUzBLBgNVHR8ERDBCMECgPqA8hjpodHRwOi8vY3JsLTMu dHJ1c3QudGVsaWFzb25lcmEuY29tL3RlbGlhc29uZXJhcm9vdGNhdjEuY3JsMB0GA1UdJQQWMBQG CCsGAQUFBwMCBggrBgEFBQcDBDAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFBx7GZ6XnHasID3Y 3OORauPbLaZTMB8GA1UdIwQYMBaAFPCPWTgAs/WPmpYM1ev6e6oX6BMSMA0GCSqGSIb3DQEBCwUA A4ICAQBQWGvx1Yw7tC6rV0PIjKfDyxaanIX+NZLEGOkdQLKGW2gVLtDUJQEPRs5QtaZiObNHCZ7m mSNMVek4lkt/0dqfVIFutVw/QkyFGwC99ZmNwXSX9z+OoMyoEBHGvw5RY6vRlZrj0uKvdASzYL4K MaB7m3NwurNDmmNbG52suRIZ76wBOEOddRZcZiTy50ZkBqYnnl2t3D3oBX2NZCQysshUcqRdUbkS 13HTCIChMuTV9W0tzPXUOJoJlJlU9nd91IikhGEOrPwfixWms+C8sF0r9qN1uJGx6ELPOiFrLfNt cMNMMbAqRHwpSLxe3wcNkJGxv9T8LswLi1UrRIQ85AKjqzBnLSsjRGgbMgJ+xKtngmvEA155JmoK fUD7DRbP6Kp14/Y9XFbR/WuDj84bYNKXe4HdDc1P+UMYm16m2L6LkIIoRlx0A5mi+K7jewuGqzFK kaPNmJ0RLCi+4d4/47Zs3DC3PUNOxdOEEHf4kkdWOaSIuj3TQYhNv+LsgF0uijiBmaz2zUFDa2bc IkKakDZfAFM4HoHz8K2BZRaHKWhd3dZua/tlSiqokUFX2DxmHmZ1n5HM9OiaAIXP/Zo2x10j/Yb1 mM3i0bqGahxlHYzl/QyEG/dujp3lewuVjCI0mPDkZGphvxyqp4Jo8qS94EnOqBvxOgftYug7OY9E KY+WkDGCAv8wggL7AgEBMFowRzELMAkGA1UEBhMCU0UxETAPBgNVBAoMCEVyaWNzc29uMSUwIwYD VQQDDBxFcmljc3NvbiBOTCBJbmRpdmlkdWFsIENBIHYzAg8BdaxPA4a6s7niAn6U30wwCQYFKw4D AhoFAKCCAXowGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjEwMjA1 MTA0ODI5WjAjBgkqhkiG9w0BCQQxFgQUnhSON/c+IKa/8iZ+bWfPMhgjp9YwQwYJKoZIhvcNAQkP MTYwNDAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAhow aQYJKwYBBAGCNxAEMVwwWjBHMQswCQYDVQQGEwJTRTERMA8GA1UECgwIRXJpY3Nzb24xJTAjBgNV BAMMHEVyaWNzc29uIE5MIEluZGl2aWR1YWwgQ0EgdjMCDwF1rE8DhrqzueICfpTfTDBrBgsqhkiG 9w0BCRACCzFcoFowRzELMAkGA1UEBhMCU0UxETAPBgNVBAoMCEVyaWNzc29uMSUwIwYDVQQDDBxF cmljc3NvbiBOTCBJbmRpdmlkdWFsIENBIHYzAg8BdaxPA4a6s7niAn6U30wwDQYJKoZIhvcNAQEB BQAEggEAKPHkqBeTJph5vjbc7ORRYvmqaZzZUk1IwOr2vg2slrve64j6hk/0yemfE+olCstikrr/ EBxd/oQ19wGoHIDVSttrRIXsehQzjNk11zvNfMfX2ydbC8lswQcI3PuE8u3z1Jv4RSIdDW627fXD dxqs4sDnjlslRWBKwssTNXHGWR1UhX8tHs/fkFHyPk+3r3vj0nZQQxghuuYCngCzr1YO1pMZlpxc yyQuDmNlKYc+mI2Qq3k5+is6GdX2qNOciQ6e5lVF0V6Wkiho5nXeYW3D4Edo5j56giZxiqaK/xPa lhxPvRgQsPBUVEIMC5+lW3sEaRnzRc2Iy6JW/qiTUDYHkwAAAAAAAA== ------=_NextPart_000_021A_01D6FBB4.D25C55C0-- From nobody Fri Feb 5 06:58:43 2021 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CC1493A11F6 for ; Fri, 5 Feb 2021 06:58:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.434 X-Spam-Level: X-Spam-Status: No, score=-1.434 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p9spw-qEeHtC for ; Fri, 5 Feb 2021 06:58:38 -0800 (PST) Received: from mail-ssdrsserver2.hosting.co.uk (mail-ssdrsserver2.hosting.co.uk [185.185.84.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CD67E3A11F5 for ; Fri, 5 Feb 2021 06:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=MU+UIimK2SQ7CKoqPf6lj5X4yc4Hh2gaUWg3RBi6o3U=; b=AOqzbFb0te0M2miOFC3fOMo818 fViNmv32FUOsdgbFl9kmwGSM+uJ4iDiErh2bmVUeckVl/IwHPkLOLT5attO8FGznwYaFrkKzNoFoY hcAMqzvlaiTGNjBZsCtan6Em4zcNdtKXjITtkTMdtxLOHTWyVhOLPqLAkpRB75jhWo5O9oWf9e8UE Q31Cvh0yr73m4jLE5CFX7q9uhOA8HheSGCZDW0j45Z2RIgXkGYbsL85l++AeDQDsFQ6sVLNIG3u5T 1gacNQg7byuTTJ2q47tk2AyUxNkPvlr6ZHTJV697iS3PTx33DV6NpzeT5Z22wkngydsNGdPfthA+0 HtuOM6Ow==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:56086 helo=[192.168.1.11]) by ssdrsserver2.hosting.co.uk with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1l82Z9-0003hX-Gm; Fri, 05 Feb 2021 14:58:35 +0000 To: Ingemar Johansson S , "Tilmans, Olivier (Nokia - BE/Antwerp)" Cc: TCP Prague List References: <1ddb42c7-596b-031c-6215-84e675d08b7d@bobbriscoe.net> From: Bob Briscoe Message-ID: Date: Fri, 5 Feb 2021 14:58:34 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hosting.co.uk X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hosting.co.uk: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hosting.co.uk: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpPrague] Pacing IW X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 14:58:41 -0000 @Olivier (I've responded to your response in line tagged [BB]) TL;DR: Your point about already pacing over half the RTT estimate probably means the code is good enough already. @Ingemar, Thx for this. The data I mentioned that Mark Handley presented was for fixed networks. But yes, of course, I should have mentioned that mobile networks exhibit these (different) extra delays when getting a flow started. more in response to Olivier below... On 05/02/2021 10:48, Ingemar Johansson S wrote: > On the pacing matters based on (initial RTT). > > It is true that an initial RTT based on e.g. 3WHS can give values that are > higher than the actual values when for instance cellular access is used. > There are a few reasons to this : > 1) DRX (Discontinous reception) : A battery saving feature that makes the cell > phone (a.k.a UE) sleep periodically. This feature is noticeable e.g. when ping > measurements are done > 2) UL scheduling : To transmit in uplink, an UE must first send a scheduling > request (SR) to the base station, the base station then transmits a grant a > short while after. The grant may however be too small, in which case the UE > must piggy back a buffer status report (BSR) to the transmitted data. > > All this gives the potential that the RTT may become higher than the true > value > > /Ingemar > >> -----Original Message----- >> From: Tilmans, Olivier (Nokia - BE/Antwerp) > labs.com> >> Sent: den 2 februari 2021 14:08 >> To: Bob Briscoe >> Cc: TCP Prague List >> Subject: Re: [tcpPrague] Pacing IW >> >> Hi Bob, >> >> > I notice you've pushed code to TCP Prague, to optionally pace out IW over >>> the estimated rtt (srtt). >> > >> https://github.com/L4STeam/linux/commit/aea3ff36380bec82804eba8e3f99 >> e5f14 >> > 342978c >> >> Note that this is disabled by default: this approach is quite fragile as you >> rightfully point later on; it enables quick experimentation. I'd expect >> someone willing to use this in production to have a finer-grained approach, >> e.g., using the destination cache to get recent srtt/cwnd values. >> >> > [...] >> > - If the initial srtt estimate is too low, there will be a queuing delay >> > spike, >> but it will never be worse than just sending IW at line rate. >> >> Agreed--I would expect this to happen, e.g., with PEPS terminating a >> connection. >> >> > - If srtt is too high, a short flow would take longer to complete because >> > >> the first ACK will return before the IW has finished sending. >> >> Agreed. More generally, that tradeoff is extremely application dependent, >> so applications sensitive to this extra delay would either not use this or >> control the pacing rate used themselves by overwriting the socket max >> pacing rate. [BB] The tradeoff is between potentially delaying others with a queue spike vs. pacing too slowly and delaying your own flow completion a little. in most cases, getting this tradeoff wrong either way should only lead to slight extra delay (relative to the base RTT). So, the goal of my email was merely to try to reach a better default position. Then few (if any) apps would have to fiddle with socket options. >> >> > [...] >> > So, possible solutions (either or both): >> > >> > 1. In tcp_prague.c, multiply the rate returned from > >> prague_update_pacing_rate() by N (where N=2, say). So IW is at least > >> paced over the first half of the RTT on average. Then, if the initial > >> srtt >> estimate is longer than reality, the error has to be 2x before IW > will be >> delayed more than a whole round. >> >> This is already the case, given the connection is seen as in slowstart ( >> ssthresh=infinity), the pacing rate will be (2 * IW/srtt) >> https://github.com/L4STeam/linux/blob/testing/net/ipv4/tcp_prague.c#L30 >> 6 [BB] Good point. So, I suspect it could be a good enough default as it stands. It might be useful to allow the app (or the user via a sysctl?) to set a different value of N during IW than during slow start (for people who want to find a good default N empirically rather than assuming N=2 is good enough). Whether N is the same as for SS or different, this is a nice simple mechanism. >> >> > 2. Introduce a third sysctl option (tcp_pace_iw = 2) that only paces IW >> > if >> srtt was initialized using the destination cache? >> >> I'd happily merge any addition or better ways to do this. >> >> > Altho this is in TCP Prague now, I guess eventually it could be taken up >> > >> by any transport with any CC. >> >> Indeed, this was the quickest way to get it to work. It can definitely be >> altered to use the destination cache, and/or to use sk_max_pacing_rate in a >> smarter way (e.g.,!= infinity indicates the application explicitly requested >> it >> before queuing data for its IW, which could/should be preferred to any >> generic in-kernel computation/assumption), and/or to be CCA independent. [BB] I just thought of a possible third element to a default solution. TL;DR: I don't think it's worth implementing by default - too little benefit unless you have cached info, in which case you wouldn't need it anyway. For completeness, here's my thought processes anyway. I was trying to think what sort of app might not be happy with a default that paces IW over 1/2 of the estimated RTT (or whatever fraction is decided as default, e.g. 1/4). I could only think of a transactional app with fewer than 10 packets in the send buffer. The risk of building a queue diminishes as the number of packets buffered reduces. So it might want to pace faster, the fewer packets it has buffered. Below is the algorithm for how fast to pace while keeping the queue below a certain (constant) delay. But first some terminology, with units in [square brackets]: _Environment_ R [s]: base RTT estimate (from handshake or dst cache) N []: no of full size segments worth of data in the send buffer (the algorithm only applies when N < IW). W [segment]: window that would fill capacity at the bottleneck (unknown) _Goal_: avoid qdelay exceeding R/L, where L []: a constant     e.g. L=40 would mean the goal is for the queue not to exceed R/40.     That's a queue limit of 0.5ms when R=20ms or 2.5ms when R=100ms. _Unknown variable_ T [s]: inter-packet departure time. _Solution_ T/R = 1/W - 1/L/N     with a floor at line rate, of course. If the sender had an estimate of W in its dst cache, it could use that. Otherwise it would have to use a conservative guess as another constant. For conservative (low) values of W, T/R doesn't vary much with N. So the algo would only really be useful if the sender had a value of W cached. but then it could cache the RTT as well, so it wouldn't need this algo. Bob >> >> >> Best, >> Olivier >> >> > >> > Cheers >> > >> > >> > Bob >> > >> > PS. Defensive coding nit: In tcp_output.c, I noticed you've kept the > >> condition >> > if ( ... || tp->data_segs_out >= 10 ) >> > Surely this shouldn't have been a hard-coded '10' in the first place? >> > >> > >> > -- >> > >> __________________________________________________________ >> ______ >> > Bob Briscoe http://bobbriscoe.net/ -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/