Subject: [Wpack] package: URL scheme

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 10 Jun 2020 15:00:01 -0700

Hi all,

I wanted to raise awareness of a discussion about the URL scheme for
addressing resources within bundles (draft-yasskin-wpack-bundled-exchanges).

We seem to be heading toward a URL of the form
package:<encoded-package-url>$<encoded-resource-uri>, which for a package
URL of https://distributor.example/package.wbn and resource URI of
https://publisher.example/page.html?q=query would lead to a URL of:

*package:https:,,distributor.example,package.wbn;q=query$https:,,publisher.example/page.html?q=query*

This arises from several considerations:
1. A bundle is served from a URL.
2. After a user downloads the bundle, it gets a new URL, often file:///...
3. We can also hash the bundle to get a URI that stays stable across
transfers.
4. Resources inside a bundle are named by URIs (which, since the bundle has
an index, are also URLs even if, like urn:uuid:..., they wouldn't normally
be locators).
5. Once a user downloads a bundle, for web browsers to give its content
storage that's persistent across reloads, as requested in
https://github.com/WICG/webpackage/issues/498, the content needs to be
assigned a non-opaque origin.

I'm updating one of the documents about this in
https://github.com/WICG/webpackage/pull/584 and would welcome comments here
or there.

The URLs are obviously gross, so
https://github.com/WICG/webpackage/pull/560 suggests
that browsers avoid showing them to users in most cases.

We could potentially simplify things if packages named things with just
paths instead of full URIs. We'd then name things based on the bundle's
origin. However, this loses archiving use cases.

This is all further discussed in the following documents and issues, but
you shouldn't feel responsible to read everything here:

*
https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit#
*
https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80
* https://github.com/WICG/webpackage/issues/583
*
https://github.com/WICG/webpackage/blob/master/explainers/navigation-to-unsigned-bundles.md#urls-for-bundle-components
* https://lists.w3.org/Archives/Public/uri/2019Nov/0000.html

Thanks,
Jeffrey

---

From: Ted Hardie <[email-masked]>
Date: Wed, 10 Jun 2020 15:43:32 -0700

Hi Jeffrey,

On Wed, Jun 10, 2020 at 3:00 PM Jeffrey Yasskin <jyasskin=
[email-masked]> wrote:

> RFC 3986, section 2.2 notes:

   If data for a URI component would conflict with a reserved
   character's purpose as a delimiter, then the conflicting data must be
   percent-encoded before the URI is formed.

Wouldn't that apply to a couple of the delimiters in the package example
above, e.g. the  ":" in "https:"?  If so, that seems like it would change
the display form a bit (though not the basic idea).  If that is needed, I
think it reinforces the notion that showing these to users or expecting the
users to grok them is a non-starter.

regards,

Ted Hardie

This arises from several considerations:
> 4. Resources inside a bundle are named by URIs (which, since the bundle
> has an index, are also URLs even if, like urn:uuid:..., they wouldn't
> normally be locators).
> https://github.com/WICG/webpackage/pull/584 and would welcome comments
> here or there.
>
> https://github.com/WICG/webpackage/pull/560 suggests that browsers avoid
> showing them to users in most cases.
>
> https://chromium-review..googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80
> <https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80>
> _______________________________________________
> Wpack mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/wpack
>

---

From: Larry Masinter <[email-masked]>
Date: Thu, 11 Jun 2020 11:39:28 -0700

I think this would work and avoid some of the squirrely awkward encodings , for / etc

pkg+https://distributor.example/package.wbn#cid=https://publisher.example?q=query 

Define pkg+originalscheme to always returning an application/wbn content-type which accepts a fragment identifier which identifies the package component.

This form avoids encoding or transforming any part of the original except the scheme and the fragment identifier.

You can even apply fragment identifier to the publisher URL.

From: Wpack <[email-masked]> On Behalf Of Ted Hardie
Sent: Wednesday, June 10, 2020 3:44 PM
To: Jeffrey Yasskin <jyasskin=[email-masked]>
Cc: WPACK List <[email-masked]>
Subject: Re: [Wpack] package: URL scheme

Hi Jeffrey,

On Wed, Jun 10, 2020 at 3:00 PM Jeffrey Yasskin <jyasskin=[email-masked] <mailto:[email-masked]> > wrote:

Hi all,

I wanted to raise awareness of a discussion about the URL scheme for addressing resources within bundles (draft-yasskin-wpack-bundled-exchanges).

We seem to be heading toward a URL of the form package:<encoded-package-url>$<encoded-resource-uri>, which for a package URL of https://distributor.example/package.wbn and resource URI of https://publisher.example/page.html?q=query would lead to a URL of:

package:https:,,distributor.example,package.wbn;q=query$https:,,publisher.example/page.html?q=query

Wouldn't that apply to a couple of the delimiters in the package example above, e.g. the  ":" in "https:"?  If so, that seems like it would change the display form a bit (though not the basic idea).  If that is needed, I think it reinforces the notion that showing these to users or expecting the users to grok them is a non-starter.

regards,

Ted Hardie

3. We can also hash the bundle to get a URI that stays stable across transfers.

4. Resources inside a bundle are named by URIs (which, since the bundle has an index, are also URLs even if, like urn:uuid:..., they wouldn't normally be locators).

5. Once a user downloads a bundle, for web browsers to give its content storage that's persistent across reloads, as requested in https://github..com/WICG/webpackage/issues/498 <https://github.com/WICG/webpackage/issues/498> , the content needs to be assigned a non-opaque origin.

I'm updating one of the documents about this in https://github.com/WICG/webpackage/pull/584 and would welcome comments here or there.

The URLs are obviously gross, so https://github.com/WICG/webpackage/pull/560 suggests that browsers avoid showing them to users in most cases. 

We could potentially simplify things if packages named things with just paths instead of full URIs. We'd then name things based on the bundle's origin. However, this loses archiving use cases.

This is all further discussed in the following documents and issues, but you shouldn't feel responsible to read everything here:

* https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit# <https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit> 

* https://chromium-review..googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80 <https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80> 

* https://github.com/WICG/webpackage/blob/master/explainers/navigation-to-unsigned-bundles.md#urls-for-bundle-components

Thanks,

Jeffrey

_______________________________________________
Wpack mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/wpack

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 11 Jun 2020 11:47:25 -0700

> users to grok them is a non-starter.

I looked at that, and I think it's ok to leave the `:` unescaped.
Specifically, since `:` is only used as a delimiter to separate the scheme,
the later `:`s in the package and resource URLs don't conflict. blob: URLs
<https://w3c.github.io/FileAPI/#blob-url> already do this, and go a bit
farther in leaving their `/`s unescaped too.

We could probably get away without escaping the `/`s here too: the
origin-finding algorithm would split on `$` instead of looking for the
first `/`, but I worry that it would be easier to write bugs in that case,
which could give packaged resources access to their distributor's storage.

Jeffrey

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 11 Jun 2020 11:59:34 -0700

This use of a fragment matches the way the TAG's packaging draft identified
subresources:
https://www.w3.org/TR/2015/WD-web-packaging-20150115/#fragment-identifiers.
I have some discussion of the option at
https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit#heading=h.jkcl1u8boyar,
but I realize I didn't include the part that worries me most there:

We need to make the nested URL contribute to the resource's origin, or else
when folks create a bundle of multiple sites for distribution together,
their storage will collide. Including part of the fragment into the origin
would be novel and potentially break assumptions and cause bugs. However,
it could be the best option anyway.

Jeffrey

On Thu, Jun 11, 2020 at 11:39 AM Larry Masinter <[email-masked]> wrote:

> pkg+
> https://distributor.example/package.wbn#cid=https://publisher.example?q=query
>
>
>
> Define pkg+originalscheme to always returning an application/wbn
> content-type which accepts a fragment identifier which identifies the
> package component.
>
>
>
> *From:* Wpack <[email-masked]> *On Behalf Of *Ted Hardie
> *Sent:* Wednesday, June 10, 2020 3:44 PM
> *To:* Jeffrey Yasskin <jyasskin=[email-masked]>
> *Cc:* WPACK List <[email-masked]>
> *Subject:* Re: [Wpack] package: URL scheme
>
>
>
> Hi Jeffrey,
>
>
>
> https://github..com/WICG/webpackage/issues/498
> <https://github.com/WICG/webpackage/issues/498>, the content needs to be
> <https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit>
>
> *

---

From: Ted Hardie <[email-masked]>
Date: Thu, 11 Jun 2020 14:06:57 -0700

Howdy,

> farther in leaving their `/`s unescaped too.

It's certainly the case that some schemes permit ":" after the scheme to be
presented unescaped (URNs require one, separating the NID from NSS).  My
concern here is really that these constructions, including the later ":"s ,
are actually being used in ways that might be read as conflicting with
their original purpose as delimiters.  "https:" occurs multiple times with
contextual cues on meaning, in essence.  That might be okay, but I think
the percent encoded version might match expected practice  bit better.
It's unreadable to the end user, but I think that true either way.

Is your concern with that approach that the origin-matching code might have
forgiving  algorithms for matching against percent-encoded elements of URIs?

regards,

Ted

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 11 Jun 2020 14:33:03 -0700

On Thu, Jun 11, 2020 at 2:07 PM Ted Hardie <[email-masked]> wrote:

>> the later `:`s in the package and resource URLs don't conflict. blob:
>> URLs <https://w3c.github.io/FileAPI/#blob-url> already do this, and go a
>> bit farther in leaving their `/`s unescaped too.
>>
>
> It's certainly the case that some schemes permit ":" after the scheme to
> be presented unescaped (URNs require one, separating the NID from NSS).  My
> Is your concern with that approach that the origin-matching code might
> have forgiving  algorithms for matching against percent-encoded elements of
> URIs?
>

I think there are two questions here:

1) Do we need to encode the embedded ":"s at all?
2) Should we encode by replacing with subdelimiters (
https://tools.ietf.org/html/rfc3986#section-2.2), or by percent-encoding?

I'm fairly neutral on encoding ":"s at all. I think it's unnecessary, but
not particularly harmful.

On percent-encoding vs replacement, I think there are several levels of
readability. It's true that we shouldn't expect end-users to read these at
all (and that Larry's suggestion is a win on that front), but when
developers are trying to debug problems, I think 'package:https!,,' is
easier to check than 'package:https%3A%2F%2F'.

I'm not currently worried that origin-matching code will be too forgiving
about percent-encoded elements, and I don't think that worry would
distinguish our proposals. Even my current proposal can result in
percent-encoding inside the origin: a bundle distributed from `
https://example/path;param/bundle` would have resource URLs starting with
`package:https:,,example,path%3Bparam,bundle$` to distinguish it from the
bundle at `https://example/path?param/bundle`.

Jeffrey

---

From: Larry Masinter <[email-masked]>
Date: Thu, 11 Jun 2020 14:36:27 -0700

The bigger problem is that WHATWG’s URL living standard has forked from the IETF specifications, where there is no visible plan for reconciliation. The specs aren’t so wildly different, but the devil’s in the details.

--

https://LarryMasinter.net

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 12 Jun 2020 10:01:29 +1000

I have a bunch of concerns about this approach, but let's start with a fairly major one:

This buries the authority.  If your authority is truly publisher.example, then that should be the authority component.  If you regard the rest as a split between the remainder of the identity of the resource itself and some secondary information about how that resource might be obtained, then you might have something closer to how you might want to structure a URI.

On Thu, Jun 11, 2020, at 08:00, Jeffrey Yasskin wrote:
> package:<encoded-package-url>$<encoded-resource-uri>, which for a 
> package URL of https://distributor.example/package.wbn and resource URI 
> of https://publisher.example/page.html?q=query would lead to a URL of:
> 
> https://github.com/WICG/webpackage/pull/560 suggests that browsers 
> avoid showing them to users in most cases. 
> 
> This is all further discussed in the following documents and issues, 
> but you shouldn't feel responsible to read everything here:
> 
> * 
> https://chromium-review..googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80 <https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80>

---

From: Larry Masinter <[email-masked]>
Date: Thu, 11 Jun 2020 17:47:32 -0700

PDF allows embedded files. There was a requirement to allow URLs pointing to
embedded files and to allow identification of fragments..

>From https://tools.ietf.org/html/rfc8118#section-3 Fragment identifiers

   ef=<name>
      Identifies the embedded file where the parameter string <name>
      matches a file specification dictionary in the EmbeddedFiles name
      tree.  If the "ef" parameter is not at the end of the fragment
      identifier, then the rest of the fragment identifier (after the
      ampersand or hash delimiter) is applied to the embedded file
      according to its own media type.  This allows identification of
      content within the embedded file (which itself might be a
      PDF file)

      NOTE: When attempting to open a PDF file that is not from a
      trusted source, the processor may choose to prompt the user or
      even prevent the file from being opened.

There was no need to define a new scheme.

Sorry I'd forgotten about this.
--
https://LarryMasinter.net https://going-remote.info

---

From: Mallory Knodel <[email-masked]>
Date: Fri, 12 Jun 2020 08:34:23 -0400

> This buries the authority.  If your authority is truly publisher.example, then that should be the authority component.  If you regard the rest as a split between the remainder of the identity of th…

Agree that the user needs an indication of publisher, and that hiding 
the URL is unhelpful to the user. Simplification sounds like the right 
thing to do, because too much information can have the same effect as none.

-Mallory

-- 
Mallory Knodel
CTO, Center for Democracy and Technology
gpg fingerprint :: E3EB 63E0 65A3 B240 BCD9 B071 0C32 A271 BD3C C780

---

From: Larry Masinter <[email-masked]>
Date: Fri, 12 Jun 2020 08:20:07 -0700

In the archival use case, isn't the distributer the authority, not the
publisher?
The publisher could be long gone.

> -----Original Message-----
> From: Wpack <[email-masked]> On Behalf Of Mallory Knodel
> Sent: Friday, June 12, 2020 5:34 AM
> To: Martin Thomson <[email-masked]>; [email-masked]
> > I have a bunch of concerns about this approach, but let's start with a
fairly
> major one:
> >
> > This buries the authority.  If your authority is truly
publisher.example, then
> that should be the authority component.  If you regard the rest as a split
> between the remainder of the identity of the resource itself and some
> secondary information about how that resource might be obtained, then you
> might have something closer to how you might want to structure a URI.
> 
> Agree that the user needs an indication of publisher, and that hiding the
URL
> is unhelpful to the user. Simplification sounds like the right thing to
do,
> because too much information can have the same effect as none.
> 
> -Mallory
> 
> > We could potentially simplify things if packages named things with
> > just paths instead of full URIs. We'd then name things based on the
> > bundle's origin. However, this loses archiving use cases.
> >
> https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi
> 1r8Y
> > 0pLaFJQoo/edit#
> > *
> > https://chromium-
> review..googlesource.com/c/chromium/src/+/2226248/7#m
> > essage-0a3efda5aff84770a1729422a5b26aeca3ee4e80
> > <https://chromium-
> review.googlesource.com/c/chromium/src/+/2226248/7#m
> > essage-0a3efda5aff84770a1729422a5b26aeca3ee4e80>
> https://github.com/WICG/webpackage/blob/master/explainers/navigation-t
> > o-unsigned-bundles.md#urls-for-bundle-components
> CTO, Center for Democracy and Technology gpg fingerprint :: E3EB 63E0
> 65A3 B240 BCD9 B071 0C32 A271 BD3C C780
>

---

From: Jeffrey Yasskin <[email-masked]>
Date: Fri, 12 Jun 2020 16:07:35 -0700

Hi Martin and Mallory,

As Larry mentioned, when the bundle is unsigned, it's the entity serving
the bundle -- distributor.example here -- who has the ability to change the
content, so they're probably the "true" authority. They're quoting the
original publisher of the content, but there's no way for the user to
verify that the quote is accurate.

Do you support the general goal of giving pairwise-distinct origins to all
of:

1. https://foo.example/page.html
2. https://bar.example/page.html
3. A resource in the bundle at https://foo.example/bundle.wbn named
https://bar.example/page.html.
4. A resource in the bundle at https://foo.example/bundle.wbn named
https://quux.example/page.html.
5. A resource in the bundle at https://foo.example/otherbundle.wbn named
https://bar.example/page.html.

If we say (1) and (3) get the same origin, it means the result can't help
the Internet Archive serve their pages more safely, and we force (3), (4),
and (5) to have the same origins.
If we say (2) and (3) get the same origin, we break the entire web origin
security model. :-D
If we say (3) and (4) get the same origin, it means that El Paquete Semanal
couldn't safely put multiple websites into the same bundle without risking
them stepping on each other's storage. It could put them in separate files,
but then they'd have trouble linking to each other.
If we say (3) and (5) get the same origin, it means that if an archive
stores multiple versions of the same website, but those versions use
storage differently, users couldn't easily try more than one version.

Abandoning some of these use cases definitely makes the URL design easier,
if there's consensus to go that direction.

However, if we want to keep the use cases, and we want to put the bundle's
server in the authority position of the URL, we get something like Larry's
suggestion: pkg+
https://foo.example/bundle.wbn?query#https://bar.example/page.html?q=query%23fragment.
To give (3)-(5) distinct origins, the origin algorithm
<https://url.spec.whatwg.org/#origin> for pkg+https needs to take the
fragment into account, returning something like ("pkg+https",
"foo.example/bundle.wbn?query#https://bar.example", null, null). This
design also makes it possible to resolve relative URLs relative to a
pkg+https:// base URL, and it gives what's probably the wrong
answer, moving relative to the bundle instead of the active subresource.
That's probably ok: links inside a bundle need to explicitly search the
bundle so that absolute references search the bundle first.

If we want to prevent package URLs from being used as base URLs, we could
move to something like "package:
https://foo.example/bundle.wbn?query#https://bar.example/page.html?q=query%23fragment",
which has a similar syntax to blob: URLs and is still more readable than my
original proposal.

Jeffrey

On Thu, Jun 11, 2020 at 5:02 PM Martin Thomson <[email-masked]> wrote:

> fairly major one:
> This buries the authority.  If your authority is truly publisher.example,
> then that should be the authority component.  If you regard the rest as a
> split between the remainder of the identity of the resource itself and some
> > https://chromium-review..
> googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80
> <

---

From: Larry Masinter <[email-masked]>
Date: Sat, 13 Jun 2020 20:40:45 -0700

The problem I keep coming back to is that there is no firm line between the “archival” case and the immediate use case.

As soon as you get a representation of the publisher’s resource, it could change. 

I looked through all of the links you provided and couldn’t find any definition
of what a “signed bundle” is, or at least what is a bundle that isn’t “unsigned”.

Who signs what, when? And what happens over time with a signed bundle
when sites change, certs expire? How are signatures validated?

You should look at https://url.spec.whatwg.org/#url-rendering for the rules you’ll need to modify to get your “address bar’ conventions.

==

 <https://larrymasinter.net/> https://LarryMasinter.net  <https://going-remote.info/> https://going-remote.info

From: Wpack <[email-masked]> On Behalf Of Jeffrey Yasskin
Sent: Friday, June 12, 2020 4:08 PM
To: Martin Thomson <[email-masked]>; [email-masked]
Cc: WPACK List <[email-masked]>
Subject: Re: [Wpack] package: URL scheme

Hi Martin and Mallory,

As Larry mentioned, when the bundle is unsigned, it's the entity serving the bundle -- distributor.example here -- who has the ability to change the content, so they're probably the "true" authority. They're quoting the original publisher of the content, but there's no way for the user to verify that the quote is accurate.

Do you support the general goal of giving pairwise-distinct origins to all of:

3. A resource in the bundle at https://foo.example/bundle.wbn named https://bar.example/page.html.

4. A resource in the bundle at https://foo.example/bundle.wbn named https://quux.example/page.html.

5. A resource in the bundle at https://foo.example/otherbundle.wbn named https://bar.example/page.html.

If we say (1) and (3) get the same origin, it means the result can't help the Internet Archive serve their pages more safely, and we force (3), (4), and (5) to have the same origins.

If we say (2) and (3) get the same origin, we break the entire web origin security model. :-D

If we say (3) and (4) get the same origin, it means that El Paquete Semanal couldn't safely put multiple websites into the same bundle without risking them stepping on each other's storage. It could put them in separate files, but then they'd have trouble linking to each other.

If we say (3) and (5) get the same origin, it means that if an archive stores multiple versions of the same website, but those versions use storage differently, users couldn't easily try more than one version.

Abandoning some of these use cases definitely makes the URL design easier, if there's consensus to go that direction.

However, if we want to keep the use cases, and we want to put the bundle's server in the authority position of the URL, we get something like Larry's suggestion: pkg+https://foo.example/bundle.wbn?query#https://bar.example/page.html?q=query%23fragment. To give (3)-(5) distinct origins, the origin algorithm <https://url.spec.whatwg.org/#origin>  for pkg+https needs to take the fragment into account, returning something like ("pkg+https", "foo.example/bundle.wbn?query#https://bar.example", null, null). This design also makes it possible to resolve relative URLs relative to a pkg+https:// base URL, and it gives what's probably the wrong answer, moving relative to the bundle instead of the active subresource. That's probably ok: links inside a bundle need to explicitly search the bundle so that absolute references search the bundle first.

If we want to prevent package URLs from being used as base URLs, we could move to something like "package:https://foo.example/bundle.wbn?query#https://bar.example/page.html?q=query%23fragment", which has a similar syntax to blob: URLs and is still more readable than my original proposal.

Jeffrey

On Thu, Jun 11, 2020 at 5:02 PM Martin Thomson <[email-masked] <mailto:[email-masked]> > wrote:

On Thu, Jun 11, 2020, at 08:00, Jeffrey Yasskin wrote:
> https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit# <https://docs.google.com/document/d/1BYQEi8xkXDAg9lxm3PaoMzEutuQAZi1r8Y0pLaFJQoo/edit> 
> * 
> https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80 <

https://chromium-review.googlesource.com/c/chromium/src/+/2226248/7#message-0a3efda5aff84770a1729422a5b26aeca3ee4e80>

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 15 Jun 2020 11:37:25 +1000

On Sat, Jun 13, 2020, at 09:07, Jeffrey Yasskin wrote:
> If we say (1) and (3) get the same origin, it means the result can't 
> help the Internet Archive serve their pages more safely, and we force 
> (3), (4), and (5) to have the same origins.

Yes, I think that having 1 and 3 being distinct is an important property.  I tend to stop at this point and not try to add too much more, but there you go.

> origin security model. :-D

(1) and (2) more so.

Part of what we're trying to do is find a way to bring (2) and (3) closer (and (5) too). That doesn't necessarily need to mean that we treat them identically.  Though we might find a path to doing so.

> If we say (3) and (4) get the same origin, it means that El Paquete 
> Semanal couldn't safely put multiple websites into the same bundle 
> without risking them stepping on each other's storage. It could put 
> them in separate files, but then they'd have trouble linking to each 
> other.

What cannot be solved with one level of abstraction might be solved with two.

If the distinctive characteristic of a bundle is that it creates a new origin, then a bundle that contains other bundles can be used to isolate content from different origins.  The El Paquete Semanal as a whole could be assembled from multiple bundles from different "real" origins.

As you say, that complicates the process of inter-origin references.  Say the El Paquete Semanal bundle was served by non-determined means, then you can't rely on having a shared identifier for the outer bundle, unless you have some means of minting one.  (This is a case where you can't refer to the outer bundle from the inner one using content identification, for reasons that I hope are obvious.)  So you mint one (a signing key might suffice) and then you can refer from one inner bundle to another inner bundle via that identifier.

The reference chain might become convoluted, but I don't see a need to constrain URL design to fit that use case.  As long as references are possible, then we can work on making it more usable.  If you have siteA.example and siteB.example served from bundle.example from distributor.example, then my hope would be that your inter-site references are of the form:

<something>://siteX.example/<a hint that mentions bundle.example and maybe distributor.example, but the latter might be best left implied>

I say that because while siteA and siteB might expect to be served from the same meta-bundle, there shouldn't need to be a strong binding to that situation.  If one is and the other not, then I would hope that the identifiers would support that without requiring strong changes.

> storage differently, users couldn't easily try more than one version.

I think that this is a key question.  My assertion is that maintaining a clean abstraction for bundles is important and that would dictate having (3) and (5) distinct.  At least initially.  Mechanisms that allow transfer of state or content between origins might allow an upgrade to occur from (3) to (5).

> Abandoning some of these use cases definitely makes the URL design 
> easier, if there's consensus to go that direction.
> 
> However, if we want to keep the use cases, and we want to put the 
> bundle's server in the authority position of the URL, we get something 
> like Larry's suggestion: 
> pkg+https://foo.example/bundle.wbn?query#https://bar.example/page.html?q=query%23fragment. To give (3)-(5) distinct origins, the origin algorithm <https://url.spec.whatwg.org/#origin> for pkg+https needs to take the fragment into account, returning something like ("pkg+https", "foo.example/bundle.wbn?query#https://bar.example", null, null). This design also makes it possible to resolve relative URLs relative to a pkg+https:// base URL, and it gives what's probably the wrong answer, moving relative to the bundle instead of the active subresource. That's probably ok: links inside a bundle need to explicitly search the bundle so that absolute references search the bundle first.

Rather than try to work the identity of the serving entity into the origin, it might be better to look to the limits of the origin concept.  We're seeing browsers move more toward placing origins within a greater context.  Double-keying storage by top-level browsing context shows us how maybe the origin isn't able to capture the entire context.  The same might apply here.  Maybe it is important that this bundle originally came from foo.example, but it might not be important to the concept of the origin model.

Relative references within a bundle should be simple and possible, but I fear that expecting fragments to support that is likely to run afoul of all sorts of existing expectations.  Using fragments for bundles is likely good if you don't want to mint a new URI scheme, but if you are in the business of defining a new scheme, then go for it.

If you are defining a new media type, then the fragment can do whatever you like.  With no new URI scheme needed:

https://foo.example/bundle.wbn?query#<use whatever you feel like, but don't expect internal references to work>

Similarly, once you are defining a new URI scheme, then you have a lot of options available.  The URI standard is perhaps unnecessarily narrow in its definition of authority, but there's a lot of room for creative interpretation: a registered name doesn't need to be domain name, and there are no real mechanism that ensure that it is "registered" (whatever that means).

pkg://<the authority for the above bundle, which might not be bar.example>/page.html?q=query%23fragment

The definition of origin necessarily becomes more flexible.  Rather than insist on the reduction to a simple tuple for an origin, we should regard these having a same-origin comparison operation, which sometimes allow two origins to be regarded the same, even with vastly different inputs.  The reduction to a tuple is useful, but potentially quite confining.

If pkg://<bar> and https://<foo> happen to be same origin because we decide that is useful (and safe), then we should be free to define that as we choose, within the constraints of the origin tuple or not.

Despite it being quite clearly invalid for domain name and port, this is completely valid URI if we so choose: <pkg://$:1000000/>. That doesn't mean that it couldn't be same-origin with <https://example.com/>, but it might be a little tricky to reduce to an origin tuple.

---

From: Jeffrey Yasskin <[email-masked]>
Date: Mon, 15 Jun 2020 11:16:56 -0700

Replies inline.

On Sun, Jun 14, 2020 at 6:38 PM Martin Thomson <[email-masked]> wrote:

> Yes, I think that having 1 and 3 being distinct is an important property.
> I tend to stop at this point and not try to add too much more, but there
> you go.
>

I think distinguishing (1) and (3) requires a new scheme.

> Part of what we're trying to do is find a way to bring (2) and (3) closer
> (and (5) too). That doesn't necessarily need to mean that we treat them
> identically.  Though we might find a path to doing so.
>

I think the signing or transfer effort described
in draft-thomson-wpack-content-origin
and draft-yasskin-http-origin-signed-responses tries to get (2) and (3)
closer together, but here I'm focusing on use cases that don't need the
browser to recognize (2) and (3) as related at all.

> If the distinctive characteristic of a bundle is that it creates a new
> origin, then a bundle that contains other bundles can be used to isolate
> content from different origins.  The El Paquete Semanal as a whole could be
> assembled from multiple bundles from different "real" origins.
>
> As you say, that complicates the process of inter-origin references.  Say
> the El Paquete Semanal bundle was served by non-determined means, then you
> can't rely on having a shared identifier for the outer bundle, unless you
> have some means of minting one.  (This is a case where you can't refer to
> the outer bundle from the inner one using content identification, for
> reasons that I hope are obvious.)  So you mint one (a signing key might
> suffice) and then you can refer from one inner bundle to another inner
> bundle via that identifier.
>
> The reference chain might become convoluted, but I don't see a need to
> constrain URL design to fit that use case.  As long as references are
> possible, then we can work on making it more usable.  If you have
> siteA.example and siteB.example served from bundle.example from
> distributor.example, then my hope would be that your inter-site references
> are of the form:
>
> <something>://siteX.example/<a hint that mentions bundle.example and maybe
> distributor.example, but the latter might be best left implied>
>
> I say that because while siteA and siteB might expect to be served from
> the same meta-bundle, there shouldn't need to be a strong binding to that
> situation.  If one is and the other not, then I would hope that the
> identifiers would support that without requiring strong changes.
>

There's a downside that this still requires rewriting all the cross-origin
links inside the documents, but as El Paquete Semanal is already rewriting
links to put everything on a filesystem, maybe that's not a big problem.

I'm curious what you think the hint could look like. You've sketched it in
the path area, but these URLs also have paths, so we'd need something to
separate the two parts of the path in the same way my first attempt
separated two parts of the authority.

> I think that this is a key question.  My assertion is that maintaining a
> clean abstraction for bundles is important and that would dictate having
> (3) and (5) distinct.  At least initially.  Mechanisms that allow transfer
> of state or content between origins might allow an upgrade to occur from
> (3) to (5).
>
> To give (3)-(5) distinct origins, the origin algorithm <
> https://url.spec.whatwg.org/#origin> for pkg+https needs to take the
> pkg+https:// base URL, and it gives what's probably the wrong answer,
> moving relative to the bundle instead of the active subresource. That's
> probably ok: links inside a bundle need to explicitly search the bundle so
> that absolute references search the bundle first.
>
> Rather than try to work the identity of the serving entity into the
> origin, it might be better to look to the limits of the origin concept.
> We're seeing browsers move more toward placing origins within a greater
> context.  Double-keying storage by top-level browsing context shows us how
> maybe the origin isn't able to capture the entire context.  The same might
> apply here.  Maybe it is important that this bundle originally came from
> foo.example, but it might not be important to the concept of the origin
> model.
>

This is an interesting point: we could use the new storage keys
<https://storage.spec.whatwg.org/#storage-keys> being created by
https://github.com/privacycg/storage-partitioning to declare that an
environment settings object with a bundle attached uses a storage key
derived from both the bundle's URL and the subresource's URL, instead of
trying to encapsulate both into a single origin.

Relative references within a bundle should be simple and possible, but I
> fear that expecting fragments to support that is likely to run afoul of all
> sorts of existing expectations.  Using fragments for bundles is likely good
> if you don't want to mint a new URI scheme, but if you are in the business
> of defining a new scheme, then go for it.
>
> https://foo.example/bundle.wbn?query#<use whatever you feel like, but
> don't expect internal references to work>
>
> Similarly, once you are defining a new URI scheme, then you have a lot of
> options available.  The URI standard is perhaps unnecessarily narrow in its
> definition of authority, but there's a lot of room for creative
> interpretation: a registered name doesn't need to be domain name, and there
> are no real mechanism that ensure that it is "registered" (whatever that
> means).
>
> pkg://<the authority for the above bundle, which might not be
> bar.example>/page.html?q=query%23fragment
>

This is how I got
to package:https:,,distributor.example,package.wbn;q=query$https:,,publisher.example/page.html?q=query
(or package://...). It claims that the "authority" for a subresource inside
a bundle consists of both the bundle's location and the subresource's
origin. arcp:// did the same, but without the subresource's origin, which
would have the effect of unifying (3) and (4).

The definition of origin necessarily becomes more flexible.  Rather than
> insist on the reduction to a simple tuple for an origin, we should regard
> these having a same-origin comparison operation, which sometimes allow two
> origins to be regarded the same, even with vastly different inputs.  The
> reduction to a tuple is useful, but potentially quite confining.
>
> If pkg://<bar> and https://<foo> happen to be same origin because we
> decide that is useful (and safe), then we should be free to define that as
> we choose, within the constraints of the origin tuple or not.
>
> Despite it being quite clearly invalid for domain name and port, this is
> completely valid URI if we so choose: <pkg://$:1000000/>. That doesn't mean
> that it couldn't be same-origin with <https://example.com/>, but it might
> be a little tricky to reduce to an origin tuple.
>

I don't mind diverging from the origin-tuple concept, but so far I haven't
found a need to.

Thanks,
Jeffrey