Subject: [quicwg/base-drafts] Cache git repo, draft references in Circle (#3009)

From: Mike Bishop <[email-masked]>
Date: Fri, 06 Sep 2019 08:50:16 -0700

This appears to shave almost a minute off of build times.
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/3009

-- Commit Summary --

  * Cache git repo between Circle runs
  * Cache draft references too

-- File Changes --

    M .circleci/config.yml (28)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/3009.patch
https://github.com/quicwg/base-drafts/pull/3009.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009

---

From: Mike Bishop <[email-masked]>
Date: Fri, 06 Sep 2019 08:57:42 -0700

@MikeBishop pushed 1 commit.

c642ae92f3b78642e666d76701a1606d87446e8d  Only gc if called for

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009/files/b8e45fba31c52c5a9bd9dda9b26e8b9443fa4fb5..c642ae92f3b78642e666d76701a1606d87446e8d

---

From: Martin Thomson <[email-masked]>
Date: Sun, 08 Sep 2019 18:08:31 -0700

martinthomson commented on this pull request.

I was concerned that this wouldn't be very much good.

The `{{ .Revision }}` key means that that cache entry is only going to be good for rebuilding that revision, something that we don't do, except when we are tagging a release.  The tag for `{{ .Branch }}` is probably good, but not for `master`, which will be baking in as soon as this lands (if it hasn't already).

The `{{ epoch }}` tag is only good if we run builds within a second of each other.

How about an idea that I just had.  We can run arbitrary code to generate a file.  If we were to write something to a file and then use `{{ checksum ".reference-cache-key" }}` we could ensure that the value rolls over daily so that we have a hot reference cache.  We could do the same for the git cache so that it is mostly at most a certain date old.  You can even cascade caches by reading from `{{ checksum ".cache-today" }}` and `{{ checksum ".cache-yesterday" }}`.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#pullrequestreview-285247588

---

From: Mike Bishop <[email-masked]>
Date: Mon, 09 Sep 2019 08:12:25 -0700

The important piece there are the multi-tiered keys in restore:
```
      - restore_cache:
          name: "Restoring cache - Git"
          keys:
            - v1-cache-git-{{ .Branch }}-{{ .Revision }}
            - v1-cache-git-{{ .Branch }}
            - v1-cache-git-

      - restore_cache:
          name: "Restoring cache - References"
          keys:
            - v1-cache-references-{{ epoch }}
            - v1-cache-references-
```
The revision will never match, so it picks up the most recent git repo cached for that branch.  The epoch will almost never match, so it picks up the most recent reference cache generated.  We could drop the first line of each restore_cache keys directive and get the same behavior.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#issuecomment-529524438

---

From: Mike Bishop <[email-masked]>
Date: Mon, 09 Sep 2019 08:31:57 -0700

And too, the proof is in the pudding -- the first build on this branch took 1:48; git caching brought it to 0:56, and reference caching brought it to 0:37.
- Code checkout goes from 36 seconds to sub-second (shows as zero)
- Draft build time goes from 32 seconds to 6 seconds
- Saving the git cache costs 5-6 seconds
- Saving the reference cache is sub-second

There's a few seconds' variation around each of those -- the last build on the branch went back up to 0:43.  But still, I stand by my claim that we improve by nearly a minute per build.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#issuecomment-529534484

---

From: Mike Bishop <[email-masked]>
Date: Mon, 09 Sep 2019 08:36:08 -0700

56e9e1a6515ac1ea6e882a83c48d7149070bdd17  Add master to fallback path for new branches

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009/files/c642ae92f3b78642e666d76701a1606d87446e8d..56e9e1a6515ac1ea6e882a83c48d7149070bdd17

---

From: Martin Thomson <[email-masked]>
Date: Mon, 09 Sep 2019 17:19:21 -0700

That's great for now, while there are relatively few new commits to add and the references are fresh.  However, we are creating caches for things that won't ever be used again (``{{ .Revision }}` and `{{ epoch }}`), which takes time that doesn't pay back.

More seriously, the `{{ .Branch }}` thing is a one-time event.  We will always be picking up the cache that you create the first time `master` uses this build.  And the references cache will be old in less than a week now, but that is the cache that will be used in perpetuity.  Sure, this is good in the sense that the build will survive networking blips (assuming that we don't add more citations), but those references will be ancient.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#issuecomment-529715739

---

From: Mike Bishop <[email-masked]>
Date: Tue, 10 Sep 2019 11:43:44 -0700

> We will always be picking up the cache that you create the first time `master` uses this build. And the references cache will be old in less than a week now, but that is the cache that will be used in perpetuity.

That would be true if we created a cache once and never wrote an updated one.  But we're not -- we're creating a fresher one each time we run.  We'll be picking up the cache from the *last* time master built, not the first.

Because each key is immutable once written, the keys *have* to be unique (epoch and revision).  But when we're searching by prefix, that doesn't mean it's going to use the oldest possible match.

>From https://circleci.com/docs/2.0/caching/#source-caching:
> If there are multiple matches, the **most recently generated cache** will be used.

Again, look at the actual behavior for builds on this branch:
- Build 12474 uses both caches from build 12461
- Build 12461 uses both caches from build 12459
- Build 12459 uses both caches from build 12457
- Build 12457 uses both caches from build 12455
- Build 12455 uses both caches from build 12453
- Build 12453 uses the git cache from build 12451 and creates the reference cache
- Build 12451 uses the git cache from build 12449
- Build 12449 creates the git cache

Each build creates a new cache instance, and each run uses:
- The most recent git cache for the current revision, or branch, or for master, or overall
- The most recent reference cache from any previous run

Sure, odds are that each stored cache only gets used once and then sits there collecting dust until Circle expires it 30 days later, which means we're not amortizing the time to create the cache over multiple future runs.  But the combined time to store a new cache instance and retrieve the latest cache instance is massively outweighed by the time saved by having that cache.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#issuecomment-530068800

---

From: Martin Thomson <[email-masked]>
Date: Tue, 10 Sep 2019 15:44:49 -0700

How did I completely miss the prefix-matching thing.  This is good.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3009#issuecomment-530148995