`bazel`-based continuous integration flow by matthew-levan · Pull Request #21 · urbit/vere

matthew-levan · 2022-11-22T18:30:59Z

This PR implements CI workflows for:

Binary builds (all four supported platforms: macos/linux, x86/arm)
Tests (i.e., c3, ent, noun, ur, and vere tests)
Releases (binary uploads to GCP) for pre-release and release tags
Docker builds
Ship tests
Caching

Workflows from urbit/urbit have been refactored for improved clarity, and nix derivations previously used in CI have been extracted and implemented with bazel.

Resolves #8.
Resolves #9.
Resolves #25 via #28.
Resolves #26 via #34.
Resolves #29.
Resolves #30.
Resolves #32 via #33.
Resolves #35 via #37.

mcevoypeter

This is only a partial review. I haven't looked through the CI scripts or the Docker markdown doc yet.

…/8/ci

mcevoypeter

LGTM

matthew-levan · 2023-01-09T16:13:54Z

When branching this runtime-specific repo off from the urbit monorepo, we initially based off of next/vere in the monorepo. However, during the course of fall 2022, buggy work ended up on next/vere. Because of this, the release branch for version 1.14 of the runtime branched off of master, not next/vere, leading to a non-trivial divergence between next/vere and master. To that end, we have to rewrite the history of this repo, which will potentially obfuscate the commit SHAs and diffs of the already merged PRs in this repo. To provide a trail of breadcrumbs, attached is a patch of the diff represented by this PR before history was rewritten.

PR21.patch

@quodss

Reverts #827 This PR causes an instant segfault in the serf after upgrading on `~dinleb-rambep`, `~norsyr-torryn` and probably all pre-existing ships. The segfault has the following stacktrace: ``` urbit 3.4-0969b6de10 boot: home is /root/dinleb-rambep disk: loaded epoch 0i575532513 loom: mapped 2048MB boot: protected loom live: mapped: GB/1.945.878.528 live: loaded: KB/16.384 boot: installed 1362 jets loom: image backup complete disk: created epoch 575532513 loom: mapped 2048MB lite: arvo formula 4ce68411 lite: core 641296f lite: final state 641296f disk: loaded epoch 0i575532513 loom: mapped 2048MB boot: protected loom live: mapped: GB/1.945.878.528 live: loaded: KB/16.384 boot: installed 1362 jets vere: checking version compatibility loom: external fault: 0 (0x200000000 : 0x280000000) Breakpoint 1, u3m_fault (adr_v=0x0, ser_i=<optimized out>) at pkg/noun/manage.c:1967 1967 pkg/noun/manage.c: No such file or directory. (gdb) bt #0 u3m_fault (adr_v=0x0, ser_i=<optimized out>) at pkg/noun/manage.c:1967 #1 0x00000000014b9544 in sigsegv_handler (sig=<optimized out>, sip=<optimized out>, ucp=0x1840b40 <Sigstk+6896>) at /home/runner/.cache/zig/p/N-V-__8AAHQhHwCCMyKUtQhO0AMsY182In-wQIwXVhenOql5/src/handler-unix.c:269 #2 <signal handler called> #3 _n_burn (pog_u=0x1ffffe121ffffe5a, bus=<optimized out>, mov=1 '\001', off=-1 '\377') at pkg/noun/nock.c:2100 #4 0x0000000001273653 in _n_burn_out (bus=1950065936, pog_u=0x1fffffff4) at pkg/noun/nock.c:2814 #5 _n_burn_on (bus=1950065936, bus@entry=3460980861, fol=<optimized out>) at pkg/noun/nock.c:2837 #6 u3n_nock_on (bus=1950065936, bus@entry=3460980861, fol=<optimized out>) at pkg/noun/nock.c:2851 #7 0x0000000001271938 in u3v_poke_raw (sam=50403342) at pkg/noun/vortex.c:293 #8 u3v_poke (ovo=ovo@entry=3459182210) at pkg/noun/vortex.c:285 #9 0x0000000001262948 in u3m_soft_top (mil_w=mil_w@entry=0, pad_w=pad_w@entry=1048576, fun_f=0x12718d0 <u3v_poke>, arg=3459182210) at pkg/noun/manage.c:1311 #10 0x00000000012630ad in u3m_soft (mil_w=4294967284, mil_w@entry=0, fun_f=0x1d0ee947, arg=1950064560) at pkg/noun/manage.c:1577 #11 0x000000000161888d in _serf_poke (sef_u=<optimized out>, cap_c=<optimized out>, mil_w=0, job=job@entry=3455400141) at pkg/vere/serf.c:556 #12 0x0000000001617804 in _serf_work (sef_u=0x183ec20 <u3V>, mil_w=0, job=3455400141) at pkg/vere/serf.c:604 #13 u3_serf_work (sef_u=sef_u@entry=0x183ec20 <u3V>, mil_w=mil_w@entry=0, job=3455400141) at pkg/vere/serf.c:692 #14 0x0000000001618666 in u3_serf_writ (sef_u=0x183ec20 <u3V>, wit=3462698097, pel=pel@entry=0x7fffffff9a0c) at pkg/vere/serf.c:1160 #15 0x000000000126141d in _cw_serf_writ (vod_p=<optimized out>, len_d=113, byt_y=0x7ffff7ff8780 "\001\377\356Mn\235\001\b") at pkg/vere/main.c:1079 #16 0x00000000014ddf08 in _newt_meat_poke (mot_u=0x183ed90 <inn_u>, met_u=0x7ffff7ff8770) at pkg/vere/newt.c:62 #17 _newt_meat_next_sync (mot_u=0x183ed90 <inn_u>) at pkg/vere/newt.c:75 #18 _newt_read_sync_cb (str_u=0x183ed90 <inn_u>, len_i=<optimized out>, buf_u=<optimized out>) at pkg/vere/newt.c:236 #19 0x00000000014ccfe7 in uv__read (stream=0x183ed90 <inn_u>) at /home/runner/.cache/zig/p/N-V-__8AAH34QwB6wi5eQK_lFbfDGSN3hRE8l-6Ep198ZsGg/src/unix/stream.c:1148 #20 uv__stream_io (loop=<optimized out>, w=0x183ee18 <inn_u+136>, events=1) at /home/runner/.cache/zig/p/N-V-__8AAH34QwB6wi5eQK_lFbfDGSN3hRE8l-6Ep198ZsGg/src/unix/stream.c:1208 #21 0x00000000014c6b94 in uv__io_poll (loop=loop@entry=0x1864148 <default_loop_struct>, timeout=-1) at /home/runner/.cache/zig/p/N-V-__8AAH34QwB6wi5eQK_lFbfDGSN3hRE8l-6Ep198ZsGg/src/unix/linux.c:1565 #22 0x00000000014c31f1 in uv_run (loop=loop@entry=0x1864148 <default_loop_struct>, mode=mode@entry=UV_RUN_DEFAULT) at /home/runner/.cache/zig/p/N-V-__8AAH34QwB6wi5eQK_lFbfDGSN3hRE8l-6Ep198ZsGg/src/unix/core.c:460 #23 0x000000000125e791 in _cw_serf_commence (argc=<optimized out>, argv=<optimized out>) at pkg/vere/main.c:1299 #24 _cw_utils (argc=argc@entry=11, argv=argv@entry=0x7fffffffdf08) at pkg/vere/main.c:3250 #25 0x000000000125c7d9 in main (argc=11, argv=0x7fffffffdf08) at pkg/vere/main.c:3275 ``` @quodss @joemfb

matthew-levan marked this pull request as draft November 22, 2022 18:31

matthew-levan linked an issue Nov 22, 2022 that may be closed by this pull request

Set up bazel-based continuous integration flow #8

Closed

matthew-levan force-pushed the issue/8/ci branch 10 times, most recently from 1027e53 to a29e8bb Compare November 29, 2022 15:23

matthew-levan force-pushed the issue/8/ci branch 5 times, most recently from 8dd1cfe to 5997e90 Compare December 5, 2022 20:36

This was linked to issues Dec 6, 2022

Select proper system include paths on macOS #26

Closed

Use bazel caching to improve CI performance #29

Closed

matthew-levan self-assigned this Dec 6, 2022

This was linked to issues Dec 13, 2022

Build for macos-x86_64 without sandboxing #30

Closed

Build and release Docker images #25

Closed

mcevoypeter reviewed Dec 14, 2022

View reviewed changes

This was linked to issues Dec 14, 2022

Remove support for openbsd and windows platforms #32

Closed

Reduce CI log noise/verbosity on build static binary and run tests jobs #35

Closed

mcevoypeter force-pushed the issue/1/bazel-build branch from 07d9308 to 3aed4db Compare December 15, 2022 21:20

Simple build/test for linux-x86_64 host/target

0ed3c50

matthew-levan force-pushed the issue/8/ci branch from 0600202 to 0ed3c50 Compare December 16, 2022 00:30

matthew-levan added 2 commits December 16, 2022 13:33

Pull arvo using git; use next/vere pill

f66edcc

Quiet zip and disable tests on macos-x86_64 builds

f4d4fb6

matthew-levan added 5 commits January 4, 2023 12:20

Merge branch 'issue/8/ci' of https://github.com/urbit/vere into issue…

b2a55e9

…/8/ci

Cleanup non-runtime language from docs

9d2420a

Fetch arvo with versioned_http_archive rule

9bccf55

Update arvo archive

cda49b8

vere.yml -> shared.yml

eab82e4

matthew-levan mentioned this pull request Jan 5, 2023

Migrate from Urbit monorepo #39

Closed

21 tasks

matthew-levan and others added 12 commits January 5, 2023 09:58

Add arvo to boot-fake-ship deps

4d2a489

Replace arvo http_archive with urbit http_file

2d289bc

Replace $(location) with $(execpath)

5ab0ef1

Use long options in fake ship tests

2b31df6

Simplify loopback port lookup

e89fac6

Run unit tests on macOS in CI

3131297

Cache only musl, not bazel

39b24dd

Use brew_clang toolchain for macOS tests

f947230

Tweak wording in GitHub workflows

c65ce5c

Don't upload binaries when running build workflow

d6bb798

Rename GCS_PROJECT as GCP_PROJECT

0b0717f

Use switch statement in build and test step

c6e674e

mcevoypeter force-pushed the issue/8/ci branch from 0702fa6 to 1e31302 Compare January 5, 2023 17:44

Reorder key and path in cache step

1e31302

mcevoypeter approved these changes Jan 5, 2023

View reviewed changes

Comment thread pkg/vere/BUILD.bazel Outdated

matthew-levan merged commit 8f6334a into issue/1/bazel-build Jan 5, 2023

matthew-levan deleted the issue/8/ci branch January 5, 2023 18:43

matthew-levan mentioned this pull request Jan 5, 2023

Review and update continuous deploy flow #9

Closed

matthew-levan linked an issue Jan 5, 2023 that may be closed by this pull request

Review and update continuous deploy flow #9

Closed

zalberico unassigned matthew-levan Jan 11, 2023

This was referenced Jan 20, 2023

macos-x86_64 build fails because it can't find brew's clang #148

Closed

Always build with either system-native or brew-installed clang on macOS #151

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`bazel`-based continuous integration flow#21

`bazel`-based continuous integration flow#21
matthew-levan merged 24 commits into
issue/1/bazel-buildfrom
issue/8/ci

matthew-levan commented Nov 22, 2022 •

edited

Loading

Uh oh!

mcevoypeter left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcevoypeter left a comment

Uh oh!

Uh oh!

matthew-levan commented Jan 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

matthew-levan commented Nov 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcevoypeter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcevoypeter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matthew-levan commented Jan 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthew-levan commented Nov 22, 2022 •

edited

Loading