Skip to content

Distributed compilation support #274

Description

@waruqi

Features

  • Cross-platform support
  • Support msvc, clang, gcc and cross-compilation
  • Support to build android, ios, linux, win, macOS programs
  • No dependencies other than the compilation toolchain
  • Support for load balancing scheduling
  • Support for real-time large file compression transfer (use lz4)
  • Almost zero configuration cost, no shared file system required, more convenient and secure

支持特性

  • 跨平台支持
  • 支持 msvc, clang, gcc 和交叉编译工具链
  • 支持构建 android, ios, linux, win, macOS 程序
  • 除了编译工具链,无任何其他依赖
  • 支持编译服务器负载均衡调度
  • 支持大文件实时压缩传输 (lz4)
  • 几乎零配置成本,无需共享文件系统,更加方便和安全

Auto-generate server/client configuration.

$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<remote_build_server>: listening 0.0.0.0:9691 ..
<distcc_build_server>: listening 0.0.0.0:9692 ..
$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            ndk = { }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}
$cat ~/.xmake/service/client.conf 
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9692",
                token = "590234653af52e91b9e438ed860f1a2b"
            }
        }
    }
}

Start service

$ xmake service
<distcc_build_server>: listening 0.0.0.0:9692 ..

Start server as daemon

$ xmake service --start

Connect project to server

$ cd projectdir
$ xmake service --connect --distcc
<client>: connect 127.0.0.1:9692 ..
<client>: 127.0.0.1:9692 connected!

Build project for distributed build

$ xmake
...
[ 93%]: ccache compiling.release src/demo/network/unix_echo_client.c         ----> local job
[ 93%]: ccache compiling.release src/demo/network/ipv6.c
[ 93%]: ccache compiling.release src/demo/network/ping.c
[ 93%]: distcc compiling.release src/demo/network/unix_echo_server.c.         ----> distcc job
[ 93%]: distcc compiling.release src/demo/network/http.c
[ 93%]: distcc compiling.release src/demo/network/unixaddr.c
[ 93%]: distcc compiling.release src/demo/network/ipv4.c
[ 94%]: distcc compiling.release src/demo/network/ipaddr.c
[ 94%]: distcc compiling.release src/demo/math/fixed.c
[ 94%]: distcc compiling.release src/demo/libm/float.c
[ 95%]: ccache compiling.release src/demo/libm/double.c
[ 95%]: ccache compiling.release src/demo/other/test.cpp
[ 98%]: archiving.release libtbox.a
[ 99%]: linking.release demo
[100%]: build ok!

Default jobs count

the default jobs count of cpu core

local default_njob = math.ceil(ncpu * 3 / 2)

the max jobs counts

local maxjobs = local_maxjobs + server_maxjobs (server count * default_njob)

Modify local jobs count

$ xmake -jN

Modify server jobs count

$cat ~/.xmake/service/client.conf 
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9692",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 8   <------- modify here
            },
            {
                connect = "192.168.01:9692",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 4
            }
        }
    }
}

Build android project

We need set NDK root path in server

$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            ndk = {
                ndk = "~/files/android-ndk-r21e"   <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Build project

$ xmake f -p android --ndk=~/files/xxxx
$ xmake

Cross-compilation

We need set cross-compilation toolchain sdk path in server

$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            cross = {
                sdkdir = "~/files/xxxx/sdk"   <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Build project

$ xmake f -p cross --sdk=~/files/xxxx
$ xmake

we can also set other configurations, e.g. bindir, cross

cross = {
    sdkdir = "~/files/xxxx/sdk",
    bindir = "~/files/xxxx/sdk/bin",
    cross = "arm-linux-"
}

Build ios project

$ xmake f -p iphoneos
$ xmake

More

More service-related configuration notes are available at: #622

Activity

  1. added this to the v2.2.5 milestone on Dec 5, 2018
  2. modified the milestones: v2.2.6, v2.2.7 on May 8, 2019
  3. modified the milestones: v2.2.7, todo on Jun 15, 2019
  4. pinned this issue on Nov 20, 2019
  5. unpinned this issue on Apr 9, 2021
  6. pinned this issue on Apr 20, 2022
  7. unpinned this issue on Apr 20, 2022
  8. modified the milestones: todo, v2.6.6 on May 15, 2022
  9. waruqi commented on May 15, 2022

    @waruqi
    MemberAuthor

    I have supported it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions