Lei Mao bio photo

Lei Mao

Machine Learning, Artificial Intelligence, Computer Science.

Twitter Facebook LinkedIn GitHub   G. Scholar E-Mail RSS

Introduction

Building and deploying applications on IoT devices are sometimes slow due to the SoC on the IoT devices are usually not as fast as the CPU we normally used on our desktop. To make the building faster, we could download cross-compilation tools for certain platforms, run cross-platform compilations on our desktop targeting certain platforms, and copy the binaries to target device once the compilation is done.


For Docker container users on IoT devices, such cross-platform building solution from source code is sometimes inconvenient, since they not only have to build the applications but also create a software environment for the same platform inside Docker container.


Fortunately, Docker has created a building tool Buildx that emulates the Docker image building process on targeting platforms. This means, for example, we could build an arm64 Docker image from our local amd64 desktop, push the Docker image to Docker Hub, and download the Docker image from Docker Hub to the target arm64 device.


In this blog post, I would like to show how to use Docker Buildx to build cross-platform Docker images.

Cross-Platform Buildx Build

Install Docker

For completeness, I have included the protocol for Docker installation on Ubuntu on amd64 platform. The complete protocol could be found from the official Docker installation instructions.

$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker 

Install Emulation Dependencies

To emulate the compilation and building on other platforms, we have to install QEMU.

$ sudo apt-get install binfmt-support qemu-user-static

Select Base Image

To build a Docker image that is compatible with platforms other than our local amd64 platform, we have to select the base image carefully. If the base image was not built on the target device, the Docker build emulation will not be successful.


For an example, the PyTorch 1.7.0 image pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel from Docker Hub was only built on the amd64 platform. So definitely we could not use this as a base image for creating a Docker image for the arm64 platform.

$ docker manifest inspect --verbose pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel
{
	"Ref": "docker.io/pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel",
	"Descriptor": {
		"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
		"digest": "sha256:837e6964e5db6e5b35f4d5e98e9cac073ab757766039b9503f39c14beafb0e98",
		"size": 2848,
		"platform": {
			"architecture": "amd64",
			"os": "linux"
		}
	},
	"SchemaV2Manifest": {
		"schemaVersion": 2,
		"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
		"config": {
			"mediaType": "application/vnd.docker.container.image.v1+json",
			"size": 10699,
			"digest": "sha256:f20d42e5d606f02b790edccc1e6741e0f287ee705a94998fd50c160e96301823"
		},
		"layers": [
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 26701612,
				"digest": "sha256:171857c49d0f5e2ebf623e6cb36a8bcad585ed0c2aa99c87a055df034c1e5848"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 852,
				"digest": "sha256:419640447d267f068d2f84a093cb13a56ce77e130877f5b8bdb4294f4a90a84f"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 162,
				"digest": "sha256:61e52f862619ab016d3bcfbd78e5c7aaaa1989b4c295e6dbcacddd2d7b93e1f5"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 7213532,
				"digest": "sha256:2a93278deddf8fe289dceef311ed19e8f2083a88eba6be60d393842fd40697b0"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 10326692,
				"digest": "sha256:c9f080049843544961377a152d7d86c34816221038b8da3e3dc207ccddb72549"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 1002,
				"digest": "sha256:8189556b23294579329c522acf5618c024520b323d6a68cdd9eca91ca4f2f454"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 1134108738,
				"digest": "sha256:c306a0c97a557ede3948263983918da203f1837a354a86fcb5d6270b0c52b9ad"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 970886836,
				"digest": "sha256:4a9478bd0b2473c3d7361f9a0a8e98923897103b9b2eb55097db2b643f50c13e"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 1057726095,
				"digest": "sha256:19a76c31766d36601b8c4a57ea5548a4e22f69846ac653c5ca2bea5eb92b759d"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 1013655,
				"digest": "sha256:1d18e0f6b7f66fdbaba1169a3439577dc12fd53b21d7507351a9098e68eb6207"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 2316494468,
				"digest": "sha256:d8015a90b67c809145c04360809eba130365a701a96319f8fc2c3c786434c33a"
			},
			{
				"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
				"size": 138,
				"digest": "sha256:211a7eed3486a96a5e8ba778a64f46475a7131e3b66ccc4ee3af57e334fb534f"
			}
		]
	}
}

The Ubuntu 20.04 image from Docker Hub, however, has multiple manifests. It has builds for amd64, arm-v7, arm64-v8, etc. So we could use ubuntu:20.04 as a base image for creating a Docker image for the aarch64 platform.

$ docker manifest inspect --verbose ubuntu:20.04
[
	{
		"Ref": "docker.io/library/ubuntu:20.04@sha256:4e4bc990609ed865e07afc8427c30ffdddca5153fd4e82c20d8f0783a291e241",
		"Descriptor": {
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"digest": "sha256:4e4bc990609ed865e07afc8427c30ffdddca5153fd4e82c20d8f0783a291e241",
			"size": 943,
			"platform": {
				"architecture": "amd64",
				"os": "linux"
			}
		},
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 3316,
				"digest": "sha256:f643c72bc25212974c16f3348b3a898b1ec1eb13ec1539e10a103e6e217eb2f1"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 28563271,
					"digest": "sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 847,
					"digest": "sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 162,
					"digest": "sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601"
				}
			]
		}
	},
	{
		"Ref": "docker.io/library/ubuntu:20.04@sha256:be2aa2178e05b3d1930b4192ba405cb1d260f6a573abab4a6e83e0ebec626cf1",
		"Descriptor": {
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"digest": "sha256:be2aa2178e05b3d1930b4192ba405cb1d260f6a573abab4a6e83e0ebec626cf1",
			"size": 943,
			"platform": {
				"architecture": "arm",
				"os": "linux",
				"variant": "v7"
			}
		},
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 3314,
				"digest": "sha256:bcf7edb34eae8d169db3f671029269af8f20bb485b648b35771104747d602f5f"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 24041185,
					"digest": "sha256:a3a20c5d313f7c8b8e8c7633f51e0ecfd29c87d093315961a38a46da87c3b2be"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 852,
					"digest": "sha256:67966bd810cedcb4375094e9af4a7197675edc270d8d16004ab58a8daacbe918"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 188,
					"digest": "sha256:c8282ed6897607dd46838e55e3fabf03616b42c2275226012a008da3cf081051"
				}
			]
		}
	},
	{
		"Ref": "docker.io/library/ubuntu:20.04@sha256:42c332a4493b201f8a5e3d4019e464aa2f5c6e6ef8fedccd0b6d3a7ac0912670",
		"Descriptor": {
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"digest": "sha256:42c332a4493b201f8a5e3d4019e464aa2f5c6e6ef8fedccd0b6d3a7ac0912670",
			"size": 943,
			"platform": {
				"architecture": "arm64",
				"os": "linux",
				"variant": "v8"
			}
		},
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 3316,
				"digest": "sha256:1c28a1589115c49dd786ca8009770815b4529a7de27b75f1f94fa1ff520fcfb0"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 27168047,
					"digest": "sha256:a970164f39c1a46f71b3615bc9d5b6710832766b530d9179db8e36563f705abb"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 848,
					"digest": "sha256:e9c66f1fb5a2d6587841797a3b0d4c2d0fd0b7ccd867e55a1314cee2e90ad47d"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 188,
					"digest": "sha256:94362ba2c285844f83a1b1e2dac5217b0426427f8bb809af534b5f4d751e298c"
				}
			]
		}
	},
	{
		"Ref": "docker.io/library/ubuntu:20.04@sha256:21f4dd9e02054a3ef9048a2f1384f64ba6368dc85fecbb1fb0d5592b75173e4d",
		"Descriptor": {
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"digest": "sha256:21f4dd9e02054a3ef9048a2f1384f64ba6368dc85fecbb1fb0d5592b75173e4d",
			"size": 943,
			"platform": {
				"architecture": "ppc64le",
				"os": "linux"
			}
		},
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 3318,
				"digest": "sha256:4fb14a3bd67fd596ce1b6a26a9848209197e7841748d973a534d7b56176c0ff2"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 33278639,
					"digest": "sha256:0162c086f3047a91e0409b0f80e741887de3c5e5490b9be62d74a5c054c7d6b0"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 855,
					"digest": "sha256:61108ece12882b4998d113ea62b7f4d286877fef610d4aae5b2d2f17d68bf8a4"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 188,
					"digest": "sha256:9a92bd8a59839464b3f3c73564e77cb7f1868ac3e96fcc07c1a77b85c51ba1b1"
				}
			]
		}
	},
	{
		"Ref": "docker.io/library/ubuntu:20.04@sha256:54585b0cee318ba7997bf4d1342f27754889ebf7be8c2f3a3f59752e856a7904",
		"Descriptor": {
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"digest": "sha256:54585b0cee318ba7997bf4d1342f27754889ebf7be8c2f3a3f59752e856a7904",
			"size": 943,
			"platform": {
				"architecture": "s390x",
				"os": "linux"
			}
		},
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 3316,
				"digest": "sha256:a376249dbc0054e2cbe74803b8d4c10aa1e51d7add19b8e1f54710c2c0e5beeb"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 27206899,
					"digest": "sha256:d777bd6e5a8d1fc6019fe013e0f29e35d2de6a93848ec43ec4b7bcabe67c7d1a"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 846,
					"digest": "sha256:c2fc12e4949d381d9eebf1d32c7547d9d4ebaed0142486b9f416d944257c44c3"
				},
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 186,
					"digest": "sha256:a15ab9f4a460426ab1fe52891f15e15e67d5b2f246f51b1a6db49e628d6b56f4"
				}
			]
		}
	}
]

In the latest Docker, buildx was installed by default. We could also examine the Docker images using the following command.

$ docker buildx imagetools inspect ubuntu:20.04
Name:      docker.io/library/ubuntu:20.04
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c
           
Manifests: 
  Name:      docker.io/library/ubuntu:20.04@sha256:4e4bc990609ed865e07afc8427c30ffdddca5153fd4e82c20d8f0783a291e241
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/library/ubuntu:20.04@sha256:be2aa2178e05b3d1930b4192ba405cb1d260f6a573abab4a6e83e0ebec626cf1
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      docker.io/library/ubuntu:20.04@sha256:42c332a4493b201f8a5e3d4019e464aa2f5c6e6ef8fedccd0b6d3a7ac0912670
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8
             
  Name:      docker.io/library/ubuntu:20.04@sha256:21f4dd9e02054a3ef9048a2f1384f64ba6368dc85fecbb1fb0d5592b75173e4d
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le
             
  Name:      docker.io/library/ubuntu:20.04@sha256:54585b0cee318ba7997bf4d1342f27754889ebf7be8c2f3a3f59752e856a7904
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x

Prepare Dockerfile

In this example, we would like to build an OpenCV Docker image for both the amd64 and the arm64 platform from our local amd64 computer.

FROM ubuntu:20.04

ARG OPENCV_VERSION=4.5.0
ARG NUM_JOBS=8

ENV DEBIAN_FRONTEND noninteractive

# Install package dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        autoconf \
        automake \
        libtool \
        pkg-config \
        ca-certificates \
        wget \
        git \
        curl \
        libjpeg-dev \
        libpng-dev \
        language-pack-en \
        locales \
        locales-all \
        python3 \
        python3-dev \
        python3-pip \
        python3-numpy \
        python3-setuptools \
        libprotobuf-dev \
        protobuf-compiler \
        zlib1g-dev \
        swig \
        vim \
        gdb \
        valgrind \
        libsm6 \
        libxext6 \
        libxrender-dev \
        cmake \
        unzip \
        sudo
RUN apt-get clean

RUN cd /usr/local/bin && \
    ln -s /usr/bin/python3 python && \
    ln -s /usr/bin/pip3 pip && \
    pip install --upgrade pip setuptools wheel

# System locale
# Important for UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

# Install OpenCV
# OpenCV-Python dependencies
RUN apt-get update && apt-get install -y python3-dev python3-numpy
RUN apt-get update && apt-get install -y libavcodec-dev libavformat-dev libswscale-dev
RUN apt-get update && apt-get install -y libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev
RUN apt-get update && apt-get install -y libgtk-3-dev 
RUN apt-get update && apt-get install -y libpng-dev libopenexr-dev libtiff-dev libwebp-dev

RUN cd /tmp && \
    wget -O opencv.zip https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
    wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/${OPENCV_VERSION}.zip && \
    unzip opencv.zip && \
    unzip opencv_contrib.zip && \
    mkdir -p build && cd build && \
    cmake -DOPENCV_EXTRA_MODULES_PATH=../opencv_contrib-${OPENCV_VERSION}/modules ../opencv-${OPENCV_VERSION} && \
    cmake --build . --parallel ${NUM_JOBS} && \
    make install
RUN rm -rf /tmp/*

Build Docker Image

To build Docker image on other platforms, we have to create a builder first.

$ docker buildx create --use --name cross-platform-build
cross-platform-build

The new builder supports emulating the building on multiple platforms, such as amd64, arm64, and riscv64, etc.

$ docker buildx inspect --bootstrap cross-platform-build
[+] Building 2.9s (1/1) FINISHED                                                                                                                                                                           
 => [internal] booting buildkit                                                                                                                                                                       2.9s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                    1.2s
 => => creating container buildx_buildkit_cross-platform-build0                                                                                                                                       1.6s
Name:   cross-platform-build
Driver: docker-container

Nodes:
Name:      cross-platform-build0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

With this builder, we can specify the platform we want to build the image on. Once the build is complete, the Docker images with the amd64 and the arm64 manifests will be uploaded to Docker Hub. To make sure the image upload is successful, please make sure we have logged in Docker Hub using the following command.

$ docker login

To run Docker image build and push the built images, please run the following command.

$ docker buildx build -f opencv.Dockerfile --platform linux/amd64,linux/arm64 -t leimao/opencv:4.5.0 --push .
$ docker buildx imagetools inspect leimao/opencv:4.5.0
Name:      docker.io/leimao/opencv:4.5.0
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:aa3fb98c6c3e88e7d1d58293eab55c8b32596b04dc20bae107031018a0fda4db
           
Manifests: 
  Name:      docker.io/leimao/opencv:4.5.0@sha256:9a876a93bb0118376018b986d72a6370ea790aa44e8b7418455a8922c0afdc07
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/leimao/opencv:4.5.0@sha256:f2df19fc509d369ff09bbab1491e2c3e5720e895fbfef0e1f5305410b8f33933
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

Once the Docker images has been pushed, we could run the Docker container on both the amd64 and arm64 platform.

$ docker pull leimao/opencv:4.5.0
$ docker run -it --rm leimao/opencv:4.5.0

To test if OpenCV has been built and installed successfully, please run the following command.

$ cd /tmp
$ wget -O opencv_extra.zip https://github.com/opencv/opencv_extra/archive/${OPENCV_VERSION}.zip && \
$ unzip opencv_extra.zip
$ export OPENCV_TEST_DATA_PATH="/tmp/opencv_extra-4.5.0/testdata"
$ cd /tmp/build/bin
$ ./opencv_test_core
...
[==========] 11587 tests from 244 test cases ran. (65537 ms total)
[  PASSED  ] 11587 tests.

To check if OpenCV-Python has been built and installed successfully, please run the following command.

$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.__version__)
4.5.0

References