Contribute To ONNX

Introduction

The ONNX library defines neural network operators and is commonly supported by multiple deep learning libraries in the community. Because the deep learning field is moving extremely fast, it is common that deep learning researchers and engineers will have to contribute to the ONNX library.

In this blog post, I would like to quickly discuss how to contribute to the ONNX library.

ONNX Development

Create ONNX Development Dockerfile

Create a development environment is often very difficult for any libraries. Here I have created a Docker file for building and developing the ONNX library.

onnx-dev.onn

FROM ubuntu:22.04

ARG CMAKE_VERSION=3.23.2
ARG NUM_JOBS=8

ENV DEBIAN_FRONTEND noninteractive

# Install package dependencies.
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        software-properties-common \
        autoconf \
        automake \
        libtool \
        pkg-config \
        ca-certificates \
        wget \
        git \
        curl \
        libjpeg-dev \
        libpng-dev \
        language-pack-en \
        locales \
        locales-all \
        python3 \
        python3-py \
        python3-dev \
        python3-pip \
        python3-numpy \
        python3-pytest \
        python3-setuptools \
        # The protobuf library installed via apt is sometimes not
        # compatible with the ONNX library.
        # We will build them from source instead.
        # libprotobuf-dev \
        # protobuf-compiler \
        zlib1g-dev \
        swig \
        vim \
        gdb \
        valgrind \
        libsm6 \
        libxext6 \
        libxrender-dev \
        cmake \
        unzip && \
    apt-get clean

RUN cd /usr/local/bin && \
    ln -s /usr/bin/python3 python && \
    ln -s /usr/bin/pip3 pip && \
    pip install --upgrade pip setuptools wheel

# System locale
# Important for UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

# Install CMake.
RUN cd /tmp && \
    wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-x86_64.sh && \
    bash cmake-${CMAKE_VERSION}-linux-x86_64.sh --prefix=/usr/local --exclude-subdir --skip-license

# Install Protobuf.
RUN cd /tmp && \
    git clone https://github.com/protocolbuffers/protobuf.git && \
    cd protobuf && \
    git checkout v3.20.2 && \
    git submodule update --init --recursive && \
    mkdir build_source && cd build_source && \
    cmake ../cmake -Dprotobuf_BUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release && \
    make -j$(nproc) && \
    make install

# Install ONNX development dependencies.
RUN cd /tmp && \
    wget https://raw.githubusercontent.com/onnx/onnx/main/requirements-dev.txt && \
    # Enfore re-install all the requirements even if they already exists.
    # This is a work-around to make pytest accessible from the $PATH.
    pip install --upgrade --force-reinstall -r requirements-dev.txt

# Set ONNX build CMake argument.
ENV CMAKE_ARGS "-DONNX_USE_LITE_PROTO=ON"

# Install ONNX.
# RUN cd /tmp && \
#     git clone https://github.com/onnx/onnx.git && \
#     cd onnx && \
#     git submodule update --init --recursive && \
#     # Optional: prefer lite proto
#     export CMAKE_ARGS=-DONNX_USE_LITE_PROTO=ON && \
#     pip install -e .

Build Docker Image

Build the ONNX development Docker image using the following command.

$ docker build -f onnx-dev.Dockerfile --tag onnx-dev:0.0.1 .

Run Docker Container

Pull the main branch of the ONNX repository to the host computer and start the Docker container for development.

$ git clone --recurse-submodules https://github.com/onnx/onnx.git
$ cd onnx/
$ docker run -it --rm -v $(pwd):/mnt -w /mnt onnx-dev:0.0.1

Build ONNX Library

The ONNX library needs to be built from source in development mode.

$ cd /mnt/
$ pip install -e .

Run Unit Tests

After adding or modifying the code in the ONNX repository from the host, we will have to run unit tests from the ONNX root directory in the Docker container to ensure that the changes we made are correct.

$ pytest

To only test a specific subset, we could use the -k argument for prefix patten matching. For example,

$ pytest -k test_gridsample

Regenerate Documentations and Test Files

For most of the changes related to ONNX scopes and specs, the ONNX documentations and unit test files have to be updated using the following script.

$ bash tools/update_doc.sh

Format Code

lintrunner is recommended to be performed from the ONNX root directory on the host outside the Docker container.

$ lintrunner

References

• ONNX Contributing Guideline
• GridSample ND