TensorRT Static Plugin VS Dynamic Plugin

Introduction

When we use the trtexec tool to build a TensorRT engine, we can specify the static and dynamic plugin libraries to be loaded.

=== System Options ===
  --device=N                  Select cuda device N (default = 0)
  --useDLACore=N              Select DLA core N for layers that support DLA (default = none)
  --staticPlugins             Plugin library (.so) to load statically (can be specified multiple times)
  --dynamicPlugins            Plugin library (.so) to load dynamically and may be serialized with the engine if they are included in --setPluginsToSerialize (can be specified multiple times)
  --setPluginsToSerialize     Plugin library (.so) to be serialized with the engine (can be specified multiple times)
  --ignoreParsedPluginLibs    By default, when building a version-compatible engine, plugin libraries specified by the ONNX parser
                              are implicitly serialized with the engine (unless --excludeLeanRuntime is specified) and loaded dynamically.
                              Enable this flag to ignore these plugin libraries instead.

It is often confusing to understand the difference between static and dynamic plugins. In fact, it is all about the life time and registration of the plugin creators. In this blog post, we will explain the difference between static and dynamic plugins and discuss their implementations.

TensorRT Static Plugin VS Dynamic Plugin

TensorRT Plugin Lifetime

The convention of static and dynamic plugins could be confusing from their names. First of all, the static plugin library and the dynamic plugin library files are both shared libraries, which means they are actually dynamic libraries with the .so extension on Linux. The difference between static and dynamic plugins is not about the file type, but about how the plugin creators are created, registered, deregistered, and destroyed.

The static plugin creator is created and registered when the static plugin library is loaded and it is deregistered and destroyed when the program exits. There are two ways to load the static plugin library, either the plugin library is linked to the program at compile time and loaded when the program starts, or the plugin library is loaded at runtime via dlopen. Either way, the static plugin creator is created and registered when the static plugin library is loaded. It should be noted that if the static plugin library is loaded via dlopen, dlclose early in the program while the plugins are still being used will cause undefined behavior. Therefore, the lifetime and registration of static plugins are tightly coupled with the lifetime of the plugin library.

In contrast, dynamic plugin creator is created, registered, deregistered, and destroyed when certain functions are called in the middle of the program. But similar to the static plugin library, the dynamic plugin library is loaded when the program starts or dynamically via dlopen. Therefore, the lifetime and registration of dynamic plugins are not tightly coupled with the lifetime of the plugin library and the dynamic plugins are operated within the lifetime of the plugin library.

TensorRT Plugin Static Registration

The REGISTER_TENSORRT_PLUGIN preprocessor implemented in the static plugin library is used to register the plugin creator statically.

#define REGISTER_TENSORRT_PLUGIN(name) static nvinfer1::PluginRegistrar<name> pluginRegistrar##name {}

Essentially, the REGISTER_TENSORRT_PLUGIN macro creates a static variable pluginRegistrar##name of type nvinfer1::PluginRegistrar<name> in the static plugin library. The constructor of nvinfer1::PluginRegistrar<name> will create and register the plugin creator. The implementation of nvinfer1::PluginRegistrar is as follows:

template <typename T>
class PluginRegistrar
{
public:
    PluginRegistrar()
    {
        getPluginRegistry()->registerCreator(instance, "");
    }

private:
    //! Plugin instance.
    T instance{};
};

Note that the namespace where the plugin creator is registered is static constant and could not be configured at the runtime.

If the static plugin library is linked to the program at compile time, because nvinfer1::PluginRegistrar<name> is static, the plugin creator will be created and registered when the program starts, and the plugin creator is destroyed when the program exits.

If the static plugin library is loaded at runtime via dlopen, we have to make sure the plugin creator is always valid before TensorRT runtime completes using it. This means dlclose should not be called before TensorRT runtime completes using the plugin creator.

The trtexec can load static libraries via the --staticPlugins option. The static plugin library is loaded via a loadLibrary function call, which returns a pointer to a DynamicLibrary object, a “static plugin” wrapped in a “dynamic library” object.

inline std::unique_ptr<DynamicLibrary> loadLibrary(std::string const& path)
{
    // make_unique not available until C++14 - we still need to support C++11 builds.
    return std::unique_ptr<DynamicLibrary>(new DynamicLibrary{path});
}

The DynamicLibrary class is a wrapper around the dlopen function and it is used to load the static plugin library.

class DynamicLibrary
{
public:
    explicit DynamicLibrary(std::string const& name)
        : mLibName{name}
    {
#if defined(_WIN32)
        mHandle = LoadLibraryA(name.c_str());
#else // defined(_WIN32)
        int32_t flags{RTLD_LAZY};
#if ENABLE_ASAN
        // https://github.com/google/sanitizers/issues/89
        // asan doesn't handle module unloading correctly and there are no plans on doing
        // so. In order to get proper stack traces, don't delete the shared library on
        // close so that asan can resolve the symbols correctly.
        flags |= RTLD_NODELETE;
#endif // ENABLE_ASAN

        mHandle = dlopen(name.c_str(), flags);
#endif // defined(_WIN32)

        if (mHandle == nullptr)
        {
            std::string errorStr{};
#if !defined(_WIN32)
            errorStr = std::string{" due to "} + std::string{dlerror()};
#endif
            throw std::runtime_error("Unable to open library: " + name + errorStr);
        }
    }

    DynamicLibrary(DynamicLibrary const&) = delete;
    DynamicLibrary(DynamicLibrary const&&) = delete;

    //!
    //! Retrieve a function symbol from the loaded library.
    //!
    //! \return the loaded symbol on success
    //! \throw std::invalid_argument if loading the symbol failed.
    //!
    template <typename Signature>
    std::function<Signature> symbolAddress(char const* name)
    {
        if (mHandle == nullptr)
        {
            throw std::runtime_error("Handle to library is nullptr.");
        }
        void* ret;
#if defined(_MSC_VER)
        ret = static_cast<void*>(GetProcAddress(static_cast<HMODULE>(mHandle), name));
#else
        ret = dlsym(mHandle, name);
#endif
        if (ret == nullptr)
        {
            std::string const kERROR_MSG(mLibName + ": error loading symbol: " + std::string(name));
            throw std::invalid_argument(kERROR_MSG);
        }
        return reinterpret_cast<Signature*>(ret);
    }

    ~DynamicLibrary()
    {
        try
        {
#if defined(_WIN32)
            ASSERT(static_cast<bool>(FreeLibrary(static_cast<HMODULE>(mHandle))));
#else
            ASSERT(dlclose(mHandle) == 0);
#endif
        }
        catch (...)
        {
            sample::gLogError << "Unable to close library: " << mLibName << std::endl;
        }
    }

private:
    std::string mLibName{}; //!< Name of the DynamicLibrary
    void* mHandle{};        //!< Handle to the DynamicLibrary
};

TensorRT Plugin Dynamic Registration

Because the plugin creator can be created and registered dynamically, it is possible to defer the plugin creator creation and registration until the program needs it.

If the dynamic plugin library is linked to the program at compile time, the dynamic plugin library is loaded when the program starts. But we could create and register the plugin creator after the program starts. We could instantiate the plugin creator using the declaration of the plugin creator class from the header file. The registration of the plugin creator is also performed using the nvinfer1::IPluginRegistry::registerCreator function. But this time, the user could decide when to create, register, deregister, and destroy the plugin creator. This is especially useful and flexible in a custom TensorRT builder and inference application. In addition, the namespace where the plugin creator is registered can also be configured at the runtime, which is not possible for static plugins.

The user could also decide when to load the dynamic plugin library. The dynamic plugin library is loaded via dlopen and the plugin creator is created and registered when the program needs it. As long as the plugin creator is deregistered and destroyed before dlclose is called for the dynamic plugin library, there will be no undefined behavior.

If the user does not want to instantiate the plugin creator using the declaration of the plugin creator class from the header file, the user could also register the plugin creator to TensorRT runtime using the nvinfer1::IPluginRegistry::loadLibrary function in the TensorRT API. But because there is no static plugin creator creation and registration, TensorRT would not know how to create the plugin creator without the user providing more information. So TensorRT ask the user to implement the following two C extern functions in the dynamic plugin library:

extern "C" void setLoggerFinder(ILoggerFinder* finder);
extern "C" IPluginCreatorInterface* const* getCreators(int32_t& nbCreators)

A typical example of the implementations of these two functions could be found in the TensorRT plugin example.

#include "vfcCommon.h"
#include "NvInfer.h"
#include "NvInferPlugin.h"
#include "roiAlignPlugin/roiAlignPlugin.h"
#include <vector>

using namespace nvinfer1;
using nvinfer1::plugin::ROIAlignV3PluginCreator;

namespace nvinfer1::plugin
{

class ThreadSafeLoggerFinder
{
private:
    ILoggerFinder* mLoggerFinder{nullptr};
    std::mutex mMutex;

public:
    ThreadSafeLoggerFinder() = default;

    //! Set the logger finder.
    void setLoggerFinder(ILoggerFinder* finder)
    {
        std::lock_guard<std::mutex> lk(mMutex);
        if (mLoggerFinder == nullptr && finder != nullptr)
        {
            mLoggerFinder = finder;
        }
    }

    //! Get the logger.
    ILogger* getLogger() noexcept
    {
        std::lock_guard<std::mutex> lk(mMutex);
        if (mLoggerFinder != nullptr)
        {
            return mLoggerFinder->findLogger();
        }
        return nullptr;
    }
};

ThreadSafeLoggerFinder gLoggerFinder;

ILogger* getPluginLogger()
{
    return gLoggerFinder.getLogger();
}

} // namespace nvinfer1::plugin

extern "C" TENSORRTAPI IPluginCreatorInterface* const* getCreators(int32_t& nbCreators)
{
    nbCreators = 1;
    static ROIAlignV3PluginCreator sRoiAlignCreator;
    static IPluginCreatorInterface* const kPLUGIN_CREATOR_LIST[] = {&sRoiAlignCreator};
    return kPLUGIN_CREATOR_LIST;
}

extern "C" TENSORRTAPI void setLoggerFinder(nvinfer1::ILoggerFinder* finder)
{
    nvinfer1::plugin::gLoggerFinder.setLoggerFinder(finder);
}

The trtexec can also load dynamic libraries via the --dynamicPlugins option. The dynamic plugin library is loaded via a nvinfer1::IPluginRegistry::loadLibrary TensorRT API call.

for (auto const& pluginPath : mDynamicPlugins)
{
    mRuntime->getPluginRegistry().loadLibrary(pluginPath.c_str());
}

Conclusions

Because the library can be a static .a library and a dynamic .so library, the dynamic .so library can be loaded at the beginning of the program or at runtime via dlopen, the plugin creator object can be created statically or dynamically, etc., the word “static” and “dynamic” for TensorRT static library and dynamic plugin library are often confusing. In fact, the static plugin library just means the plugin creators in the library are registered immediately when the library is loaded, and the dynamic plugin library just means the plugin creators in the library are registered after the library is loaded with some level of the control from the user.

References

• Use Plugins - trtexec
• Plugin Shared Libraries - TensorRT 10.7
• Plugin Shared Libraries - TensorRT 8.6