TensorRT Plugin Version and Namespace

Introduction

TensorRT plugins are essential for extending the capabilities of TensorRT by allowing developers to implement custom layers and operations. However, when the inference becomes complex, multiple plugins may exist and be used, the mapping of plugins to custom layers can also become complex. In this case, TensorRT plugins of the same name can be used for different custom layers, and the same custom layer may want to use different plugins in different scenarios. To handle these situations, TensorRT uses versions and namespaces, in addition to the plugin name, to create a mapping from a custom layer to a TensorRT plugin.

In this blog post, I would like to explain how TensorRT uses plugin versions and namespaces to handle conflicts and ensure that the correct plugin is used for each custom layer.

TensorRT Plugin Version and Namespace

TensorRT plugins, when registered, have three attributes used for the mapping to custom layers:

• Plugin Name: A string that identifies the plugin.
• Plugin Version: An string that indicates the version of the plugin.
• Plugin Namespace: A string that provides a namespace for the plugin, allowing for grouping and avoiding conflicts with plugins of the same name. The default namespace is an empty string "".

IPluginCreator and IPluginV2 Interface

Prior to TensorRT 10, TensorRT plugin creator, which is used for creating plugins, and plugin uses the IPluginCreator and IPluginV2 interfaces, respectively. A TensorRT plugin creator can be uniquely queried by its name, version, and namespace using the nvinfer1::IPluginRegistry::getPluginCreator API.

The plugin name and plugin version are usually hard-coded in the plugin implementation and cannot be changed duringt the runtime. They can be queried using the nvinfer1::IPluginCreator::getPluginName and nvinfer1::IPluginCreator::getPluginVersion APIs, respectively, by TensorRT Runtime. The plugin namespace can be set when the plugin is registered using the nvinfer1::IPluginRegistry::registerCreator API, and can be queried using the nvinfer1::IPluginCreator::getPluginNamespace API by TensorRT Runtime. Before registering the plugin, make sure the plugin namespace is also set to the plugin creator via the nvinfer1::IPluginCreator::setPluginNamespace API.

IPluginCreatorV3One and IPluginV3 Interface

Since TensorRT 10, IPluginCreator and IPluginV2 have been deprecated, and the new interfaces IPluginCreatorV3One and IPluginV3 are introduced for TensorRT plugin creator and plugin, respectively.

In terms of plugin name, version, and namespace, compared to IPluginCreator and IPluginV2, the main difference is that the plugin namespace becomes hard-coded as well, just like the plugin name and plugin. The user cannot call setPluginNamespace to change the plugin namespace at runtime anymore. The plugin namespace in this case is more like the namespace in C++ which is used for denoting the unique source organization or project of the plugin.

TensorRT Engine Building

Usually, TensorRT engines are created from ONNX models. The ONNX parser automatically attempts to import unrecognized nodes as plugins. If a plugin with the same plugin name as the node is found in the plugin registry, the parser forwards the attributes of the node to the plugin creator as plugin field parameters in order to create the plugin. The plugin version and plugin namespace are described in the ONNX node attributes plugin_version and plugin_namespace, respectively. If these attributes are not set, the parser uses "1" as the plugin version and "" as the plugin namespace by default. The ONNX node attributes plugin_version and plugin_namespace can be specified using ONNX graph surgeon or PyTorch ONNX export, examples could be found on my GitHub.

In trtexec, --dynamicPlugins seems to register the plugin creators to the namespace queried using getPluginNamespace API, --staticPlugins registers the plugin creators to the default namespace "", and --safePlugins registers the plugin creators to the namespace the user specifies in the command.

During serialization, the TensorRT engine internally stores the plugin type, plugin version, and namespace (if it exists) for all plugins.

TensorRT Engine Execution

During deserialization, TensorRT looks up the plugin creator from the plugin registry using the plugin type, version, and namespace.

Examples

TensorRT Plugins of The Same Name and Version

If we have two plugins with the same name and version, which may or may not have the same interface and functionality, and we want to use them for specific custom layers unambiguously, we can should register them with different namespaces. In the custom layers, the plugin namespace must be specified. In this case, the ONNX custom layer attributes plugin_namespace should be set to the namespace of the desired plugin.

This sometimes happens when we cannot change the plugin name or version from the source code.

TensorRT Plugins of The Same Name

It is possible to have multiple TensorRT plugins to work with one ONNX custom layer. This could be because the TensorRT plugins have been developed by different teams, or because the plugin implementation has been improved over time.

If the plugin versions, which are hard coded in the plugin implementation, are completely different between those plugins, we could specify the plugin version in the ONNX custom layer attributes plugin_version to select the desired plugin.

If the plugin versions are the same, as mentioned above, we must register the plugins with different namespaces. In this case, we can specify the plugin namespace in the ONNX custom layer attributes plugin_namespace to select the desired plugin.

References

• TensorRT Static Plugin VS Dynamic Plugin
• Using Custom Layers When Importing a Model with a Parser