Setting Up Environment Variables In SSH Sessions Over TCP On Runpod

Introduction

Runpod has a bug that prevents the setting of environment variables in the custom Docker container that was connected to via full SSH over TCP, which is necessary for setting up remote development environments using IDEs. This bug, however, does not exist in the same Docker container that was connected to via basic SSH.

In this blog post, I would like to quickly discuss how to work around this bug in SSH sessions over TCP on Runpod.

Setting Up Environment Variables In SSH Sessions Over TCP On Runpod

Configure Environment Variables In Templates

There are a few ways to configure environment variables in templates on Runpod, which has been described in details in the Runpod documentation “Environment variables”.

Environment Variables Are Visible In Basic SSH Sessions

When connecting to the custom Docker container via basic SSH, the environment variables configured in the template are visible in the SSH session.

For example, I configured an environment variable GIT_REPO_ACCESS_TOKEN in a template on Runpod. After a pod instance was instantiated using this template, I connected to the pod instance via basic SSH and ran the command env to list all environment variables, and I could see the configured environment variable GIT_REPO_ACCESS_TOKEN in the output.

$ ssh 4hlesiqw1k88e5-644111dc@ssh.runpod.io -i ~/.ssh/id_ed25519

-- RUNPOD.IO --
Enjoy your Pod #4hlesiqw1k88e5 ^_^

root@03013cc93cd4:/# env
NV_LIBCUBLAS_VERSION=12.6.4.1-1
NVIDIA_VISIBLE_DEVICES=all
NV_NVML_DEV_VERSION=12.6.77-1
NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6
NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1
HOSTNAME=03013cc93cd4
NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551
NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1
NV_NVTX_VERSION=12.6.77-1
NV_CUDA_CUDART_DEV_VERSION=12.6.77-1
NV_LIBCUSPARSE_VERSION=12.5.4.2-1
GIT_REPO_ACCESS_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
NV_LIBNPP_VERSION=12.3.1.54-1
NCCL_VERSION=2.23.4-1
PWD=/
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1
NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1
NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1
NVIDIA_PRODUCT_NAME=CUDA
RUNPOD_CPU_COUNT=9
NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6
NV_CUDA_CUDART_VERSION=12.6.77-1
HOME=/root
JUPYTER_PASSWORD=rqe6x0xvn0mlxzy5jn5g
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
CUDA_VERSION=12.6.3
RUNPOD_POD_ID=4hlesiqw1k88e5
NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1
NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1
NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1
NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6
RUNPOD_MEM_GB=50
RUNPOD_PUBLIC_IP=69.30.85.179
NV_LIBNPP_DEV_VERSION=12.3.1.54-1
RUNPOD_GPU_COUNT=1
TERM=xterm
NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1
RUNPOD_POD_HOSTNAME=4hlesiqw1k88e5-644111dc
LIBRARY_PATH=/usr/local/cuda/lib64/stubs
RUNPOD_DC_ID=CA-MTL-1
SHLVL=1
NV_CUDA_LIB_VERSION=12.6.3-1
NVARCH=x86_64
RUNPOD_GPU_NAME=NVIDIA+RTX+A5000
NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1
RUNPOD_TCP_PORT_22=22174
NV_NVPROF_VERSION=12.6.80-1
RUNPOD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NV_LIBNCCL_PACKAGE_NAME=libnccl2
NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1
PUBLIC_KEY=ssh-ed25519 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx@xxxxx.com
_=/usr/bin/env

Environment Variables Are Not Visible In Full SSH Sessions Via TCP

However, when I connected to the same pod instance via full SSH over TCP, the environment variables configured in the template GIT_REPO_ACCESS_TOKEN were not visible in the SSH session. In addition, other environment variables from the original Docker container, such as PATH, were also missing.

$ ssh root@69.30.85.179 -p 22174 -i ~/.ssh/id_ed25519
The authenticity of host '[69.30.85.179]:22174 ([69.30.85.179]:22174)' can't be established.
ED25519 key fingerprint is SHA256:sohNIHzDM77rTNptF7qNmASbIdLAMN4K/G+oq9Pymi4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[69.30.85.179]:22174' (ED25519) to the list of known hosts.
Welcome to Ubuntu 24.04.1 LTS (GNU/Linux 5.15.0-102-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@03013cc93cd4:~# env
SHELL=/bin/bash
PWD=/root
LOGNAME=root
HOME=/root
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
SSH_CONNECTION=98.207.105.156 56160 192.168.0.2 22
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=root
SHLVL=1
SSH_CLIENT=98.207.105.156 56160 22
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SSH_TTY=/dev/pts/0
_=/usr/bin/env

Inspiration of a Workaround

I happened to find that Runpod will add a few environment variables, including PUBLIC_KEY and JUPYTER_PASSWORD, in the template for the pod instance, even if the user does not explicitly configure these environment variables in the template. Note that these two environment variables are only visible in the basic SSH session, but not in the full SSH session via TCP.

In addition, the “Container Start Command” used for instantiating SSH daemon in the pod instance uses the environment variable PUBLIC_KEY.

# Instantiating SSH daemon.
# https://docs.runpod.io/pods/configuration/use-ssh#full-ssh-via-public-ip-with-key-authentication
apt update;
DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;
mkdir -p ~/.ssh;
cd $_;
chmod 700 ~/.ssh;
echo "$PUBLIC_KEY" >> authorized_keys;
chmod 700 authorized_keys;
service ssh start;
sleep infinity;

This inspired me to think that environment variables configured in the template must be visible when pod instance is firstly instantiated. If we could save the environment variables to files when the pod instance is instantiated, later we could read the environment variables from these files in the full SSH session via TCP.

Save Environment Variables to Files When Pod Instance Is Instantiated

Instead of using the recommended “Container Start Command” for instantiating SSH daemon in the pod instance, I modified it to save the environment variables to files first, and then instantiate the SSH daemon. The modified “Container Start Command” is as follows.

# Save environment variables to files.
mkdir -p ~/.env_vars;
env > ~/.env_vars/env_vars.txt;
# Instantiating SSH daemon.
# https://docs.runpod.io/pods/configuration/use-ssh#full-ssh-via-public-ip-with-key-authentication
apt update;
DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;
mkdir -p ~/.ssh;
cd $_;
chmod 700 ~/.ssh;
echo "$PUBLIC_KEY" >> authorized_keys;
chmod 700 authorized_keys;
service ssh start;
sleep infinity;

Formally, the one line version of the modified “Container Start Command” used for templates on Runpod is as follows.

bash -c 'mkdir -p ~/.env_vars;env > ~/.env_vars/env_vars.txt;apt update;DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $_;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorized_keys;service ssh start;sleep infinity'

Set Up Environment Variables In Full SSH Sessions Via TCP

Because the environment variables have been saved to files when the pod instance is instantiated, after we connect to the pod instance via full SSH over TCP, we could read the environment variables from these files and export them to the current SSH session using the following command.

while IFS= read -r line; do
  export "$line"
done < ~/.env_vars/env_vars.txt

References

• Environment Variables In Direct SSH - Answer Overflow