How to do remote debugging

In this section, we will introduce how to use vscode to remotely access the Docker container for c++/python development/debugging.

To complete this, there are two main steps:

• Step 1: Configure the Docker on the remote host
• Step 2: Configure vscode

Step 1: Configure Docker on the remote host

1. Start Docker

To allow vscode to SSH into the Docker container, we need to assign a port number when starting the Docker container. The specific startup method is as follows: (8022 is the host port, which can be modified to avoid conflicts. 22 is the fixed port inside Docker and can be left as default.)

docker_image_name=nvcr.io/nvidia/pytorch:22.12-py3
docker run -it --gpus all --ipc=host   --shm-size 1G  --ulimit memlock=-1 --ulimit stack=67108864 -p 8022:22 -v `pwd`:/workspace  -w/workspace  --name="debug_tp"  --cap-add=SYS_PTRACE $docker_image_name  bash

2. Add password inside Docker

passwd

3. Install openssh

apt-get update -y
apt-get install -y  openssh-server  openssh-client vim gdb

Modify system configuration

vim /etc/ssh/sshd_config

Edit the following three lines:

Port 22
PermitRootLogin yes
PubkeyAuthentication yes

Start the service: /etc/init.d/ssh start or /etc/init.d/ssh restart

After starting the service, you can perform connection verification: ssh -p 22 root@127.0.0.1. If the startup fails, it will report a "Connection refused" error. If successful, it will prompt you to enter a password and display a "Welcome" message.

6. Compile torchpipe in debug mode

Change directory to torchpipe:

## debug模式，
DEBUG=1 pip3 install -e .
## debug模式， 并且子线程异常不转发到主线程，就地宕机，以便定位位置
DEBUG=1 NCATCH_SUB=1 pip3 install -e .

Step 2: Configure vscode

1. Connect vscode to remote Docker

Configure the ssh config file as follows, note that User is root, start vscode, install the Remote-SSH plugin, and configure the config as follows:

Host docker_debug  #docker_debug can be any name
    HostName xxxx #server IP
    User root
    Port 8022 #the port number in Step 1

2. Install Extensions in VS Code

In the extension marketplace:

• Search for and install the corresponding language extension for C++

• Search for and install the corresponding language extension for Python

3. Configure VS Code

Click run - open configurations, and complete the configuration of the launch.json file according to the following scenario.

3.1 Scenario One: Debugging Only Python Code

Configuration File:

{
    "name": "Python: Current File",
    "type": "python",
    "request": "launch",
    "program": "${file}", // or modify to specify a certain file
    "console": "integratedTerminal",
    "justMyCode": true,
    "cwd": "${workspaceFolder}/examples", // modify to appropriate directory if file is not specified with full path
    "env": {
        "DEBUG": "1", // used for debugging jit-compiled c++ code
        "LD_LIBRARY_PATH": "${LD_LIBRARY_PATH}:/path/to/you/want", // used for loading additional dynamic libraries
    }
}

Debugging Steps: Select the file to run, set breakpoints using the mouse, and start "Python: Current File" to begin debugging! Note: If there is a bug in the program where the relative path cannot be found, you can modify the cwd configuration like this: "cwd": "${fileDirname}"

Additional Method: Quick Python Debugging Without Starting IDE:

If it is a simple Python code, you can also use pdb for debugging without starting the IDE. Set breakpoints at the desired location with the following code.

import pdb;

pdb.set_trace()

3.2 Scenario 2: Debugging Only C++ Code

Configuration File

// Modify "program": "/usr/bin/python" according to the result of "which python"
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Attach",
            "type": "cppdbg",
            "request": "attach",
            "program": "/usr/bin/python", /* My virtual env */
            "processId": "${command:pickProcess}",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Debugging Steps:

You can directly execute your Python file, such as python3 a.py, and then start "(gdb) Attach" to bind to the Python process. If the Python process exits too quickly, you can sleep for a period of time to begin debugging.

3.3 Scenario 3: Debugging Python and C++ Together

Reference: Example debugging mixed Python C++ in VS Code

Configuration File

Simply configure both of the above configuration files.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": true,
            "cwd": "${fileDirname}",
            "env":{
                "DEBUG":"1",
            }
        },

        {
            "name": "(gdb) Attach",
            "type": "cppdbg",
            "request": "attach",
            "program": "/usr/bin/python", /* My virtual env */
            "processId": "${command:pickProcess}",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Debugging Steps:

1. First, start "Python: Current File". Wait until the breakpoint is hit.
2. Start "(gdb) Attach" to complete the debugging process for Python and C++.

Additional Method: Quick Debugging Without Starting IDE

If you do not want to start the IDE, you can use the gdb command line to quickly start debugging.

## Debugging with gdb
gdb python
# set environment DEBUG=1 # <= If you need to JIT compile C++ code, you can compile it in debug mode with this command.
run your.py

Trace the stack after a crash: bt

4. VSCode Header File Autocomplete (Optional)

Update the project configuration in ./.vscode/c_cpp_properties.json by adding the appropriate header file paths.

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**",
                "/opt/conda/lib/python3.8/site-packages/torch/include",
                "/usr/local/cuda/targets/x86_64-linux/include"
            ],
            "defines": [],
            "compilerPath": "/usr/bin/gcc",
            "cStandard": "gnu17",
            "cppStandard": "gnu++14",
            "intelliSenseMode": "linux-gcc-x64"
        }
    ],
    "version": 4
}