Often when doing some computationally heavy processing at least two machines are involved, one for the local development (laptop?), with test runs, and one for running the full processing.

Setting up and maintaining the correct environment across the two machines can be complex and take time.

Switching between local development and remote development is not easy, maybe you want to tweak one line of code on your local machine and re-run the full processing again on the remote machine? That one change will often cause you to re-package the code, send it to the remote, access the remote and then run it there, tedious work!

I have developed Reacher/ReacherDocker - reach out to a remote, pypi github, that automates much of the problems listed above. Reacher can be used together with any VMs in the cloud or with some hardware that you already own. For ReacherDocker docker must be installed on the remote and ssh enabled, besides that, ReacherDocker takes care of the rest.

To install

pip install reacher

From your local machine, ReacherDocker will,

build a docker image on the remote (docker must already be installed and ssh enabled) according to specifications
set up container on the remote with port port-forwarding and enviroment variables
execute your local code on the remote, in the container, with printouts shown as your were running it locally
upload/download files from the container to your local machine
set up port-forwarding between the remote and your local machine
put and get files to and from remote

and for your local machine, Reacher will,

execute your local code on the remote, with printouts shown as your were running it locally
upload/download files from the remote to your local machine
set up port-forwarding between the remote and your local machine
put and get files to and from remote

Getting started…

To get started,

pip install reacher
pip install python-dotenv

First we must setup a connection to the remote, RemoteClient will create a ssh connection between the local and remote machine.

from reacher.reacher import Reacher, ReacherDocker, RemoteClient
from dotenv import dotenv_values
config = dotenv_values()  # take environment variables from .env.
client = RemoteClient(
    host=config["HOST"],
    user=config["USER"],
    password=config["PASSWORD"],
    ssh_key_filepath=config["SSH_KEY_PATH"]
)

the connection is sent to ReacherDocker together with the name of the image that we want to build and the name of the container.

reacher = ReacherDocker(
    client=client,
    build_name="base",
    image_name="base",
    build_context="dockercontext",
)

or send in the aruments for RemoteClient direcly to Reacher

reacher = ReacherDocker(
    build_name="base",
    image_name="base",
    build_context="dockercontext",
    host=config["HOST"],
    user=config["USER"],
    password=config["PASSWORD"],
    ssh_key_filepath=config["SSH_KEY_PATH"]
)

build_context should contain everything for building the docker image on the remote. It might look like,

$ ls dockercontext/
Dockerfile  requirements.txt

Once ReacherDocker has been setup we can build the image on the remote. ReacherDocker will send the build_context to the remote and trigger docker to build an image according to the specifications in the Dockerfile

reacher.build()

[+] Building 0.0s (0/1)                                                         
[+] Building 0.2s (2/3)                                                         
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 528B                                       0.0
 ...
 ...

and thereafter we can setup the docker container. Reacher will make sure this container is running until we have explicitly deleted it.

reacher.setup(ports=[8888, 6666], envs=dotenv_values(".env"))

If you want to execute the code direcly on the remote, not in a container, go for Reacher. Reacher will use the remote enviroment, as is, when executing the commands.

reacher = Reacher(
    build_name="base",
    host=config["HOST"],
    user=config["USER"],
    password=config["PASSWORD"],
    ssh_key_filepath=config["SSH_KEY_PATH"],
    prefix_cmd="PATH=...."
)

prefix_cmd is useful to set when we want to specify some PATH or when we want to source an venv before running a command.

Port-forwarding between remote and local

If you want to access some service on the remote, you can forward the traffic on the ports of the remote to the local machine.

reacher.add_port_forward(remote_port=6006, local_port=5998, paramiko=True)

this will spin of a seperate daemon thread handling the port-forwading. Set paramiko to False to trigger a system call for the port forwarding instead.

Put and getting files

Supports list of files or single files

reacher.put(
    path=["setup.py", "build.sh", "reacher", "src"],
    destination_folder=None,
    excluded_exts=[".pyc"],
)

reacher.get(["setup.py", "build.sh", "reacher"], <destination>)

Use reacher.ls(path) to list files on the remote.

Running code on the remote

Running a code-snippet

Now we have built the docker image on the remote and have a container ready to execute whatever code that we want to run.

A “Hello World” test can be triggered from a notebook.

First we create the python module that we want to execute,

%%writefile simple_test.py
import time
while 1:
    print("Hello from remote")
    time.sleep(1)

and then we execute it on the remote inside our controlled docker enviroment.

reacher.execute(
    context=["simple_test.py"],
    command="python simple_test.py",
    named_session="simple_test",
    # clean the container from previous runs.
    cleanup_before=True, 
)

Hello from remote
Hello from remote
...

simple_test will continue to run in the background (even if you kill the cell/script that you instantiated the reacher.execute from) until we explicitly have killed it.

With list_named_sessions you can get all currently running sessions.

reacher.list_named_sessions()
There is a screen on:
	22.simple_test	(03/19/23 18:20:07)	(Attached)
1 Socket in /run/screen/S-root.

we can always attach to a named session to continue to get printouts.

reacher.attach_named_session("simple_test")

or kill it

reacher.kill_named_session("simple_test")

Running with dependencies

To execute some code that depends on other modules inside a src directory, simply add src as a context when calling reacher.execute.

%%writefile dependency_test.py
from dependency import Dependency
d = Dependency()

reacher.execute(
    context=["src", "dependency_test.py"],
    command="python dependency_test.py",
    named_session="dependency_test",
)

Hello from class Dependency
[screen is terminating]

Generate artifact

In many cases we want to run some code on the remote and afterwards collect some artifacts.

Everything that is saved in the build path will be available for us to download to the local machine using reacher.get.

Running bash commands

We can always execute bash commands directly on the remote in the container, for examples we can start a jupyter notebook server on the remote inside the container,

reacher.execute_command(
    "jupyter notebook --ip 0.0.0.0 --allow-root --port 8888",
    named_session="jupyter"
)

Serving notebooks from local directory: /workspace
[I 14:56:39.436 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 14:56:39.436 NotebookApp] http://0278e1fd8511:8888/?token=50c00460ecada75b1e5eb28ae62211c547d5c487dcfe05e3
[I 14:56:39.436 NotebookApp]  or http://127.0.0.1:8888/?token=50c00460ecada75b1e5eb28ae62211c547d5c487dcfe05e3
[I 14:56:39.436 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 14:56:39.438 NotebookApp] No web browser found: could not locate runnable browser.
[C 14:56:39.438 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-194-open.html
    Or copy and paste one of these URLs:
        http://0278e1fd8511:8888/?token=50c00460ecada75b1e5eb28ae62211c547d5c487dcfe05e3
     or http://127.0.0.1:8888/?token=50c00460ecada75b1e5eb28ae62211c547d5c487dcfe05e3

this session will be running in the background on the remote in the container under the named session “jupyter”.

Getting started…#

Port-forwarding between remote and local#

Put and getting files#

Running code on the remote#

Running a code-snippet#

Running with dependencies#

Generate artifact#

Running bash commands#