backend.ai

Backend.AI's core engine uses many open source software components and is itself developed as open source.

When using Backend.AI, if you encounter any inconveniences or discover bugs, enterprise customers can use customer support and technical support channels for issue tracking and assistance. However, others can also contribute directly to the open source project.

There are two main ways to contribute: creating detailed issues that explain problems or improvement ideas, and making direct code contributions through pull requests.

This article introduces what you should know in advance to communicate more effectively with the development team during the contribution process.

GitHub Repository Introduction

If you want to submit an issue, the first place to look is the Backend.AI meta-repository.

We call the repository with the project name "backend.ai" a meta-repository because it doesn't contain code for actual functionality, but rather serves as a project management hub.

All Backend.AI server and Client SDK related issues are managed here, and it provides links to other projects through the README. Additionally, this repository's packages provide component version mapping for Backend.AI's major releases (e.g., version 21.03 of the backend.ai package has dependencies registered for version 21.03 of backend.ai-manager and backend.ai-agent packages).

When creating new issues, we provide two basic templates: bug report and feature request, but you don't need to strictly follow these formats. However, given Backend.AI's complexity and various usage environments, following these templates makes it easier to provide context for problem identification.

Backend.AI Component Relationships

Figure 1 shows a diagram of the relationships between Backend.AI's major components. All components have repository and package names in the format backend.ai-xxx-yyy. Here's a brief summary of what each component does:

• backend.ai-manager (Manager): Core service responsible for monitoring cluster compute resources and session scheduling, providing APIs for user authentication and session execution
• backend.ai-agent (Agent): Service installed on compute nodes to manage and control containers
• backend.ai-common (Common): Library that collects functions and data formats commonly or frequently used by multiple server-side components
• backend.ai-client-py (Client SDK for Python): Official command-line interface and library providing API wrapper functions and classes for Python
• backend.ai-client-js (Client SDK for Javascript): Library providing API wrapper functions and classes for JavaScript environments
• backend.ai-storage-proxy (Storage Proxy): Service that enables user web browsers or Client SDKs to perform large-scale I/O directly from network storage
• backend.ai-webserver (Web Server): HTTP service that provides routing for Web UI and SPA (single-page app) implementation and web session-based user authentication
• backend.ai-webui (Web UI & Desktop App): Web component-based implementation of the actual user interface. Also supports Electron-based desktop app builds and includes a local lightweight version of app proxy that allows users to directly access application ports running inside containers.

Backend.AI Version Management Method

Backend.AI has major releases every 6 months (March and September each year) and provides post-release support for about 1 year. Therefore, version numbers follow the CalVer format in YY.0M.micro style (e.g., 20.09.14, 21.03.8).

However, due to Python packaging system version number normalization, wheel package versions use the YY.MM.micro format without zero-padding in the month part (e.g., 20.9.14, 21.3.8). Some sub-components that have different update cycles from the main release cycle may follow the general SemVer format.

Development Environment Setup Method

To make actual code contributions, you need to write pull requests. For non-trivial contributions beyond simple typo fixes or documentation changes, you need to modify code and test it directly, so setting up a development environment is essential.

Because Backend.AI consists of multiple components working together, simply cloning one repository and creating a Python virtual environment with editable install¹ is insufficient.

At minimum, you need to configure and run manager, agent, storage-proxy, webserver, and wsproxy to see a functioning GUI. For CLI environments, you also need to separately install the client SDK. Additionally, you need to run Redis, PostgreSQL, and etcd servers for manager operation and communication with agents.

To automate this complex installation process, we created the scripts/install-dev.sh script in the meta-repository. This script does the following:

• Checks installation status of pyenv, docker, python, etc., and provides automatic installation or installation guidance
• Installs all the various components mentioned above in their respective directories and Python virtual environments in editable state
  • Components needed for other component operations like common and accelerator-cuda are also installed in editable state in the dependent component's virtual environment as well as their own virtual environment. For example, when you change common source code, it will be reflected when running manager and agent as well.
• Adds database/etcd fixtures including basic port settings for components to communicate with each other and example authentication keys
• Creates and runs postgresql, redis, and etcd services using docker-compose under the name "halfstack"

When the install-dev script completes successfully, it outputs commands for running service daemons like manager and agent, along with default configured example account information. Follow the instructions and use terminal multiplexers like tmux or screen, or multi-tab features of terminal apps to run service daemons in separate shells. Once you confirm that hello world examples work, you're ready to develop and test Backend.AI.

NOTE

Currently, this method only supports Intel (amd64/x86_64) based macOS and Ubuntu/CentOS Linux environments, and Windows 10 WSL v2 environment. WSL requires some additional configuration.

When first using this install-dev script, it often stops due to various errors or pre-check failures and needs to be run again. In such cases, you can use the scripts/delete-dev.sh script to conveniently perform the deletion procedure.

These installation/deletion scripts use an environment ID to allow installing and testing multiple versions or multiple copies of Backend.AI on one computer. When running the install-dev script, you can specify an ID with the -e option, and if not specified, it generates and uses a random one. When running the delete-dev script, you must specify the -e option. By using the same value you specified when running the install-dev script (or the generated value shown at the beginning of the installation log), you can delete that development environment.

Installing and Developing Multiple Versions of Backend.AI on One Computer

Using these install-dev and delete-dev scripts with the environment ID described above, you can run multiple versions of Backend.AI alternately on one computer. For example, let's create Backend.AI development environments with two versions: dev2103 and dev2109.

NOTE

When installing with the install-dev script, you can also specify specific release branches like 20.09, 21.03 instead of the default main branch using --server-branch and --client-branch options to install different versions.

First, clone the meta repository to obtain the install-dev script:

$ git clone https://github.com/lablup/backend.ai meta

Install the dev2103 environment first:

$ meta/scripts/install-dev.sh \ -e dev2103 \ --server-branch=21.03 \ --client-branch=21.03 \ --install-path=./dev2103

Now install the dev2109 environment, but first you need to stop the docker-compose container set created for the dev2103 environment so that the second environment's install-dev script can create a new docker-compose container set. At the time of writing, version 21.09 is being developed on the main branch, so the branch name is specified as "main" rather than a version number.

$ docker-compose \ -f ./dev2103/backend.ai/docker-compose.halfstack.dev2103.yml \ -p dev2103 \ stop $ meta/scripts/install-dev.sh \ -e dev2109 \ --server-branch=main \ --client-branch=main \ --install-path=./dev2109

Later, if you want to switch between dev2103 and dev2109 environments, first stop all running Backend.AI service daemons from the previous environment and stop and start docker-compose container sets as follows (swap dev2103 and dev2109 depending on the switching direction):

$ docker-compose \ -f ./dev2109/backend.ai/docker-compose.halfstack.dev2109.yml \ -p dev2109 \ stop $ docker-compose \ -f ./dev2103/backend.ai/docker-compose.halfstack.dev2103.yml \ -p dev2103 \ start

The compose file name in docker-compose's -f option and the value in the -p option correspond to the environment ID specified with the -e option in the install-dev script, and parts like ./dev2103 in the directory path correspond to the --install-path option in the install-dev script.

Installing CUDA Plugin

When installed as above, Backend.AI is installed by default in a state where only CPU can be used. To run CUDA GPU accelerated environments, you need to install the CUDA plugin on the agent. Currently, the open source version provides an open source version CUDA plugin that can allocate by CUDA device unit to containers. If you give the --enable-cuda option when running the install-dev script, it will be installed together with the agent, and you can specify the plugin version with the --cuda-branch option. For compatibility between specific Backend.AI release versions and CUDA plugin versions, please refer to this table.

Pull Request Writing Guidelines

To actually send specific bug patches or feature implementations as PRs, you first need to upload them to GitHub. There are several methods, but we recommend the following approach:

• Fork from the GitHub repository page (if you have direct commit permissions, we recommend creating a branch directly without forking)
• Point your local working copy to that fork repository with git remote
  • Following convention, it's good to name Lablup's original repository as upstream and the newly created fork repository as origin
  • If you first installed with install-dev rather than cloning after forking, the original repository will be origin, so you'll need to rename the remote
• Create a new branch
  • For branch names, use fix/ prefix for bug fixes and feature/ prefix for feature additions or improvements, followed by a kebab-case summary of the topic (e.g., feature/additional-cluster-env-vars, fix/memory-leak-in-stats). Other prefixes like docs/, refactor/ are also used
  • While it's possible to write PRs by directly modifying the main branch, during PR review and modification periods, if additional changes occur in the main branch, you'll need to rebase or merge every time you sync with the upstream repository, which is more troublesome. Having a separate branch allows you to rebase and merge when you want
• Commit changes to that branch
  • Commit messages should preferably follow conventional commit style. Like branch names, use title prefixes like fix:, feat:, refactor:, docs:, release:, and Backend.AI specifically uses additional prefixes like setup: for dependency-related commits and repo: for cases like gitignore updates or repository directory structure changes. Components in parentheses may also be marked (e.g., fix(scripts/install-dev): Update for v21.03 release)
  • Commit messages must be written in English
• Push the branch and create a PR
  • For PRs with separate issues, you must include the issue number in the PR body. To reference meta-repository issues, write in the format lablup/backend.ai#123 and GitHub will automatically create links
  • While we don't require a specific format for PR bodies, it's good to describe what problem you're trying to solve, what principles you used to write it, what tools or libraries you utilized, and why you made those choices
  • PR titles and bodies can be written in English or Korean
  • When you create a PR, you can see various automated checking tools running. In particular, CLA (contributor license agreement) must be signed (GitHub username registration) for review to proceed
  • You must pass all basic coding style and coding rule checks for each language (flake8, mypy, etc. for Python code)
  • In repositories with a changes directory and towncrier checks, after creating a PR and receiving its number, create a file named changes/<PR number>.<modification type> and write a one-line English sentence summarizing the changes in Markdown syntax. (For relatively simple content or when there's a separate existing issue, this content sometimes serves as the PR body) Modification types include fix, feature, breaking, misc, deprecation, doc, and project-specific parts are defined in each repository's pyproject.toml. You can refer to files like CHANGELOG.md or CHANGES.md to see how existing messages were written
• Proceed with the review process
  • When completed, reviewers usually organize the commit log in squash-merge format to merge as a single commit
  • Therefore, don't feel burdened about frequently making small modification commits during the review process and feel free to make commits whenever you think of something

NOTE

Tools like GitHub CLI, SourceTree, and GitKraken used together with git commands are even better.

Summary

We've now looked at Backend.AI's overall component structure and repository structure, development environment setup methods, and pull request writing guidelines. We hope this guide helps you take one step closer to Backend.AI's source code.