At Backend.AI, we use GraphQL. In this post, we're going to talk about three main topics. First, we'll give a brief explanation of GraphQL and how we use it. Second, we'll discuss the technical challenges we faced while using GraphQL and how we solved them. Finally, we'll take some time to explain Pagination.
GraphQL Overview
GraphQL is a query language created by Facebook. A Facebook page displays a wide variety of information, such as personal info, timelines, and notification messages. If you were to query each of these individually, multiple connections would be created for each query. Fetching all of this with a JSON REST API would result in a very high number of round trips (leading to problems like over-fetching, under-fetching, and managing numerous REST endpoints)1. GraphQL helps create faster and lighter web applications by overcoming these shortcomings of REST APIs.
How Does Lablup Use GraphQL?
At Backend.AI, we use GraphQL not only for the advantage of reducing round-trip calls mentioned earlier, but also because it allows for flexible implementation of resolvers. This flexibility can be explained in two ways. First, for fields that existed in an older version of an object schema but have been removed in a new version, we can use a separate resolver method after object creation to populate the legacy field content using data from the current version's object. Second, it allows us to combine fields belonging to a single object from different data sources to provide a consistent data model.
As an example of the first point, if you add a separate resolver for a field that has been removed in a new version, like the resolve_legacy_compute_session() method below, you can populate the legacy table's content using data from the current version's table. This makes it easier to support backward compatibility even as versions are updated and new fields or object types are added.
async def resolve_legacy_compute_session(
executor: AsyncioExecutor,
info: graphene.ResolveInfo,
sess_id: str,
*,
domain_name: str = None,
access_key: AccessKey = None,
status: str = None,
) -> Optional[LegacyComputeSession]:
graph_ctx: GraphQueryContext = info.context
loader = graph_ctx.dataloader_manager.get_loader(
graph_ctx,
'LegacyComputeSession.detail',
domain_name=domain_name,
access_key=access_key,
status=status
)
matches = await loader.load(sess_id)
if len(matches) == 0:
return None
elif len(matches) == 1:
return matches[0]
else:
raise TooManyKernelsFoundAs a second example, we designed our system so that row objects fetched from the DB via a query are converted using the from_row() method as shown below, so that the GraphQL engine can understand them.
@classmethod
def from_row(
cls,
ctx: GraphQueryContext,
row: Row,
) -> Agent:
mega = 2 ** 20
return cls(
id=row['id'],
status=row['status'].name,
status_changed=row['status_changed'],
region=row['region'],
...
# legacy fields
mem_slots=BinarySize.from_str(row['available_slots']['mem']) // mega,
cpu_slots=row['available_slots']['cpu'],
gpu_slots=row['available_slots'].get('cuda.device', 0),
tpu_slots=row['available_slots'].get('tpu.device', 0),
...
)In cases where a field is defined in the object but cannot be fully populated during the conversion, a separate resolver like the resolve_live_stat() method below can be used to fetch the data from Redis instead of the DB. Therefore, although fields may belong to the same object in the GraphQL schema, their values can come from the DB or from Redis. In summary, you can implement a conceptual data model in GraphQL, and the actual data that populates this model can be fetched from various data sources (like a SQL database or Redis).
async def resolve_live_stat(self, info: graphene.ResolveInfo) -> Any:
ctx: GraphQueryContext = info.context
rs = ctx.redis_stat
live_stat = await redis.execute_with_retries(
lambda: rs.get(str(self.id), encoding=None)
)
if live_stat is not None:
live_stat = msgpack.unpackb(live_stat)
return live_statThe Problem of Inefficient Database Queries with GraphQL and How to Solve It Through Batching2
For example, if you want to fetch data for 10 users and you write a SQL query inside a GraphQL resolver, you will end up sending 10 separate queries. This might not be a problem on a small scale, but it becomes very inefficient as the number of queries increases. To solve this, we use a library called DataLoader. When DataLoader resolves the same GraphQL object multiple times (e.g., passing 10 different user.ids as arguments to the same resolve function), it doesn't process the queries individually. It groups the input arguments (in this case, user.id) into a list and then processes them in batches. Of course, the developer can decide the size of the batch (at Backend.AI, we process them in batches of 128). Consequently, the WHERE clause of the resolver's query statement changes as follows.
WHERE users.id == $user_idWHERE users.id IN $user_idsThus, the DataLoader library has the effect of batching SQL queries. Since it can reduce the various tasks a database has to perform for each individual SQL query (like parsing the SQL statement and creating a query plan) to a single operation, making good use of this library reduces the number of round trips to the actual SQL database. While GraphQL reduces the number of round trips between the client and the API server, if queries are not batched, the number of round trips increases with the number of queries, offsetting the advantages of GraphQL. In Backend.AI's case, we use a library that wraps Graphene, a GraphQL engine for Python. Since Graphene supports both synchronous and asynchronous operations, we use aiodataloader for batching in our asynchronous resolvers.
To maximize the batching effect by reusing the same DataLoader for the same resolver within a single API request as much as possible, we made additional optimizations to the DataLoader library. Specifically, it ensures that the same dataloader is consistently reused even when multiple GraphQL queries are sent in a single API request, or when the same type of object is accessed from different branches of nested query trees. For optimization, Backend.AI created a class called DataLoaderManager and designed it to create one DataLoaderManager object per API request and cache it for repeated calls.
class DataLoaderManager:
cache: Dict[int, DataLoader]
def __init__(self) -> None:
self.cache = {}
self.mod = sys.modules['ai.backend.manager.models']
@staticmethod
def _get_key(otname: str, args, kwargs) -> int:
key = (otname, ) + args
for item in kwargs.items():
key += item
return hash(key)
def get_loader(self, context: GraphQueryContext, objtype_name: str, *args, **kwargs) -> DataLoader:
k = self._get_key(objtype_name, args, kwargs)
loader = self.cache.get(k)
if loader is None:
objtype_name, has_variant, variant_name = objtype_name.partition('.')
objtype = getattr(self.mod, objtype_name)
if has_variant:
batch_load_fn = getattr(objtype, 'batch_load_' + variant_name)
else:
batch_load_fn = objtype.batch_load
loader = DataLoader(
apartial(batch_load_fn, context, *args, **kwargs),
max_batch_size=128,
)
self.cache[k] = loader
return loaderPagination
Although the official GraphQL documentation describes a recommended method for implementing pagination3, it's not a mandatory standard, so the rules differ for each implementation. We adopted a method using limit and offset. If you have 10-20 objects, you can fetch them all at once, but when the number grows to hundreds or thousands, the load on the DB becomes significant, which is why we use pagination. When we initially implemented Backend.AI with GraphQL, we used graphene.List without pagination and just added appropriate filter conditions. We sent the SQL query all at once using a method called load_all() to fetch the data. However, as the number of agents increased to dozens and cloud users to thousands, the need for pagination arose. To address this, we defined a new object type called AgentList, which follows the PaginatedList interface.
class AgentList(graphene.ObjectType):
class Meta:
interfaces = (PaginatedList, )
items = graphene.List(Agent, required=True)class Item(graphene.Interface):
id = graphene.ID()
class PaginatedList(graphene.Interface):
items = graphene.List(Item, required=True)
total_count = graphene.Int(required=True)We created the AgentList object type and also implemented its resolver. resolve_agent_list() takes limit, offset, filter, and order as common arguments. limit and offset have the same meaning as in SQL syntax for limiting the range of query results, while filter and order are expressions that allow specifying conditions for arbitrary fields. total_count signifies the total number of agents matching the conditions, obtained via the load_count() function, and agent_list refers to the populated list of agents fetched according to the limit and offset conditions via the load_slice() function.
async def resolve_agent_list(
executor: AsyncioExecutor,
info: graphene.ResolveInfo,
limit: int,
offset: int,
*,
filter: str = None,
order: str = None,
scaling_group: str = None,
status: str = None,
) -> AgentList:
total_count = await Agent.load_count(
info.context,
scaling_group=scaling_group,
raw_status=status,
filter=filter,
)
agent_list = await Agent.load_slice(
info.context, limit, offset,
scaling_group=scaling_group,
raw_status=status,
filter=filter,
order=order,
)
return AgentList(agent_list, total_count)
Summary
In this post, we have discussed the following topics:
- An overview of GraphQL
- How we use GraphQL at Backend.AI
- Problems when using GraphQL and how to solve them with batching
- The need for Pagination and how to implement it
We hope this article serves as a helpful reference for those looking to adopt and optimize GraphQL.
Footnotes
-
https://www.howtographql.com/basics/1-graphql-is-the-better-rest/ ↩
-
Batching refers to the task of processing requests in a bundle, rather than processing them as soon as they arrive. ↩
