How we built our data pipeline, Kirin API
At Qraft Technologies, we’ve been refactoring our data-driven API service, Kirin API. Kirin API was initially developed by AI Research team providing different kinds of financial datas integrated with many other widely used database, covering corporate market data such as S&P Global’s Compustat database, Thomson Reuters’ Datastream and economic macro datas like FRED or Quandl APIs at one endpoint.
As the demands of this project grows internally and externally, it has expanded into a commercial project from an internal pilot project, in charge of my (Research Platform) team into highly scalable and available service that can handle more traffics and more client.
Data Flow
We started from the most simple architecture as the diagram below.
Different kinds of financial datas are loaded in different databases and back-end layer layers provides different APIs to the clients. In our case, the client-side is the utility library that provides interface to python to call the API, and we have a roadmap to support other programming languages or platforms in the future.
There are some major problems with this architecture.
Majority of end-point users are in-house python developers working on internally. However this requirement eventually would change to support not just Python but different interfaced clients such as R lanaguage or an Excel plugin to load data in spreadsheets. Another concern is the maintainence issue to support different versions of client interfaces in a single point.
Needs for Agnostic Service layer
Groups of clients have different requirements on how they interact with the data they want. Thus, the back-end layer should handle those complex requirements of each client and this cause serveral issues such as agility issues within the team, and high maintenance cost as the project proceeds.
There are some existing microservice design patterns and solutions for similar problems and those are Backend-For-Frontend(BFF) and GraphQL.
BFF and GraphQL
BFF design pattern was one of the benchmark model for our API service. I am not going to cover the details about this, but the key aspect is that instead of providing a generic backend service, make a specific API that shares the business logic which is actually a part of client rather than backend.
If BFF solved the problem in client-side’s aspect, GraphQL on the other hand, remains this design implementation on the server side. According to the official documentation, GraphQL is defined as:
GraphQL is a query language for your API and a server-side runtime for executing queries by using a type system you define for your data.
GraphQL provides simplicity of the implementation of the backend server by providing a query language that client can request which specifies exact resources that client wants.
BFF and GraphQL, both of them are looking for API layer how to provide an API that can be used by multiple types of clients requirements. The main difference is that BFF is geared to full-stack implementation, (Coupled BFF and client as one moduled component), GraphQL handles the complexity by proving an agnostic query and grants high degree of freedom to the client side. Our team has decided to use GraphQL.
So Why GraphQL?
First of all, BFF has several issues to be applied:
Agility issues across BFF and client (these roles are split in our case)
Hard to predict capacity required by individual client
(especially for heavy requests such as daily/monthly tickers for large number of securities)Single point of failure
Additional deployment complexity
On the other hand, GraphQL helps dealing with evolving APIs since high degree of freedom is granted to the client groups as each groups can declaratively interact with data in AST(Abstract Syntax Tree) structure.
Querying Compustat database through graphQL
Implementation
Currently, Kirin API service architecture using GraphQL looks as below.
Each client group requests data to API gateway(GraphQL server) and corresponding resolvers fetch data from original datasource. When data is fetched, postprocesses are handled by the asynchronous service workers and returns the result to the clients.
Our CICD, Automated pipeline runs scheduled batches to check any updates on the datasources and validates integrity of the data and ensures each group of client versions are use up-to-date.
Cache Policies
Resolvers in GraphQL runs independently so duplicated contexts between requests must be preliminarily checked. We inquiry if the data exists in the cache, and only fallbacks to original datasources when the data is not available in the cache.