Feature(core): Query Result Batching
Problem Statement
Currently when a query is executed all the result data is first collected, then transformed to JSON and then sent to the client. There are a few problems with this.
- If the query result data size is larger than the available memory of the microservice, or the complete server, a crash will occur.
- If the query result is large, it will take a long time for anything to be visible in the frontend, meaning a reduced user experience.
- Currently query results are forwarded via the message queue, which has a 128MB message limit, and any reasonably sized dataset will go over that limit.
Who will benefit?
This will greatly benefit every user that has a database that is larger than 128MB. And in general will benefit every user of the application as latencies will be reduced and the time for query results to become visible will be drastically reduced.
Benefits and risks
Pros
- Reduced query latencies
- Large queries supported (larger than 128MB result size)
Cons
- Will be challenging to implement
- Requires large architectural changes in both front- and backend
- Does not solve frontend data overflow issues (too much data being sent to frontend, have to be solved by clustering or something similar)
Proposed solution
Send the query result in batches. The database libraries often already return the query result in batches, which we could easily transform and send to the frontend. In the frontend these batches would have to be combined somehow, to create the full query result.
Examples
Priority/Severity
-
High (This will bring a huge increase in performance/productivity/usability/legislative cover) -
Medium (This will bring a good increase in performance/productivity/usability) -
Low (anything else e.g., trivial, minor improvements)
Edited by Behrisch, M. (Michael)