From eb080834b4e853c85d28023b1a26f1e164a154f5 Mon Sep 17 00:00:00 2001 From: tschw Date: Sat, 20 Jul 2013 16:39:40 -0700 Subject: [PATCH] Created Thoughts On Fast Bitmap Image Rendering (markdown) --- Thoughts-On-Fast-Bitmap-Image-Rendering.md | 147 +++++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 Thoughts-On-Fast-Bitmap-Image-Rendering.md diff --git a/Thoughts-On-Fast-Bitmap-Image-Rendering.md b/Thoughts-On-Fast-Bitmap-Image-Rendering.md new file mode 100644 index 0000000..faba722 --- /dev/null +++ b/Thoughts-On-Fast-Bitmap-Image-Rendering.md @@ -0,0 +1,147 @@ +### Introduction + +Rendering bitmap images in WebGL is accomplished by issuing +surface geometry that is then textured within the _Fragment_ +_Shader_. + +The WebGL API requires texture memory resources _(Texture_ +_Objects)_ to be assigned to _Texturing_ _Units_ as part of +the rendering state. This limitation is often (unless for +very recent desktop graphic cards) hardware-driven. + +The number of WebGL Texture Objects that can be used within +one _Batch_ is limited. Rendering many different images, +such as icons, each presented to WebGL as a single Texture +Object therefore complicates the rendering process and +requires costly CPU intervention. + +### Tiled Texture + +This problem is typically solved by using a single Texture +Object that represents a tiled image consisting of the +individual images to be rendered and appropriate texture +coordinates to select those sub-images. + +### Caching + +Uploading data to the GPU from the host system is costly (and +intentionally bandwidth-limited for consumer hardware within +PC systems other than for high-end workstation cards without +this limitation). Therefore we want to keep as much data as +possible on the GPU between frames in order to keep up +interactive and very competitive frame rates. + +In order to accomplish this we are building a _Cache_: +Whenever an image is needed, we check whether it has been +uploaded already. If so, we can use it. Otherwise when there +are free tiles we can use one of those tiles to upload it. +If there are no free tiles we have to evict an image that is +not required to render the current batch or frame. Ideally we want +to pick an image that is unlikely to be needed in the next +couple of batches and frames (as could be the case just using +_RNU_). _LRU_ and _LFU_ strategies are suitable for more or less +repetitively used images, respectively. + +Whenever images are evicted from the cache, rendering data +referencing it becomes invalid. + +We assume the cache uses one tile size for all images (that +may or may not use the entire space provided by the tile). + +### Differently Sized Tiles + +Using just a single tile size may waste too much GPU memory +when image sizes differ too much. In the following, we briefly +discuss four alternatives that could be used to approach this +problem. Those are (ordered by sophistication and implementation +complexity): + +1. Instantiation of multiple image caches (each for a predefined + image size) using different Texture Objects, +2. Instantiation of multiple image caches (each for a predefined + image size) using different areas of the same Texture Object, +3. Using layered image caches, where the area used by one is + allocated from a layer for larger-sized tiles (again using + predefined image sizes, but dynamically adjusting to the + needs) +4. Implementing one image cache that allows allocation of + relatively small-sized, consecutive tiles that performs + defragmentation so larger blocks are kept available. + +The first two approaches are trivial to implement. Assigning +Texture regions to caches manually can be impractical on the +maintenance side, however, allowing it to be possible can be +useful in order to evolve towards alternative three. + +The third approach can -to some extent- dynamically adjust to +the image sizes needed by the application. When images come in +widely varying sizes at unforeseeable number, it is clearly +much better than the previously discussed alternatives. Care must +be taken to avoid that underpopulated sub-caches occupy too +many large tiles (the kind of fragmentation encountered with +this allocation scheme) e.g. by forcibly removing underpopulated +sub-caches when there is enough free space for tiles of their +size elsewhere. + +The fourth approach most effectively manages memory but carries +a significant overhead for defragmentation (while moving the data +can be done on the GPU, the CPU has to determine the permutation, +reordering tiles in Z-curve arrangement according to the eviction +criterion). Also the texture coordinates used for rendering have +to be updated, requiring the coordinates to be uploaded again. + +### Notes on Broader Applicability + +Not every raster image must necessarily originate from an URL: +Labels or individual Unicode characters may be rendered on the +client side. The API of the image cache should make no assumptions +about how the image is generated or uploaded (we're also able to +use the texture as a render target, then). +Furthermore images may be encoded as scalable, monochrome images +(very attractive for text, maybe for "instancing vector graphics" +as well, see +[this paper](http://www.valvesoftware.com/publications/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf)) +that occupy just one or two 8-bit color channels of the texture - +we might want to allow sharing tiles in this case in the future. + +### Glossary + +***GPU*** - **G**raphics **P**rocessing **U**nit + +***CPU*** - **C**entral **P**rocessing **U**nit; processor +steering a computer + +***Shader*** - a routine that runs at a specific stage of a +programmable rendering pipeline of a _GPU_ + +***Fragment Shader*** - the routine within the currently active +graphics pipeline routine that is executed on a per-pixel basis + +***Texturing Unit*** - custom hardware within the _GPU_ for +addressing, sampling and converting input data from a +certain region of memory represented by a _Texture_ _Object_. + +***Texture Object*** - a certain, uniquely identifiable region of +_Texture_ _Memory_ associated with a 1-3 dimensional image of +data tuples + +***Texture Memory*** - a region of a GPU's memory that is optimized +for 2D spatial access patterns + +***Batch (OpenGL Rendering)*** - bulk-processed data to be rendered +under a shared rendering state + +***Cache*** - quickly accessible storage for data that has been +acquired in the past (typically from slower memory) and is now +kept ready for future access + +***Eviction Strategy (Caching)*** - determination of data to be removed +from a full _Cache_ when space for new data is needed + +***RNU (Eviction Strategy for Caching)*** - **R**ecently **N**ot **U**sed + +***LRU (Eviction Strategy for Caching)*** - **L**east **R**ecently **U**sed + +***LFU (Eviction Strategy for Caching)*** - **L**east **F**requently **U**sed + +