Spandan Ghosh 11 Yash Margaje 16 Akshay Kanade 17 Shamika Mhatre 19 Durva Kavathkar 22 Shreyasi Patil 35 Parth Gadekar 36 Suyog
Morankar 40
DEEP LEARNING TECHNIQUES FOR SUPER-
RESOLUTION IN VIDEO GAMES
Dept. of Information Technology,
A. C. Patil College of Engineering,
Kharghar.Mumbai-410210
Source : Deep Learning Techniques for Super-Resolution in Video Games
Abstract: The computational cost of video game graphics is
increasing, and graphics processing hardware is struggling to
keep up. This means that computer scientists must be creative
in new ways to improve the performance of the graphics
processing hardware. The super resolution of video enables
video games to enjoy high quality graphics and offsets a large
part of the computational costs. These recent technologies
enable consumers to improve the performance and enjoyment
of video games, and the has the potential to become the
standard within the game development industry.
Keywords : deep learning, super resolution, video games,
neural networks
INTRODUCTION
Video game graphics are improving. Complex physics and ray-
traced light effects, where environmental lighting is rendered in
real time as individual rays, have become the standard within
major game development studios. For users to play games with
these graphics on their computers, they need dedicated
hardware called graphics processing units (GPUs). At the same
time increases in GPU processing power are beginning to
plateau as the size of transistors reach the atomic level.
Traditionally performance was improved by shrinking the size
of transistors on a microchip and doubling the count. Now
computer scientists are devising new methods to improve the
performance of GPUs using emerging tech such as the use of
deep neural networks to enhance video game graphics in real
time through a technique called Video Super Resolution (SR).
This has the potential to reduce the GPU performance required
to play video games while improving the graphics quality
according to Wang et al.
BACKGROUND
Gaming enthusiasts appreciate the stunning graphics as well as
the high performance of their games, which is a difficult balance
to maintain. There is a performance cost to playing video games
at resolutions like 4k, with lots of graphics effects. It can reduce
the frame rate to a level at which the game cannot be played in
a pleasant way. Fast-paced action games require a high frame
rate for a smooth gaming experience, especially in competitive
scenarios where fractions of a second can decide between
victory and defeat. Research by Claypool et al show that in a
first-person shooter game, players performed 7 times better at
60 fps compared to reduced frame rate of 3 to 7 fps.
While consumers can increase performance by using lower
quality graphic settings, it is undesirable for both the player and
game developer as their product is not being experienced as
intended. Using Deep Learning Video SR allows consumers
with lower performance GPUs run games at a higher perceived
visual quality and performance than what is usually possible.
This makes games more accessible whilst improving
performance on high end systems.
The leader in a consumer solution to this problem is NVIDIA
with the "Deep Learning Super Sampling (DLSS)" rendering
method which claims a 2-3 times performance improvement at
4K. Alternatively, there are promising open-source solutions
from companies like AMD with their "FidelityFX" technology
and Microsoft with their upcoming application programming
interface (API) called DirectML which they have demonstrated
to upscale videos from 540p to 1080p in real time.
QUESTIONS
There are questions to be addressed with this emerging
technology which will determine it’s potential to become the
industry standard used by the majority of major game
development studios. What are the percentage increases in
frame rate using deep learning video SR? How accurate is the
upscaled image compared to the original? Additional
Performance cost? Hardware required to make use at a
consumer level and how accessible are these solutions for the
game developers?
They can be answered by looking at the research for deep
learning super resolution by Wang et al
FRAME RATE
The most substantial source of data on frame rate increase using
deep learning video SR is NVIDIA’s official Deep Learning
Super-Sampling (DLSS) information page, they display figures
showing the average frames per second (fps) with and without
DLSS on four different videogames with the frame rate for
every resolution on each of the tested GPUs. At 1080p the fps
increases by a range of 51-63 across the various GPUs, with the
smallest increase coming from the then flagship RTX 2080 Ti
GPU and an average of 77.8 fps increase from the cheapest
RTX 2060 GPU going from an average of 41.8 fps with DLSS
off to 68.1 fps with At 4k the frame rate increase is drastically
higher with an increase ranging from 159-360 across the GPUs,
with the RTX 2080 Ti increasing from an average of 25.8 fps
From this data we can see that the FPS gains are sizeable, with
percentage increases from games with lower initial average FPS
with DLSS disabled. The results show that Deep Learning SR
can be highly effective in improving game performance with
NVIDIA's DLSS, it is able to make the cheaper RTX 2060 GPU
perform better than their flagship RTX 2080 Ti.
ACCURACY
Convolutional neural networks: They are an efficient
technique for super-resolution deep learning. CNN requires less
pre-processing than traditional SR imaging techniques. it can
be trained to identify and distinguish between key features in
each frame. For SR-Video, CNN can receive the neighbouring
frames as input, it helps the neural network learn the spatial and
temporal information of the frame, enhancing CNN's ability to
produce high quality results
Motion Compensation: By providing motion vectors to CNN,
the algorithm can compensate for motion between frames and
produce high quality SR video
Deep Learning Super Sampling: NVIDIA's method for SR
video takes low resolution alias frame images and their motion
vectors, which contain information about the direction of
movement of all pixels in the following frame, and puts them
into a CNN, called a convolutional auto encoder, which is
trained on a supercomputer to use these inputs to generate a
high-resolution version of the frame image on a frame-by-
frame basis
Image Quality Assessment: The results of Deep Learning SR
can be assessed using performance metrics called Image
Quality Assessments (IQAs). Objective IQA methods use
computer modelling to compare images. The problem with
these methods is that the computer does not perceive as a human
does, so the results of the assessment can easily contradict the
results of the subjective IQA. This makes it difficult to calculate
the performance measurement task effectively, but SR video
with deep learning can provide effective results based on
subjective and objective IQA methods.
PERFORMANCE COSTS
An advantage of SR video over single-frame SR is that adjacent
fields can be used with the current field and motion vectors to
reduce the computation required to generate the current field.
This dramatically increases the speed of the algorithm with
negligible performance loss.
ACCESSIBILITY
Consumers: The only widely available solution for the
consumer at present is NVIDIA's DLSS which requires
dedicated specific hardware in the form of one of the RTX
series graphics processing units (GPUs). This price is only for
the GPU which is only one component of a personal computer
(PC), which makes the total price of the entry much higher. This
means that the DLSS solution is currently only available to
enthusiastic PC owners with mid to high end PCs. However, as
this hardware is replaced with new hardware, prices will drop,
making the more accessible to consumers on reduced budgets.
Developers: For deep learning video SRs to become a standard
technique for game development, these techniques must be
widely available to game developers worldwide. According to
NVIDIA, "DLSS 2.0 is now available to Unreal Engine 4
developers through the DLSS Developer program, which will
accelerate deployment to one of the world's most popular game
engines". Unreal Engine 4 and one of the most popular game
engines, which means that these tools are widely used
internationally. Additionally, AMD's FidelityFX and
Microsoft's DirectML are open-source tools that anyone can
use, although they are still in their infancy compared to
NVIDIA's solution.
CONCLUSION
There is the potential for deep learning super resolution
techniques to become the standard in the video game industry,
allowing consumers to achieve substantial increases in
performance from their GPUs while maintaining high images.
These emerging technologies are still in their infancy and there
is currently only NVIDIA Deep Learning Super Sampling
(DLSS) support for a limited number of games. The adjacent
images and motion vectors available as input to CNN allow
greater accuracy of the resulting video, with satisfactory results
based on subjective AQI methods. The performance cost of the
deep learning algorithms is negligible, with the performance
gains exceeding the resulting slight side grade in perceivable
graphics quality.
FUTURE WORK
These questions could be further answered with future research
that could be conducted by independently testing NVIDIA's
DLSS solution with Microsoft and AMD solutions in a fair
environment that could control variables - a single-blind study,
conducted on a large number of subjects and which could
provide a fair analysis of any of the video super resolution
methods based on observation without bias.
REFERENCES
A. G. E. Moore, "Progress in digital integrated electronics [Technical literature,
Copyright 1975 IEEE. Reprinted, with permission.Technical Digest. International
Electron Devices Meeting, IEEE, 1975, pp. 11-13.]," IEEE Solid-State Circuits
Society Newsletter, vol. XI, no. 3, pp. 36-37, 2006.
B. R. Williams, "What's next? [The end of Moore's law]," Comping in Science &
Engineering, vol. XIX, no. 2, pp. 7-13, 2017.
C. Z. Wang, J. Chen and S. C. Hoi, "Deep learning for image super-resolution: A
survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. II,
2020.
D. M. Claypool, K. Claypool and F. Damaa, "The effects of frame rate and resolution
on users playing first person shooter games," Proceedings of SPIE - The
International Society for Optical Engineering, vol. 6071, 2006.
E. NVIDIA DLSS 2.0: A Big Leap In AI Rendering | GeForce News | NVIDIA
F. Radeon™ Software | FidelityFX | AMD
G. microsoft/DirectML: DirectML is a high-performance, hardware-accelerated
DirectX 12 library for machine learning. DirectML provides GPU acceleration for
common machine learning tasks across a broad range of supported hardware and
drivers, including all DirectX 12-capable GPUs from vendors such as AMD, Intel,
NVIDIA, and Qualcomm. (github.com)
H. Kappeler, S. Yoo, Q. Dai and K. Katsaggelos, "Video super resolution with
convolutional neural networks," IEEE Transactions on Computational Imaging, vol.
II, no. 2, pp. 109-122, 2016.
I. X. Tao, H. Gao, R. Liao, J. Wang and J. Jia, "Detail-revealing deep video super-
resolution," IEEE International Conference on Computer Vision (ICCV), pp. 4482-
4490, 2017.
J. Z. Zhang and V. Sze, "FAST: A framework to accelerate super resolution
processing on compressed videos," in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Workshops, 2017.
K. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z.
Wang, "Real-time single image and video super-resolution using an efficient sub-
pixel convolutional neural network," 2016 IEEE Conference on Computer Vision
and Pattern Recognition, pp. 1874-1883, 2016.
L. Christopoulou and S. Xinogalos, "Overview and comparative analysis of game
engines for desktop and mobile devices," International Journal of Serious Games,
vol. IV, pp. 21-36, 2017