Award from the society like ACMC graph computer graphics or word as well as the homework

research award and today we all have our audience here on the campus we all have probably
hundreds of audience in zoom so welcome everyone to actually use the this seminar and we
also or we have a doctor ivan palm of from UCSB department who was actually a former PHD
students of professor heydrich back when professor heidrich talk as let's welcome professor
heidrich to give us stop of the kind introduction Oh yeah so I will be talking about learned
imaging systems but before that I want to tell you a little bit about cost because I know it's not
we're not a very old university we're only 15 years old so maybe some of you haven't heard of
us yet so the the university isn't Saudi Arabia on the Red Sea on the Red Sea this is actually new
new from the from the campus so it's actually quite nice and we're about an hour north of
Jeddah and uh here are some stats so where of intended to be a very small university graduate
students only very international so you can see that we have a about 120 nationalities
represented on campus with a student body of about only 1500 students so it's right it's it's very
international very multicultural however you know chinese is one one of the largest and women
in Saudi Arabia I want to bridge basically just highlight that even though I mean we have even
though we are graduate university only only in science and technology we actually have about a
third of our students our women OK so that's actually a very good internationally if you look it's
actually a very good racial for graduate schools um now what I want to talk about today is
basically my research area so computational imaging and display the way I look at it is it's
basically a hardware software Co design problem and So what do I mean by that so in a classical
optical system you would have an optical engineer who builds the best optical device possible
then you would have an electronic engineer who builds the best sensor possible and then you
have a either no compute afterwards or just a little bit of cleanup of noise saying denoising
deblurring things like that and I'm going to argue today that basically this traditional design
pipeline leaves a lot of the design space unaccessible and so instead we want to basically push
for a new paradigm where we think of the optics as an encoder of information OK so the where
there's information in the real world that we want to capture uh the optics and courses
information somehow onto a 2D plane uh where it's sampled and space-time wavelength and
whatever other dimension you might might need and then afterwards it will be decoded and
software OK so encoder decoder you want to do this design the two parts together there's a
goal to that which is computational display I will not talk much about display today but I do
want to start with it because it is actually attack me thinking about this hardware software for
design approach OK so this is an old project don't worry I will not be talking only about old stuff
this is just the first few slides OK so the the idea that we when we set out when I was still at UBC
about 20 years ago now um was that we wanted to build a display that is able to reproduce the
range of intensities that we see in the real world OK so in the real world we have you know if
you go between direct sunlight and Starlight you have about 16 orders of magnitude of
brightness change brightness change right and the human eye somehow has adapted to really
deal with this huge range so it's not simultaneously right there's an adaptation process at any
given point in time with York about five orders of magnitude rather than 16 but then if we are
exposed to a certain lighting environment for long enough we adapt to see in the sliding
environment whether it's direct sunlight or whether it's just Starlight out in the desert now this
place at the time where more in the range of three orders of magnitude like 100 contrast of 100
to 1200 to 1300 to 1 something like that and of course there's a big gap between that and the
ability to represent an outdoor an outdoor world OK so basically the the concept behind this
high dynamic range display technology is very simple I can explain it to you in one slide OK so
here's the slide so basically you have an LCD panel which is low contrast by various for for
various reasons that has to do with the technology it's relatively low contrast it has and it's it's
basically a programmable transparency OK so you can you have a a uniform light source behind
it and you program for every pixel how much of that light is transmitted for each of the color
channels and again for various technical reasons that's relatively limited in contrast so now what
you do is rather than use a uniform light as a backlog as a as a background you actually
introduce a low frequency modulated light in the background so it could just be an array of LED
and in fact that's what ultimately we have so we have a low resolution area of LED's optically
these are blurred and then we put the slide on top of it and the transparency on top of it the
two contrasts multiply OK and now you have a high contrast display and it's really that
multiplication that gets you out of it's not linear it's a it's a product right that that gets you out
of this OK so this is by the way the technology behind the display that I have here on this laptop
one of the reasons I wanted to use my because it's uh so this is actually something that we had
a there was a startup and it was bought by Dolby and then I had a Dolby research chair but
basically in the end this is the if you buy apple XDR display that's basically the technology we did
years ago uh so that different prototypes or one with projector one with these LED arrays so the
the one that commercially won out in the end is basically the one where you have these LED
low relatively low resolution LED arrays behind the LCD panel and and that's that's basically the
setup now what started my my idea of this cool design is really when we looked at the image
processing software for that OK so when you now want to actually show an image on this
hardware that I just described to you you basically at some high level you need to solve some
kind of an optimization problem so the optimization problem is that you the target image target
high dynamic range image and you have to choose what to put on the LCD panel and you have
to choose what you have to want to put on the on the LED grid on the low resolution LED grid
right and so basically then you need to somehow minimize this to to to to optimize some loss
but it met image metric or whatever you want to call it and So what happens physically here is
that the you know basically the the LED values that we that we choose they get blurred optically
which you can describe for the matrix and then they get pixel by pixel multiplied with that what
you put whatever you choose to put on the LCD panel right and you solve this minimization
problem and you find the best way to control the display OK now what what's interesting
though is that once you actually start thinking about not just building one prototype of this
display but making it more commercial and finding good design points is you wonder OK how
do we we can have some control over this matrix a right so how do we how do we actually
choose the correct optical blur that will make all of this work best right and so actually in the
design phase we actually solve basically the same optimization problem but now the the matrix
a this optical design space is no longer given to us it's difficult to choose OK and that's really
very powerful and that's basically what what computational imaging and computational display
is all about is that we don't live with whatever optical you know function and optical design I
guess that but we get to design it we get to choose it for the best performance of our energy OK
so with that I want to talk about the camera side for the rest of the presentation and so basically
what I want to talk about is end to end designed camera systems so we have and and so the just
in terms of terminology computational cameras are in my terminology there are cameras that
rely inherently on a computational aspect in order to provide the features that they're supposed
to provide and an end to end design is the is uh you know a subset of computational cameras in
which we we basically jointly optimize the hardware and the software so that you get the best
performance you can also do computational cameras in a different way where you basically
manually design but an end to end design we're basically want to jointly and automatically solve
both the optimized together both the hardware aspect and the whatever reconstruction level
we have now you might ask why why we why we need this because isn't isn't deep learning
fixing everything for us we can just take a regular camera right and we can basically extract
whatever we want right and if we look at the computer vision literature there is actually a lot of
log on high dynamic range emit estimating high dynamic range images from single from low
dynamic range images estimating hyperspectral images from RGB images estimating monocular
depth images right stuff like that and what I argue is that basically these are very limited
methods OK and let me just choose the you know you know an example from high dynamic
from previous learning example from high dynamic range and so on the left hand side you know
this is the image your camera sees and on the right hand side these guys down here at this
paper where they called it high dynamic range image hallucination which is a term I very much
like and I'll get back to that basically where they in painted this saturated region with a plausible
texture right and that is kind of for in many situations this is very this is very good you know this
is a very valid if you want to just maybe have a movie and you want to prepare for one of these
high dynamic range displays that I talked about that's perfectly reasonable to put a texture like
this in because it is plausible and it looks good right with deep learning of course it could be
could could look even better now what you don't want to do is use this method in a self driving
car right because if you exit a tunnel and the outside of the tunnel is all blah blown out with
white you don't want to imagine what are plausible ways of the for the road to look like you
want to know what the road actually looks like and whether there are cars there or not OK so
basically that's why I think that's why I like this terminology high dynamic range image
hallucination OK we have to understand that for certain application we can use deep learning to
fill in material to fill in information that we haven't measured OK but for other situations we
really need to understand what the limits are what we've actually measured and we need to
make sure that we don't make decisions based on information we don't have OK and you can
make the same argument for hyperspectral you know for hyperspectral imaging right I mean if
you do have a Wayne Wayne Finder you know one vein finding is a good application for
hyperspectral imaging but of course you want to find where the veins actually are not where
they could possibly be right and so the same thing goes on for depth and for other kinds of
other kinds of you know modalities OK so the key part here is that this these are hallucination
methods and we need to understand what's possible what the information is that we actually
capture and that we can actually rely on for this particular image OK now what we uh when we
talk about cameras we want to basically solve a few things in in in computational cameras
typically so one of them is that we may want to shrink optical systems OK so here is a an image
of this is a redrawn schematic from a cannon patent and basically it shows you a really very
good optical system but it's extremely complex OK so why is it so complex you might ask and
the reason is because if we have a higher more and more demand for higher resolution images
on the sensor here right and we're going from