[Fanisko]

XROS Fellowship 2023

3D portal view of Video/ 360 image

Name and Contact Information

Name: Prakhar Agarwal
Github username: Prakhar-Agarwal-byte
Email: prakharagarwal3031@gmail.com
Alternative Mail: prakhar.agarwal.mec20@itbhu.ac.in
Discord username: @Prakhar#9227
Availability: 30-40 hours a week
University info
● University Name: Indian Institute of Technology (BHU) Varanasi
● Major: Mechanical Engineering
● Current year and expected graduation date: 3rd year ( junior).
Expected graduation will be around 30th November 2024
Degree: B.Tech. (Bachelor of Technology (4 years))
Website: prakharcodes.tech
LinkedIn: prakhar-agarwal-byte
Country: India
Time Zone: IST (UTC+05:30)
 Fanisko | 2

About Me
I am Prakhar Agarwal, a junior year student, currently pursuing
Mechanical Engineering at the Indian Institute of Technology (BHU)
Varanasi and am expected to graduate in the year 2024. I am a
Full-Stack developer with proficiency in HTML, CSS, Javascript,
Reactjs, Nodejs, Python, Flask, C/C++, Threejs, PlayCanvas and
Github. I love exploring new tech and have worked with various
technologies and languages. I love contributing to the open-source
community and building something other developers can use.

Google Summer of Code 2021 with Purr Data

I was selected for Google Summer of Code 2021 with Purr Data
organization. Purr Data is a graphical data-flow programming
environment based on Pure Data written in C++. During the project, I
improved the overall usability of the Purr Data Web Application to
make it more intuitive to use, improve its stability by fixing many
bugs and also improve the UI to better fit the web app. Technologies
I used in this project - Javascript, HTML, CSS, NW.js, C++,
Bootstrap.
The details of my work on the GSoC project can be found at
https://github.com/Prakhar-Agarwal-byte/purr-data

Contributor at Amahi
I have contributed to the Amahi Android app codebase and also
received a letter of recommendation for its co-founder, Carlos
Puchol. I worked on improving the UI/UX of the android app, fixed
bugs, and improved many features in it. Technologies I used in this
project - Java, Kotlin, Android
 Fanisko | 3

Contributions to other open source projects

I’ve contributed to the Meshery and Layer-5 which is a cloud-native

management plane offering lifecycle, configuration, and
performance management of service meshes. I’ve also contributed
to crypto wallets such as Padawan Wallet where I learned more
about blockchain and bitcoins. Here I fixed issues, reviewed PRs,
and learned how to work collaboratively.

Internships at many startups

Apart from this, I have done many internships at startups as a
Full-Stack developer where I got to work in a team and enhance my
technical expertise with the help of my mentors. Technologies I
used: Javascript, Typescript, Reactjs, Nodejs, Python, Flask,
Github, Expressjs, HTML, CSS, and Threejs

List of my open source contributions:

Organization Contributions
Amahi ● PR
● Issues
Purr Data ● PR
● Issues
Padawan Wallet ● PR
● Issues
Meshery ● PR
● Issues
Layer-5 ● PR
 Fanisko | 4

Table of Contents
● Name and Contact Information
● About Me
● Table of Contents
● Title
● Synopsis
● Project Plan
1. Import the 3D model into PlayCanvas
2. Use Three.js to create a texture
3. Use JavaScript to add interactivity to the portal
4. Optimization
● Project Timeline
● Future Deliverables
● Benefits to Community
● Why Me?

Title
3D portal view of Video/ 360 image

Synopsis
This project's main purpose is to launch or make a user view
something on a whole new level. The user will be instructed to enter
the portal in AR, and once Doing so, he will be presented with a
video streaming or 360 Degree picture. The initial screen is
equipped with the rich graphics of having the Digital portal in 3D
view. Upon entering it, the user will be presented with video
streaming material or a 360-degree image that they may rotate.
 Fanisko | 5

Project Plan
PlayCanvas Setup: To begin this project, the first step is to set up a
new project in PlayCanvas. Next, import the 3D model and texture
maps into the project. The scene should be set up by positioning the
camera and lighting to achieve the desired look and feel.

Three.js Integration: To integrate Three.js into the project, the video

or 360-degree image must be added as a texture to the inside of the
portal mesh. The texture should be mapped to the inside of the
portal mesh using Three.js. Any additional features such as lighting
or particle effects can also be added using Three.js.

JavaScript Interactivity: Adding JavaScript interactivity to the portal

is crucial to creating a rich user experience. This can include
allowing the user to rotate the 360-degree image or click on
hotspots to access different parts of the video. Testing and refining
these interactivity features is necessary to ensure they work as
intended.

Optimization and Deployment: Once the project is complete, it's

important to optimize it for performance and loading times. This can
include compressing textures and minimizing the number of assets
used. The project can then be deployed to a web server or hosting
platform for users to access. User testing and feedback can be
gathered to further improve the project.

.
 Fanisko | 6

Import 3D model into PlayCanvas

● Set up a PlayCanvas account and create a new project.

● Import the 3D model of the portal created in software like

Blender or Maya. Ensure that the model is correctly exported in
a format compatible with PlayCanvas, such as .fbx or .obj.

● Add the imported 3D model to the PlayCanvas scene. This can

be done by dragging and dropping the file into the project's
asset panel, and then dragging the asset onto the scene.

● Position and scale the portal model in the scene so that it is the
correct size and orientation. We can use the scene editor in
PlayCanvas to do this.

● Create any additional objects, such as lights or cameras, that

are needed to make the portal scene look realistic.

● Set up the physics properties of the portal mesh, such as its

mass and shape. This can be done using the physics editor in
PlayCanvas.

● Set up any materials or shaders that are needed for the portal
mesh. This can be done using the material editor in
PlayCanvas.

● Once the portal model is set up in the scene, we can test it by

running the project in PlayCanvas and interacting with the
portal.
 Fanisko | 7

Rough technical implementation to create a basic PlayCanvas

project and import a 3D model:

// Initialize the PlayCanvas engine

var canvas =
document.getElementById('application-canvas');
var app = new pc.Application(canvas);
app.start();

// Load the 3D model

var url = 'models/portal.fbx'; // replace with own
model file path
app.assets.loadFromUrl(url, 'model', function(err,
asset) {
if (err) {
console.error(err);
return;
}

// Create an entity to hold the 3D model

var entity = new pc.Entity();
entity.addComponent('model', {
type: 'asset',
asset: asset.id
});
app.root.addChild(entity);
});
 Fanisko | 8

Threejs Integration
● Create a Three.js texture object using the video or 360-degree
image file. We can use the following code to create a texture
from a video:
const video = document.createElement('video');
video.src = 'video-url-here.mp4';
video.autoplay = true;
video.loop = true;

const videoTexture = new THREE.VideoTexture(video);

videoTexture.minFilter = THREE.LinearFilter;
videoTexture.magFilter = THREE.LinearFilter;
videoTexture.format = THREE.RGBFormat;

● Create a material using the Three.js texture object:

const material = new THREE.MeshBasicMaterial({
map: videoTexture
});

● Find the mesh of the portal object in the scene and set its
material to the one we just created:
const portalMesh =
scene.findByName('portal-mesh-name');
portalMesh.material = material;

Now the video or 360-degree image will be displayed as a texture

inside the portal mesh. We can customize the appearance and
behavior of the portal using additional Three.js functionality, such as
adding lighting or animations.
 Fanisko | 9

Javascript Interactivity

● Rotating the 360-degree image:

To allow the user to rotate the 360-degree image, we can use
the Three.js OrbitControls library. Here's an example of how to
use it:
// create the OrbitControls object
var controls = new THREE.OrbitControls(camera,
renderer.domElement);

// set options for the controls

controls.enableDamping = true;
controls.dampingFactor = 0.05;
controls.rotateSpeed = 0.5;

// update the controls in the animation loop

function animate() {
requestAnimationFrame(animate);
controls.update();
renderer.render(scene, camera);
}
animate();

● Adding hotspots to the video:

To add hotspots to the video, we can create clickable objects
in Three.js and then listen for mouse events. Here's an
example:
// create a clickable object
var hotspot = new THREE.Mesh(
new THREE.SphereGeometry(0.5, 32, 32),
new THREE.MeshBasicMaterial({ color: 0xff0000 })
);
 Fanisko | 10

hotspot.position.set(1, 0, 0);
scene.add(hotspot);

// listen for mouse events

window.addEventListener('click', function(event) {
// calculate mouse position in normalized device
coordinates
var mouse = new THREE.Vector2();
mouse.x = (event.clientX / window.innerWidth) * 2
- 1;
mouse.y = -(event.clientY / window.innerHeight) *
2 + 1;

// create a raycaster from the camera and mouse

position
var raycaster = new THREE.Raycaster();
raycaster.setFromCamera(mouse, camera);

// check for intersection with the hotspot

var intersects =
raycaster.intersectObject(hotspot);
if (intersects.length > 0) {
// do something when the hotspot is clicked
console.log('Hotspot clicked!');
}
});

These are just a few examples of the types of interactivity we can

add with JavaScript and Three.js. The possibilities are endless, and
we can use our creativity to come up with unique and engaging
interactions for our users.

Optimization
 Fanisko | 11

We can optimize our view by using the following techniques:

● Use a lower polygon count: Reducing the polygon count of 3D

models can improve performance by reducing the number of
calculations needed to render the scene. This can be achieved
by simplifying the models in a 3D modeling software like
Blender or Maya before importing them into PlayCanvas.

● Optimize textures: Large texture sizes can also impact

performance, so it's important to optimize them by
compressing them and reducing their resolution. PlayCanvas
provides built-in compression settings for textures, which can
be adjusted to optimize the performance.

● Implement culling techniques: Culling involves hiding objects

that are outside of the camera's view or occluded by other
objects in the scene. This can be achieved by implementing
techniques like frustum culling and occlusion culling to improve
performance.

● Use efficient code: Ensuring that JavaScript code is efficient

and optimized can also help improve performance. This can be
achieved by using best practices like minimizing function calls,
using efficient algorithms, and minimizing memory usage.

By implementing these optimizations, the 3D portal view of

video/360 image can run smoothly and efficiently, providing a better
user experience.
 Fanisko | 12

Project Timeline

Project Period (March 25, 2023 - June 25, 2023)

Mar 25, 2023 - Apr 1, 2023 ● Set up PlayCanvas project

and import 3D model and
textures.
● Position camera and
lighting in the scene

Apr 2, 2023 - Apr 16, 2023 ● Research and implement

Three.js integration for
video or 360-degree
image
● Start documenting the
project along with the
code.

Apr 17, 2023 - Apr 24, 2023 ● Map texture to inside of

portal mesh
● Add lighting or particle
effects as needed

Apr 25, 2023 - May 2, 2023 ● Use JavaScript to add

interactivity to portal, such
as rotation or hotspots

May 3, 2023 - May 9, 2023 ● Test and refine interactivity

features
● Keep on documenting my
work

May 10, 2023 - May 17, 2023 ● Design and implement

navigation features, such
as menus or toolbars
● Test and refine UI/UX
features

May 18, 2023 - Jun 1, 2023
Jun 2, 2023 - Jun 9, 2023
Jun 10, 2023 - Jun 17, 2023
Jun 18, 2023 - Jun 25, 2023
Fanisko | 13
● Optimize project for
performance and loading
times
● Conduct user testing and
gather feedback for further
improvement
● Improve on the feedback
by mentor
● Discuss the stretch goal
with the mentor and
finalize what to do.
● Work on the stretch goal.
● Extensive testing, catching
bugs, fixing them and
further improving the code
coverage.
● The final week has been
left free for completing any
remaining work (if any).
This provides sufficient
cushion for making sure
that the timeline is
followed. If everything gets
completed smoothly
before this period then this
will be utilized for
optimizing the backend.
 Fanisko | 14

Future Deliverables
I intend to stay with the community after the XROS Fellowship and
will try to polish the UX of the 3D view. I’ll keep on contributing to
improve the features of Fanisko. It would give me immense pleasure
to be a part of the Fanisko community and improve the lives of those
who use the products of Fanisko.

Benefits to Community
The benefit to the community from this project is the enhancement
of user experience and engagement through the use of cutting-edge
technology like AR, video streaming, and 360-degree imagery. By
creating a 3D portal view of video/360 image users can immerse
themselves in a new level of interactive content that is both visually
stunning and engaging. This technology has the potential to be used
in a variety of fields, including gaming, education, and entertainment,
and can revolutionize the way we experience digital content.

Why Me?
I’m passionate about open source, and when I tried Fanisko for the
first time, I instantly fell in love with it. I know the required
technologies, and as I shared some of my accomplishments above,
I’ve hands-on working experience with all of those. My passion for
open source and never give up attitude when it comes to software
development sets me apart from others. I am excited about the
opportunity to work on this project and will work as hard as I have to
make this project a grand success.