Summer at UPenn: Building AI, 3D Scanning Rooms, and Playing Tennis

This summer, I had the chance to dive deep into artificial intelligence at one of the top programs in the country: the Engineering Summer Academy at the University of Pennsylvania (ESAP). Over three immersive weeks, I joined a cohort of high school students from around the world to explore the math, theory, and hands-on coding behind today’s most exciting AI breakthroughs.

What I Built:

For my final project at the University of Pennsylvania’s engineering summer program, I built Reality Layer a system that captures real-world spaces and transforms them into fully explorable video game environments. My aim was to create a seamless pipeline where reality could be scanned, processed, and dropped directly into an interactive 3D world.

The process starts with scanning a location using photogrammetry, capturing hundreds of overlapping images to reconstruct the environment as a detailed 3D mesh with realistic textures. Once the mesh is generated, it’s cleaned, optimized, and exported in a format Unity can read, such as GLB or FBX.

From there, Unity takes over. The 3D model is imported into a game scene, where I set up collision meshes so players can walk around without clipping through walls. Lighting is baked to match the original environment’s mood, and physics are applied so objects behave realistically. A first-person controller allows the user to explore the scanned space as if they were actually there.

The final product is more than a static model — it’s a functional, interactive environment. This opens exciting possibilities: museums could create virtual tours of rare exhibits, urban planners could test layouts before construction, or gamers could explore real places inside entirely new storylines.

What I Learned from the Project:

Building Reality Layer showed me how engineering, art, and game design can merge into something both immersive and practical. By turning real-world data into a digital playground, I saw firsthand how technology can reshape the way we experience, share, and preserve our environments. Working alongside others taught me the importance of dividing responsibilities, communicating ideas clearly, and leveraging different skill sets to achieve a shared vision.

Our 3d Scanner with no color

Overcoming Failure and Limitations:

Reality Layer went through multiple iterations shaped by trial and error. Our first approach used LiDAR scanning(a method that measures distance with lasers) which gave fast results but lacked color, making the environments feel lifeless. To improve realism, we switched to photogrammetry (using multiple 2D images from different angles to reconstruct a 3D model) but this required far more computing power than expected, forcing us to move processing from a phone to a computer.

Even then, the workflow was slow and inconsistent, so we turned to a premade photogrammetry app (Polycam) for all scans, which streamlined image capture and processing. However, the large, high-detail models caused Unity to lag or crash. We optimized them by reducing polygon counts and baking textures while keeping visual quality intact.

Initially, loading models into Unity required using the terminal and placing files in a specific folder. We solved this by integrating StandaloneFileBrowser, allowing models to be loaded at the click of a button.

Each challenge — from hardware limitations to workflow bottlenecks — pushed us to adapt and simplify. These setbacks ultimately made the system faster, more reliable, and more accessible for end users.

Our game environment(a room at UPenn) with our game environment and object placement

What I Learned: The Math Behind the Magic

ESAP isn’t just about coding — it’s about understanding how AI works under the hood. We covered:

• Linear algebra & matrix operations for image and data transformation

• Probability and statistics for model training and error correction

• Neural networks and the backpropagation algorithm

• Training image classifiers using TensorFlow

• Using AI to recognize faces, objects, and gestures

We also got to see real research happening at Penn Engineering and how these tools are being applied in fields like healthcare, robotics, and autonomous systems.

Life Outside the Lab: Tennis Every Night

While the days were packed with lectures, labs, and coding sessions, the evenings were a different kind of fun. I played tennis every night — a perfect way to wind down, meet other students, and keep up my competitive spirit. The campus courts were incredible, and I got to challenge players from around the world.

Why This Mattered

This wasn’t just another summer camp. ESAP gave me:

• A stronger foundation in AI theory and math

• Real experience building a vision-based application

• Exposure to college-level research and academics

• The confidence to explore real-world applications of AI, like my guide dog distraction detection project

Most importantly, it deepened my passion for using AI to solve meaningful problems — from assistive tech to spatial computing.

Up Next

I’m currently working on refining my room scanner into a prototype that could plug into game engines. I’m also preparing to present my AI + guide dog research at a European assistive tech conference this fall (see my previous post for that story).