The free Luma AI iPhone application, is one of the best free tools that can make fully rendered video from 2D images using NeRF technology. Luma AI is an iPhone app that lets you capture and edit NeRFs, which are 3D models of objects, landscapes and scenes that can be viewed from any angle. You can also create cinematic videos of your NeRFs using a camera path editor or by moving through the NeRF in AR preview mode. Luma AI is a great way to create realistic 3D scenes with your iPhone, using the power of AI and NeRF.

You can create videos with impossible camera moves for Social Networks, or just relive the moment. Luma AI is compatible with iPhones as old as the iPhone 11, and does not require any LiDAR sensor or fancy capture equipment. With just your iPhone, you can create incredible and lifelike videos of whatever you want, wherever you are. 

Using the App, you can record a video of your surrounding scene, or around a certain object (recording at 3 different heights, towards the object’s centre).  The app takes care of the rest by uploading the video to Luma AI's servers. As a result, an ultra-smooth orbit around the NeRF can be downloaded as a video, and a 3D textured object can be generated within a few minutes. You can also share the generated NeRF as an interactive Web view.

How to use Luma AI iPhone application:

To use Luma AI iPhone application, you need to follow these steps:

• Download Luma AI from the App Store and sign up with your Apple or Google account.

• Choose Object or Scene that you want to capture in 3D. Make sure it has enough details, textures, and lighting, and avoid transparent, reflective, or thin objects.

• Tap on the Capture button and follow the AR instructions on the screen. Try to move slowly and steadily, and avoid occluding the object or scene with your hand or body.

• After you finish capturing, you can preview your scene or 3D model.

• You can either use a preset path or create your own to gain total control of the "flight" path, by tapping Reshoot, then Custom. This will open a camera path editor based on keyframes. We can move our viewpoint to a new position and tap the Add Keyframe button to record that position. Pinch and drag to move to another location and camera angle, then click the Plus button at the right to add another keyframe.  Luma AI App places keyframes two seconds apart by default, and pressing the Play button shows how it smoothly moves the virtual camera through those two points. We can continue to add keyframes to make a drone-like path. It's laborious compared to walking and moving around in AR mode (more on this in the next section below), but using keyframes allows for greater precision and smoother path.

• Dragging the purple, diamond-shaped keyframes below the camera view allows us to fine-tune the timing. The stopwatch icon allows us to adjust the total length to speed up or slow down the different video sections. We can tap the Overview tab at the top of the screen to see the camera path as a purple line with the camera icons at each keyframe location.

• When we’re satisfied with my flight path, we can use the options at the bottom of the screen to set the aspect ratio and other settings. When first clicking on Render button, we can select the video resolution (SD, HD or Full HD) and the number of frames per second, then we have just to click on Render again, and voila!

  

  • Luma AI App will upload these details to Luma AI’s servers to create the video. Your iPhone isn't needed for this step. You can close the app, or record another video. When the render is complete, you can download the video.

  • You can share a web link of your NeRF through Luma AI’s website, where anyone can view it in 3D.

The following video is a NeRF scan done recently, on a Beautiful Sunny Day ☀️. The result is quite impressive.

AR Mode: 

Luma AI also has an AR mode, which lets you travel through a NeRF (Neural Radiance Field) with your iPhone, revisiting places that you have previously captured. You can simulate a drone fly-through with your iPhone, creating impossible camera moves and perspectives.. Luma AI’s AR mode is a new way to explore and relive your memories in 3D. 

When first opening the app, you will find a gallery of NeRFs  previously captured. Tapping any item will place you into the Cinematic Render view. Tapping the AR button at the top right, places the NeRF within your physical space. In your real space, you can see either the extracted object resting on the floor, or the entire NeRF scene filling your surroundings as an immersive view. You can even record a new video of the scene while navigating in AR mode, it’s as you’re revisiting the same place again!. Please note that you must stay aware of your surroundings, because your physical space doesn't match the NeRF generated environment.

The following video is recorded in AR Mode:

To finish recording, just tap the shutter button again. The video can be saved immediately, but there won't be any stabilization applied to the video. But, if you click on Reshoot button, then the magic happens! Now you can choose how much stabilization to apply. You have also the option to specify the aspect ratio of the video to be rendered, from 9:16, 3:4, 1:1, 4:3, or 16:9.

When you first click on Render, you will get more options like HD/Full HD, or the number of Frames Per Second, then you have to click Render again.

The video will process for few seconds (or more depending on the length of the video), but you can continue do other tasks while the video is rendering.

You can refer to this Luma AI’s X post (Twitter), when they first introduced this AR feature.

The limitations of NeRFs:

When using Luma AI App, we sometimes see "floaters," the equivalent of noise or artifacts. If we move slowly and steadily enough while capturing the NeRF, this is minimized, and the details are much sharper.

When reshooting a camera path for a video, there's a limit to how far and where you can move without these distortions affecting the video. As I move further from the center of the NeRF, the scene distorts and blurs due to a lack of detail, the scene gets blurred if you venture too far.

We hope this will improve, as NeRF and AI models evolves, and Luma AI has already significantly enhanced the quality of their AI models.

Luma AI Capturing Best Practices:

For best results, move the phone slowly and try to avoid rapid movements, especially rotation, as much as possible.

For best results, the object or scene should be captured from as many unique viewpoints as possible. Additionally, it is better to move the phone around (in 3D space) rather than rotating it from a stationary position when capturing. Standing in the same place and capturing outwards in a sphere typically does not work well.

Currently, the app struggles with complex reflections (e.g., curved mirror-like surfaces), curved transparent objects (e.g., car windows or plastic water bottles), and very large textureless surfaces (e.g. white walls). Most other materials work well.

The app can capture objects in most lighting conditions as long as textures can still be identified (i.e., not washed out or completely dark). Lighting conditions will be baked in, so the scene should be lit however you would like it to appear in the final result.

Any movement in the scene during capture may degrade the quality of the final result. For example, tree leaves moving in the wind may result in loss of detail, and people moving around in the background could introduce artifacts.

If using video capture, it is very important that video stabilization is off, since it causes the frames to have unstable camera intrinsics; this is particularly important on Android devices. Also avoid using the "HDR video" option on iOS.

Please refer here, for more details.

Flythroughs iPhone App:

Luma AI has recently simplified the drone footage feature into a new iPhone app called "Flythroughs" which it is marketing explicitly to the real estate industry, that anyone can download from the App Store. 

According to Luma AI, the app can replace expensive professional interior shots for real estate sales.

Luma AI shows high-quality examples of flythroughs generated with the app, like this one:

Please note that exporting the video is a paid feature in this App.

Summary:

Nerf have been around for some time, but was difficult to implement and use. Now it has finally become accessible for anyone to use. Luma AI makes NeRFs almost as easy as shooting a video. We can capture a scene in three dimensions, then reshoot it as a video later using a virtual camera.

The possibilities are endless, simplifying the creation of 3D objects and scenes.

Please share with us your experiences with NeRFs and write down in the comment section any information you would like to add.

Have you ever wondered how to turn your 2D photos into a digital 3D scene? With the latest advances in artificial intelligence, you can now do it in a matter of seconds. In this post, I hope you will know more about NeRF, a neural rendering technique that can generate novel views of complex 3D scenes, based on a partial set of 2D images.

NeRF stands for Neural Radiance Field, which is a fully-connected neural network. NeRF learns to represent and render realistic 3D scenes by optimizing a continuous volumetric scene function using a rendering loss that reproduces input views of a scene.

To use NeRF, you need to provide a sparse set of input views that capture a scene from different angles. NeRF then trains a neural network to predict the colour and opacity of any point in the scene, given its location and viewing direction. By sampling many points along each viewing ray, NeRF can synthesize novel views of the scene using volume rendering.

NeRF can produce high-quality results that are indistinguishable from real images, even for scenes with complex geometry and lighting.

If you want to learn more about NeRF and how it works, you can check out this 2020 research paper on NeRF, by researchers from UC Berkeley, UC San Diego, and Google Research. 

NeRF limitless possibilities:

After generating the 3D scene with NeRF, we can create any camera motion path (I mean virtually) that would both loop perfectly and allow us to create the many different versions of the video we want to render, this is because the camera is created, placed, and controlled mathematically in post-production. That’s why we can easily create looping videos. Even with the best motion control gear, this kind of perfect loop is hard to achieve.

During a regular film shooting, for Commercial advertisements for example, sometimes we can’t get a smooth video like we wish to have, since you only get the frames you shoot and finish in post processing, and also that’s going to be limited by the time frame you have and budget you have. With NeRFs that’s not the case, because your generated video is rendered from a virtual scene, and the length of the video isn’t in any way restrained by the available video frames.

This means it’s always possible to create multiple different versions of the video at different aspect ratios, and the results will never suffer. For example from the same original video footage (or 2D images set), we can generate a 1:1 aspect ratio video for our social network feed, another 9:16 video for our social network story and 16:9 for our main video.

NeRFs are incredibly powerful, and I think we are very close to being ready for the production of daily social media videos, and I think we will not wait for so long until we can use NeRF for production movies with the huge and unprecedented advances of artificial intelligence that are happening these days.

NeRF Applications:

NeRF is a powerful tool for creating synthetic data, which can be used for various applications: 

Robotics: NeRF can be used to train robots and self-driving cars to understand the size and shape of real-world objects by capturing 2D images or video footage of them. 

Urban mapping: NeRF can be used to create 3D maps of cities or landscapes from aerial or satellite images, which can help with planning, navigation, or tourism. 

Virtual reality/augmented reality: NeRF can be used to create immersive and realistic 3D environments from photos or videos, which can enhance the user experience and enable new forms of storytelling. 

Entertainment: NeRF can be used to generate 3D models of celebrities, characters, or scenes from movies or games.

NVIDIA Instant NeRF:

But NeRF is also very computationally intensive, and processing complex scenes can take a lot of time. Fortunately, there are new algorithms that can dramatically improve the performance of NeRF, such as NVIDIA Instant NeRF, which can train and render NeRF models in seconds.

It is one of the fastest NeRF techniques to date, achieving more than 1000x speedups in some cases. It also supports VR mode, which allows users to navigate the 3D scene in virtual reality. You can download the software from here.

Conclusion:

Neural Radiance Fields provide an exciting new way to understand and represent the world around us, by capturing the essence and beauty of reality in 3D.

With continued research and development, Neural Radiance Fields will continue to improve, opening up new world of possibilities in Artificial Intelligence.

NeRF is an amazing example of how AI can transform 2D images into 3D scenes or 3D objects, in the blink of an eye. With NeRF, you can unleash your creativity and turn your photos into immersive experiences!

Augmented reality (AR) augments our physical world by combining it with a layer of digital information. Using an AR application gives you a view of the physical world superimposed with computer-generated 3D objects.

For example, you can use AR to see how a piece of furniture would look in your room before buying it, or to play a game that uses your surroundings as part of the gameplay. AR can also provide additional information about your environment, such as directions, reviews or historical facts.

AR works by using a device, such as a smartphone, tablet or glasses. The device can detect your location, orientation and movement, and overlay relevant digital information on top of the real world. Some AR applications also use machine vision, object recognition and gesture recognition to create more immersive experiences.

AR, just needs a handheld AR-enabled devices (smartphones) or, head-mounted AR glasses. Since you’ll be fully aware of your environment, you won’t need to clear the room to enjoy the experience. Since AR requires less equipment and fewer safety precautions, it’s also more accessible.

Types of AR based Applications:

This is an overview of the common types of augmented reality applications to consider:

• Marker-based AR.  

Marker-based AR, based on Image Recognition, detects an object (a “marker”) in front of the device camera and provides information about the object on the screen. When a device detects that marker, the app replaces the marker on the screen with a 3D object. Then the user can view the object like in real world, from different angles. 

• Markerless AR.  

Markerless AR doesn’t need to be triggered by a specific object in the real world. Instead, the user can place a virtual object anywhere on vertical or horizontal planes. You can easily manipulate (rotate, move…) the object.

• Location-based AR.  

Location-based AR is a type of markerless AR that uses geographic location to display digital content at specific locations. Pokemon Go is an example of location-based AR. 

• Projection-based AR.

Projection-based AR involves projecting synthetic light onto physical surfaces. This allows users to view AR content with their naked eyes without needing any external devices. Common examples include the holograms. 

Where Augmented Reality is used:

AR can be applied into different industries and domains. Here are some of them:

1-Furniture: AR furniture apps like IKEA Place that allow customers to virtually see how furniture would look in their homes before buying it. The app uses marker-based AR, which uses a physical marker (the catalog) to trigger the digital content. The user can also snap a picture of the furniture in the room.

2-Out of home advertising: AR out of home advertising campaigns that use large screens or billboards to display immersive and interactive content to the public. For example, Pepsi used a bus shelter screen in London to show realistic scenes of aliens, tigers and robots invading the street.

3-Healthcare: AR healthcare applications that can assist physicians, surgeons, nurses and patients in various ways. For example, AR can be used to visualize CT and MRI scans during interventional procedures, to guide needle placement, to display vital signs or patient information, to provide training or education, or to enhance patient engagement and satisfaction.

4-Education: AR education applications that can enhance learning outcomes and engagement for students of all ages and levels. For example, AR can be used to create interactive textbooks, to display 3D models of objects or concepts, to provide simulations or experiments, or to enable remote collaboration.  

5-Entertainment: AR entertainment applications that can provide fun and immersive experiences for users. For example, AR can be used to create games that use the real environment as part of the gameplay, such as Pokemon Go, to create filters or effects that modify the user's appearance or voice, such as Snapchat or Instagram, to create stories or animations that interact with the user's surroundings, such as Wonderscope.

6-E-commerce: AR e-commerce applications that can help customers make better purchasing decisions and increase conversions for businesses. For example, AR can be used to let customers try on clothes, accessories, makeup or glasses virtually, such as Warby Parker's virtual try-on app, or to let customers preview how products would look in their homes or spaces, such as Amazon's AR View or Wayfair.

7-Tourism: AR tourism applications that can enhance the travel experience and provide useful information for tourists. For example, AR can be used to display historical facts, cultural insights, directions or reviews about landmarks or attractions, to create immersive tours or guides that showcase the past or present of a location, or to create interactive games or challenges that encourage exploration and discovery.

8-Business cards: AR business cards that can display more information than a traditional card, such as a personal portfolio site or a voice-over. The user can scan the card with their smartphone and see the digital content overlaid on the card.

These are just some of the examples of AR applications in different industries. There are many more possibilities and use cases for AR technology that are being developed and explored every day.

If we take IKEA Place as an example. it’s is an AR app that allows customers to virtually see how furniture would look in their homes before buying it.

To use IKEA Place, you need a smartphone with a camera and the IKEA Place app installed. You need a physical marker, such as the IKEA catalog or any flat surface.

To start using IKEA Place, you need to open the app and scan your surrounding area with your camera. The app will then display a list of furniture items that you can choose from.

To place a furniture item in your room, you need to tap on it and drag it to the desired location. You can also rotate and move the item to adjust it to your space.

IKEA Place also uses realistic 3D models and lighting effects to make the furniture look as natural as possible in your room. You can also snap a picture of the furniture in your room and share it with others.

IKEA Place is one of the many AR apps that can help customers make better purchasing decisions and increase conversions for businesses by combining the real world and digital content.

How Augmented Reality is impacting our lives:

AR is impacting our lives in various ways, such as:

1-Enhancing learning and education: AR can provide interactive and immersive experiences that can improve learning outcomes and engagement for students of all ages and levels. AR can also enable remote collaboration and access to educational resources that may not be available otherwise.

2-Improving work and productivity: AR can help workers perform tasks more efficiently and effectively by providing relevant information, guidance and feedback in real time. AR can also facilitate virtual meetings and communication that can reduce travel costs and environmental impact.

3-Enriching entertainment and social interaction: AR can provide fun and creative experiences that can enhance our enjoyment and connection with others. AR can also allow us to customize and transform our surroundings according to our preferences and moods.

These are some of the ways that augmented reality is impacting our lives. However, there are also some challenges and limitations that need to be addressed, such as technical issues, ethical concerns, privacy risks, social norms and user acceptance.

Challenges and limitations of Augmented Reality

Some of the challenges and limitations of AR are:

1-Cost: Implementing AR technology can be expensive, depending on the type and quality of the devices, software and content. Customizing and maintaining AR solutions can also require significant financial resources and expertise.

2-User adoption: AR technology is still relatively new and unfamiliar to many users, who may not have access to or interest in using AR devices or apps. User adoption can also be influenced by factors such as usability, functionality, design, privacy and security.

3-Technical limitations: AR technology still faces some technical challenges, such as latency, accuracy, resolution, battery life, connectivity and compatibility. These challenges can affect the performance and quality of the AR experience and cause frustration or discomfort for the users.

4-Content availability: AR technology requires a lot of content to create immersive and realistic experiences for the users. However, creating and updating AR content can be time-consuming and complex, requiring specialized skills and tools. There may also be a lack of standards or guidelines for AR content creation and distribution.

These are some of the challenges and limitations of AR that need to be addressed before AR technology can become more widespread and beneficial for various domains and purposes.

Final thought about Augmented Reality and its future adoption

I think that augmented reality is a very promising and exciting technology because it can create immersive and interactive experiences that can enhance our learning, work, entertainment and social interaction. For example, AR can help us visualize complex data, access relevant information, collaborate with others, create personalized content, and also have fun and creative experiences. I think that AR technology can also open up new possibilities and opportunities for various domains and purposes, such as education, healthcare, tourism, e-commerce, marketing and more. For example, AR can help us access educational resources that may not be available otherwise, provide medical assistance or guidance, explore new places or cultures, make better purchasing decisions, and engage with brands or products.

I hope that AR technology will enhance our lives and not replace them. I hope that users will use AR technology wisely and moderately, and that developers will design AR technology with user needs and values in mind.

Hello and welcome to my new blog, where I will share with you my passion and expertise in augmented reality (AR) and artificial intelligence (AI). If you are wondering what AR and AI are and why they matter, you have come to the right place. 

In this blog, I will explain the concepts and principles behind these technologies, as well as showcase some of the amazing applications and projects that I have been involved in or inspired by.

I have been working in the AR industry for many years, developing innovative solutions for various domains such as entertainment, marketing and more. I have witnessed the evolution of AR from a niche technology to a mainstream phenomenon that is changing the way we interact with the world. AR is a technology that overlays digital information or graphics onto the physical environment, creating a mixed reality that enhances our perception. AR can be experienced through different devices such as smartphones, tablets, glasses or headsets.

AI is a technology that enables machines or systems to perform tasks that normally require human intelligence, such as learning, reasoning, decision making or natural language processing. AI can be integrated with AR to create more intelligent and interactive experiences that adapt to the user's context and preferences.

I believe that AR and AI are powerful technologies that can enrich our lives in many ways. They can provide us with new forms of entertainment, education, communication, collaboration and creativity. In this blog, I will explore the latest trends and developments in AR and AI, as well as the challenges and opportunities they present for the future. I hope you will join me on this journey and discover how AR and AI can transform our world for the better!