May_Sora｜Visual Generative AI Technology and Application Development Overview(Next)

Published On: 2024/05/31|Categories: 科技(Technology)|

Currently, most of the visual generative AI belongs to multimodal model, which realizes cross-domain information processing and interaction by learning multimodal connections from data, including text-to-graphic, text-to-3D object, text-to-movie, image-to-movie, image-to-text, and so on, and the following is a summary of the development of the index vendors.

Google :

Image: In 2021, Disco Diffusion, an AI image generation tool on the Google Colab platform, and Dream Fields, an AI 3D object generation tool, will be released, which utilizes the NeRF architecture to generate 3D objects, and then determines whether the generated objects are realistic through the CLIP architecture. 2022 Launch of AI Image Generation Tools Imagen and the 3D object tool Dream Fusion, the former adopts the Diffusion architecture, and will be followed by the development of Imagen 2 and Imagen FX in 2023 and 2024, while the latter is an evolved version of Dream Fields, which can create 3D objects without the need for a large amount of data due to the use of Imagen to generate 2D images.
Video : In 2022, Imagen Video and Phenaki and other two AI-generated video tools, Imagen Video's mode of operation is first embedded in the user input text through the T5 model, and then by the Diffusion framework at a rate of 3 frames per second to generate a resolution of 24 × 48 video, and then TSR and SSR to generate a total of 128 frames and a resolution of 1280 × 768 3-second video. 128 frames and a resolution of 1280 × 768 of the 3-second video. Phenaki can be based on different combinations of text descriptions to generate a coherent and diverse video, although the image quality is not as good as Imagen Video, but the video presented is very close to the text descriptions, but also based on a series of text to generate a coherent multi-video. 2023, in collaboration with the Stanford University research team to launch the A In 2023, we collaborated with Stanford University's research team to launch W.A.L.T, an AI-generated video tool that uses the Diffusion Transformer framework to generate a 3-second video with a resolution of 512×896 from this text and image. In the same year, the company also launched a self-developed AI-generated video tool VideoPoet, in addition to text to generate a video, image to generate a video, video style conversion, video repair, video to generate audio and other five functions, it can also be a second image prediction of the succeeding video, and then generate a longer video through the successive prediction.

OpenAI.

Image: DALL-E, the first AI-generated image tool, will be released in 2021, and has been developed into the third generation, which has the functions of text-generated image, edit image, fill-in-the-blanks, and remake-change, etc. Text-generated image generates corresponding images based on the input commands, edit image modifies the image based on the input commands, fill-in-the-blanks generates images other than the original one to fill in the picture, and remake-change generates five additional change images according to the user's choices. In 2022, Point-E, an AI-generated 3D object tool, was released with Glide and Image-to-3D modes, the former generating objects from textual descriptions, and the latter being trained by using images and related 3D objects to learn to generate corresponding point clouds with coarser textures. 2023 saw the release of Shap-E, an upgraded version of the AI-generated 3D object tool. An upgraded version of the AI-generated 3D object tool, Shap-E, is introduced in 2023, which generates implicit functional parameters for rendering using the NeRF architecture to produce realistic 3D objects with faster correction speeds and better object quality than Point-E in higher dimensions and multiple representations of the output space.
Video : In 2024, we announced the launch of Sora, an AI video generation tool that can generate a 1-minute high-resolution video from input text in a few seconds, far exceeding the industry's current video generation time of a few seconds, making it the most powerful video generation tool available today. However, only the result video has been released and is expected to be uploaded by the end of 2024, and the video will not be equipped with sound effects at this stage, so the function is not yet complete.

Meta.

Imaging : MCC, an AI-generated 3D object tool, will be launched in 2023. it uses the Transformer architecture to generate a code from an incoming RGB-D image, and then a decoder to generate 3D point data to predict the occupancy and RGB color of that point, with better versatility than mesh and stereoscopic metrics.
Video: In 2023, we will launch Make-A-Video and Emu Video, AI video generation tools. The former first uses existing models to generate images that match text descriptions, and then assembles the images into a coherent video through spatio-temporal modeling, which has the advantages of speeding up the training process of the models, eliminating the need for matching textual and video data, and generating videos that are aesthetically diverse to meet the aesthetic needs of different users. The advantages include faster model training, no need for matching textual movie data, and aesthetic diversity to satisfy different users' needs. The latter is a simple and efficient video generation tool that generates images based on textual cues, and then generates a 4-second video with a resolution of 512×512 through the Diffusion framework for easy distribution on social media.

Apple.

Image: In June 2021, we released Object Capture, an AI-generated 3D object tool that allows you to take multiple photos from your iPhone or iPad and stitch them together into a 3D image in USDZ format for subsequent professional software optimization. in July 2022, we will release Gaudi, an AI-generated 3D scene tool that operates on the principle of potential and network parameter optimization, and then learns to generate a 3D scene using a Diffusion architecture. Gaudi, a 3D scene generation tool for AI, will be launched in July 2022.

Runway.

Video : In 2023, Gen-1 and Gen-2, AI generated video tools, will be launched. The former adopts the Diffusion framework, which is mainly used to convert existing videos, allowing users to modify the color of objects in the video, graphic style and other features with text commands. It provides five modes, including: (a).Stylization: transfer any image or cue style to each frame of the video, (b).Storyboard: turn the model into a fully stylized and animated rendering, (b).Mask: isolate the subject matter in the video and modify it using simple text cues, (d).Render: apply textless textures to the image or cue by inputting the image or cue, (e). Render: Turn textureless rendering into realistic output by applying input images or prompts, (e).Customization: Customize the model to get higher quality image results; the latter allows users to generate 4~16 second videos with text descriptions. The latter allows users to generate a 4~16 second video with text descriptions. Compared to Gen-1, three new functions have been added: Text to Video, Text + Image to Video, and Image to Video, which are to synthesize a video in the desired style using only text prompts.

Luma AI

Image : AI-based 3D object generation tool based on the ReNF architecture has been launched, allowing users to generate 3D images by taking photos or typing text. the web version of Luma will be available in October 2022, and the iOS version of the Luma App will be launched in January 2023, allowing users to upload images from their iPhones to generate 3D images, which supports three modes: imported, guided, and free, etc. The imported mode is the same as the web version of Luma. It supports three modes: import, guide, and free. The import mode is the same as the web version of Luma, the guide mode requires the app to prompt for a 360-degree shooting angle, and the free mode supports general shooting where the app does not prompt for an angle.
Imaging : Flythroughs, an AI-generated video tool launching in August 2023, allows users to create 3D flight videos simulating drone footage via their iPhones.