This product has been discontinued.


Introduction

TMPGEnc Reader SDK for Unity is an asset that provides the facilities to play common files such as MP4 (H.264/AAC) directly on Unity without using the MovieTexture. Since this asset utilizes Microsoft Media Foundation for importing and decoding video and audio files, you can use generic formats without installing external CODECs.

  • Allows playing common files (e.g.MP4) directly on Unity.
  • Easy handling on Unity since it imports video as Texture2D and audio as AudioClip.
  • Allows high-speed rendering with Hardware Acceleration. *Depends on GPU capability.
  • This asset does not contain Copyleft license such as GPL or LGPL.

Demo Movie

Requirements

Supported Platforms

  • Windows 7 / Windows 8.1 / Windows 10 (x86 / x64)
    • Windows Runtime (UWP) is not supported.
    • Visual C++ 2015 install is required. (The required merge modules for the installer are included in this asset.)
  • Unity 5.3.5f1 and later.
    • GraphicsDevice class supports only Direct3D11.

Supported File Formats

  • Formats supported by Microsoft Media Foundation.
    • Video - H.264, H.265 (Windows 10), WMV and others
    • Audio - AAC, WMA and others
    • System - MP4, MKV (Windows 10), ASF and others
 

* Only supports the first video and audio stream in a file.

Usage

Setup

  1. Launch Unity Editor with the Direct3D11 mode.
  2. Set the Graphics API only to Direct3D11 in PlayerSettings (Edit > Project Settings > Player > PlayerSettings > Other Settings).
  3. Import the unitypackage of the plugin to an arbitrary project.  
    • The required folders are "Plugins", "TMPGEncReaderSDK/Scripts" and other folders located under "TMPGEncReaderSDK/Scripts"; please import them.
    • The unitypackage contains the Scene file which is a sample application. If you do not want to use it, please exclude it from importing the files.
    • The "TMPGEncReaderSDK/Redist" contains Visual C++ runtime merge modules. Please install them into a developed application when you deliver it.
  4. Generate a GameObject of the "AVReaderPluginManager".
    Select GameObject > TMPGEnc Reader > AVReaderPluginManager, then generate the GameObject.
    • The AVReaderPluginManager simplifies the initializing and releasing of the video engine on an OS.
    • Please place one AVReaderPluginManager on a first or previous position where the AVPlayer reads for the first time. The first scene where the application starts and reads is recommended.
    • The AVReaderPluginManager continues existing since DontDestroyOnLoad is applied to it.
             

Play Movie

     
  1. Generate a GameObject of the ""AVPlayer".
    Select GameObject > TMPGEnc Reader > AVPlayer, then generate the GameObject.
  2. Set the following settings when opening a source file at starting the script.
    * If you want to open a source file at any time, please execute OpenAsync method separately from the script.
    1. Check "Load Source File on Start".
    2. Specifies a source file to play at "Source File Path" of "AV Player" in the Inspector.  
      • Checking "Load from StreamingAssets Folder" registers files in "StreamingAssets" and under folders as relative path.
    3. Implement callback event. It requires registering the following event handers (at least 2 or more) in an arbitrary MonoBehavior script.
      • OnOpenSource(AVPlayer, OpenResult) event: This is notified when the open source process completes. Implement the code to set the Material for the GameObject as the rendering target when OpenResult is Success. You can get the information of video renewed Texture and format from "AV Player".
      • "OnPreviewCloseSource(AVPlayer)" event: This is notified immediately before processing to close the source. Please release the reference to the Texture when you set at Open. If you do not release the reference, which makes Unity unstable since the Texture which is contained in the plugin will be discarded at closing the source.
    4. If you want to open and then play a source file, please check "Start Playback after Source File Loading".
          
               

About Texture

UV axis (0.0, 0.0) becomes top left and (1.0, 1.0) becomes top right in the video of the Texture which is passed from the plugin.
Occasionally it is displayed rotated or reversed unintentionally depending on the vertex setting of the applied Mesh. In such a case, please properly adjust the setting of the Mesh or by the Shader. Since aspect ratio adjustment is not processed, please adjust the rendering position by considering the size and pixel ratio of the video. For more details, please refer to the API references.
The effectiveness of the Texture is maintained only between Open and Close. So please be careful of managing the reference for the Texture.

API References

High Level API

Classes that inherit MonoBehaviour and can be used directly in a GameObject.


Pegasys.AVReader.AVReaderPluginManager

Keeps the effective status of the plugin.
(Executes PluginLoader.AVReaderInitialize in Awake, and executes AVReaderFinalize in OnDestroy.)
The options for AVReaderPluginSetting can be set in the Inspector.


Field / Property

AllowHardwareDecoder

Specifies whether to allow the use of the hardware decoders.
For more details, please refer to "Pegasys.AVReader.AVReaderPluginSetting" below.


AllowDirect3D11Video

Specifies whether to allow the use of Direct3D 11 Video API as the video API.
For more details, please refer to "Pegasys.AVReader.AVReaderPluginSetting" below.


AudioBufferLengthSec

Sets the audio buffer length in seconds.
For more details, please refer to " Pegasys.AVReader.AVReaderPluginSetting" below.

     

LogType

Sets the options to export logs displayed in Unity Console for debugging.
For more details, please refer to " Pegasys.AVReader.LogType" below.

     

Method

AVReaderInitialize

Initializes the plugin.
This can be executed any number of times, but actual initialization is processed only on the first time. The functions of the plugin can be used in the status of which the initialization is being done.

     

AVReaderFinalize

Releases the plugin.
It requires executing at the same time that AVReaderInitialize was executed. The actual releasing is done when the last AVReaderFinalize is called.

     

Pegasys.AVReader.AVPlayer

The class of the movie player.
The video is rendered as a Texture of Unity.
This class plays audio which is imported as AudioClip in AudioSource.
It is required that the AudioSource is being registered in the GameObject because this class is depending on the AudioSource.


Field / Property

OpenOnStart

Sets whether to open the source file when the Start event is called.


OpenOnStartSourcePath

The path of the source file to open when OpenOnStart = true.


OpenOnStartLoadFromStreamingAssets

Specifies whether the path which is specified in OpenOnStartSourcePath is under "StreamingAssets".
When true is set, the path is handled as a file under StreamingAssets and is opened as a related path.
When false is set, it requires setting the source file path as an absolute path.
If you want to play a movie in an application, please place the movie under "StreamingAssets" and set the related path to OpenOnStartSourcePath after setting to true.


OpenOnStartRequestVideo

The RequestVideo flag to be specified when OpenOnStart = true.
For more details, please refer to OpenAsync method below.


OpenOnStartRequestAudio

The RequestAudio flag to be specified when OpenOnStart = true.
For more details, please refer to the OpenAsync method below.


OnOpenSource

Registers the event handler to notify the result of the OpenAsync method.
The result is notified regardless of success or failure.
When displaying video, please set the settings of Material, etc. by using this event.


OnPreviewCloseSource

Registers an event handler which notifies that it is going to close the source such as CloseAsync method.
Close cannot be canceled. Please release the reference to VideoTexture in this event.
Although the process of closing the source is also called from OnDestroy, even if such a case happens, this event is notified.


OnCloseSource

Registers an event handler which notifies that it has closed the source such as the CloseAsync method.
Although the process of closing the source is also called from OnDestroy, even if such a case happens, this event is notified.


OnEndOfSource

Registers a callback which notifies when the current playing source reaches the end of its length.
The callback is notified even if it is set as Loop = true (loops after the notification).


PlayOnOpenSucceed

Sets whether to start playing also if opening the movie succeeds.


Loop

Specifies whether to play the movie in a loop.


IsOpened

Retrieves whether a file is being opened.


OpenResult

Retrieves the results after opening the file. If a file is not opened, HasValue returns false.


HasVideo

Retrieves video source existence. If a file is opened in requestVideo = false, this returns false even if a video source actually exists in the file.


HasAudio

Retrieves audio source existence. If a file is opened in requestAudio = false, this returns false even if an audio source actually exists in the file.


CurrentStatus

Retrieves the current player status.


IsSeeking

Retrieves whether the seeking is being processed.


CurrentPosition

Retrieves the current playing position (time). This cannot be set. * The position setting can be set in SeekAsync method.


Duration

Retrieves the length of the source.


VideoFormat

Retrieves the video format.


AudioFormat

Retrieves the audio format.


VideoTexture

Texture for video. The decoded video is refreshed on this texture.        

           
  • VideoTexture is discarded internally when the source file is closed. Please release the reference by using the OnPreviewCloseSource event when it has a reference.
  • The texture size applies the number of pixels of the source file as it is.
    For example, when the source file size is 1440x1080 16:9, the texture size becomes 1440x1080 in size, not 1920x1080.
  • The aspect ratio can be retrieved in VideoFormat. Please adjust the ratio to be displayed properly by using any method based on the retrieved parameters.
    * The aspect ratio which is retrieved in VideoFormat is pixel based aspect ratio.
  •        


AudioClip

Retrieves AudioClip of the audio source. This can be used when you want to get the audio samples and process them separately.


Method

OpenAsync

Opens the source file. This is asynchronous behavior. The result is notified asynchronously to the callback which is determined in the OnOpenSource field.

  • sourcePath: Source file path. Specify the full path of a source file.
  • requestVideo: Sets whether to request the video source. If you want to open only an audio stream in a file which also contains a video stream, please set to false.
  • requestAudio: Sets whether to request the audio source. If you want to open only a video stream in a source file which also contains an audio stream, please set to false.


CloseAsync

Closes the source file. This is asynchronous behavior. The result is notified asynchronously to the callback which is determined in the OnPreviewCloseSource field prior to actual closing. After closing, the result is also notified asynchronously to the callback which is determined in the OnCloseSoure field.


SeekAsync

Moves the current playing position to the specified position (time). This is asynchronous behavior.

  • pos: Destination time


PlayAsync

Starts playing. This is asynchronous behavior.


Pause

Pauses playing.


Low Level API

Classes that do not inherit MonoBehaviour. Normally, please use High Level API.


Pegasys.AVReader.AVReaderPluginSetting

This class sets the settings in connection with the entire plugin. You can process the actual access from the references which can be retrieved from Default properties.


Field / Property

IsAvailablePlugin

Retrieves whether the plugin is valid. If it is invalid, it may be caused by the following conditions.

  • The initialization is not processed. AVReaderPluginManager does not exist, etc.
  • Graphics API in Unity is not set to Direct3D 11.
  • System does not meet the requirements.


AllowHardwareDecoder

Specifies whether to allow the use of the hardware decoders.

  • When true is set, uses the hardware decoder functions if possible.
  • When false is set, always disables the hardware acceleration and uses software decoders forcibly.
Normally, set as true.


AllowDirect3D11Video

Specifies whether to allow the use of Direct3D 11 Video API as the video API.

  • When true is set, uses the Direct3D 11 Video API if possible.
  • When false is set, always uses Direct3D 9 Video API.
The performance of Direct3D 11 Video API is better than other APIs, so please set as true in most cases.
Since Direct3D 11 Video API cannot be used in Windows 7 environments, Direct3D 9 Video is always used.


AudioBufferLengthSec

Sets the audio buffer length in seconds.
This is the length of buffer which maintains decoded audio samples in the reader. In other words, it is the size which can maintain decoded samples without any processing.
Usually you don't have to change the value when using AVPlayer.


LogType

Sets the options to export logs to Unity Console.
You can turn on and off the options for the plural items by bit flag.
Please set Error, Warning, Trace_Initialize in most cases.
Since Trace_GetSample and Trace_Render are called at high frequency, the performance of Unity itself may become slow.

  • Warning: Exports the warnings. The errors that occur while AVReader is working are exported as warnings.
  • Trace_Initialize: Exports the trace log at initializing and releasing.
  • Trace_GetSample: Exports the trace log of processing which is related to getting samples while AVReader is working.
  • Trace_Render: Exports the trace log of processing which is related to rendering video while AVReader is working.


Default

Retrieves the instance of AVReaderPluginSetting.


Pegasys.AVReader.AVReader

This class is to read a source from a source file and decode it. This is the direct wrapper of the native plugin.
Since all methods of this are synchronization call, it requires an alternative action to desynchronize them. AVPlayer works in asynchronous mode, so please use AVPlayer normally.


Field / Property

  

IsOpened

Retrieves whether a source file is being opened.


  

Duration

Retrieves the length of the source.


  

HasVideo

Retrieves video source existence. If a file is opened in requestVideo = false, this returns false even if a video source actually exists in the file.


  

HasAudio

Retrieves audio source existence. If a file is opened in requestAudio = false, this returns false even if an audio source actually exists in the file.


  

VideoFormat

Retrieves the video format.


  

AudioFormat

Retrieves the audio format.


  

CachedVideoSampleCount

The number of video samples which are currently being decoded and cached.


  

CachedAudioSampleCount

The number of audio samples which are currently being decoded and cached.


Open

Opens the source file. This can be called from outside of the main thread.

  • sourcePath: Source file path. Specify the full path of a source file.
  • requestVideo: Sets whether to request the video source. If you want to open only an audio stream in a source file which also contains a video stream, please set to false.
  • requestAudio: Sets whether to request the audio source. If you want to open only a video stream in a source file which also contains an audio stream, please set to false.
  • Return value: The results of the operation.


  

Close

Closes the source file.


  

GetNativeRenderTargetTexture

Retrieves Texture of the native plugin.
Generates Texture internally at first call after Open.
Calling from outside of the main thread may make it invalid in status. Please use CreateRenderTargetTexture in most cases.


  

CreateRenderTargetTexture

Generates Texture for rendering video.
This can be processed only when the video source exists after Open.


  

Seek

Seeks the specified time and gets the samples from the position.
All existing caches are cleared. This can be called from outside of the main thread.


  

SetRenderVideoPosition

Specifies the time to render the video samples.
The video which corresponds with the specified time is reflected to Texture by the RenderVideo method. This can be called from outside of the main thread.


  

NextVideoSample

Decodes and retrieves the video samples from the source file, and stores the cache data. Nothing is processed if the cache size already reaches the maximum value.
When the return is true, NextVideoSample can still be processed because it is not the end of the source. This can be called from outside of the main thread.


  

NextAudioSample

Decodes and retrieves the video samples from the source file, and stores the cache data. There is a possibility that the cache data will overflow. To reduce overflow, it requires reading the data accordingly from the cache through ReadAudioSample.
When the return is true, NextAudioSample still can be processed because it is not the end of the source. When the return is false, it means the current position is the end of the source, or the cache buffer overflows. This can be called from outside of the main thread.


  

RenderVideo

Refreshes video which corresponds with the specified time in SetRenderVideoPosition on Texture. Also, this discards samples which are never referred to from cache such as samples located prior to the specified time.


  

ReadAudioSample

Reads audio samples from the cache.
The samples are read from the beginning of the cache, and the already read samples are deleted from the cache. This can be called from outside of the main thread.

  • dst: The array to store the data.
  • offsetSampleCount: Specifies the offset of the read start position of the dst array.
  • readSampleCount: The number of samples to read.


Copyright information

      

Copyright (c) Pegasys Inc, All rights reserved.

The sample movie contained within the SDK is from the Elephants Dream licensed under the Creative Commons Attribution 2.5. Copyright 2006, Blender Foundation / Netherlands Media Art Institute / www.elephantsdream.org