Introduction
TMPGEnc Reader SDK for Unity is an asset that provides the facilities to play common files such as MP4 (H.264/AAC) directly on Unity without using the MovieTexture. Since this asset utilizes Microsoft Media Foundation for importing and decoding video and audio files, you can use generic formats without installing external CODECs.
- Allows playing common files (e.g.MP4) directly on Unity.
- Easy handling on Unity since it imports video as Texture2D and audio as AudioClip.
- Allows high-speed rendering with Hardware Acceleration. *Depends on GPU capability.
- This asset does not contain Copyleft license such as GPL or LGPL.
Demo Movie
Requirements
Supported Platforms
- Windows 7 / Windows 8.1 / Windows 10 (x86 / x64)
- Windows Runtime (UWP) is not supported.
- Visual C++ 2015 install is required. (The required merge modules for the installer are included in this asset.)
- Unity 5.3.5f1 and later.
- GraphicsDevice class supports only Direct3D11.
Supported File Formats
- Formats supported by Microsoft Media Foundation.
- Video - H.264, H.265 (Windows 10), WMV and others
- Audio - AAC, WMA and others
- System - MP4, MKV (Windows 10), ASF and others
* Only supports the first video and audio stream in a file.
Usage
Setup
- Launch Unity Editor with the Direct3D11 mode.
- Set the Graphics API only to Direct3D11 in PlayerSettings (Edit > Project Settings > Player > PlayerSettings > Other Settings).
- Import the unitypackage of the plugin to an arbitrary project.
- The required folders are "Plugins", "TMPGEncReaderSDK/Scripts" and other folders located under "TMPGEncReaderSDK/Scripts"; please import them.
- The unitypackage contains the Scene file which is a sample application. If you do not want to use it, please exclude it from importing the files.
- The "TMPGEncReaderSDK/Redist" contains Visual C++ runtime merge modules. Please install them into a developed application when you deliver it.
- Generate a GameObject of the "AVReaderPluginManager".
Select GameObject > TMPGEnc Reader > AVReaderPluginManager, then generate the GameObject.- The AVReaderPluginManager simplifies the initializing and releasing of the video engine on an OS.
- Please place one AVReaderPluginManager on a first or previous position where the AVPlayer reads for the first time. The first scene where the application starts and reads is recommended.
- The AVReaderPluginManager continues existing since DontDestroyOnLoad is applied to it.
Play Movie
- Generate a GameObject of the ""AVPlayer".
Select GameObject > TMPGEnc Reader > AVPlayer, then generate the GameObject. - Set the following settings when opening a source file at starting the script.
* If you want to open a source file at any time, please execute OpenAsync method separately from the script.- Check "Load Source File on Start".
- Specifies a source file to play at "Source File Path" of "AV Player" in the Inspector.
- Checking "Load from StreamingAssets Folder" registers files in "StreamingAssets" and under folders as relative path.
- Implement callback event. It requires registering the following event handers (at least 2 or more) in an arbitrary MonoBehavior script.
- OnOpenSource(AVPlayer, OpenResult) event: This is notified when the open source process completes. Implement the code to set the Material for the GameObject as the rendering target when OpenResult is Success. You can get the information of video renewed Texture and format from "AV Player".
- "OnPreviewCloseSource(AVPlayer)" event: This is notified immediately before processing to close the source. Please release the reference to the Texture when you set at Open. If you do not release the reference, which makes Unity unstable since the Texture which is contained in the plugin will be discarded at closing the source.
- If you want to open and then play a source file, please check "Start Playback after Source File Loading".
About Texture
UV axis (0.0, 0.0) becomes top left and (1.0, 1.0) becomes top right in the video of the Texture which is passed from the plugin.
Occasionally it is displayed rotated or reversed unintentionally depending on the vertex setting of the applied Mesh. In such a case, please properly adjust the setting of the Mesh or by the Shader.
Since aspect ratio adjustment is not processed, please adjust the rendering position by considering the size and pixel ratio of the video. For more details, please refer to the API references.
The effectiveness of the Texture is maintained only between Open and Close. So please be careful of managing the reference for the Texture.
API References
High Level API
Classes that inherit MonoBehaviour and can be used directly in a GameObject.
Pegasys.AVReader.AVReaderPluginManager
Keeps the effective status of the plugin.
(Executes PluginLoader.AVReaderInitialize in Awake, and executes AVReaderFinalize in OnDestroy.)
The options for AVReaderPluginSetting can be set in the Inspector.
Field / Property
AllowHardwareDecoder
Specifies whether to allow the use of the hardware decoders.
For more details, please refer to "Pegasys.AVReader.AVReaderPluginSetting" below.
AllowDirect3D11Video
Specifies whether to allow the use of Direct3D 11 Video API as the video API.
For more details, please refer to "Pegasys.AVReader.AVReaderPluginSetting" below.
AudioBufferLengthSec
Sets the audio buffer length in seconds.
For more details, please refer to " Pegasys.AVReader.AVReaderPluginSetting" below.
LogType
Sets the options to export logs displayed in Unity Console for debugging.
For more details, please refer to " Pegasys.AVReader.LogType" below.
Method
AVReaderInitialize
Initializes the plugin.
This can be executed any number of times, but actual initialization is processed only on the first time. The functions of the plugin can be used in the status of which the initialization is being done.
AVReaderFinalize
Releases the plugin.
It requires executing at the same time that AVReaderInitialize was executed. The actual releasing is done when the last AVReaderFinalize is called.
Pegasys.AVReader.AVPlayer
The class of the movie player.
The video is rendered as a Texture of Unity.
This class plays audio which is imported as AudioClip in AudioSource.
It is required that the AudioSource is being registered in the GameObject because this class is depending on the AudioSource.
Field / Property
OpenOnStart
Sets whether to open the source file when the Start event is called.
OpenOnStartSourcePath
The path of the source file to open when OpenOnStart = true.
OpenOnStartLoadFromStreamingAssets
Specifies whether the path which is specified in OpenOnStartSourcePath is under "StreamingAssets".
When true is set, the path is handled as a file under StreamingAssets and is opened as a related path.
When false is set, it requires setting the source file path as an absolute path.
If you want to play a movie in an application, please place the movie under "StreamingAssets" and set the related path to OpenOnStartSourcePath after setting to true.
OpenOnStartRequestVideo
The RequestVideo flag to be specified when OpenOnStart = true.
For more details, please refer to OpenAsync method below.
OpenOnStartRequestAudio
The RequestAudio flag to be specified when OpenOnStart = true.
For more details, please refer to the OpenAsync method below.
OnOpenSource
Registers the event handler to notify the result of the OpenAsync method.
The result is notified regardless of success or failure.
When displaying video, please set the settings of Material, etc. by using this event.
OnPreviewCloseSource
Registers an event handler which notifies that it is going to close the source such as CloseAsync method.
Close cannot be canceled. Please release the reference to VideoTexture in this event.
Although the process of closing the source is also called from OnDestroy, even if such a case happens, this event is notified.
OnCloseSource
Registers an event handler which notifies that it has closed the source such as the CloseAsync method.
Although the process of closing the source is also called from OnDestroy, even if such a case happens, this event is notified.
OnEndOfSource
Registers a callback which notifies when the current playing source reaches the end of its length.
The callback is notified even if it is set as Loop = true (loops after the notification).
PlayOnOpenSucceed
Sets whether to start playing also if opening the movie succeeds.
Loop
Specifies whether to play the movie in a loop.
IsOpened
Retrieves whether a file is being opened.
OpenResult
Retrieves the results after opening the file. If a file is not opened, HasValue returns false.
HasVideo
Retrieves video source existence. If a file is opened in requestVideo = false, this returns false even if a video source actually exists in the file.
HasAudio
Retrieves audio source existence. If a file is opened in requestAudio = false, this returns false even if an audio source actually exists in the file.
CurrentStatus
Retrieves the current player status.
IsSeeking
Retrieves whether the seeking is being processed.
CurrentPosition
Retrieves the current playing position (time). This cannot be set. * The position setting can be set in SeekAsync method.
Duration
Retrieves the length of the source.
VideoFormat
Retrieves the video format.
AudioFormat
Retrieves the audio format.
VideoTexture
Texture for video. The decoded video is refreshed on this texture.
- VideoTexture is discarded internally when the source file is closed. Please release the reference by using the OnPreviewCloseSource event when it has a reference.
- The texture size applies the number of pixels of the source file as it is.
For example, when the source file size is 1440x1080 16:9, the texture size becomes 1440x1080 in size, not 1920x1080. - The aspect ratio can be retrieved in VideoFormat. Please adjust the ratio to be displayed properly by using any method based on the retrieved parameters.
* The aspect ratio which is retrieved in VideoFormat is pixel based aspect ratio.
AudioClip
Retrieves AudioClip of the audio source. This can be used when you want to get the audio samples and process them separately.
Method
OpenAsync
Opens the source file. This is asynchronous behavior. The result is notified asynchronously to the callback which is determined in the OnOpenSource field.
- sourcePath: Source file path. Specify the full path of a source file.
- requestVideo: Sets whether to request the video source. If you want to open only an audio stream in a file which also contains a video stream, please set to false.
- requestAudio: Sets whether to request the audio source. If you want to open only a video stream in a source file which also contains an audio stream, please set to false.
CloseAsync
Closes the source file. This is asynchronous behavior. The result is notified asynchronously to the callback which is determined in the OnPreviewCloseSource field prior to actual closing. After closing, the result is also notified asynchronously to the callback which is determined in the OnCloseSoure field.
SeekAsync
Moves the current playing position to the specified position (time). This is asynchronous behavior.
- pos: Destination time
PlayAsync
Starts playing. This is asynchronous behavior.
Pause
Pauses playing.
Low Level API
Classes that do not inherit MonoBehaviour. Normally, please use High Level API.
Pegasys.AVReader.AVReaderPluginSetting
This class sets the settings in connection with the entire plugin. You can process the actual access from the references which can be retrieved from Default properties.
Field / Property
IsAvailablePlugin
Retrieves whether the plugin is valid. If it is invalid, it may be caused by the following conditions.
- The initialization is not processed. AVReaderPluginManager does not exist, etc.
- Graphics API in Unity is not set to Direct3D 11.
- System does not meet the requirements.
AllowHardwareDecoder
Specifies whether to allow the use of the hardware decoders.
- When true is set, uses the hardware decoder functions if possible.
- When false is set, always disables the hardware acceleration and uses software decoders forcibly.
AllowDirect3D11Video
Specifies whether to allow the use of Direct3D 11 Video API as the video API.
- When true is set, uses the Direct3D 11 Video API if possible.
- When false is set, always uses Direct3D 9 Video API.
Since Direct3D 11 Video API cannot be used in Windows 7 environments, Direct3D 9 Video is always used.
AudioBufferLengthSec
Sets the audio buffer length in seconds.
This is the length of buffer which maintains decoded audio samples in the reader. In other words, it is the size which can maintain decoded samples without any processing.
Usually you don't have to change the value when using AVPlayer.
LogType
Sets the options to export logs to Unity Console.
You can turn on and off the options for the plural items by bit flag.
Please set Error, Warning, Trace_Initialize in most cases.
Since Trace_GetSample and Trace_Render are called at high frequency, the performance of Unity itself may become slow.
- Warning: Exports the warnings. The errors that occur while AVReader is working are exported as warnings.
- Trace_Initialize: Exports the trace log at initializing and releasing.
- Trace_GetSample: Exports the trace log of processing which is related to getting samples while AVReader is working.
- Trace_Render: Exports the trace log of processing which is related to rendering video while AVReader is working.
Default
Retrieves the instance of AVReaderPluginSetting.
Pegasys.AVReader.AVReader
This class is to read a source from a source file and decode it. This is the direct wrapper of the native plugin.
Since all methods of this are synchronization call, it requires an alternative action to desynchronize them. AVPlayer works in asynchronous mode, so please use AVPlayer normally.
Field / Property
IsOpened
Retrieves whether a source file is being opened.
Duration
Retrieves the length of the source.
HasVideo
Retrieves video source existence. If a file is opened in requestVideo = false, this returns false even if a video source actually exists in the file.
HasAudio
Retrieves audio source existence. If a file is opened in requestAudio = false, this returns false even if an audio source actually exists in the file.
VideoFormat
Retrieves the video format.
AudioFormat
Retrieves the audio format.
CachedVideoSampleCount
The number of video samples which are currently being decoded and cached.
CachedAudioSampleCount
The number of audio samples which are currently being decoded and cached.
Open
Opens the source file. This can be called from outside of the main thread.
- sourcePath: Source file path. Specify the full path of a source file.
- requestVideo: Sets whether to request the video source. If you want to open only an audio stream in a source file which also contains a video stream, please set to false.
- requestAudio: Sets whether to request the audio source. If you want to open only a video stream in a source file which also contains an audio stream, please set to false.
- Return value: The results of the operation.
Close
Closes the source file.
GetNativeRenderTargetTexture
Retrieves Texture of the native plugin.
Generates Texture internally at first call after Open.
Calling from outside of the main thread may make it invalid in status. Please use CreateRenderTargetTexture in most cases.
CreateRenderTargetTexture
Generates Texture for rendering video.
This can be processed only when the video source exists after Open.
Seek
Seeks the specified time and gets the samples from the position.
All existing caches are cleared. This can be called from outside of the main thread.
SetRenderVideoPosition
Specifies the time to render the video samples.
The video which corresponds with the specified time is reflected to Texture by the RenderVideo method. This can be called from outside of the main thread.
NextVideoSample
Decodes and retrieves the video samples from the source file, and stores the cache data.
Nothing is processed if the cache size already reaches the maximum value.
When the return is true, NextVideoSample can still be processed because it is not the end of the source. This can be called from outside of the main thread.
NextAudioSample
Decodes and retrieves the video samples from the source file, and stores the cache data.
There is a possibility that the cache data will overflow. To reduce overflow, it requires reading the data accordingly from the cache through ReadAudioSample.
When the return is true, NextAudioSample still can be processed because it is not the end of the source. When the return is false, it means the current position is the end of the source, or the cache buffer overflows. This can be called from outside of the main thread.
RenderVideo
Refreshes video which corresponds with the specified time in SetRenderVideoPosition on Texture. Also, this discards samples which are never referred to from cache such as samples located prior to the specified time.
ReadAudioSample
Reads audio samples from the cache.
The samples are read from the beginning of the cache, and the already read samples are deleted from the cache. This can be called from outside of the main thread.
- dst: The array to store the data.
- offsetSampleCount: Specifies the offset of the read start position of the dst array.
- readSampleCount: The number of samples to read.
Copyright information
Copyright (c) Pegasys Inc, All rights reserved.
The sample movie contained within the SDK is from the Elephants Dream licensed under the Creative Commons Attribution 2.5. Copyright 2006, Blender Foundation / Netherlands Media Art Institute / www.elephantsdream.org