Top Description Fields Constructors Methods
jogamp.opengl.util.av.impl

public Class FFMPEGMediaPlayer

extends GLMediaPlayerImpl
Class Inheritance
Imports
java.io.IOException, java.nio.ByteBuffer, java.security.AccessController, .PrivilegedAction, com.jogamp.opengl.GL, .GL2ES2, .GLException, com.jogamp.common.util.IOUtil, .VersionNumber, com.jogamp.gluegen.runtime.ProcAddressTable, com.jogamp.opengl.util.TimeFrameI, .GLPixelStorageModes, com.jogamp.opengl.util.av.AudioSink, .AudioSink.AudioFormat, .AudioSinkFactory, .GLMediaPlayer, com.jogamp.opengl.util.texture.Texture, jogamp.common.os.PlatformPropsImpl, jogamp.opengl.GLContextImpl, jogamp.opengl.util.av.AudioSampleFormat, .GLMediaPlayerImpl, .VideoPixelFormat

Implementation utilizes Libav or FFmpeg which are ubiquitous available and usually pre-installed on Unix platforms.

Due to legal reasons we cannot deploy binaries of it, which contains patented codecs.

Besides the default BSD/Linux/.. repositories and installations, precompiled binaries can be found at the listed location below.

Implementation specifics

The decoded video frame is written directly into an OpenGL texture on the GPU in it's native format. A custom fragment shader converts the native pixelformat to a usable RGB format if required. Hence only 1 copy is required before bloating the picture from YUV* to RGB, for example.

Implements pixel format conversion to RGB via fragment shader texture-lookup functions:

Libav Specifics

Utilizes a slim dynamic and native binding to the Lib_av libraries:

LibAV Compatibility

Currently we are binary compatible w/:

libav / ffmpeglavclavflavulavr/lswr FFMPEG* class
0.8 53 53 51 FFMPEGv08
9.0 / 1.2 54 54 52 01/00 FFMPEGv09
10 / 2.[0-3] 55 55 53/52 01/00 FFMPEGv10
11 / 2.[4-8] 56 56 54 02/01 FFMPEGv11
12 / 2.[9-x] 57 57 55 02/01 TODO

See FFmpeg:

See libav:

Check tag 'FIXME: Add more planar formats !' here and in the corresponding native code jogl/src/jogl/native/libav/ffmpeg_impl_template.c

TODO:

FFMPEG / LibAV Availability

Field Summary

Modifier and TypeField and Description
private int
private static final boolean
private AudioSink.AudioFormat
private static final int
private static final int
private static final int
private static final int
public static final String
private static final boolean
private static final boolean
private static final int
ENOSYS

POSIX ENOSYS : Function not implemented.

private long
private static final FFMPEGNatives
private final GLPixelStorageModes
private String
private static final int
private int
private String
private int
private boolean
private int
private int
private VideoPixelFormat
private int
Inherited from jogamp.opengl.util.av.GLMediaPlayerImpl:
audioSinkaudioSinkPlaySpeedSetcameraPathcameraProps

Constructor Summary

AccessConstructor and Description
public

Method Summary

Modifier and TypeMethod and Description
private final AudioSink.AudioFormat
avAudioFormat2Local(final AudioSampleFormat
ffmpeg/libav audio-sample-format, see AudioSampleFormat.
audioSampleFmt
,
final int
sample rate in Hz (1/s)
audioSampleRate
,
final int
number of channels
audioChannels
)

Returns AudioFormat as converted from the given libav/ffmpeg values.

protected final TextureSequence.TextureFrame
createTexImage(final GL gl, final int texName)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.createTexImage.

private final void
protected final void
protected final int
getNextTextureImpl(final GL
valid and current GL instance, shall be null for audio only.
gl
,
final TextureSequence.TextureFrame
the TextureFrame to store the video PTS and texture data, shall be null for audio only.
nextFrame
)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.getNextTextureImpl.

Process stream until the next video frame, i.e. TextureFrame, has been reached.
public final String
getTextureLookupFragmentShaderImpl()

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFragmentShaderImpl.

Implements com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFragmentShaderImpl.

Returns the complete texture2D lookup function code of type
  vec4 funcName(in getTextureSampler2DType() image, in vec2 texCoord) {
     vec4 texColor = do_something_with(image, texCoord);
     return texColor;
  }
public final String
getTextureLookupFunctionName(final String
desired lookup function name. If null or ignored by the implementation, a build-in name is returned.
desiredFuncName
)

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFunctionName.

Implements com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFunctionName.

This implementation simply returns the build-in function name of texture2D, if not overridden by specialization.
protected final void
initGLImpl(final GL
null for audio-only, otherwise a valid and current GL object.
gl
)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.initGLImpl.

Shall initialize all GL related resources, if not audio-only.
protected final void
initStreamImpl(final int vid, final int aid)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.initStreamImpl.

Implementation shall set the following set of data here
pack-priv final boolean
isAudioFormatSupported(final int
ffmpeg/libav audio-sample-format, see AudioSampleFormat.
audioSampleFmt
,
final int
sample rate in Hz (1/s)
audioSampleRate
,
final int
number of channels
audioChannels
)

Native callback Converts the given libav/ffmpeg values to AudioFormat and returns AudioSink#isSupported(AudioFormat).

public static final boolean
public final boolean
public final boolean
protected void
protected void
pack-priv final void
pushSound(final ByteBuffer sampleData, final int data_size, final int audio_pts)

protected final synchronized int
seekImpl(final int msec)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.seekImpl.

pack-priv void
setupFFAttributes(final int vid, final int pixFmt, final int planes, final int bitsPerPixel, final int bytesPerPixelPerPlane, final int tWd0, final int tWd1, final int tWd2, final int vW, final int vH, final int aid, final int audioSampleFmt, final int audioSampleRate, final int audioChannels, final int
in audio samples per frame and channel
audioSamplesPerFrameAndChannel
)

Native callback

pack-priv void
updateVidAttributes(final boolean isInGLOrientation, final int pixFmt, final int planes, final int bitsPerPixel, final int bytesPerPixelPerPlane, final int tWd0, final int tWd1, final int tWd2, final int vW, final int vH)

Native callback

Inherited from jogamp.opengl.util.av.GLMediaPlayerImpl:
addEventListeneraddStateEventMaskattachObjectattributesUpdatedchangeStatecreateTexFramescreateTexImageImpldestroydestroyTexFramedetachObjectgetAIDgetAttachedObjectgetAudioBitrategetAudioCodecgetAudioFramesgetAudioPTSgetAudioPTSImplgetAudioSinkgetAudioVolumegetAudioVolumeImplgetDecodedFrameCountgetDurationgetEventListenersgetFramerategetHeightgetLastTexturegetNextSingleThreadedgetNextTexturegetPerfStringgetPlaySpeedgetPresentedFrameCountgetPropIntValgetRequiredExtensionsShaderStubgetStategetStreamBitrategetStreamExceptiongetTextureCountgetTextureFormatgetTextureFragmentShaderHashCodegetTextureMinMagFiltergetTextureSampler2DTypegetTextureTargetgetTextureTypegetTextureUnitgetTextureWrapSTgetUrigetVIDgetVideoBitrategetVideoCodecgetVideoFramesgetVideoPTSgetWidthinitGLinitStreamisGLOrientedisTextureAvailablepauseplayremoveEventListenerseeksetAudioVolumesetAudioVolumeImplsetFirstAudioPTS2SCRsetIsGLOrientedsetPlaySpeedsetPlaySpeedImplsetStatesetTextureFormatsetTextureMinMagFiltersetTextureTargetsetTextureTypesetTextureUnitsetTextureWrapSTtoHexStringtoHexStringtoStringupdateAttributesvalidateTextureCount

Field Detail

audioSamplesPerFrameAndChannelback to summary
private int audioSamplesPerFrameAndChannel
availableback to summary
private static final boolean available
avChosenAudioFormatback to summary
private AudioSink.AudioFormat avChosenAudioFormat
avCodecMajorVersionCCback to summary
private static final int avCodecMajorVersionCC
avFormatMajorVersionCCback to summary
private static final int avFormatMajorVersionCC
avResampleMajorVersionCCback to summary
private static final int avResampleMajorVersionCC
avUtilMajorVersionCCback to summary
private static final int avUtilMajorVersionCC
dev_video_linuxback to summary
public static final String dev_video_linux
enableAvResampleback to summary
private static final boolean enableAvResample
enableSwResampleback to summary
private static final boolean enableSwResample
ENOSYSback to summary
private static final int ENOSYS

POSIX ENOSYS : Function not implemented. FIXME: Move to GlueGen ?!

moviePtrback to summary
private long moviePtr
nativesback to summary
private static final FFMPEGNatives natives
psmback to summary
private final GLPixelStorageModes psm
singleTexCompback to summary
private String singleTexComp
swResampleMajorVersionCCback to summary
private static final int swResampleMajorVersionCC
texHeightback to summary
private int texHeight
texLookupFuncNameback to summary
private String texLookupFuncName
texWidthback to summary
private int texWidth
usesTexLookupShaderback to summary
private boolean usesTexLookupShader
vBitsPerPixelback to summary
private int vBitsPerPixel
vBytesPerPixelPerPlaneback to summary
private int vBytesPerPixelPerPlane
vPixelFmtback to summary
private VideoPixelFormat vPixelFmt
vPlanesback to summary
private int vPlanes

Constructor Detail

FFMPEGMediaPlayerback to summary
public FFMPEGMediaPlayer()

Method Detail

avAudioFormat2Localback to summary
private final AudioSink.AudioFormat avAudioFormat2Local(final AudioSampleFormat audioSampleFmt, final int audioSampleRate, final int audioChannels)

Returns AudioFormat as converted from the given libav/ffmpeg values.

Parameters
audioSampleFmt:AudioSampleFormat

ffmpeg/libav audio-sample-format, see AudioSampleFormat.

audioSampleRate:int

sample rate in Hz (1/s)

audioChannels:int

number of channels

createTexImageback to summary
protected final TextureSequence.TextureFrame createTexImage(final GL gl, final int texName)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.createTexImage.

Annotations
@Override
destroyAudioSinkback to summary
private final void destroyAudioSink()
destroyImplback to summary
protected final void destroyImpl(final GL gl)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.destroyImpl.

Annotations
@Override
getNextTextureImplback to summary
protected final int getNextTextureImpl(final GL gl, final TextureSequence.TextureFrame nextFrame)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.getNextTextureImpl.

Doc from jogamp.opengl.util.av.GLMediaPlayerImpl.getNextTextureImpl.

Process stream until the next video frame, i.e. TextureFrame, has been reached. Audio frames, i.e. AudioSink.AudioFrame, shall be handled in the process.

Video frames shall be ignored, if getVID() is STREAM_ID_NONE.

Audio frames shall be ignored, if getAID() is STREAM_ID_NONE.

Method may be invoked on the StreamWorker decoding thread.

Implementation shall care of OpenGL synchronization as required, e.g. glFinish()/glFlush()!

Parameters
gl:GL

valid and current GL instance, shall be null for audio only.

nextFrame:TextureSequence.TextureFrame

the TextureFrame to store the video PTS and texture data, shall be null for audio only.

Returns:int

the last processed video PTS value, maybe TimeFrameI#INVALID_PTS if video frame is invalid or n/a. Will be TimeFrameI#END_OF_STREAM_PTS if end of stream reached.

Annotations
@Override
getTextureLookupFragmentShaderImplback to summary
public final String getTextureLookupFragmentShaderImpl() throws IllegalStateException

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFragmentShaderImpl.

Implements com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFragmentShaderImpl.

Doc from jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFragmentShaderImpl.

Returns the complete texture2D lookup function code of type

  vec4 funcName(in getTextureSampler2DType() image, in vec2 texCoord) {
     vec4 texColor = do_something_with(image, texCoord);
     return texColor;
  }

funcName can be negotiated and queried via getTextureLookupFunctionName(String).

Note: This function may return an empty string in case a build-in lookup function is being chosen. If the implementation desires so, getTextureLookupFunctionName(String) will ignore the desired function name and returns the build-in lookup function name.

This implementation simply returns an empty string since it's using the build-in function texture2D, if not overridden by specialization. Depending on the pixelformat, a specific conversion shader is being created, e.g. YUV420P to RGB. Otherwise the call is delegated to it's super class.
Annotations
@Override
Exceptions
IllegalStateException:

Doc from com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFragmentShaderImpl.

if instance is not initialized

getTextureLookupFunctionNameback to summary
public final String getTextureLookupFunctionName(final String desiredFuncName) throws IllegalStateException

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFunctionName.

Implements com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFunctionName.

Doc from jogamp.opengl.util.av.GLMediaPlayerImpl.getTextureLookupFunctionName.

This implementation simply returns the build-in function name of texture2D, if not overridden by specialization. If this implementation generates a specialized shader, it allows the user to override the default function name ffmpegTexture2D. Otherwise the call is delegated to it's super class.

Parameters
desiredFuncName:String

Doc from com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFunctionName.

desired lookup function name. If null or ignored by the implementation, a build-in name is returned.

Returns:String

Doc from com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFunctionName.

the final lookup function name

Annotations
@Override
Exceptions
IllegalStateException:

Doc from com.jogamp.opengl.util.texture.TextureSequence.getTextureLookupFunctionName.

if instance is not initialized

initGLImplback to summary
protected final void initGLImpl(final GL gl) throws IOException, GLException

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.initGLImpl.

Doc from jogamp.opengl.util.av.GLMediaPlayerImpl.initGLImpl.

Shall initialize all GL related resources, if not audio-only.

Shall also take care of AudioSink initialization if appropriate.

Parameters
gl:GL

null for audio-only, otherwise a valid and current GL object.

Annotations
@Override
initStreamImplback to summary
protected final void initStreamImpl(final int vid, final int aid) throws IOException

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.initStreamImpl.

Doc from jogamp.opengl.util.av.GLMediaPlayerImpl.initStreamImpl.

Implementation shall set the following set of data here

Annotations
@Override
isAudioFormatSupportedback to summary
pack-priv final boolean isAudioFormatSupported(final int audioSampleFmt, final int audioSampleRate, final int audioChannels)

Native callback Converts the given libav/ffmpeg values to AudioFormat and returns AudioSink#isSupported(AudioFormat).

Parameters
audioSampleFmt:int

ffmpeg/libav audio-sample-format, see AudioSampleFormat.

audioSampleRate:int

sample rate in Hz (1/s)

audioChannels:int

number of channels

isAvailableback to summary
public static final boolean isAvailable()
pauseImplback to summary
public final boolean pauseImpl()

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.pauseImpl.

Annotations
@Override
playImplback to summary
public final boolean playImpl()

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.playImpl.

Annotations
@Override
postNextTextureImplback to summary
protected void postNextTextureImpl(final GL gl)

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.postNextTextureImpl.

Annotations
@Override
preNextTextureImplback to summary
protected void preNextTextureImpl(final GL gl)

Overrides jogamp.opengl.util.av.GLMediaPlayerImpl.preNextTextureImpl.

Annotations
@Override
pushSoundback to summary
pack-priv final void pushSound(final ByteBuffer sampleData, final int data_size, final int audio_pts)
seekImplback to summary
protected final synchronized int seekImpl(final int msec)

Implements abstract jogamp.opengl.util.av.GLMediaPlayerImpl.seekImpl.

Annotations
@Override
setupFFAttributesback to summary
pack-priv void setupFFAttributes(final int vid, final int pixFmt, final int planes, final int bitsPerPixel, final int bytesPerPixelPerPlane, final int tWd0, final int tWd1, final int tWd2, final int vW, final int vH, final int aid, final int audioSampleFmt, final int audioSampleRate, final int audioChannels, final int audioSamplesPerFrameAndChannel)

Native callback

Parameters
audioSamplesPerFrameAndChannel:int

in audio samples per frame and channel

updateVidAttributesback to summary
pack-priv void updateVidAttributes(final boolean isInGLOrientation, final int pixFmt, final int planes, final int bitsPerPixel, final int bytesPerPixelPerPlane, final int tWd0, final int tWd1, final int tWd2, final int vW, final int vH)

Native callback