JOGL 2.3.2 · com.jogamp.opengl.util.av.GLMediaPlayer

GLMediaPlayer interface specifies a TextureSequence state machine using a multiplexed audio/video stream as it's source.

Audio maybe supported and played back internally or via an AudioSink implementation.

Audio and video streams can be selected or muted via initStream(Uri, int, int, int) using the appropriate stream id's.

Camera input can be selected using the CameraInputScheme Uri.

StreamWorker Decoding Thread

Most of the stream processing is performed on the decoding thread, a.k.a. StreamWorker:

Stream initialization triggered by initStream(..) - User gets notified whether the stream has been initialized or not via attributesChanges(..).
Stream decoding - User gets notified of a new frame via newFrameAvailable(...).
Caught exceptions on the decoding thread are delivered as StreamExceptions.

StreamWorker generates it's own GLContext, shared with the one passed to initGL(GL). The shared GLContext allows the decoding thread to push the video frame data directly into the designated TextureFrame, later returned via getNextTexture(GL) and used by the user.

StreamWorker Error Handling

Caught exceptions on StreamWorker are delivered as StreamExceptions, which either degrades the State to State#Uninitialized or State#Paused.

An occurring StreamException triggers a EVENT_CHANGE_ERR event, which can be listened to via GLMediaEventListener#attributesChanged(GLMediaPlayer, int, long).

An occurred StreamException can be read via getStreamException().

GLMediaPlayer Lifecycle

Action	`State` Before	`State` After	`Event`
`initStream(Uri, int, int, int)`	`Uninitialized`	`Initialized`¹, `Uninitialized`	`EVENT_CHANGE_INIT` or ( `EVENT_CHANGE_ERR` + `EVENT_CHANGE_UNINIT` )
`initGL(GL)`	`Initialized`	`Paused`, , `Uninitialized`	`EVENT_CHANGE_PAUSE` or ( `EVENT_CHANGE_ERR` + `EVENT_CHANGE_UNINIT` )
`play()`	`Paused`	`Playing`	`EVENT_CHANGE_PLAY`
`pause(boolean)`	`Playing`	`Paused`	`EVENT_CHANGE_PAUSE`
`seek(int)`	`Paused`, `Playing`	`Paused`, `Playing`	none
`getNextTexture(GL)`	`Paused`, `Playing`	`Paused`, `Playing`	none
`getLastTexture()`	`Paused`, `Playing`	`Paused`, `Playing`	none
`END_OF_STREAM`	`Playing`	`Paused`	`EVENT_CHANGE_EOS` + `EVENT_CHANGE_PAUSE`
`StreamException`	ANY	`Paused`, `Uninitialized`	`EVENT_CHANGE_ERR` + ( `EVENT_CHANGE_PAUSE` or `EVENT_CHANGE_UNINIT` )
`destroy(GL)`	ANY	`Uninitialized`	`EVENT_CHANGE_UNINIT`

Audio and video Stream IDs

value	request	get
`STREAM_ID_NONE`	mute	not available
`STREAM_ID_AUTO`	auto	unspecified
≥0	specific stream	specific stream

Current implementations (check each API doc link for details):

Implementations of this interface must implement:

   public static final boolean isAvailable();

to be properly considered by GLMediaPlayerFactory#create(ClassLoader, String) and GLMediaPlayerFactory#createDefault().

Timestamp Accuracy

Timestamp type and value range has been chosen to suit embedded CPUs and characteristics of audio and video streaming. See TimeFrameI.

Audio and video synchronization

The class follows a passive A/V synchronization pattern. Audio is being untouched, while getNextTexture(GL) delivers a new video frame only, if its timestamp is less than MAXIMUM_VIDEO_ASYNC ahead of time. If its timestamp is more than MAXIMUM_VIDEO_ASYNC ahead of time, the previous frame is returned. If its timestamp is more than MAXIMUM_VIDEO_ASYNC after time, the frame is dropped and the next frame is being fetched.

https://en.wikipedia.org/wiki/Audio_to_video_synchronization

  d_av = v_pts - a_pts;

Recommendation of audio/video pts time lead/lag at production:

Overall: +40ms and -60ms audio ahead video / audio after video
Each stage: +5ms and -15ms. audio ahead video / audio after video

Recommendation of av pts time lead/lag at presentation:

TV: +15ms and -45ms. audio ahead video / audio after video.
Film: +22ms and -22ms. audio ahead video / audio after video.

Test Streams