android - Handle MediaCodec video with dropped frames

Question

I'm currently doing fast precise seeking using MediaCodec. What I currently do to skip frame by frame is, I first get the total frames:

mediaInfo.totalFrames = videoTrack.getSamples().size();

Then I get the length of the video file:

mediaInfo.durationUs = videoTrack.getDuration() * 1000 *1000 / timeScale;

//then calling:
public long getDuration() {
    if (mMediaInfo != null) {
        return (int) mMediaInfo.durationUs / 1000; // to millisecond
    }
    return -1;
}

Now, when I want to get the next frame I call the following:

mNextFrame.setOnClickListener(new View.OnClickListener() {

        @Override
        public void onClick(View view) {
            int frames = Integer.parseInt(String.valueOf(getTotalFrames));
            int intervals = Integer.parseInt(String.valueOf(mPlayer.getDuration() / frames));

            if (mPlayer.isPlaying()) {
                mPlayer.pause();
                mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);
            } else {
                mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);
            }
           
        }
    });

Here is the info about the file I'm testing with:

Frames = 466 Duration = 15523

So the interval between frames are

33,311158798283262

In other words, each time I press the next button the intervals will be rounded to 33, when I press the next button it will call mPlayer.seekTo(mPlayer.getCurrentPosition() + 33 meaning that some frames will be lost, or that is what I thought. I tested and got the following back when logging getCurrentPosition after each time the button is pressed and here is the result:

33 -> 66 -> 99 -> 132 -> 166

Going from 132 to 166 is 34ms instead of 33, so there was a compensation to make up with the frames that would have be lost.

---

The above works perfectly fine, I can skip through frames without any problem, here is the issue I facing.

Taking the same logic I used above I created a custom RangeBar. I created a method setTickCount (it's basically the same as seekbar.setMax) and I set the "TickCount" like this:

int frames = Integer.parseInt(String.valueOf(getTotalFrames));
mrange_bar.setTickCount(frames);

So the max value of my RangeBar is the amout of frames in the video.

When the "Tick" value changes I call the following:

int frames = Integer.parseInt(String.valueOf(getTotalFrames));
int intervals = Integer.parseInt(String.valueOf(mPlayer.getDuration() / frames));
mPlayer.seekTo(intervals * TickPosition);

So the above will work like this, if my tickCount position is, let's say 40:

mPlayer.seekTo(33 * 40); //1320ms

I would think that the above would work fine because I used the exact same logic, but instead the video "jump/skip" back to (what I assume is the key frame) and the continues the seeking.

Why is happening and how I can resolve this issue?

---

EDIT 1:

I mentioned above that it is jumping to the previous key frame, but I had a look again and it is calling end of stream while seeking (at spesific points during the video). When I reach end of stream I release my previous buffer so that one frame can still be displayed to avoid a black screen, by calling:

mDecoder.releaseOutputBuffer(prevBufferIndex, true);

So, for some reason, end of stream is called, where I then restart mediacodec causing a "lag/jump" effect. If I remove the above, I don't get the frame "jump", but there is still a lag while mediacodec is being initialized.

---

EDIT 2:

After digging deeper I found that readSampleData is -1:

ByteBuffer[] inputBuffers = mDecoder.getInputBuffers();
    int inIndex = mDecoder.dequeueInputBuffer(TIMEOUT_USEC);
    if (inIndex >= 0) {
        ByteBuffer buffer = inputBuffers[inIndex];
        int sampleSize = mExtractor.readSampleData(buffer, 0);
        if (sampleSize < 0) {
            mDecoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
            mIsExtractorReachedEOS = true;
        } else {
            mDecoder.queueInputBuffer(inIndex, 0, sampleSize, mExtractor.getSampleTime(), 0);
            mExtractor.advance();
        }
    }

For some reason my sampleSize is -1 at a specific point during seeking.

---

EDIT 3

This issue is definitely regarding the time that I pass, I tried 2 different approaches, the first:

mPlayer.seekTo(progress);

//position is retrieved by setting mSeekBar.setMax(mPlayer.getDuration); ....

and the second approach, I determine the frame intervals:

//Total amount of frames in video
long TotalFramesInVideo = videoTrack.getSamples().size();
//Duration of file in milliseconds
int DurationOfVideoInMs = mPlayer.getDuration();

//Determine interval between frames
int frameIntervals = DurationOfVideoInMs / Integer.parseInt(String.valueOf(TotalFramesInVideo));

//Then I seek to the frames like this:
mPlayer.seekTo(position * frameIntervals);

After trying both the above methods, I realised that the issue is related to the time being passed to mediaCodec because the "lag/jump" happens at different places.

I'm not sure why this doesn't happen when I call:

mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);

Answer 1

Waitting for answers