iphone - Using AVAssetReader to read (stream) from a remote asset

Question

My main goal is to stream a video from a server, and cut it frame by frame while streaming (so that it can be used by OpenGL). For that, I've used this code that I found everywhere on the Internet (as I recall it was from Apple's GLVideoFrame sample code):

NSArray * tracks = [asset tracks];
NSLog(@"%d", tracks.count);

for(AVAssetTrack* track in tracks) {

    NSLog(@"type: %@", [track mediaType]);

    initialFPS = track.nominalFrameRate;
    width = (GLuint)track.naturalSize.width;
    height = (GLuint)track.naturalSize.height;


    NSError * error = nil;

    // _movieReader is a member variable
    @try {
        self._movieReader = [[[AVAssetReader alloc] initWithAsset:asset error:&error] autorelease];
    }
    @catch (NSException *exception) {
        NSLog(@"%@ -- %@", [exception name], [exception reason]);
        NSLog(@"skipping track");

        continue;
    }


    if (error)
    {
        NSLog(@"CODE:%d
DOMAIN:%@
DESCRIPTION:%@
FAILURE_REASON:%@", [error code], [error domain], error.localizedDescription, [error localizedFailureReason]);                                          
        continue;
    }

    NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
    NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
    NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key]; 
    [_movieReader addOutput:[AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:track
                                                                       outputSettings:videoSettings]];
    [_movieReader startReading];
    [self performSelectorOnMainThread:@selector(frameStarter) withObject:nil waitUntilDone:NO];
}

But I always get this exception at [[AVAssetReader alloc] initWithAsset:error:].

NSInvalidArgumentException -- *** -[AVAssetReader initWithAsset:error:] Cannot initialize an instance of AVAssetReader with an asset at non-local URL 'http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8'

So my two questions are:

1. Is the exception really telling me that AVAssetReader must have a local URL? Can it be used for streaming (just like the rest of the AVFoundation classes)?
2. If the AVFoundation approach won't work, what are other suggestions to stream the video and split its frames at the same time?

Thanks a lot for your help.

Answer 1

AVFoundation does not seem to distinguish as much between local and non-local files, as it does between the KIND of files or protocols used. There is a VERY clear distinction between using mp4/mov's versus using the HTTP Live streaming protocol via m3u8's, but the differences using a local or remote mp4 are a little fuzzier.

To expand on the above:

a) If your 'remote' asset is an M3U8 (that is, you are using HTTP 'live' streaming), then no chance whatsoever. No matter if the M3U8 is in your local filesystem or on a remote server, for a multitude of reasons AVAssetReader and all AVAsset-associated functionality just does NOT work. However, AVPlayer, AVPlayerItem etc would work just fine.

b) If it is an MP4/MOV, a little further investigation is due. Local MP4/MOV's work flawlessly. While in case of remote MP4/MOV's, I'm able to create (or retrieve from an AVPlayerItem or AVPlayer or AVAssetTracks) an AVURLAsset with which I'm sometimes able to initialize an AVAssetReader successfully (I'll expand on the 'sometimes' as well, shortly). HOWEVER, copyNextSampleBuffer always returns nil in case of remote MP4's. Since several things UPTO the point of invoking copyNextSampleBuffer work, I'm not 100% sure if:

i) copyNextSampleBuffer not working for remote mp4's, after all the other steps having been successful, is intended/expected functionality.

ii) That the 'other steps' seem to work at all for remote MP4's is an accident of Apple's implementation, and this incompatibility is simply coming to the fore when we hit copyNextSampleBuffer..............what these 'other steps' are, I'll detail shortly.

iii) I'm doing something wrong when trying to invoke copyNextSampleBuffer for remote MP4's.

So @Paula you could try to investigate a little further with remote MOV/MP4's.

For reference, here are the approaches I tried for capturing a frame from videos:

a)

Create an AVURLAsset directly from the video URL.

Retrieve the video track using [asset tracksWithMediaType:AVMediaTypeVideo]

Prepare an AVAssetReaderTrackOutput using the video track as the source.

Create an AVAssetReader using the AVURLAsset.

Add AVAssetReaderTrackOutput to the AVAssetReader and startReading.

Retrieve images using copyNextSampleBuffer.

b)

Create an AVPlayerItem from the video URL, and then an AVPlayer from it (or create the AVPlayer directly from the URL).

Retrieve the AVPlayer's 'asset' property and load its 'tracks' using "loadValuesAsynchronouslyForKeys:".

Separate the tracks of type AVMediaTypeVideo (or simply call tracksWithMediaType: on the asset once the tracks are loaded), and create your AVAssetReaderTrackOutput using the video track.

Create AVAssetReader using the AVPlayer's 'asset', 'startReading' and then retrieve images using copyNextSampleBuffer.

c)

Create an AVPlayerItem+AVPlayer or AVPlayer directly from the video URL.

KVO the AVPlayerItem's 'tracks' property, and once the tracks are loaded, separate the AVAssetTracks of type AVMediaTypeVideo.

Retrieve the AVAsset from AVPlayerItem/AVPlayer/AVAssetTrack's 'asset' property.

Remaining steps are similar to approach (b).

d)

Create an AVPlayerItem+AVPlayer or AVPlayer directly from the video URL.

KVO the AVPlayerItem's 'tracks' property, and once the tracks are loaded, separate the ones of type AVMediaTypeVideo.

Create an AVMutableComposition, and initialize an associated AVMutableCompositionTrack of type AVMediaTypeVideo.

Insert the appropriate CMTimeRange from video track retrieved earlier, into this AVMutableCompositionTrack.

Similar to (b) and (c), now create your AVAssetReader and AVAssetReaderTrackOutput, but with the difference that you use the AVMutableComposition as the base AVAsset for initializing your AVAssetReader, and AVMutableCompositionTrack as the base AVAssetTrack for your AVAssetReaderTrackOutput.

'startReading' and use copyNextSampleBuffer to get frames from the AVAssetReader.

P.S: I tried approach (d) here to get around the fact that the AVAsset retrieved directly from AVPlayerItem or AVPlayer was not behaving. So I wanted to create a new AVAsset from the AVAssetTracks I already had in hand. Admittedly hacky, and perhaps pointless (where else would the track information be ultimately retrieved from if not the original AVAsset!) but it was worth a desperate try anyway.

Here's a summary of the results for different types of files:

1) Local MOV/MP4's - All 4 approaches work flawlessly.

2) Remote MOV/MP4's - The asset and tracks are retrieved correctly in approaches (b) through (d), and the AVAssetReader is initialized as well but copyNextSampleBuffer always returns nil. In case of (a), creation of the AVAssetReader itself fails with an 'Unknown Error' NSOSStatusErrorDomain -12407.

3) Local M3U8's (accessed through an in-app/local HTTP server) - Approaches (a), (b) and (c) fail miserably as trying to get an AVURLAsset/AVAsset in any shape or form for files streamed via M3U8's is a fools errand.

In case of (a), the asset is not created at all, and the initWithURL: call on AVURLAsset fails with an 'Unknown Error' AVFoundationErrorDomain -11800.

In case of (b) and (c), retrieving the AVURLAsset from the AVPlayer/AVPlayerItem or AVAssetTracks returns SOME object, but accessing the 'tracks' property on it always returns an empty array.

In case of (d), I'm able to retrieve and isolate the video tracks successfully, but while trying to create the AVMutableCompositionTrack, it fails when trying to insert the CMTimeRange from the source track into the AVMutableCompositionTrack, with an 'Unknown Error' NSOSStatusErrorDomain -12780.

4) Remote M3U8's, behave exactly the same as local M3U8's.

I'm not entirely educated on why these differences exist, or could not have been mitigated by Apple. But there you go.