Julien's Blog

Migrating a local store to iCloud in iOS 6.1

2013-01-29T00:00:00+01:00

As discussed in a previous post there’s a bug on iOS 6.0 when migrating a persistent store from iCloud storage to local storage. iOS 6.1 seems to fix this, but introduces a new bug when migrating in the opposite direction, from local to iCloud. Apple notes this in the release notes:

Known Issues
Moving data from a local sandbox to iCloud using migratePersistentStore causes a crash. Instead, manually migrate the data store by iterating over the objects in the local data store file.

I outlined the basic procedure to perform such a manual migration previously, but I recommend you use Drew McCormack’s MCPersistentStoreMigrator class found in his iCloudCoreDataTester github repository¹. This class uses your model’s NSEntityDescriptions to implement generic methods to perform a manual migration.

Here’s an example on how you can use MCPersistentStoreMigrator:

/* Load the iCloud store, optionally migrate fallback store to iCloud */
- (BOOL)loadiCloudStoreMigrate:(BOOL)migrate error:(NSError * __autoreleasing *)error
{
    BOOL success = YES;
    NSError *localError = nil;
    NSFileManager *fm = [NSFileManager defaultManager];
    NSURL *iCloudDataURL = [fm URLForUbiquityContainerIdentifier:nil];
    iCloudDataURL = [iCloudDataURL URLByAppendingPathComponent:kTransactionLogsDirectory];
    NSDictionary *options = @{
        NSPersistentStoreUbiquitousContentNameKey : kiCloudContentName,
        NSPersistentStoreUbiquitousContentURLKey : iCloudDataURL,
        NSMigratePersistentStoresAutomaticallyOption: @YES,
        NSInferMappingModelAutomaticallyOption: @YES
    };

    if (migrate) {
        NSManagedObjectModel *model = [NSManagedObjectModel mergedModelFromBundles:nil];
        MCPersistentStoreMigrator *migrator = [[MCPersistentStoreMigrator alloc] initWithManagedObjectModel:model
                                                                                             sourceStoreURL:self.fallbackStoreURL
                                                                                        destinationStoreURL:self.iCloudStoreURL];
        migrator.destinationStoreOptions = options;
        [migrator beginMigration];
        NSError *migrateError;
        /* call migrateEntityWithName:batchSize:save:error for each of your entities
        if (![migrator migrateEntityWithName:@"SomeEntity" batchSize:500 save:YES error:&localError]) {
            NSLog(@"Error migrating SomeEntity entities to iCloud: %@", migrateError);
            success = NO;
        }
        [migrator endMigration];        
    }

    if (success) {
        [self.persistentStoreCoordinator lock];
        self.iCloudStore = [self.persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType
                                                                         configuration:nil
                                                                                   URL:self.iCloudStoreURL
                                                                               options:options
                                                                                 error:&localError];
        [self.persistentStoreCoordinator unlock];
        success = (self.iCloudStore != nil);
    }

    if (!success) {
        if (localError && (error != NULL)) {
            *error = localError;
        }        
    }
    return success;
}

You should also take a look at his series of blog posts about iCloud and Core Data from May/June 2012. ↩

Creating a Tilt-shift Effect with CoreImage

2012-12-18T00:00:00+01:00

Tilt-shift

A tilt-shift effect can be used to simulate a miniature scene:

QuartzComposer

Using QuartzComposer¹ we can build a filter chain to apply this effect and play around with it.

As described in the CoreImage Documentation the first step is to setup a blur filter. Since the blur filter makes the image slightly larger, we need to crop it back to the original size.

The gradient mask is built by combining two linear gradients and is then used to blend the blurred image with the original image. Note that we need to crop the gradients to the correct size since the output image from CIGradientFilter has infinite size.

Download the QuartzComposer composition

CoreImage

This same filter chain can be implemented using CoreImage:

- (CIImage *)outputImage
{
    CGRect cropRect = self.inputImage.extent;
    CGFloat height = cropRect.size.height;

    CIFilter *blur = [CIFilter filterWithName:@"CIGaussianBlur"
                                keysAndValues:@"inputImage", self.inputImage,
                                                @"inputRadius", @(self.inputRadius), nil];
    blur = [CIFilter filterWithName:@"CICrop"
                    keysAndValues:@"inputImage", blur.outputImage,
                                    @"inputRectangle", [CIVector vectorWithCGRect:cropRect], nil];

    CIFilter *topGradient = [CIFilter filterWithName:@"CILinearGradient"
                                    keysAndValues:@"inputPoint0", [CIVector vectorWithX:0 Y:(self.inputTop * height)],
                                                    @"inputColor0", [CIColor colorWithRed:0 green:1 blue:0 alpha:1],
                                                    @"inputPoint1", [CIVector vectorWithX:0 Y:(self.inputCenter * height)],
                                                    @"inputColor1", [CIColor colorWithRed:0 green:1 blue:0 alpha:0], nil];
    CIFilter *bottomGradient = [CIFilter filterWithName:@"CILinearGradient"
                                        keysAndValues:@"inputPoint0", [CIVector vectorWithX:0 Y:(self.inputBottom * height)],
                                                        @"inputColor0", [CIColor colorWithRed:0 green:1 blue:0 alpha:1],
                                                        @"inputPoint1", [CIVector vectorWithX:0 Y:(self.inputCenter * height)],
                                                        @"inputColor1", [CIColor colorWithRed:0 green:1 blue:0 alpha:0], nil];
    topGradient = [CIFilter filterWithName:@"CICrop"
                            keysAndValues:@"inputImage", topGradient.outputImage,
                                            @"inputRectangle", [CIVector vectorWithCGRect:cropRect], nil];
    bottomGradient = [CIFilter filterWithName:@"CICrop"
                                keysAndValues:@"inputImage", bottomGradient.outputImage,
                                                @"inputRectangle", [CIVector vectorWithCGRect:cropRect], nil];
    CIFilter *gradients = [CIFilter filterWithName:@"CIAdditionCompositing"
                                    keysAndValues:@"inputImage", topGradient.outputImage,
                                                    @"inputBackgroundImage", bottomGradient.outputImage, nil];

    CIFilter *tiltShift = [CIFilter filterWithName:@"CIBlendWithMask"
                                    keysAndValues:@"inputImage", blur.outputImage,
                                                    @"inputBackgroundImage", self.inputImage,
                                                    @"inputMaskImage", gradients.outputImage, nil];

    return tiltShift.outputImage;
}

The full source code for the TiltShiftFilter can be found at https://gist.github.com/4327330.

From here on we can easily modify this. Instead of having the gradients along the Y-axis we could pass in vectors to control the direction of the gradients and have vertical or diagonal fields in focus. Another option would be to use a CIRadialGradient instead of two linear gradients and have a circular field.

You can find QuartzComposer in the Graphics Tools for Xcode package on https://developer.apple.com/downloads ↩

Migrating an iCloud CoreData Store

2012-11-25T00:00:00+01:00

The Problem

In my previous post I mentioned some issues with Core Data iCloud storage. One issue I didn’t mention is that on iOS 6 migrating a persistent store from iCloud to local storage using the migratePersistentStore:toURL:options:withType:error: method is broken (rdar://12253540). Using the method logs the following error message:

2012-11-25 15:48:49.213 Migration[27675:1903] -[NSPersistentStoreCoordinator addPersistentStoreWithType:configuration:URL:options:error:]( 1055): CoreData: Ubiquity:  Error: A persistent store which has been previously added to a coordinator using the iCloud integration options must always be added to the coordinator with the options present in the options dictionary. If you wish to use the store without iCloud, migrate the data from the iCloud store file to a new store file in local storage. file://localhost/var/mobile/Applications/02963CFC-D52A-4005-9555-B5E2377B3ADB/Documents/local.sqlite
This will be a fatal error in a future release.

Worse, trying to save changes to disc will now result in an error:

Error Domain=NSCocoaErrorDomain Code=134030 "The operation couldn’t be completed. (Cocoa error 134030.)"

Migrating in the other direction from local storage to iCloud storage works fine.

What’s happening?

The error message gives us a hint about what’s going on: CoreData seems to think that the migrated store was setup with “iCloud integration options”. Inspecting the local store’s metadata reveals that there are indeed “ubiquity” options present:

NSDictionary *metadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType
    URL:localStoreURL error:&error];
NSLog(@"metadata: %@", metadata);

2012-11-25 15:48:49.225 Migration[27675:1903] metadata: {
    /* snip */
    "com.apple.coredata.ubiquity.baseline.timestamp" = 1353854739;
    "com.apple.coredata.ubiquity.token" = <0a7f4e58 4c5693df 2081ac0a c476bc0f 6398af80>;
    "com.apple.coredata.ubiquity.ubiquitized" = 1;
}

It seems that the migration method copies the iCloud store’s metadata over to the new local store, including the ubiquity options.

The Solution

There are two methods to work around this bug: you can fix the store’s metadata, or you can manually migrate your data between two CoreData stacks.

Fixing the Metadata

Fixing the metadata is pretty easy and works on iOS 6, but might fall in a gray area regarding use of unsupported API since the names of the metadata keys are an implementation detail and not publicly documented. If you use this method you should make sure to thoroughly test migration on any new iOS releases. Note that you must remove the store from the NSPersistentStoreCoordinator before modifying its metadata.

[self dropStores];
NSError *error;
BOOL success = NO;
NSMutableDictionary *metadata;
metadata = [[NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType
    URL:localStoreURL error:&error] mutableCopy];
if (metadata) {
    [metadata removeObjectForKey:@"com.apple.coredata.ubiquity.baseline.timestamp"];
    [metadata removeObjectForKey:@"com.apple.coredata.ubiquity.token"];
    [metadata removeObjectForKey:@"com.apple.coredata.ubiquity.ubiquitized"];
    if (![NSPersistentStoreCoordinator setMetadata:metadata forPersistentStoreOfType:NSSQLiteStoreType
            URL:localStoreURL error:&error]) {
        success = NO;
        DDLogError(@"Error setting metadata for migrated fallback store: %@", error);
    } else {
        success = [self loadLocalStore:&error];
    }
}

Manual Migration

Manually migrating the data requires a lot more work. You setup a second, entirely separate, CoreData stack for your local store. You then fetch the data from your iCloud stack and create a new entity in your local stack for each iCloud entity and copy all its properties. Something along the lines of:

NSManagedObjectContext *localContext = ...
NSManagedObjectContext *iCloudContext = ...
NSArray *fetchResults = [iCloudContext executeFetchRequest:request error:&error];

for (NSManagedObject *obj in fetchResults) {
    NSManagedObject *migratedObject = [NSEntityDescription insertNewObjectForEntityForName:@"MyModelClass"
        inManagedObjectContext:localContext]
    migratedObject.property1 = obj.property1;
    migratedObject.property2 = obj.property2;
    ...
}

This is fairly simple as long as the properties are discrete values, but gets a bit tricker for relationships. WWDC 2012 Session 227 explains the procedure in more detail.

While the manual migration is more complex, it does have some advantages. First, it’s unlikely to break because of changes done to fix this bug or changes to the metadata keys. More importantly, migratePersistentStore:toURL:options:withType:error performs the migration in one go. It has to load all the data into memory and copies it into the new store in one action. If your database is really large this might not be possible. The manual migration method can split the data fetching into batches and thus keep the maximum memory needed lower (see Session 227).

Problems with Core Data iCloud Storage

2012-11-24T00:00:00+01:00

What is iCloud?

Apple uses iCloud as an umbrella term for all their online services and APIs, so it’s easy to get confused about what iCloud actually is. There’s iCloud e-mail, which is just IMAP. Calendars and contacts are synchronized using CalDAV and CardDav. Notes use the IMAP server of your email account. All these are completely independent of the iCloud APIs available to iOS and OS X developers.

There are actually three different iCloud APIs developers can use:

Ubiquitous Key Value Store: NSUbiquitousKeyValueStore is a simple key-value store with an API modelled after NSUserDefaults. While it can sometimes be a bit slow to propagate across all your devices, this works pretty well. You use this to store discrete values, for example preferences or highscores or similar simple data.
Document storage: when using document storage your app has access to a special folder called ubiquity container. Files stored in this container are synchronized across your devices. It’s not quite as simple as using the normal filesystem since the files might change at any moment, or even get deleted from under you. UIDocument handles a lot of this file coordination for you, so you’ll almost always want to use that.
Core Data storage: this builds on top of document storage. Your SQLite database isn’t stored directly inside the ubiquity container¹, instead each time you save your NSManagedObjectContext transaction logs are generated that represent the changes and these logs are then stored in the ubiquity container. Other devices download and import these transaction logs, replicating all the changes.

Ubiquity Containers and Transaction Logs

Document storage and Core Data storage both use the ubiquity container, but the key-value store is completely independent of this.

You can inspect the ubiquity containers of the apps installed on your iOS or OS X devices by logging in to developer.icloud.com or by browsing to ~/Library/Mobile\ Documents on your Mac.

Drew McCormack does a great job explaining the format of the transaction logs in a blog post. Note that you should never touch these logs directly. Removing transaction logs manually will break synchronization, unless you delete all the transaction logs and delete the SQLite databases on each device, basically resetting all the data.

Problems with Core Data iCloud Storage

Core Data in iCloud has a number of problems and hard to deal with use cases. These problems mostly stem from the fact that the API is presented to developers as a black box: you set up your persistent store with ubiquity options and everything else happens automatically in the background. There’s no way to hook into the underlying machinery dealing with the transaction logs, so when something goes wrong there’s not much you can do in your application.

When the user resets the container of your app using the iCloud settings, you need to completely drop and rebuild your Core Data stack, but there is no built-in way to detect these resets. If you don’t detect and handle it properly, synchronization just stops working. Not good. The usual method to deal with detecting resets is to use the document storage API to store an additional “sentinel” file in your ubiquity container. Since the transaction logs and the sentinel file are all stored in the same ubiquity container, you know that if the sentinel file disappears the transaction logs must have been deleted too.

You can handle the user logging out of iCloud or logging in to a different account the same way since the ubiquity container is tied to the account. Alternatively, on iOS 6 you can register for the NSUbiquityIdentityDidChangeNotification notification to detect this. Note that this notification does not fire for data resets.

Another problem is that setting up your Core Data stack can sometimes be very slow when using iCloud. The call to addPersistentStoreWithType:configuration:URL:options:error: sometimes blocks for 5 - 30 seconds. Even if you load your Core Data stack in a background queue, you can’t access the stored data until this finishes. This makes for a horrible user experience. (Thread on devforums.apple.com)

Yet another problem is that on iOS the transaction logs are not downloaded unless your app is in the foreground. So not only is the first launch of your app sometimes slow because of the time it takes to setup the Core Data stack, new data is not immediately available, but slowly trickles in as the transaction logs are downloaded and imported. (Bugreport on openradar.appspot.com)

It would also be really helpful if there was a complete sample project by Apple. Session 227 from WWDC 2012 covers the basics, but the presented sample project is incomplete and doesn’t properly handle iCloud resets.

The Future

iOS 6 brought some improvements to Core Data iCloud storage and Apple developers have acknowledged that there are still problems. Hopefully by the time iOS 7 is released Core Data iCloud storage will be a rock solid API. It isn’t unusable right now, but be prepared to run into problems when using it in your app.

You can store the database inside the ubiquity container, but it must be inside a *.nosync folder. Apple recommends you do this so that the database is automatically removed when the user logs out of iCloud or logs in to a different account. ↩

Getting Location Data for an UIImage

2012-11-23T00:00:00+01:00

The images you get from the imagePickerController:didFinishPickingMediaWithInfo: UIImagePickerController delegate method do not contain location data. To access this metadata you have to go through the AssetsLibrary framework and the latitude and longitude can then be found under the {GPS} key of the metadata dictionary.

- (void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary *)info
{
    NSURL *url = info[@"UIImagePickerControllerReferenceURL"];
    if (url) {
        ALAssetsLibrary *lib = [[ALAssetsLibrary alloc] init];
        [lib assetForURL:url resultBlock:^(ALAsset *asset) {
            NSDictionary *metadata = asset.defaultRepresentation.metadata;
            if (metadata) {
                NSNumber *longitude = metadata[@"{GPS}"][@"Longitude"];
                NSNumber *latitude = metadata[@"{GPS}"][@"Latitude"];
                if (longitude && latitude) {
                    CLLocationCoordinate2D coord = CLLocationCoordinate2DMake([latitude doubleValue],
                        [longitude doubleValue]);
                    // do something with coordinate
                }
            }  
        } failureBlock:^(NSError *error) {
            //User denied access
            NSLog(@"Unable to access image: %@", error);
        }];
    }
}

The first time assetForUrl:resultBlock: is called the user will be asked to authorize the access.

Show/Hide Invisible Chars in Vim

2012-11-22T00:00:00+01:00

listchars lets you define replacements for unprintable characters to display in list mode. List mode can then be toggled on and off using :set list!

set listchars=tab:▸,eol:¬
nnoremap <leader>i :set list!<CR>