Dealing with Google Takeout in Mac OS

Overview

After July 2019, Google Photos will no longer sync with Google Drive. Whether or not this is a good thing depends on specific use cases. Regardless, it is happening and this impacts the way that I use Photos. Generally when I take photos on my Pixel 3 phone, I just let the system handle things. The phone syncs with Google Photos in the cloud. Google Photos syncs with Google Drive and Apple’s Time Machine backs up my Google Drive onto a local disk drive. I have been comfortable with this. I am not a heavy curator of my photos. I have some albums in Google Photos but generally I want my photos organized by creation date. This is important because the files that Google syncs to my Google Drive do not include the album folders or meta data. I’m ok with this.

Some people have noted with dismay that deleting Photos on your phone can cause propagating deletes throughout the synchronized stores causing a cherished photo to disappear. If you do not back up the Google Drive like I do, you are out of luck. If you run out of space on your phone you may have no other option but to delete a bunch of photos. My pixel 3 has no SD card so I will hit this point some day.

After Google stops synchronization between Google Photos and Google Drive, the pictures I keep in Google Photos will become distinct from the ones I keep in Google Drive. The phone will continue to sync with Google Photos but new pictures will not make their way into Google Drive (and my Time Machine backup).

So how do I back up my collection of pictures? One way I have found is to use Google Takeout to download a set of zip files containing all of my pictures along with Google’s own metadata, fomatted as JSON files.

Goggle Takout was designed for the use case of someone who needs to get all of their stuff out of the Google Cloud. It’s generic enough to handle all of the data formats that Google has developed over the years. This includes Google Photos. While the original intent of the Takeout service may have been to leave the Google-sphere, the service only creates a copy of the data. It does not remove it. This makes it suitable for backup purposes.

The Takeout service is found by going to the My Account page and selecting Data and Personalization -> Download Your Data. After you marvel at all the data Google does keep, the easiest path is click the Deselect All option, then scroll down to Google Photos and select that. After that click the Next Step button and customize the download options.

When I did this I wound up with 10 zip files. Takeout limits each zip file to two GB and the files within each zip are selected based on what will fit within the limit rather than using a strictly logical organization. From there you are on your own.

Troubles Using Finder

At first I assumed that I simply needed to unzip the files and move them to a common folder. However the files are organized to fill up two GB files efficiently. This means many folders have been duplicated across the zip files. In other words the files in any particular folder may have been spread across several zip files. A given folder name may exist in multiple zip files each with a subset of the total set of files in that folder. Mac’s Finder does not deal with this in the way that I expected.

I assumed that I could drag and drop folders to a common destination folder and that a proper merge would occur. Instead Finder prompts me with a dialog that offers two options: Stop the move, or Replace the contents of the common destination folder with the contents of the source folder. Selecting Replace deletes the current contents before moving the new contents. This is not a merge.

After some internet searching I found a hidden option that performs a merge. Holding the Option key while performing the drag and drop adds a Merge option to the the dialog box that pops up when a name collision occurs. This sounds like what I want but it turns out that is is not my idea of a merge. This operation will select the newest version of an item and delete the older one. This includes folders not just files. Since the folder names are duplicated across these zip files, this so-called merge is no better than a strict replacment operation (other than the date selection). I still wind up with many files getting lost.

Using Ditto to Merge Google Takeout Zip Files

Back to the internet. I found out that Mac OS has a ditto command which performs a proper recursive merge of a file hierarchy. Furthermore, it can take multiple zip files as the source and let you skip the unpacking step.

I copied the zip files into a work folder and issued the following command:

ditto -x -k -V takeout-20190710T165610Z-001.zip \
	takeout-20190710T165610Z-002.zip \
	takeout-20190710T165610Z-003.zip \
	takeout-20190710T165610Z-004.zip \
	takeout-20190710T165610Z-005.zip \
	takeout-20190710T165610Z-006.zip \
	takeout-20190710T165610Z-007.zip \
	takeout-20190710T165610Z-008.zip \
	takeout-20190710T165610Z-009.zip \
	takeout-20190710T165610Z-010.zip \
	result 1>log.txt 2>&1

This created the target folder result and merged all of the files into their proper folders. No files were lost in this process. Now I copy these files to my local backup disk. I will have to repeat this periodically. Unfortunately there is no incremental takeout feature so I will have to repeat the full download. Then I can use the rsync command to update the backup. I’ll leave that for another day.

Leave a Reply

Your email address will not be published. Required fields are marked *