Dealing with Google Takeout in Mac OS

Overview

After July 2019, Google Photos will no longer sync with Google Drive. Whether or not this is a good thing depends on specific use cases. Regardless, it is happening and this impacts the way that I use Photos. Generally when I take photos on my Pixel 3 phone, I just let the system handle things. The phone syncs with Google Photos in the cloud. Google Photos syncs with Google Drive and Apple’s Time Machine backs up my Google Drive onto a local disk drive. I have been comfortable with this. I am not a heavy curator of my photos. I have some albums in Google Photos but generally I want my photos organized by creation date. This is important because the files that Google syncs to my Google Drive do not include the album folders or meta data. I’m ok with this.

Some people have noted with dismay that deleting Photos on your phone can cause propagating deletes throughout the synchronized stores causing a cherished photo to disappear. If you do not back up the Google Drive like I do, you are out of luck. If you run out of space on your phone you may have no other option but to delete a bunch of photos. My pixel 3 has no SD card so I will hit this point some day.

After Google stops synchronization between Google Photos and Google Drive, the pictures I keep in Google Photos will become distinct from the ones I keep in Google Drive. The phone will continue to sync with Google Photos but new pictures will not make their way into Google Drive (and my Time Machine backup).

So how do I back up my collection of pictures? One way I have found is to use Google Takeout to download a set of zip files containing all of my pictures along with Google’s own metadata, fomatted as JSON files.

Goggle Takout was designed for the use case of someone who needs to get all of their stuff out of the Google Cloud. It’s generic enough to handle all of the data formats that Google has developed over the years. This includes Google Photos. While the original intent of the Takeout service may have been to leave the Google-sphere, the service only creates a copy of the data. It does not remove it. This makes it suitable for backup purposes.

The Takeout service is found by going to the My Account page and selecting Data and Personalization -> Download Your Data. After you marvel at all the data Google does keep, the easiest path is click the Deselect All option, then scroll down to Google Photos and select that. After that click the Next Step button and customize the download options.

When I did this I wound up with 10 zip files. Takeout limits each zip file to two GB and the files within each zip are selected based on what will fit within the limit rather than using a strictly logical organization. From there you are on your own.

Troubles Using Finder

At first I assumed that I simply needed to unzip the files and move them to a common folder. However the files are organized to fill up two GB files efficiently. This means many folders have been duplicated across the zip files. In other words the files in any particular folder may have been spread across several zip files. A given folder name may exist in multiple zip files each with a subset of the total set of files in that folder. Mac’s Finder does not deal with this in the way that I expected.

I assumed that I could drag and drop folders to a common destination folder and that a proper merge would occur. Instead Finder prompts me with a dialog that offers two options: Stop the move, or Replace the contents of the common destination folder with the contents of the source folder. Selecting Replace deletes the current contents before moving the new contents. This is not a merge.

After some internet searching I found a hidden option that performs a merge. Holding the Option key while performing the drag and drop adds a Merge option to the the dialog box that pops up when a name collision occurs. This sounds like what I want but it turns out that is is not my idea of a merge. This operation will select the newest version of an item and delete the older one. This includes folders not just files. Since the folder names are duplicated across these zip files, this so-called merge is no better than a strict replacment operation (other than the date selection). I still wind up with many files getting lost.

Using Ditto to Merge Google Takeout Zip Files

Back to the internet. I found out that Mac OS has a ditto command which performs a proper recursive merge of a file hierarchy. Furthermore, it can take multiple zip files as the source and let you skip the unpacking step.

I copied the zip files into a work folder and issued the following command:

ditto -x -k -V takeout-20190710T165610Z-001.zip \
	takeout-20190710T165610Z-002.zip \
	takeout-20190710T165610Z-003.zip \
	takeout-20190710T165610Z-004.zip \
	takeout-20190710T165610Z-005.zip \
	takeout-20190710T165610Z-006.zip \
	takeout-20190710T165610Z-007.zip \
	takeout-20190710T165610Z-008.zip \
	takeout-20190710T165610Z-009.zip \
	takeout-20190710T165610Z-010.zip \
	result 1>log.txt 2>&1

This created the target folder result and merged all of the files into their proper folders. No files were lost in this process. Now I copy these files to my local backup disk. I will have to repeat this periodically. Unfortunately there is no incremental takeout feature so I will have to repeat the full download. Then I can use the rsync command to update the backup. I’ll leave that for another day.

26 thoughts on “Dealing with Google Takeout in Mac OS

  1. Would this tool be a good process for using takeout to get a copy of all my photos from the google ecosphere organized properly for upload into the Apple ecosystem? I downloaded my take-outs and have (3) large zip folders but as you mentioned with the meta data etc it is very ill organized

    1. I have a Pixel phone and I have not used Apple’s cloud to manage my photos. So I don’t know what “organized properly” means in the Apple context. However, using the “ditto” command with Takeout files will organize the files in folders by date, or album name. All of the metadata is retained in JSON files in each folder. So if you have added descriptions and location information it will be in those JSON files. Potentially an importer program could be written to use this information to create albums in another system.

  2. I have downloaded the approx. 950GB that I have been storing in Google Drive (I had a rash of bad drive issues a year ago) in the form of multiple 50GB .tgz files. I don’t have enough space on my SSD to store and unpack these files. Can I run this operation from my external 4TB drive on which these files currently reside?

  3. questions from a Terminal beginner: I’m wanting to use this process for my music instead of photos. Google Play Music is transferring to YouTube music, and its an inferior app that doesn’t do what google did. I copied your commands, but got the error messages: Unknown compression type and Couldn’t read PKZip signature. Why? What should I be doing differently?

  4. I am not good with terminal coding. I don’t understand how to get this working, can you help with step by step instructions please?

  5. Wow! Thank you for this code and write-up. Saved me headache and time and trouble. I tried so many other things but this just worked.

  6. Hey, I usually never comment on stuff but this little command saved my friggin life! I downloaded about 800GB of photos from google takeout and getting all of that sorted was a logistical nightmare. I just used a wildcard in my command to make it a bit more concise. All of my takeout zip files were from google photos and moved into a working directory like was suggested.

    ditto -x -k -V takeout-*.zip result 1>log.txt 2>&1
    then in a separate console tab I ran
    tail -f log.txt
    so I could watch what was happening in real time.

    1. Hi Brian… or anyone avail to assist
      I’m so excited to know there could be “hope” for what a whole lot of people are going through.. having the exact same issue with the download of “photos” from Google Takeout to my MacBook Pro, now I have a lot of JSON files (photos) sitting in folders, and can’t open to view photos.
      Can you please help explain step by step on how to use the code given to open the JSON files on my Mac. I’m not a techy person, but I’m basically familiarized with my Mac.

      Thank you kindly!

    2. Hey Brian,
      Thank you, as well as Mike, for uploading that extra trick. I just did it longhand, but this is such a slick process for Google Takeout that I’m sure I’m going to be doing it again for other accounts, like my dad’s, or other family/friends. Using the wildcard will be very helpful 🙂

      –Jeff

  7. Hi Mike,
    I really would like some assistance with my photos that were downloaded from google photos takeout, & sitting in 33 folders on my MacOS with “jpg.json and mp4.json” that I can’t open.
    Can you please show us step by step (layman style) for us non-tech’s, how to: for MacOS.

    Thank you, I appreciate it!!

  8. Thank you so much! The smartest approach found so far on the web! I tested it and it works really nice!

    Now, is there any way to DELETE the zip files?

    This script both extracts and stores another copy of Zip files in the newly created “result” folder. I have lots of data to process and this will take up my space limit pretty quickly.

    The ideal would be:

    1) Delete the source file immediately after extraction
    AND
    2) Don’t copy the source file in the output folder

    Any ideas?

    Funny website name – CodingStick 🙂

  9. Mike,
    You are THE MAN! Thank you so much for taking the time to publish this so that others can benefit from your hard work. I just used your instructions to retrieve all of the music that I have uploaded over the years to Google Music, which is being shut down in favor of YouTube Music (which seems to be an inferior tool for what I need to do). Your process is churning away as I type this. The “result” directory is at about 5gb at the moment, and all of the files seem to be named properly and are appearing in Finder in the “tracks” subfolder, just like clockwork.

    I couldn’t be happier! Where can I send you a beer or other token that can be retrieved for the treat of your choice?

    –Jeff

  10. I look forward to your “rsync” for working with previous result from ditto.
    This is super useful blog, thank you for sharing 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *