Overview
After July 2019, Google Photos will no longer sync with Google Drive. Whether or not this is a good thing depends on specific use cases. Regardless, it is happening and this impacts the way that I use Photos. Generally when I take photos on my Pixel 3 phone, I just let the system handle things. The phone syncs with Google Photos in the cloud. Google Photos syncs with Google Drive and Apple’s Time Machine backs up my Google Drive onto a local disk drive. I have been comfortable with this. I am not a heavy curator of my photos. I have some albums in Google Photos but generally I want my photos organized by creation date. This is important because the files that Google syncs to my Google Drive do not include the album folders or meta data. I’m ok with this.
Some people have noted with dismay that deleting Photos on your phone can cause propagating deletes throughout the synchronized stores causing a cherished photo to disappear. If you do not back up the Google Drive like I do, you are out of luck. If you run out of space on your phone you may have no other option but to delete a bunch of photos. My pixel 3 has no SD card so I will hit this point some day.
After Google stops synchronization between Google Photos and Google Drive, the pictures I keep in Google Photos will become distinct from the ones I keep in Google Drive. The phone will continue to sync with Google Photos but new pictures will not make their way into Google Drive (and my Time Machine backup).
So how do I back up my collection of pictures? One way I have found is to use Google Takeout to download a set of zip files containing all of my pictures along with Google’s own metadata, fomatted as JSON files.
Goggle Takout was designed for the use case of someone who needs to get all of their stuff out of the Google Cloud. It’s generic enough to handle all of the data formats that Google has developed over the years. This includes Google Photos. While the original intent of the Takeout service may have been to leave the Google-sphere, the service only creates a copy of the data. It does not remove it. This makes it suitable for backup purposes.
The Takeout service is found by going to the My Account page and selecting Data and Personalization -> Download Your Data. After you marvel at all the data Google does keep, the easiest path is click the Deselect All option, then scroll down to Google Photos and select that. After that click the Next Step button and customize the download options.
When I did this I wound up with 10 zip files. Takeout limits each zip file to two GB and the files within each zip are selected based on what will fit within the limit rather than using a strictly logical organization. From there you are on your own.
Troubles Using Finder
At first I assumed that I simply needed to unzip the files and move them to a common folder. However the files are organized to fill up two GB files efficiently. This means many folders have been duplicated across the zip files. In other words the files in any particular folder may have been spread across several zip files. A given folder name may exist in multiple zip files each with a subset of the total set of files in that folder. Mac’s Finder does not deal with this in the way that I expected.
I assumed that I could drag and drop folders to a common destination folder and that a proper merge would occur. Instead Finder prompts me with a dialog that offers two options: Stop the move, or Replace the contents of the common destination folder with the contents of the source folder. Selecting Replace deletes the current contents before moving the new contents. This is not a merge.
After some internet searching I found a hidden option that performs a merge. Holding the Option key while performing the drag and drop adds a Merge option to the the dialog box that pops up when a name collision occurs. This sounds like what I want but it turns out that is is not my idea of a merge. This operation will select the newest version of an item and delete the older one. This includes folders not just files. Since the folder names are duplicated across these zip files, this so-called merge is no better than a strict replacment operation (other than the date selection). I still wind up with many files getting lost.
Using Ditto to Merge Google Takeout Zip Files
Back to the internet. I found out that Mac OS has a ditto command which performs a proper recursive merge of a file hierarchy. Furthermore, it can take multiple zip files as the source and let you skip the unpacking step.
I copied the zip files into a work folder and issued the following command:
ditto -x -k -V takeout-20190710T165610Z-001.zip \
takeout-20190710T165610Z-002.zip \
takeout-20190710T165610Z-003.zip \
takeout-20190710T165610Z-004.zip \
takeout-20190710T165610Z-005.zip \
takeout-20190710T165610Z-006.zip \
takeout-20190710T165610Z-007.zip \
takeout-20190710T165610Z-008.zip \
takeout-20190710T165610Z-009.zip \
takeout-20190710T165610Z-010.zip \
result 1>log.txt 2>&1
This created the target folder result and merged all of the files into their proper folders. No files were lost in this process. Now I copy these files to my local backup disk. I will have to repeat this periodically. Unfortunately there is no incremental takeout feature so I will have to repeat the full download. Then I can use the rsync command to update the backup. I’ll leave that for another day.
Many thanks, Mike. Worked like a charm and saved me a lot of time. Cheers.
Would this tool be a good process for using takeout to get a copy of all my photos from the google ecosphere organized properly for upload into the Apple ecosystem? I downloaded my take-outs and have (3) large zip folders but as you mentioned with the meta data etc it is very ill organized
I have a Pixel phone and I have not used Apple’s cloud to manage my photos. So I don’t know what “organized properly” means in the Apple context. However, using the “ditto” command with Takeout files will organize the files in folders by date, or album name. All of the metadata is retained in JSON files in each folder. So if you have added descriptions and location information it will be in those JSON files. Potentially an importer program could be written to use this information to create albums in another system.
You are my hero!!! I have needed this for three years.
I have downloaded the approx. 950GB that I have been storing in Google Drive (I had a rash of bad drive issues a year ago) in the form of multiple 50GB .tgz files. I don’t have enough space on my SSD to store and unpack these files. Can I run this operation from my external 4TB drive on which these files currently reside?
I did exactly this (only with a bunch of 2GB zip files) on my external 4TB drive and it worked fine.
This was very helpful. Thank you so much.
Thanks. A better way: ditto -x -k -V *.zip outputFolder
Thanks for this. It worked like a charm.
Thank you! It worked like a charm.
this worked for me (extracting on my external USB SSD) fwiw.
Thanks Mike! This helped me a lot with extracting my google music library! 🙂
questions from a Terminal beginner: I’m wanting to use this process for my music instead of photos. Google Play Music is transferring to YouTube music, and its an inferior app that doesn’t do what google did. I copied your commands, but got the error messages: Unknown compression type and Couldn’t read PKZip signature. Why? What should I be doing differently?
I am not good with terminal coding. I don’t understand how to get this working, can you help with step by step instructions please?
oh, thank. you.
Wow! Thank you for this code and write-up. Saved me headache and time and trouble. I tried so many other things but this just worked.
Hey, I usually never comment on stuff but this little command saved my friggin life! I downloaded about 800GB of photos from google takeout and getting all of that sorted was a logistical nightmare. I just used a wildcard in my command to make it a bit more concise. All of my takeout zip files were from google photos and moved into a working directory like was suggested.
ditto -x -k -V takeout-*.zip result 1>log.txt 2>&1
then in a separate console tab I ran
tail -f log.txt
so I could watch what was happening in real time.
That’s exactly. what I’m gonna do
Hi Brian… or anyone avail to assist
I’m so excited to know there could be “hope” for what a whole lot of people are going through.. having the exact same issue with the download of “photos” from Google Takeout to my MacBook Pro, now I have a lot of JSON files (photos) sitting in folders, and can’t open to view photos.
Can you please help explain step by step on how to use the code given to open the JSON files on my Mac. I’m not a techy person, but I’m basically familiarized with my Mac.
Thank you kindly!
Hey Brian,
Thank you, as well as Mike, for uploading that extra trick. I just did it longhand, but this is such a slick process for Google Takeout that I’m sure I’m going to be doing it again for other accounts, like my dad’s, or other family/friends. Using the wildcard will be very helpful 🙂
–Jeff
Champion!!! You’re a legend for solving this!
So about that rsync command..? 🙂
Hi Mike,
I really would like some assistance with my photos that were downloaded from google photos takeout, & sitting in 33 folders on my MacOS with “jpg.json and mp4.json” that I can’t open.
Can you please show us step by step (layman style) for us non-tech’s, how to: for MacOS.
Thank you, I appreciate it!!
Thank you so much! The smartest approach found so far on the web! I tested it and it works really nice!
Now, is there any way to DELETE the zip files?
This script both extracts and stores another copy of Zip files in the newly created “result” folder. I have lots of data to process and this will take up my space limit pretty quickly.
The ideal would be:
1) Delete the source file immediately after extraction
AND
2) Don’t copy the source file in the output folder
Any ideas?
Funny website name – CodingStick 🙂
Mike, will the same procedure work for Play Music zip files downloaded with Takeout.
Thanks
Mike,
You are THE MAN! Thank you so much for taking the time to publish this so that others can benefit from your hard work. I just used your instructions to retrieve all of the music that I have uploaded over the years to Google Music, which is being shut down in favor of YouTube Music (which seems to be an inferior tool for what I need to do). Your process is churning away as I type this. The “result” directory is at about 5gb at the moment, and all of the files seem to be named properly and are appearing in Finder in the “tracks” subfolder, just like clockwork.
I couldn’t be happier! Where can I send you a beer or other token that can be retrieved for the treat of your choice?
–Jeff
I look forward to your “rsync” for working with previous result from ditto.
This is super useful blog, thank you for sharing 🙂
Thanks a ton!! I’ve been trying to tackle this one for a long time now!
I was super surprised that “unzip” does not work with split archives even though “zip” can generate them :O
Mike – based on the “Halleluya” chorus emanating from the Comments – I think you may be the solution that has eluded me for months. I pick things up quickly, but I have zero coding experience and haven’t seen a command line since my brother had a DOS machine when we were kids…
I’m sure this will elicit whoops of laughter and eye-rolling, but I need a bit more help. I have 84 2G zip files from Takeout, I’ve got them all in a single folder, I can get to “New Terminal at Folder”, but then the big hole in my brain where UNIX is supposed to be… announces itself. Do I type:
“ditto -x -k -V [copy/paste the full file names for all 84 zip files? takeout-20221013T205613Z-001.zip
takeout-20221013T205613Z-002.zip
takeout-20221013T205613Z-003.zip
…
takeout-20221013T205613Z-084.zip]?
Also, the final command line [“result 1>log.txt 2>&1”]?? did you name the target folder “result”, and “1>log.txt 2>&1” is the actual combine and merge command?
I know how pathetic this may sound, but I own it, want to learn and more than anything, need to get 160Gig of my life off of Google Drive.
Your advice is so appreciated.
Jonathan
Wow, that’s a lot of files. I don’t think you want to enter all of that directly in the command line. I have two suggestions.
One is to open the TextEdit app and type the command in building it up with cut and paste if you can. Note that in my example I split the command over multiple lines using the back slash “\” character. That made it look nice in the post but you can just string all the files together separated by a space for each one. When you get them all together copy that and paste it into the Terminal command line.
The other is to put all of the zip files in a directory by themselves. Then you can let the command shell do the work for you. Use the Change Directory command “cd” to the one with all the zip files. Assuming all you have in the new directory is Google’s zip files this command should work
$ ditto -x -k -V *.zip result 1>log.txt 2>&1
In the command:
⁃ “-x” and “-k” tells ditto to extract the files from the zip archives given in the command
⁃ “-V” tells ditto to be verbose in reporting. It writes a line to the “stderr” for each and every file.
⁃ “*.zip” tells the shell (bash or zsh) to expand the command line with each .zip file before ditto gets called. So this is the same as if you typed it all, just less work.
⁃ “result” is the destination directory. If this doesn’t exist yet it will create it. If it does exist it will merge the contents.
⁃ “1>log.txt 2>&1” redirects stdout (number 1) and stderr (number 2) as a merged output into the file “log.txt”
Note that you have a lot of files so your “log.txt” file will be big. If you are not going to look at it you can simplify things by not using the verbose option “-V” and skip the redirection clause. Then the output will be less. So then you could just have the command:
$ ditto -x -k *.zip result
Hope that helps
Don’t get me started… It’s all because of the change Google made to its Cloud services. For over a decade, it offered “Selective Sync”, allowing you to choose which files lived only in the cloud vs. those files that you kept locally and “sync’d” to the cloud. It was great. I had decades of digital detritus that I was quite happy keeping in the cloud and not clogging my hard drive. Then in June 2021, they revamped the service to offer only two options:
1) Keep everything in the Cloud and stream all files as/when needed; or
2) Mirror everything – so, all files had to be in both places, Cloud and local.
Of my many irate emails with the G: Drive dev team, my personal favorite was my asking them if there was anyone on the team over the age of 22 who had ever set foot outside of a large, American downtown core… The fact that they configured a global system on the assumption that every user has fast, reliable broadband left me speechless. I’ve done business in places where cell-service was a blessing…
Anyway, the 84 Zip files are the result of me having to move almost 200Gig off of G: Drive to a local archive, so I can set-up a new MBP without bloating the hard drive from day one with the above-mentioned detritus.
I’ll tryout your suggestions and let you know – thank you soooo much.
Thank you for posting this: you are a lifesaver! I have about 175 2.5GB ZIP files and was worried about having to extract them properly. On my Synology drive from my MacbookPro. Also a BIG shout out to Brian for his Terminal prompt which seems to be working very well.
Now has anyone figured out how to manage the extracted files, and what to do exactly with the JSON files? I’d like to keep as much of it as possible; over the years I’ve also noticed my file naming convention has changed from the random: “Jon and Jill skiing.jpg” to my new preferred method which is to rename the files to: “YYYY-MM-DD HH.MM.SS”. I find it organizes my photos better this way. I know how to perform the name change, but how can I match up the new name with the name in JSON?
Thank you again!