The motivation for this small bash script is that when you want to make a big
backup of files, let's say to google drive, dropbox, icloud etc., it can simply
take forever before your files are uploaded. In my case, I wanted to backup
32000 separate files (photo's) from an external drive directly to the cloud.
This can take a couple of days to be uploaded, so the idea is to create smaller
batches so that you can also turn your computer off and continue later.
So, I wanted to move / copy these separate 32000 files into separate folders of
size `N`, and then select all the files in the folder and drag-and-drop or select-and-upload
with out the risk of introducing a lot duplicate uploads because I don't exactly know
where I left off... sound familiar probably?
So you want to subdivide all these separate files into separate folders like so (forget about how `tree` sorted it :))
The motivation for this small bash script is that when you want to perform a big backup of files to an online cloud system, let's say to google drive, dropbox, icloud etc.. This can simply take forever if you try to upload all your files at once. In my case, I wanted to backup 32000 separate files (photo's) from an external drive directly to the cloud. This can take a couple of days to be uploaded, so the idea is to create smaller batches so that you can also turn your computer off and continue later.
## What the tool does
So, I wanted to move / copy these separate 32000 files into separate folders of size `N` and then select all the files in the folder and **drag-and-drop** or **select-and-upload** without the risk or annoying fact of uploading a lot of files twice. Because I don't exactly know where I left off the next day when I turned my computer off... sound familiar?
So what I want it to subdivide all these 32000 files into separate folders like so (forget about how `tree` sorted it :))
~~~shell
├── folder # (here all your files are located for example).
└── output
...
...
@@ -21,15 +14,16 @@ So you want to subdivide all these separate files into separate folders like so
├── part_1500
└── part_500
~~~
and then upload then by selecting a couple of 100 files, this is often far more manageable.
The tool itself uses the `cp` command to copy files to a new folder in given `outputDir`, so `cp inputDir/some-awesome-photo.jpg outputDir/part_0/some-awesome-photo.jpg` and it does this for all the files at -maxdepth 1 in the `inputDir` folder.
It makes copies using the `cp` command to a new folder in the `outputDir`,
so `cp inputDir/some-awesome-photo.jpg outputDir/part_0/some-awesome-photo.jpg` and it does this for all the files at level1 depth of in the `inputDir` folder. Now that you copied a bunch of files, let's see 1500 files you, you just stop the command using CTRL+C and use the `skip` at a later point in time and set it to 1500 again. It will then start at `part_1500` and start copying again. Of course you have a potential problem if you have some files in the 1500 directory and you've uploaded them leaving you with some (potential!!) duplicates. I say potential, because maybe your online cloud system filters those?
## Usage
Now that you copied a bunch of files, let's see 1500 files you, you just stop the command using CTRL+C and use the `skip` at a later point in time and set it to 1500 again. It will then start at `part_1500` and start copying again. Of course you have a potential problem if you have some files in the 1500 directory and you've uploaded them leaving you with some (potential!!) duplicates. I say potential, because maybe your online cloud system filters those?
Notice that I purposely not use the `mv` command because I don't want anyone (including me) to use this
teeny tiny tool to delete their files.. don't want that responsibility..
## Usage
Make part directories of size 10 (default is 10 per folder)
