Is it possible to adapt this to support folders?

[nbelles]

Hello,

I would like to be able to upload folders at a time instead of just individual files. Am curious if there is any capacity/support for such a thing especially because it seems as though you can supply multiple files when using curl.

Thanks!

[orhun]

Hey!

rustypaste currently does not support this. It is an interesting idea though.

Also, I'm not entirely sure how that would work. In theory, you can upload multiple files like this:

curl -F "file=@file1" -F "file=@file2" "<server_address>"

Then maybe specifying an extra header for indicating a directory in upload/ folder would be the way to go?

curl -F "file=@file1" -F "file=@file2" -H "directory:myfiles"  "<server_address>"

Then the server will create the directory if it does not exist and serve the files from there.

Still, I'm not entirely sure whether if we should implement this due to extra complexity and performance issues due to increasing directory traversal. Plus, specifying an extra header does not seem ideal to me but I can't think of anything else right now.

Any ideas?

[nbelles]

Even just having one layer of directory support would be cool, exactly how you explained it where there is an additional header that would indicate to the receiving end to put the list of files that were sent into a directory with that name. Then in rustypaste-cli you could detect if the thing being passed in is a file or a directory and if it is a directory then make sure it doesn't have any additional directories inside it.

I did some looking into how Google Drive allows for directory uploads because that's the only other file uploading system that supports folders that I can think of off the top of my head. It seems as though they have separate PUT requests for every individual file and in each request they also have a header that names a "parent" to that file.

This might be a lot to write out by hand in a curl request but in rustypaste-cli you could pretty easily create a json-like header that describes the structure of the files attached. If no additional header is received then just treat them as a bunch of separate files.

Just thinking out loud for a bit:

If the folder structure looked like this:

grandpa
├── 0001.mp4
├── 0002.mp4
├── 0003.mp4
└── father
    ├── 0004.mp4
    ├── 0005.mp4
    ├── 0006.mp4
    └── son
        ├── 0007.mp4
        ├── 0008.mp4
        └── 0009.mp4

then the json-like header would look something like this:

{
   "directory": "grandpa",
   "files": ["0001.mp4", "0002.mp4", "0003.mp4"],
   "subdirectories": [
      {
         "directory": "father",
         "files": ["0004.mp4", "0005.mp4", "0006.mp4"],
         "subdirectories": [
            {
               "directory": "son",
               "files": ["0007.mp4", "0008.mp4", "0009.mp4"],
               "subdirectories": []
            }
         ]
      }
   ]
}

[nbelles]

One thing I just thought about, there is the edge case that if you pass in a folder that has files within it with the same name at different levels, it might be hard to reconstruct them on the other side. Off the top of my head I don't see any particularly easy ways to handle that without it getting quite complex (like generating a unique id for each folder and then prepending each file name with the unique id of the folder it resides in and then removing the prepended unique id on the receiving end when reconstructing the folder structure).

Personally, if it is too complex (even just for now), I think the benefits of being able to upload folders far outweigh the downsides of not being able to send folders with similarly named files at different levels and that that could just be a warning that is provided to users of rustypaste-cli when passing in a folder.

Even being able to send single layer folders would be really cool and would love to see that implemented. Sadly I have almost no knowledge on Rust so I don't think I could be of much help on the coding side of things (otherwise I would have already started a pull request). But I am happy to provide testing services wherever necessary.

[inglor]

This is not a functionality of a pastebin in my opinion..

[nbelles]

@inglor Normally it would make sense to compress the folder that you want to send into a .zip or something similar. For me personally, I have the use case where I am trying to send folders of large files and it doesn't make sense to zip the folder before sending because of the space required locally to do such a thing (not to mention the time lost in zipping and unzipping and the processing power and space required on both ends to handle the file).

I have looked at other pastebins that exist but even if they did eventually add support for folders, they often choose to split up files into smaller chunks when sending over the network which throws performance completely out the window. This is one of the few pastebins that I've seen that has such great performance, I was even able to stream an uncompressed 4k60p video I recorded with one of my video cameras over the Internet. Thanks again @orhun for creating such a performant service.

[orhun]

I am going to put some thought into this and see if I can come up with a simpler solution/implementation. In the meantime, if someone feels like implementing it, just comment to this issue :)

[shabane]

i have an idea to do that,

ls /path/to/dir/ | cpio -o | curl <server_address> -F file=@-