multipart file upload example

[Matt Aimonetti]

I couldn't find a good example of how to do multipart file upload with some
extra params, so I wrote one:
http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/

Feel free to let me know if I missed something or did something wrong.

- Matt

[jonathan-TOYmAeewdbJWk0Htik3J/]

Instead of buffering the file into memory twice, you can leverage io.Pipe
and a goroutine to stream the part from disk:
http://play.golang.org/p/xfNSyyTd15

Jonathan

Post by Matt Aimonetti
I couldn't find a good example of how to do multipart file upload with
http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/
Feel free to let me know if I missed something or did something wrong.
- Matt

[jonathan-TOYmAeewdbJWk0Htik3J/]

Fixed two mistakes that I made (this is still completely untested):
http://play.golang.org/p/eEFBMGMNTW

[Matt Aimonetti]

Having to use a goroutine makes the example quite more complex, and withou
the goroutine the code just blocks. However, I agree that I should have use
`copy` instead of reading the content of the file in memory.
Good point also about using "file/path". I updated the blog post with your
suggestions, thanks.

Post by jonathan-TOYmAeewdbJWk0Htik3J/
http://play.golang.org/p/eEFBMGMNTW
--
You received this message because you are subscribed to a topic in the
Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/golang-nuts/WIFky_-5qpU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/groups/opt_out.

[Jonathan Rudenberg]

Having to use a goroutine makes the example quite more complex, and withou the goroutine the code just blocks.

Sure, it's a bit more complex, but it demonstrates the correct way to do this with streaming IO vs reading the entire file into memory for no good reason.

Also, I noticed another bug:

var bodyContent []byte
// …
resp.Body.Read(bodyContent)

bodyContent is a nil slice, so the Read call will read zero bytes. Instead, you can do this:

buf := &bytes.Buffer{}
_, err := buf.ReadFrom(resp.Body)

[Matt Aimonetti]

oh man, how did I miss this huge bug o_O
Thanks, I updated the blog post ince again.
I added a link to your example with the goroutine, and while it's probably
the best approach it does add more complexity.

Can you help me better understand the difference between using a pipe and
copying the memory?
It's probably a stupid question but I'm not quite sure I understand the
problem. Are you saying that allocation wise, using copy vs piping is the
same thing, but the difference is that using copy requires that we read the
entire source in memory instead of reading by small chunks? (which could be
a problem in case of big files)
If that's the case, wouldn't something like bufio helps?
Again, I realize it might sound like a stupid question but I would like to
make sure I fully understand the problem.

Thanks



On Thu, Jul 4, 2013 at 12:15 PM, Jonathan Rudenberg

Post by Matt Aimonetti
Having to use a goroutine makes the example quite more complex, and

withou the goroutine the code just blocks.
Sure, it's a bit more complex, but it demonstrates the correct way to do
this with streaming IO vs reading the entire file into memory for no good
reason.
var bodyContent []byte
// …
resp.Body.Read(bodyContent)
bodyContent is a nil slice, so the Read call will read zero bytes.
buf := &bytes.Buffer{}
_, err := buf.ReadFrom(resp.Body)

[Jonathan Rudenberg]

Are you saying that allocation wise, using copy vs piping is the same thing, but the difference is that using copy requires that we read the entire source in memory instead of reading by small chunks? (which could be a problem in case of big files)

You are copying the entire file from disk into an in-memory buffer (the bytes.Buffer that you are encoding the multipart request body into). This will fail when the file is too large to fit into memory and is also slower, because you have to wait to read the entire file into memory.

The pipe that my implementation uses never stores more than the chunk size used in io.Copy in memory, streaming it from the disk through to the socket one chunk at a time.

If that's the case, wouldn't something like bufio helps?

bufio exists to solve a different problem by combining lots of tiny read/write syscalls and providing helpers for some standard operations like reading a line at a time.