Discussion:
DumpRequestOut from net/http/httputils fails to dump
Cem Ezberci
2013-04-08 17:44:02 UTC
Permalink
I have been trying to use DumpRequestOut function in net/http/httputil
package for a small pet project. It seems like it fails to execute due to
Scheme being an empty string in URL object that is part of the Request
object.

I have put together a simple http server that I was able to test (I think)
on my local with version 1.0.3 and latest tip version I pulled on April 4
(go version devel +e77430da3316 Thu Apr 04 16:36:10 2013 +1100
linux/amd64). It seems like Scheme field is not populated in either
version. Is this a known bug or a bug at all?

http://play.golang.org/p/Rplem4q4Yy

When I run this and make a call to the server (http://127.0.0.1:8080/), I
get the following output in stdout:

2013/04/08 13:39:04 Error while dumping request: unsupported protocol
scheme ""
2013/04/08 13:39:04 Scheme:
2013/04/08 13:39:04 []

-Cem
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Brad Fitzpatrick
2013-04-09 01:34:17 UTC
Permalink
This behavior is at least consistent with Transport: a Transport (the http
Client) won't make a request without a scheme.

And the server handler's requests don't have a scheme (since it can be
unknown via which listener a request arrived on).
Post by Cem Ezberci
I have been trying to use DumpRequestOut function in net/http/httputil
package for a small pet project. It seems like it fails to execute due to
Scheme being an empty string in URL object that is part of the Request
object.
I have put together a simple http server that I was able to test (I think)
on my local with version 1.0.3 and latest tip version I pulled on April 4
(go version devel +e77430da3316 Thu Apr 04 16:36:10 2013 +1100
linux/amd64). It seems like Scheme field is not populated in either
version. Is this a known bug or a bug at all?
http://play.golang.org/p/Rplem4q4Yy
When I run this and make a call to the server (http://127.0.0.1:8080/), I
2013/04/08 13:39:04 Error while dumping request: unsupported protocol
scheme ""
2013/04/08 13:39:04 []
-Cem
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Brad Fitzpatrick
2013-04-15 16:44:27 UTC
Permalink
What's the question? How to serialize a Request/Response? encoding/gob
will do everything except the Body io.Reader field. But you can nil that
out and then gob it.
Thanks, Brad. That makes sense.
I guess I should be more clear about what I am trying to do instead. The
goal of this experiment is to see if I can build an http reverse proxy that
accepts requests, compresses the entire request (to make sure everything is
compressed including the headers -- applies to response too), streams out
to the receiving end that decompresses the data, makes the actual http call
to the origin that was intended for, retrieve the response, compress it and
send it back to the initiator which then decompresses and passes it over to
the original requester. I am looking for a sane way to pass these requests
and responses between those two where http is not the protocol that I will
be using. I do not see a reason to have request and response objects built
just to dump them in byte slices, compress and send around. Is there a
better way to do this? I am planning to use the lzma go package I found for
compression and zmq push-pull (for passing requests) and pub-sub (for
passing responses).
-Cem
Post by Brad Fitzpatrick
This behavior is at least consistent with Transport: a Transport (the
http Client) won't make a request without a scheme.
And the server handler's requests don't have a scheme (since it can be
unknown via which listener a request arrived on).
Post by Cem Ezberci
I have been trying to use DumpRequestOut function in net/http/httputil
package for a small pet project. It seems like it fails to execute due to
Scheme being an empty string in URL object that is part of the Request
object.
I have put together a simple http server that I was able to test (I
think) on my local with version 1.0.3 and latest tip version I pulled on
April 4 (go version devel +e77430da3316 Thu Apr 04 16:36:10 2013 +1100
linux/amd64). It seems like Scheme field is not populated in either
version. Is this a known bug or a bug at all?
http://play.golang.org/p/**Rplem4q4Yy<http://play.golang.org/p/Rplem4q4Yy>
When I run this and make a call to the server (http://127.0.0.1:8080/),
2013/04/08 13:39:04 Error while dumping request: unsupported protocol
scheme ""
2013/04/08 13:39:04 []
-Cem
--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Cem Ezberci
2013-04-15 21:06:00 UTC
Permalink
Unfortunately encoding/gob doesn't work for Request because it has a URL
field that has a User field, which is of type url.UserInfo and it does not
have any exported fields. Please see below for the error when I try to
encode a Request object. I set URL.User field to nil (following your advice
for Body field) before encoding but that didn't seem to have helped.

2013/04/15 16:23:51 Gob'ing: Unable to prepare request for
f0df68e386fb52fbc3a5c700555b37981d5e1a84: gob: type url.Userinfo has no
exported fields

The workaround I found is to set URL.Scheme field in Request object to
"http" (for now I'll just handle http), URL.Host to Host from Request
object, RequestURI to "" (otherwise I wouldn't be able to make a Request
using the Request object I'd reconstruct later using Request.Do) and then
call the Write method of the Request with an io.Writer (lzma instance) to
chain the write and dump the request as bytes and compress. This would need
to be put in another struct with request id and forwarder id and gob encode
and send over (so that the receiving end can return the response back to
the original forwarder and requester but that's for routing with zmq etc.
so it is not important at this point). On the receiving end, I'd gob decode
the data received, decompress the compressed request in the struct, and
then reconstruct the Request object using http.ReadRequest function. It
looks like similar method exists for Response type (Response.Write) to do
the reverse operation but I haven't found an equivalent of ReadRequest
function for Response so that I could reconstruct the Response from wire
format later on. Maybe I need to hijack the ResponseWriter and push out the
data in wire format when serving back the response to the original
requester.
Post by Brad Fitzpatrick
What's the question? How to serialize a Request/Response? encoding/gob
will do everything except the Body io.Reader field. But you can nil that
out and then gob it.
Thanks, Brad. That makes sense.
I guess I should be more clear about what I am trying to do instead. The
goal of this experiment is to see if I can build an http reverse proxy that
accepts requests, compresses the entire request (to make sure everything is
compressed including the headers -- applies to response too), streams out
to the receiving end that decompresses the data, makes the actual http call
to the origin that was intended for, retrieve the response, compress it and
send it back to the initiator which then decompresses and passes it over to
the original requester. I am looking for a sane way to pass these requests
and responses between those two where http is not the protocol that I will
be using. I do not see a reason to have request and response objects built
just to dump them in byte slices, compress and send around. Is there a
better way to do this? I am planning to use the lzma go package I found for
compression and zmq push-pull (for passing requests) and pub-sub (for
passing responses).
-Cem
Post by Brad Fitzpatrick
This behavior is at least consistent with Transport: a Transport (the
http Client) won't make a request without a scheme.
And the server handler's requests don't have a scheme (since it can be
unknown via which listener a request arrived on).
Post by Cem Ezberci
I have been trying to use DumpRequestOut function in net/http/httputil
package for a small pet project. It seems like it fails to execute due to
Scheme being an empty string in URL object that is part of the Request
object.
I have put together a simple http server that I was able to test (I
think) on my local with version 1.0.3 and latest tip version I pulled on
April 4 (go version devel +e77430da3316 Thu Apr 04 16:36:10 2013 +1100
linux/amd64). It seems like Scheme field is not populated in either
version. Is this a known bug or a bug at all?
http://play.golang.org/p/**Rplem4q4Yy<http://play.golang.org/p/Rplem4q4Yy>
When I run this and make a call to the server (http://127.0.0.1:8080/),
2013/04/08 13:39:04 Error while dumping request: unsupported protocol
scheme ""
2013/04/08 13:39:04 []
-Cem
--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Tamás Gulácsi
2013-04-16 04:43:25 UTC
Permalink
I think two viable solutions are:

a) send the original request, the wire format

b) save and send just what you need, and recreate a request on the other side from that minimal info.

If you don't really know all the possible request types beforehand, a) is better.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Loading...