Problem with increasing memory and cpu usage of a Go program

[Roger Pau Monné]

Hello,

I've wrote to this list a few days ago, about a Bittorrent client I'm trying
to develop in Go, but I came across a problem about memory usage in Go. I
will try to explain what my program does in an easy way, my English is not
very good, so let's hope it is still understandable.

Every peer is formed of 3 goroutines, 1 goroutine that reads from a channel
and writes to a socket, another one that reads from a socket and writes to a
channel, an a goroutine that manages the queue (has 1 incoming and 1
outgoing channels and a map[int] *message to store messages). When a peer
dies, the 3 goroutines return, and every element in the map is set to nil,
false, and the whole map to nil. All this goroutines are methods of a type
Peer or PeerQueue, in the case of the goroutine that manages the queue.

Also in another struct I have a list of all pieces requested to a specific
peer, in the form of "peers map[addr]map[uint64]int64", which is also
cleaned using "peers[peer_addr][piece_ref] = 0, false", and if the peer is
no longer available I do a "peers[peer_addr] = nil, false".

The problem is that if I was connected to 50 peers, and all of them closed
the connection, so I'm no longer connected to anyone, the memory used by my
program is not reduced, and if then I receive 50 new peers from the tracker
and connect to them the memory increases again, but never reduces, although I
try to free everything after it's no longer used. I've created a specific
log file for every peer, to check that all processes associated with it are
finished, and everything looks fine, but the problem is still there. You can
take a look at the full code here: http://github.com/royger/wgo

Do I have to do any specific thing to release memory? I even tried to call
runtime.GC() manually every 30s, but it has no effect. I also finish the
goroutines using return, do I have to use runtime.Goexit()? Is there any
problem if goroutines are methods of a struct? Any clue is welcome, since I
haven't got much experience with GC languages, and I've been trying to
resolve this for some days now.

Thanks, Roger.

[Jessta]

Post by Roger Pau Monné
Do I have to do any specific thing to release memory? I even tried to call
runtime.GC() manually every 30s, but it has no effect.

The current GC doesn't return memory to the operating system at all.
So the memory usage will not decrease.

- jessta

[Johann Höchtl]

Post by Jessta
The current GC doesn't return memory to the operating system at all.
So the memory usage will not decrease.

I guess this is sthg. which will change over time as the runtime
improves? Otherwise go "servlets" are not suitable for long running
server processes.

Post by Jessta
- jessta
--
=====================http://jessta.id.au

[Ryanne Dolan]

This behavior is fairly typical of gc'd languages. The assumption is that a
valid program will grow to some max size (not grow to infinity), at which
point the OS has already given you the memory and there isn't much reason to
give it back. At any rate, the kernel is more qualified to juggle memory
than each program independently.

Ryanne

- from my phone -

The current GC doesn'...

I guess this is sthg. which will change over time as the runtime
improves? Otherwise go "servlets" are not suitable for long running
server processes.

- jessta
--
=====================http://jessta.id.au

[Roger Pau Monné]

Thanks for the responses, so if the GC doesn't return memory to the
operating system, it reuses the deallocated memory to allocate new items?

Also I would like to know if there is anyway (a debugger) to see which
goroutines are running and the "name" (the function/method name) of them. I
took a look at Ogle, but I don't know how to use it, and haven't been able
to find any resources that explain how to use it.

Thanks again, Roger.

Post by Ryanne Dolan
This behavior is fairly typical of gc'd languages. The assumption is that a
valid program will grow to some max size (not grow to infinity), at which
point the OS has already given you the memory and there isn't much reason to
give it back. At any rate, the kernel is more qualified to juggle memory
than each program independently.
Ryanne
- from my phone -

The current GC doesn'...

I guess this is sthg. which will change over time as the runtime
improves? Otherwise go "servlets" are not suitable for long running
server processes.

- jessta
--
=====================http://jessta.id.au

--
To unsubscribe, reply using "remov...

[Russ Cox]

Post by Roger Pau Monné
Thanks for the responses, so if the GC doesn't return memory to the
operating system, it reuses the deallocated memory to allocate new items?

Yes.

Post by Roger Pau Monné
Also I would like to know if there is anyway (a debugger) to see which
goroutines are running and the "name" (the function/method name) of them. I
took a look at Ogle, but I don't know how to use it, and haven't been able
to find any resources that explain how to use it.

There isn't right now a way to do that.
The debugger is more a sketch than a useful tool at the moment.

A destructive way to get the stack trace of every goroutine is
to run kill -ABRT pid.

Russ

[Roger Pau Monné]

Hello,

I've done what you said Russ, I've left the program running until I had 0
peers connected and the CPU usage was 50%, which was not normal, since there
where no peers connected and the download was finished. The strack trace is
as follows:

SIGABRT: abort

Faulting address: 0x1af9a

pc: 0x1af9a


mach_semaphore_wait+0xb
/Users/royger/go/src/pkg/runtime/darwin/amd64/sys.s:196

mach_semaphore_wait()

mach_semacquire+0x1a /Users/royger/go/src/pkg/runtime/darwin/thread.c:427

mach_semacquire(0xb03, 0xffffffff00000000)

usemacquire+0x57 /Users/royger/go/src/pkg/runtime/darwin/thread.c:101

usemacquire(0xb03, 0xffffffff)

notesleep+0x2a /Users/royger/go/src/pkg/runtime/darwin/thread.c:127

notesleep(0xe50f0, 0x3260b0)

nextgandunlock+0xfc /Users/royger/go/src/pkg/runtime/proc.c:339

nextgandunlock()

scheduler+0x189 /Users/royger/go/src/pkg/runtime/proc.c:518

scheduler()

mstart+0x47 /Users/royger/go/src/pkg/runtime/proc.c:385

mstart()

_rt0_amd64+0x74 /Users/royger/go/src/pkg/runtime/amd64/asm.s:46

_rt0_amd64()


goroutine 92 [4]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runfinq+0x34 /Users/royger/go/src/pkg/runtime/mgc0.c:344

runfinq()

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 13 [1]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runtime.exitsyscall+0x89 /Users/royger/go/src/pkg/runtime/proc.c:602

runtime.exitsyscall()

syscall.Syscall+0x61 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:35

syscall.Syscall()

syscall.gettimeofday+0x3c
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:676

syscall.gettimeofday(0x74, 0x16bdb20, 0x0, 0x0, 0x4bbb69a0, ...)

syscall.Gettimeofday+0x1e
/Users/royger/go/src/pkg/syscall/syscall_darwin_amd64.go:31

syscall.Gettimeofday(0x16bdb20, 0x0, 0x10954, 0xad6f90)

os.Time+0x3c /Users/royger/go/src/pkg/os/time.go:16

os.Time(0x16bdb20, 0x16bdb20, 0x453c3, 0x19ee450, 0x4bbb69a0, ...)

time.Nanoseconds+0x13 /Users/royger/go/src/pkg/time/time.go:26

time.Nanoseconds(0xaa34a0, 0x19ee450)

time.Sleep+0x13 /Users/royger/go/src/pkg/time/sleep.go:16

time.Sleep(0x4bbb69a0, 0x288bbaf0, 0x0, 0x0)

time.wakeLoop+0x55 /Users/royger/go/src/pkg/time/tick.go:82

time.wakeLoop(0x683f8, 0x11a1fbd7742a7ee8, 0x0, 0x1226d)

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 8 [4]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742

runtime.selectgo(0x10dc580, 0xaa3cc0)

main.*PieceMgr·Run+0xa05 /Users/royger/Documents/wgo/PieceMgr.go:82

main.*PieceMgr·Run(0x10dc580, 0xaaccc0)

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 7 [4]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742

runtime.selectgo(0x40e8a0, 0xf64220)

main.*PeerMgr·Run+0x9d3 /Users/royger/Documents/wgo/PeerMgr.go:88

main.*PeerMgr·Run(0x40e8a0, 0xaac000)

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 6 [4]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742

runtime.selectgo(0x1194b80, 0xaa3620)

main.*Stats·Run+0x101 /Users/royger/Documents/wgo/Stats.go:83

main.*Stats·Run(0x1194b80, 0xaacc60)

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 3 [2]:

gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539

gosched()

runtime.exitsyscall+0x89 /Users/royger/go/src/pkg/runtime/proc.c:602

runtime.exitsyscall()

syscall.Syscall+0x61 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:35

syscall.Syscall()

MCache_Alloc+0xfb /Users/royger/go/src/pkg/runtime/mcache.c:42

MCache_Alloc(0x16e5cc0, 0x20, 0x20, 0x0, 0x4bbb69a0, ...)


goroutine 2 [3]:

runtime.entersyscall+0x50 /Users/royger/go/src/pkg/runtime/proc.c:560

runtime.entersyscall()

syscall.Syscall6+0x5 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:39

syscall.Syscall6()

syscall.kevent+0x6d
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:108

syscall.kevent(0x16b, 0x6, 0x0, 0x0, 0x393008, ...)

syscall.Kevent+0x99 /Users/royger/go/src/pkg/syscall/syscall_bsd.go:375

syscall.Kevent(0x6, 0x0, 0x0, 0x393008, 0x20000000a, ...)

net.*pollster·WaitFD+0xfb /Users/royger/go/src/pkg/net/fd_darwin.go:87

net.*pollster·WaitFD(0x6, 0x0, 0x0, 0x393008, 0xa0000000a, ...)

net.*pollServer·Run+0x9e /Users/royger/go/src/pkg/net/fd.go:232

net.*pollServer·Run(0x393000, 0x0)

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()


goroutine 1 [3]:

runtime.entersyscall+0x50 /Users/royger/go/src/pkg/runtime/proc.c:560

runtime.entersyscall()

syscall.Syscall6+0x5 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:39

syscall.Syscall6()

syscall.Select+0x5d
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:512

syscall.Select(0x5d, 0x0, 0x0, 0x0, 0x0, ...)

syscall.Sleep+0x71 /Users/royger/go/src/pkg/syscall/syscall_bsd.go:145

syscall.Sleep(0x600000000, 0x0, 0x0, 0x0)

time.Sleep+0x3b /Users/royger/go/src/pkg/time/sleep.go:19

time.Sleep(0x6fc23ac00, 0x0, 0x323740, 0x200000002)

main.main+0x97c /Users/royger/Documents/wgo/test.go:63

main.main()

mainstart+0xf /Users/royger/go/src/pkg/runtime/amd64/asm.s:54

mainstart()

goexit /Users/royger/go/src/pkg/runtime/proc.c:145

goexit()

rax 0xe

rbx 0xe50f0

rcx 0x7fff5fbff458

rdx 0x0

rdi 0xb03

rsi 0xc72d20

rbp 0xe50f0

rsp 0x7fff5fbff458

r8 0x0

r9 0x0

r10 0x0

r11 0x206

r12 0x2

r13 0x1248adee

r14 0xe5040

r15 0xe4fa8

rip 0x1af9a

rflags 0x206

cs 0x2f

fs 0x10

gs 0x48

I can recognize some of this goroutines as part of my program, I think
goroutine
1 [3] is the main goroutine, which does a time.Sleep, then goroutine 8 [4]
is the PieceMgr process, which is waiting on a select with multiple
channels, goroutine 7 [4] is the PeerMgr, which is also waiting on a select
with multiple channels and goroutine 6 [4] is the Stats goroutine, that is
also waiting in a select statement, and some other goroutines should be
Tickers. But what are the other goroutines doing? At least one of them is
consuming a lot of CPU and I don't know which one, neither why is it doing
that. Can someone help me identifying this goroutines?

Thanks, Roger.

Post by Russ Cox

Post by Roger Pau Monné
Thanks for the responses, so if the GC doesn't return memory to the
operating system, it reuses the deallocated memory to allocate new items?

Yes.

Post by Roger Pau Monné
Also I would like to know if there is anyway (a debugger) to see which
goroutines are running and the "name" (the function/method name) of them.

I

Post by Roger Pau Monné
took a look at Ogle, but I don't know how to use it, and haven't been

able

Post by Roger Pau Monné
to find any resources that explain how to use it.

There isn't right now a way to do that.
The debugger is more a sketch than a useful tool at the moment.
A destructive way to get the stack trace of every goroutine is
to run kill -ABRT pid.
Russ

[Roger Pau Monné]

I've also released that I use a lot of Time.Tick in the Peers processes to
send the keep alive messages, do I have to close them in some special way? I
just do a close(keepAlive), where keepAlive := time.Tick(interval), but I
don't know if it will be best to use time.NewTicker and close it using
Stop().

Thanks again.

Post by Roger Pau Monné
Hello,
I've done what you said Russ, I've left the program running until I had 0
peers connected and the CPU usage was 50%, which was not normal, since there
where no peers connected and the download was finished. The strack trace is
SIGABRT: abort
Faulting address: 0x1af9a
pc: 0x1af9a
mach_semaphore_wait+0xb
/Users/royger/go/src/pkg/runtime/darwin/amd64/sys.s:196
mach_semaphore_wait()
mach_semacquire+0x1a /Users/royger/go/src/pkg/runtime/darwin/thread.c:427
mach_semacquire(0xb03, 0xffffffff00000000)
usemacquire+0x57 /Users/royger/go/src/pkg/runtime/darwin/thread.c:101
usemacquire(0xb03, 0xffffffff)
notesleep+0x2a /Users/royger/go/src/pkg/runtime/darwin/thread.c:127
notesleep(0xe50f0, 0x3260b0)
nextgandunlock+0xfc /Users/royger/go/src/pkg/runtime/proc.c:339
nextgandunlock()
scheduler+0x189 /Users/royger/go/src/pkg/runtime/proc.c:518
scheduler()
mstart+0x47 /Users/royger/go/src/pkg/runtime/proc.c:385
mstart()
_rt0_amd64+0x74 /Users/royger/go/src/pkg/runtime/amd64/asm.s:46
_rt0_amd64()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runfinq+0x34 /Users/royger/go/src/pkg/runtime/mgc0.c:344
runfinq()
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runtime.exitsyscall+0x89 /Users/royger/go/src/pkg/runtime/proc.c:602
runtime.exitsyscall()
syscall.Syscall+0x61 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:35
syscall.Syscall()
syscall.gettimeofday+0x3c
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:676
syscall.gettimeofday(0x74, 0x16bdb20, 0x0, 0x0, 0x4bbb69a0, ...)
syscall.Gettimeofday+0x1e
/Users/royger/go/src/pkg/syscall/syscall_darwin_amd64.go:31
syscall.Gettimeofday(0x16bdb20, 0x0, 0x10954, 0xad6f90)
os.Time+0x3c /Users/royger/go/src/pkg/os/time.go:16
os.Time(0x16bdb20, 0x16bdb20, 0x453c3, 0x19ee450, 0x4bbb69a0, ...)
time.Nanoseconds+0x13 /Users/royger/go/src/pkg/time/time.go:26
time.Nanoseconds(0xaa34a0, 0x19ee450)
time.Sleep+0x13 /Users/royger/go/src/pkg/time/sleep.go:16
time.Sleep(0x4bbb69a0, 0x288bbaf0, 0x0, 0x0)
time.wakeLoop+0x55 /Users/royger/go/src/pkg/time/tick.go:82
time.wakeLoop(0x683f8, 0x11a1fbd7742a7ee8, 0x0, 0x1226d)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742
runtime.selectgo(0x10dc580, 0xaa3cc0)
main.*PieceMgr·Run+0xa05 /Users/royger/Documents/wgo/PieceMgr.go:82
main.*PieceMgr·Run(0x10dc580, 0xaaccc0)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742
runtime.selectgo(0x40e8a0, 0xf64220)
main.*PeerMgr·Run+0x9d3 /Users/royger/Documents/wgo/PeerMgr.go:88
main.*PeerMgr·Run(0x40e8a0, 0xaac000)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runtime.selectgo+0x352 /Users/royger/go/src/pkg/runtime/chan.c:742
runtime.selectgo(0x1194b80, 0xaa3620)
main.*Stats·Run+0x101 /Users/royger/Documents/wgo/Stats.go:83
main.*Stats·Run(0x1194b80, 0xaacc60)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
gosched+0x4e /Users/royger/go/src/pkg/runtime/proc.c:539
gosched()
runtime.exitsyscall+0x89 /Users/royger/go/src/pkg/runtime/proc.c:602
runtime.exitsyscall()
syscall.Syscall+0x61 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:35
syscall.Syscall()
MCache_Alloc+0xfb /Users/royger/go/src/pkg/runtime/mcache.c:42
MCache_Alloc(0x16e5cc0, 0x20, 0x20, 0x0, 0x4bbb69a0, ...)
runtime.entersyscall+0x50 /Users/royger/go/src/pkg/runtime/proc.c:560
runtime.entersyscall()
syscall.Syscall6+0x5 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:39
syscall.Syscall6()
syscall.kevent+0x6d
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:108
syscall.kevent(0x16b, 0x6, 0x0, 0x0, 0x393008, ...)
syscall.Kevent+0x99 /Users/royger/go/src/pkg/syscall/syscall_bsd.go:375
syscall.Kevent(0x6, 0x0, 0x0, 0x393008, 0x20000000a, ...)
net.*pollster·WaitFD+0xfb /Users/royger/go/src/pkg/net/fd_darwin.go:87
net.*pollster·WaitFD(0x6, 0x0, 0x0, 0x393008, 0xa0000000a, ...)
net.*pollServer·Run+0x9e /Users/royger/go/src/pkg/net/fd.go:232
net.*pollServer·Run(0x393000, 0x0)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
runtime.entersyscall+0x50 /Users/royger/go/src/pkg/runtime/proc.c:560
runtime.entersyscall()
syscall.Syscall6+0x5 /Users/royger/go/src/pkg/syscall/asm_darwin_amd64.s:39
syscall.Syscall6()
syscall.Select+0x5d
/Users/royger/go/src/pkg/syscall/zsyscall_darwin_amd64.go:512
syscall.Select(0x5d, 0x0, 0x0, 0x0, 0x0, ...)
syscall.Sleep+0x71 /Users/royger/go/src/pkg/syscall/syscall_bsd.go:145
syscall.Sleep(0x600000000, 0x0, 0x0, 0x0)
time.Sleep+0x3b /Users/royger/go/src/pkg/time/sleep.go:19
time.Sleep(0x6fc23ac00, 0x0, 0x323740, 0x200000002)
main.main+0x97c /Users/royger/Documents/wgo/test.go:63
main.main()
mainstart+0xf /Users/royger/go/src/pkg/runtime/amd64/asm.s:54
mainstart()
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()
rax 0xe
rbx 0xe50f0
rcx 0x7fff5fbff458
rdx 0x0
rdi 0xb03
rsi 0xc72d20
rbp 0xe50f0
rsp 0x7fff5fbff458
r8 0x0
r9 0x0
r10 0x0
r11 0x206
r12 0x2
r13 0x1248adee
r14 0xe5040
r15 0xe4fa8
rip 0x1af9a
rflags 0x206
cs 0x2f
fs 0x10
gs 0x48
I can recognize some of this goroutines as part of my program, I think goroutine
1 [3] is the main goroutine, which does a time.Sleep, then goroutine 8 [4]
is the PieceMgr process, which is waiting on a select with multiple
channels, goroutine 7 [4] is the PeerMgr, which is also waiting on a select
with multiple channels and goroutine 6 [4] is the Stats goroutine, that is
also waiting in a select statement, and some other goroutines should be
Tickers. But what are the other goroutines doing? At least one of them is
consuming a lot of CPU and I don't know which one, neither why is it doing
that. Can someone help me identifying this goroutines?
Thanks, Roger.

Post by Roger Pau Monné

Post by Roger Pau Monné
Thanks for the responses, so if the GC doesn't return memory to the
operating system, it reuses the deallocated memory to allocate new

items?
Yes.

Post by Roger Pau Monné
Also I would like to know if there is anyway (a debugger) to see which
goroutines are running and the "name" (the function/method name) of

them. I

Post by Roger Pau Monné
took a look at Ogle, but I don't know how to use it, and haven't been

able

Post by Roger Pau Monné
to find any resources that explain how to use it.

There isn't right now a way to do that.
The debugger is more a sketch than a useful tool at the moment.
A destructive way to get the stack trace of every goroutine is
to run kill -ABRT pid.
Russ

[ygl]

Post by Roger Pau Monné
I've also released that I use a lot of Time.Tick in the Peers processes to
send the keep alive messages, do I have to close them in some special way? I
just do a close(keepAlive), where keepAlive := time.Tick(interval), but I
don't know if it will be best to use time.NewTicker and close it using
Stop().

time.Tick() is for a ticker which you never want to shutdown (http://
golang.org/pkg/time/#Tick). For a ticker with one-shot or a few shots,
we could use "tick := NewTicker(ns); tick.Stop()"

Regards
Yigong

[Russ Cox]

It's very hard to say from the stack trace what was
happening dynamically. Instead of sending an ABRT
you can also run the profiler for a little while:

6prof -p pid -t 10 -d 1

[Roger Pau Monné]

The problem was coming from using time.Tick over and over in the same
variable, now I've switched to time.NewTicker and I do a Stop() before
overwriting the variable, that seems to fix the problem. Anyway how can I
use the profiler? Sorry for my ignorance, but I cannot find the 6prof
command. Do I have to build it in some special way?

Thanks, Roger.

Post by Russ Cox
It's very hard to say from the stack trace what was
happening dynamically. Instead of sending an ABRT
6prof -p pid -t 10 -d 1

[Russ Cox]

I cannot find the 6prof command.

You must be on OS X.
The debugging programs need to be installed separately
there, because they need to be setgid procmod to do
things like attach to running processes:

cd $GOROOT/src
./sudo.bash

Russ

[Roger Pau Monné]

Hello,

Yes, I'm on OS X, thanks for the help installing it. When I attach 6prof to
my process, it outputs this:

prof: too many threads; limiting to 0
threadstopped thread_info -0x40000b5c: mach: invalid argument
threadstopped thread_info -0x40000c5c: mach: invalid argument
5699 samples (avg 3e+01 threads)
1674.49% mach_semaphore_wait
1225.25% syscall.Syscall
298.96% syscall.Syscall6
0.42% write
0.14% scanblock

And then my program crashes with this error:

panic: runtime error: slice bounds out of range

panic PC=0x3272e0
runtime.panic+0x7c /Users/royger/go/src/pkg/runtime/proc.c:982
runtime.panic(0x0, 0x171d9)
panicstring+0x60 /Users/royger/go/src/pkg/runtime/runtime.c:85
panicstring(0x9b388, 0xca1110)
runtime.panicslice+0x1c /Users/royger/go/src/pkg/runtime/runtime.c:52
runtime.panicslice()
runtime.slicearray+0x1f0 /Users/royger/go/src/pkg/runtime/slice.c:155
runtime.slicearray(0x200016b, 0x3273b0, 0xa00000008, 0x4ac15, 0x38f008, ...)
net.*pollster·WaitFD+0x1c2 /Users/royger/go/src/pkg/net/fd_darwin.go:97
net.*pollster·WaitFD(0x38f008, 0xa, 0x200200016b, 0x38f008, 0xa0000000a,
...)
net.*pollServer·Run+0x9e /Users/royger/go/src/pkg/net/fd.go:232
net.*pollServer·Run(0x38f000, 0x49328829c8)
goexit /Users/royger/go/src/pkg/runtime/proc.c:145
goexit()

I'm pretty confident the error is related to attaching 6prof to my program,
since it only happens when I do so, but I don't know if it is caused because
of bad usage or if it's a bug (it seems something related to the scheduler).

Thanks, Roger.

Post by Russ Cox

I cannot find the 6prof command.

You must be on OS X.
The debugging programs need to be installed separately
there, because they need to be setgid procmod to do
cd $GOROOT/src
./sudo.bash
Russ

[Jonathan Wills]

I think you might be suffering from issue
536<http://code.google.com/p/go/issues/detail?id=536>
.
Basically if you have a select statement that gets reached
regularly but there is a case within the select that never gets
used you will have a memory leak.

Post by Roger Pau Monné
Hello,
I've wrote to this list a few days ago, about a Bittorrent client I'm
trying to develop in Go, but I came across a problem about memory usage in
Go. I will try to explain what my program does in an easy way, my English is
not very good, so let's hope it is still understandable.
Every peer is formed of 3 goroutines, 1 goroutine that reads from a channel
and writes to a socket, another one that reads from a socket and writes to a
channel, an a goroutine that manages the queue (has 1 incoming and 1
outgoing channels and a map[int] *message to store messages). When a peer
dies, the 3 goroutines return, and every element in the map is set to nil,
false, and the whole map to nil. All this goroutines are methods of a type
Peer or PeerQueue, in the case of the goroutine that manages the queue.
Also in another struct I have a list of all pieces requested to a specific
peer, in the form of "peers map[addr]map[uint64]int64", which is also
cleaned using "peers[peer_addr][piece_ref] = 0, false", and if the peer is
no longer available I do a "peers[peer_addr] = nil, false".
The problem is that if I was connected to 50 peers, and all of them closed
the connection, so I'm no longer connected to anyone, the memory used by my
program is not reduced, and if then I receive 50 new peers from the tracker
and connect to them the memory increases again, but never reduces,
although I try to free everything after it's no longer used. I've created
a specific log file for every peer, to check that all processes associated
with it are finished, and everything looks fine, but the problem is still
http://github.com/royger/wgo
Do I have to do any specific thing to release memory? I even tried to call
runtime.GC() manually every 30s, but it has no effect. I also finish the
goroutines using return, do I have to use runtime.Goexit()? Is there any
problem if goroutines are methods of a struct? Any clue is welcome, since I
haven't got much experience with GC languages, and I've been trying to
resolve this for some days now.
Thanks, Roger.