Ruby에서 메모리 누수의 원인 찾기

Ruby에서 메모리 누수의 원인 찾기

---

Rails 코드에서 메모리 누수를 발견했습니다. 즉, 어떤 코드가 누수 되는지 는 찾았 지만 왜 누수 되는지는 알아 냈습니다. Rails가 필요없는 테스트 케이스로 축소했습니다.

require 'csspool'
require 'ruby-mass'

def report
    puts 'Memory ' + `ps ax -o pid,rss | grep -E "^[[:space:]]*#{$$}"`.strip.split.map(&:to_i)[1].to_s + 'KB'
    Mass.print
end

report

# note I do not store the return value here
CSSPool::CSS::Document.parse(File.new('/home/jason/big.css'))

ObjectSpace.garbage_collect
sleep 1

report

ruby-mass를 사용하면 메모리에있는 모든 객체를 볼 수 있습니다. CSSPool을 기반으로 CSS 파서입니다 racc . /home/jason/big.css는 1.5MB CSS 파일 입니다.

결과는 다음과 같습니다.

Memory 9264KB

==================================================
 Objects within [] namespace
==================================================
  String: 7261
  RubyVM::InstructionSequence: 1151
  Array: 562
  Class: 313
  Regexp: 181
  Proc: 111
  Encoding: 99
  Gem::StubSpecification: 66
  Gem::StubSpecification::StubLine: 60
  Gem::Version: 60
  Module: 31
  Hash: 29
  Gem::Requirement: 25
  RubyVM::Env: 11
  Gem::Specification: 8
  Float: 7
  Gem::Dependency: 7
  Range: 4
  Bignum: 3
  IO: 3
  Mutex: 3
  Time: 3
  Object: 2
  ARGF.class: 1
  Binding: 1
  Complex: 1
  Data: 1
  Gem::PathSupport: 1
  IOError: 1
  MatchData: 1
  Monitor: 1
  NoMemoryError: 1
  Process::Status: 1
  Random: 1
  RubyVM: 1
  SystemStackError: 1
  Thread: 1
  ThreadGroup: 1
  fatal: 1
==================================================

Memory 258860KB

==================================================
 Objects within [] namespace
==================================================
  String: 7456
  RubyVM::InstructionSequence: 1151
  Array: 564
  Class: 313
  Regexp: 181
  Proc: 113
  Encoding: 99
  Gem::StubSpecification: 66
  Gem::StubSpecification::StubLine: 60
  Gem::Version: 60
  Module: 31
  Hash: 30
  Gem::Requirement: 25
  RubyVM::Env: 13
  Gem::Specification: 8
  Float: 7
  Gem::Dependency: 7
  Range: 4
  Bignum: 3
  IO: 3
  Mutex: 3
  Time: 3
  Object: 2
  ARGF.class: 1
  Binding: 1
  Complex: 1
  Data: 1
  Gem::PathSupport: 1
  IOError: 1
  MatchData: 1
  Monitor: 1
  NoMemoryError: 1
  Process::Status: 1
  Random: 1
  RubyVM: 1
  SystemStackError: 1
  Thread: 1
  ThreadGroup: 1
  fatal: 1
==================================================

당신이가는 메모리 볼 수있는 방법을 위로. 카운터 중 일부는 올라가지 만 CSSPool에 특정한 개체가 없습니다. 다음과 같은 참조가있는 객체를 검사하기 위해 ruby-mass의 "index"메소드를 사용했습니다.

Mass.index.each do |k,v|
    v.each do |id|
        refs = Mass.references(Mass[id])
        puts refs if !refs.empty?
    end
end

그러나 다시 말하지만 이것은 CSSPool과 관련된 어떤 것도 제공하지 않으며 보석 정보 등 만 제공합니다.

또한 "GC.stat"출력을 시도했습니다 ...

puts GC.stat
CSSPool::CSS::Document.parse(File.new('/home/jason/big.css'))
ObjectSpace.garbage_collect
sleep 1
puts GC.stat

결과:

{:count=>4, :heap_used=>126, :heap_length=>138, :heap_increment=>12, :heap_live_num=>50924, :heap_free_num=>24595, :heap_final_num=>0, :total_allocated_object=>86030, :total_freed_object=>35106}
{:count=>16, :heap_used=>6039, :heap_length=>12933, :heap_increment=>3841, :heap_live_num=>13369, :heap_free_num=>2443302, :heap_final_num=>0, :total_allocated_object=>3771675, :total_freed_object=>3758306}

내가 이해했듯이 객체가 참조되지 않고 가비지 수집이 발생하면 해당 객체를 메모리에서 지워야합니다. 그러나 그것은 여기서 일어나는 일이 아닌 것 같습니다.

C 레벨 메모리 누수에 대해서도 읽었으며 CSSPool은 C 코드를 사용하는 Racc를 사용하기 때문에 이것이 가능성이라고 생각합니다. Valgrind를 통해 코드를 실행했습니다.

valgrind --partial-loads-ok=yes --undef-value-errors=no --leak-check=full --fullpath-after= ruby leak.rb 2> valgrind.txt

결과는 여기에 있습니다 . 루비가 Valgrind가 이해하지 못하는 메모리로 일을한다는 것을 읽었 기 때문에 이것이 C 레벨 누출을 확인하는지 확실하지 않습니다.

사용 된 버전 :

• Ruby 2.0.0-p247 (내 Rails 앱이 실행되는 것입니다)
• Ruby 1.9.3-p392-ref (ruby-mass 테스트 용)
• 루비 질량 0.1.3
• CSSPool 4.0.0 from here
• CentOS 6.4 and Ubuntu 13.10

---

It looks like you are entering The Lost World here. I don’t think the problem is with c-bindings in racc either.

Ruby memory management is both elegant and cumbersome. It stores objects (named RVALUEs) in so-called heaps of size of approx 16KB. On a low level, RVALUE is a c-struct, containing a union of different standard ruby object representations.

So, heaps store RVALUE objects, which size is not more than 40 bytes. For such objects as String, Array, Hash etc. this means that small objects can fit in the heap, but as soon as they reach a threshold, an extra memory outside of the Ruby heaps will be allocated.

This extra memory is flexible; is will be freed as soon as an object became GC’ed. That’s why your testcase with big_string shows the memory up-down behaviour:

def report
  puts 'Memory ' + `ps ax -o pid,rss | grep -E "^[[:space:]]*#{$$}"`
          .strip.split.map(&:to_i)[1].to_s + 'KB'
end
report
big_var = " " * 10000000
report
big_var = nil 
report
ObjectSpace.garbage_collect
sleep 1
report
# ⇒ Memory 11788KB
# ⇒ Memory 65188KB
# ⇒ Memory 65188KB
# ⇒ Memory 11788KB

But the heaps (see GC[:heap_length]) themselves are not released back to OS, once acquired. Look, I’ll make a humdrum change to your testcase:

- big_var = " " * 10000000
+ big_var = 1_000_000.times.map(&:to_s)

And, voilá:

# ⇒ Memory 11788KB
# ⇒ Memory 65188KB
# ⇒ Memory 65188KB
# ⇒ Memory 57448KB

The memory is not released back to OS anymore, because each element of the array I introduced suits the RVALUE size and is stored in the ruby heap.

If you’ll examine the output of GC.stat after the GC was run, you’ll find that GC[:heap_used] value is decreased as expected. Ruby now has a lot of empty heaps, ready.

The summing up: I don’t think, the c code leaks. I think the problem is within base64 representation of huge image in your css. I have no clue, what’s happening inside parser, but it looks like the huge string forces the ruby heap count to increase.

Hope it helps.

---

Okay, I found the answer. I am leaving my other answer up because that information was very difficult to gather, it is related, and it could help someone else searching for a related issue.

Your problem, however, appears to be due to the fact that Ruby actually does not release memory back to the Operating System once it has acquired it.

Memory Allocation

While Ruby programmers do not often worry about memory allocation, sometimes the following question comes up:

Why did my Ruby process stay so big even after I’ve cleared all references to big objects? I’m /sure/ GC has run several times and freed my big objects and I’m not leaking memory.

A C programmer might ask the same question:

I free()-ed a lot of memory, why is my process still so big?

Memory allocation to user space from the kernel is cheaper in large chunks, thus user space avoids interaction with the kernel by doing more work itself.

User space libraries/runtimes implement a memory allocator (e.g.: malloc(3) in libc) which takes large chunks of kernel memory2 and divides them up into smaller pieces for user space applications to use.

Thus, several user space memory allocations may occur before user space needs to ask the kernel for more memory. Thus if you got a large chunk of memory from the kernel and are only using a small part of that, that large chunk of memory remains allocated.

Releasing memory back to the kernel also has a cost. User space memory allocators may hold onto that memory (privately) in the hope it can be reused within the same process and not give it back to the kernel for use in other processes. (Ruby Best Practices)

So, your objects may very well have been garbage collected and released back to Ruby's available memory, but because Ruby never gives back unused memory to the OS, the rss value for the process remains the same, even after garbage collection. This is actually by design. According to Mike Perham:

...And since MRI never gives back unused memory, our daemon can easily be taking 300-400MB when it’s only using 100-200.

It’s important to note that this is essentially by design. Ruby’s history is mostly as a command line tool for text processing and therefore it values quick startup and a small memory footprint. It was not designed for long-running daemon/server processes. Java makes a similar tradeoff in its client and server VMs.

---

This could be due to the "Lazy Sweeping" feature in Ruby 1.9.3 and above.

Lazy sweeping basically means that, during garbage collection, Ruby only "sweeps" away enough objects to create space for the new objects it needs to create. It does this because, while the Ruby garbage collector runs, nothing else does. This is known as "Stop the world" garbage collection.

Essentially, lazy sweeping reduces the time that Ruby needs to "stop the world." You can read more about lazy sweeping here.

What does your RUBY_GC_MALLOC_LIMIT environment variable look like?

Here is an excerpt from Sam Saffron's blog concerning lazy sweeping and the RUBY_GC_MALLOC_LIMIT:

The GC in Ruby 2.0 comes in 2 different flavors. We have a "full" GC that runs after we allocate more than our malloc_limit and a lazy sweep (partial GC) that will run if we ever run out of free slots in our heaps.

The lazy sweep takes less time than a full GC, however only performs a partial GC. It's goal is to perform a short GC more frequently thus increasing overall throughput. The world stops, but for less time.

The malloc_limit is set to 8MB out of the box, you can raise it by setting the RUBY_GC_MALLOC_LIMIT higher.

Is your RUBY_GC_MALLOC_LIMIT extremely high? Mine is set to 100000000 (100MB). The Default is around 8MB, but for rails apps they recommend it to be quite a bit higher. If yours is too high it could be preventing Ruby from deleting garbage objects, because it thinks it has plenty of room to grow.

---

Building on @mudasobwa's explanation, I finally tracked down the cause. The code in CSSPool was checking the very long data URI for escape sequences. It would call scan on the URI with a regexp that matched an escape sequence or a single character, map those results to unescape, and then join it back into a string. This was effectively allocating a string for every character in the URI. I modified it to gsub the escape sequences, which seems to have the same results (all the tests pass) and greatly reduces the ending memory used.

Using the same test case as originally posted (minus the Mass.print output) this is the result before the change:

Memory 12404KB
Memory 292516KB

and this is the result after the change:

Memory 12236KB
Memory 19584KB

ReferenceURL : https://stackoverflow.com/questions/20385767/finding-the-cause-of-a-memory-leak-in-ruby