django - Out of memory: Kill process (gunicorn) score or sacrifice child -
i'm running django 1.8 project on digitalocean vps (512 mb ram, 1 cpu, 20gb ssd). have nginx proxying traffic gunicorn. here gunicorn command (run via supervisor):
gunicorn my_web_app.wsgi:application --worker-class gevent --bind 127.0.0.1:8001 --timeout=1200 i noticed when upload ~3-5 mb image web app, gunicorn worker crashes error:
jan 16 12:39:46 dev-1 kernel: [663264.917312] out of memory: kill process 31093 (gunicorn) score 589 or sacrifice child jan 16 12:39:46 dev-1 kernel: [663264.917416] killed process 31093 (gunicorn) total-vm:560020kb, anon-rss:294888kb, file-rss:8kb i monitored output top, shows memory usage steadily increasing:
top output before upload image (baseline):
top - 13:19:45 7 days, 16:54, 2 users, load average: 0.00, 0.03, 0.05 tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie %cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st kib mem: 501780 total, 298384 used, 203396 free, 17112 buffers kib swap: 0 total, 0 used, 0 free. 72048 cached mem pid user pr ni virt res shr s %cpu %mem time+ command 902 my_user 20 0 145336 32332 3900 s 0.0 6.4 0:01.00 gunicorn about minute upload (half way mark):
top - 13:22:00 7 days, 16:56, 2 users, load average: 0.05, 0.03, 0.05 tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie %cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st kib mem: 501780 total, 313976 used, 187804 free, 18100 buffers kib swap: 0 total, 0 used, 0 free. 77196 cached mem pid user pr ni virt res shr s %cpu %mem time+ command 902 my_user 20 0 171492 40076 5352 s 0.0 8.0 0:01.33 gunicorn moments before gunicorn crashing:
top - 13:23:14 7 days, 16:57, 2 users, load average: 0.19, 0.07, 0.06 tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie %cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st kib mem: 501780 total, 341836 used, 159944 free, 18236 buffers kib swap: 0 total, 0 used, 0 free. 90228 cached mem pid user pr ni virt res shr s %cpu %mem time+ command 902 my_user 20 0 239184 52492 5568 r 80.9 10.5 0:01.65 gunicorn and finally, moment of crash:
top - 13:23:15 7 days, 16:57, 2 users, load average: 0.19, 0.07, 0.06 tasks: 99 total, 4 running, 95 sleeping, 0 stopped, 0 zombie %cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st kib mem: 501780 total, 495800 used, 5980 free, 176 buffers kib swap: 0 total, 0 used, 0 free. 31564 cached mem pid user pr ni virt res shr s %cpu %mem time+ command 902 my_user 20 0 545520 284012 5264 r 80.1 56.6 0:02.74 gunicorn what causing gunicorn worker spiking in memory usage of > 200mb, though i'm uploading < 5mb file?
what gunicorn server doing image once receives it? once encountered specific corrupt jpeg image triggered memory leak in (now old) version of imagemagick crashed servers.
you can test if issue gunicorn vs code creating cli entry point same code accepts file command line argument. if still memory leak loading file cli, issue application code.
if don't memory leak in cli test, issue gunicorn configuration.
also, go ahead add swap machine safety buffer. add 1 gig. instructions adding swap on digital ocean. doing may allow task finish , practice anyway.
Comments
Post a Comment