After Reboot - docker-compose is throwing OOM-Killer

Question

I have a single board computer running. (1GByte - RAM) I started 11 different Docker containers via a Docker compose file and this worked! To be clear. I can docker-compose down - docker-compose up at any time. No Problem.

---

This is the docker stats for overview names replaced with xxx

CONTAINER ID        NAME    CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             
aabfe39a92f7        xxx     0.00%               5.055MiB / 980.7MiB     0.52%               24.7kB / 25.4kB    
8d48601b0347        xxx     0.00%               2.988MiB / 980.7MiB     0.30%               23.3kB / 24.1kB     
c02b231ac2e4        xxx     99.63%              3.141MiB / 980.7MiB     0.32%               23.6kB / 24.4kB     
f7644e87fdf9        xxx     0.00%               3.27MiB / 980.7MiB      0.33%               23.5kB / 24.1kB     
23961cfd579b        xxx     0.00%               3.066MiB / 980.7MiB     0.31%               24.4kB / 25.8kB    
f6681673e839        xxx     0.00%               3.238MiB / 980.7MiB     0.33%               42.3kB / 46.8kB    
6c362ec43ebd        xxx     0.04%               2.965MiB / 980.7MiB     0.30%               24.7kB / 24.6kB     
0424597b9ec9        xxx     0.07%               3.051MiB / 980.7MiB     0.31%               25.2kB / 24.8kB    
b249062c3ef1        xxx     0.10%               32.83MiB / 980.7MiB     3.35%               16.5kB / 12.9kB     
be6357b5cc6f        xxx     0.22%               30.27MiB / 980.7MiB     3.09%               20.6kB / 13.9kB     
1b63c0788f8b        xxx     0.04%               1.008MiB / 980.7MiB     0.10%               144kB / 73.4kB

---

Now i added a restart-policy (always OR unless-stopped)

However, I get the following error message when my device reboots.

[   32.329313] audit: type=1300 audit(1607527641.136:92): arch=c00000b7 syscall=206 success=yes exit=32 a0=c a1=4000e27b20 a2=20 a3=0 items=0 ppid=1 pid=3718 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" key=(null)
[   32.356160] audit: type=1327 audit(1607527641.136:92): proctitle=2F7573722F62696E2F646F636B657264002D67002F646174612F646F636B6572002D480066643A2F2F
[   34.963417] exe invoked oom-killer: gfp_mask=0x26080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=-500
[   35.007927] exe cpuset=/ mems_allowed=0
[   35.011944] CPU: 1 PID: 5039 Comm: exe Tainted: G           O    4.9.51-1.1.0+ga523a59977a6 #1
[   35.020560] Hardware name: Freescale i.MX8MQ CloudplugEdge-Plus V1.2 (DT)

---

[   35.187065] Mem-Info:
[   35.269260] active_anon:178030 inactive_anon:18828 isolated_anon:0
[   35.269260]  active_file:128 inactive_file:148 isolated_file:22
[   35.269260]  unevictable:0 dirty:0 writeback:0 unstable:0
[   35.269260]  slab_reclaimable:4098 slab_unreclaimable:12028
[   35.269260]  mapped:9970 shmem:20408 pagetables:2000 bounce:0
[   35.269260]  free:16178 free_pcp:145 free_cma:10572
[   35.305050] Node 0 active_anon:712120kB inactive_anon:75312kB active_file:364kB inactive_file:252kB unevictable:0kB isolated(anon):0kB isolated(file):88kB mapped:39556kB dirty:0kB writeback:0kB shmem:81632kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 268288kB writeback_tmp:0kB unstable:0kB pages_scanned:44 all_unreclaimable? no
[   35.334821] DMA free:55912kB min:22528kB low:28160kB high:33792kB active_anon:714924kB inactive_anon:75312kB active_file:488kB inactive_file:4808kB unevictable:0kB writepending:0kB present:1048576kB managed:1004212kB mlocked:0kB slab_reclaimable:16400kB slab_unreclaimable:48112kB kernel_stack:8448kB pagetables:8004kB bounce:0kB free_pcp:712kB local_pcp:220kB free_cma:33192kB
[   35.372993] lowmem_reserve[]: 0 0 0
[   35.482077] DMA: 503*4kB (UMEHC) 252*8kB (UMEHC) 145*16kB (UEHC) 79*32kB (UMEHC) 21*64kB (UMEHC) 5*128kB (EC) 5*256kB (UMEC) 4*512kB (UMEH) 1*1024kB (E) 14*2048kB (MEC) 1*4096kB (C) 0*8192kB 0*16384kB 0*32768kB = 47980kB
[   35.586488] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[   35.609213] 22216 total pagecache pages
[   35.614348] 0 pages in swap cache
[   35.617931] Swap cache stats: add 0, delete 0, find 0/0
[   35.641266] Free swap  = 0kB
[   35.651498] Total swap = 0kB
[   35.662283] 262144 pages RAM
[   35.685127] 0 pages HighMem/MovableOnly
[   35.831080] 11091 pages reserved
[   35.871178] 81920 pages cma reserved
[   35.885236] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[   36.000916] [ 1842]     0  1842     8488     1262      17       3        0             0 systemd-journal
[   36.061116] [ 1867]     0  1867    74580      578      13       5        0             0 rngd
[   36.104337] [ 2599]     0  2599     3505      219       8       4        0         -1000 systemd-udevd
[   36.155494] [ 2654]     0  2654     3439      155      10       4        0             0 systemd-udevd
[   36.215883] [ 2655]     0  2655     3439      157       9       4        0             0 systemd-udevd
[   36.316312] [ 2656]     0  2656     4709      153      10       4        0             0 systemd-udevd
[   36.410753] [ 2657]     0  2657     3439      155      10       4        0             0 systemd-udevd
[   36.454639] [ 2658]     0  2658     3439      153      10       4        0             0 systemd-udevd
[   36.493610] [ 2660]     0  2660     3439      161       9       4        0             0 systemd-udevd
[   36.504237] [ 2661]     0  2661     3439      155      10       4        0             0 systemd-udevd
[   36.524043] [ 2662]     0  2662     3439      155      10       4        0             0 systemd-udevd
[   36.533427] [ 2663]     0  2663     3439      187       9       4        0             0 systemd-udevd
[   36.543217] [ 2664]     0  2664     3439      153       9       4        0             0 systemd-udevd
[   36.552995] [ 3210]   994  3210    20374       90      10       3        0             0 systemd-timesyn
[   36.562790] [ 3221]   995  3221     1892      103       8       3        0             0 systemd-resolve
[   36.572643] [ 3281]   998  3281     1163      170       6       4        0          -900 dbus-daemon
[   36.582324] [ 3295]     0  3295     1249       25       6       3        0             0 agetty
[   36.591232] [ 3296]     0  3296     1830      108      10       4        0             0 systemd-logind
[   36.600859] [ 3324]     0  3324     1983      198       7       4        0             0 ofonod
[   36.609715] [ 3345]     0  3345     1249       25       6       3        0             0 agetty
[   36.618549] [ 3351]     0  3351    61812      740      22       4        0             0 NetworkManager
[   36.628046] [ 3375]     0  3375      797      160       6       4        0             0 klogd
[   36.636773] [ 3401]     0  3401    78286      367      21       4        0             0 ModemManager
[   36.646095] [ 3436]     0  3436      797       23       6       3        0             0 syslogd
[   36.654991] [ 3456]     0  3456   139516     1129      26       4        0             0 mender
[   36.663793] [ 3468]   999  3468     1242      120       6       4        0             0 avahi-daemon
[   36.673182] [ 3494]   999  3494     1177       60       6       4        0             0 avahi-daemon
[   36.682621] [ 3498]     0  3498     1044       45       7       3        0             0 weston-launch
[   36.692105] [ 3533]     0  3533    52979      890      39       3        0             0 weston
[   36.700990] [ 3534]     0  3534     1780       70       7       3        0             0 systemd-hostnam
[   36.710557] [ 3566]     0  3566     1779       70       8       4        0             0 systemd-rfkill
[   36.720151] [ 3593]     0  3593    56214     4644      46       4        0             0 weston-keyboard
[   36.729792] [ 3602]     0  3602    53226     1749      40       3        0             0 weston-desktop-
[   36.739430] [ 3640] 65534  3640      571       32       5       3        0             0 dnsmasq
[   36.748388] [ 3702]     0  3702     1579       35       7       3        0             0 mender-inventor
[   36.758042] [ 3703]     0  3703     1579       56       5       3        0             0 mender-inventor
[   36.767915] [ 3704]     0  3704   187930     7199      55       6        0             0 docker
[   36.776799] [ 3705]     0  3705      797       19       5       4        0             0 awk
[   36.785406] [ 3706]     0  3706      797       20       6       4        0             0 grep
[   36.794109] [ 3707]     0  3707      797       19       6       4        0             0 tr
[   36.802630] [ 3708]     0  3708      797       19       5       4        0             0 sed
[   36.811252] [ 3718]     0  3718   629571    13552     120       7        0          -500 dockerd
[   36.820203] [ 3747]     0  3747   478477     7659      81       6        0          -500 containerd
[   36.829404] [ 4194]     0  4194   119090      892      20       5        0          -500 docker-proxy
[   36.838808] [ 4200]     0  4200   174451     1118      26       5        0          -999 containerd-shim
[   36.848402] [ 4238]     0  4238   192884     1089      28       4        0          -999 containerd-shim
[   36.858178] [ 4254]     0  4254   119580     1632      23       4        0          -999 runc
[   36.866919] [ 4256]     0  4256   174387     1090      26       5        0          -999 containerd-shim
[   36.876561] [ 4288]     0  4288   155956     1405      23       5        0          -500 docker-proxy
[   36.885935] [ 4289]     0  4289   119228     1615      21       4        0          -999 runc
[   36.894704] [ 4337]     0  4337    76418     1062      16       2        0             0 java
[   36.903356] [ 4350]     0  4350   100795     1219      18       4        0             0 runc:[2:INIT]
[   36.912837] [ 4365]     0  4365   174451     1089      26       4        0          -999 containerd-shim
[   36.922412] [ 4390]     0  4390   174879     1106      27       5        0          -999 runc
[   36.931068] [ 4399]     0  4399   100731     1220      18       4        0             0 runc:[2:INIT]
[   36.940468] [ 4419]     0  4419   172667     9183      61       5        0          -999 exe
[   36.949037] [ 4427]     0  4427   137523      893      21       5        0          -500 docker-proxy
[   36.958343] [ 4428]     0  4428   172667     8670      61       5        0          -999 exe
[   36.966869] [ 4433]     0  4433   174803     1089      28       5        0          -999 containerd-shim
[   36.976415] [ 4507]     0  4507   119164     1899      19       4        0             0 runc:[2:INIT]
[   36.985788] [ 4516]     0  4516   156370     2113      25       5        0          -999 containerd-shim
[   36.995354] [ 4555]     0  4555   138013     1634      23       5        0          -999 runc
[   37.004021] [ 4585]     0  4585   154234     9187      59       4        0          -999 exe
[   37.012561] [ 4594]     0  4594   174451     1088      26       5        0          -999 containerd-shim
[   37.022102] [ 4595]     0  4595   138013     1107      23       4        0          -999 runc
[   37.030847] [ 4611]     0  4611   174451     2127      28       5        0          -999 containerd-shim
[   37.040568] [ 4627]     0  4627   174527     1109      27       4        0          -999 runc
[   37.049271] [ 4707]     0  4707   174451     1091      26       5        0          -999 containerd-shim
[   37.058960] [ 4720]     0  4720   119580     1898      22       4        0             0 runc:[2:INIT]
[   37.068355] [ 4736]     0  4736   100795     1220      19       4        0             0 ru

Answer 1

I found the problem myself. In my Docker compose file, I specified an order of containers via "depends_on". This order prevented my memory from filling up at once. However, after a reboot/crash the "restart:always" function kicked in and started all 11 containers at once.

Note: As docker-compose I also used a docker container (aarch64 - problems) which could possibly lead to this problem.