text:
Vespa proton taking 20GB upon start-up

Describe the bug
Vespa proton taking 20GB upon start-up with zero documents
To Reproduce
Steps to reproduce the behavior:
deploy attached schema. We have total 39 schemas.
Expected behavior
Expect to use minimal memory with zero documents.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):

OS: linux
Infrastructure: self-hosted via podman containers
Podman Version: 4.2.0

Vespa version
dockerhub.site.gs.com:8413/vespaengine/vespa:8.82.24
Additional context
See attached
1. services.xml
services.xml.txt
2. session logs -
[vespa@content-5 /]$ kill -SIGPROF 452
[vespa@content-5 /]$ vespa-logfmt -S searchnode
[2023-03-07 15:12:19.165] WARNING searchnode       stderr       SignalHandler 27 caught
[2023-03-07 15:12:19.166] WARNING searchnode       stderr       DataSegment at 0x7f91b49942f8(8389832), AllocPool at 0x7f91b51947c0(2912), ThreadList at 0x7f91b51953a0(9568288)
[2023-03-07 15:12:19.166] WARNING searchnode       stderr       Start at 0x10000000000, End at 0x104d9c00000(0xf83000) size(20833107968) partialExtension(0) NextLogLimit(11000000000) logLevel(2)
[2023-03-07 15:12:19.166] WARNING searchnode       stderr       SizeClass -1(SYSTEM) has    88 blocks with  184549376 bytes
[2023-03-07 15:12:19.167] WARNING searchnode       stderr       SizeClass  0 has   125 blocks with  262144000 bytes
[2023-03-07 15:12:19.167] WARNING searchnode       stderr       SizeClass  1 has    15 blocks with   31457280 bytes
[2023-03-07 15:12:19.167] WARNING searchnode       stderr       SizeClass  2 has   104 blocks with  218103808 bytes
[2023-03-07 15:12:19.167] WARNING searchnode       stderr       SizeClass  3 has    17 blocks with   35651584 bytes
[2023-03-07 15:12:19.167] WARNING searchnode       stderr       SizeClass  4 has    30 blocks with   62914560 bytes
[2023-03-07 15:12:19.168] WARNING searchnode       stderr       SizeClass  5 has    17 blocks with   35651584 bytes
[2023-03-07 15:12:19.168] WARNING searchnode       stderr       SizeClass  6 has    15 blocks with   31457280 bytes
[2023-03-07 15:12:19.168] WARNING searchnode       stderr       SizeClass  7 has    92 blocks with  192937984 bytes
[2023-03-07 15:12:19.168] WARNING searchnode       stderr       SizeClass  8 has   137 blocks with  287309824 bytes
[2023-03-07 15:12:19.168] WARNING searchnode       stderr       SizeClass  9 has   343 blocks with  719323136 bytes
[2023-03-07 15:12:19.169] WARNING searchnode       stderr       SizeClass 10 has   424 blocks with  889192448 bytes
[2023-03-07 15:12:19.169] WARNING searchnode       stderr       SizeClass 11 has    14 blocks with   29360128 bytes
[2023-03-07 15:12:19.169] WARNING searchnode       stderr       SizeClass 12 has   461 blocks with  966787072 bytes
[2023-03-07 15:12:19.169] WARNING searchnode       stderr       SizeClass 13 has  4356 blocks with 9135194112 bytes
[2023-03-07 15:12:19.169] WARNING searchnode       stderr       SizeClass 14 has    37 blocks with   77594624 bytes
[2023-03-07 15:12:19.170] WARNING searchnode       stderr       SizeClass 15 has     3 blocks with    6291456 bytes
[2023-03-07 15:12:19.170] WARNING searchnode       stderr       SizeClass 16 has  3656 blocks with 7667187712 bytes
[2023-03-07 15:12:19.170] WARNING searchnode       stderr       Usage: Total=20833107968(100%), admin=184549376(0%), unused=0(0%), allocated=20648558592(99%)
[2023-03-07 15:12:19.170] WARNING searchnode       stderr       GlobalPool getChunks(19728, 174036) allocChunksList(22):
[2023-03-07 15:12:19.171] WARNING searchnode       stderr       SC  0(        32) GetAlloc(   563) GetFree(   648) ExChangeAlloc( 73436) ExChangeFree(  8729) ExactAlloc(     0) Returned(     0) Malloc(   125)
[2023-03-07 15:12:19.171] WARNING searchnode       stderr       SC  1(        64) GetAlloc(    15) GetFree(   648) ExChangeAlloc( 76423) ExChangeFree( 72936) ExactAlloc(     0) Returned(     0) Malloc(    15)
[2023-03-07 15:12:19.171] WARNING searchnode       stderr       SC  2(       128) GetAlloc(   107) GetFree(   648) ExChangeAlloc(107710) ExChangeFree( 94530) ExactAlloc(     0) Returned(     0) Malloc(   104)
[2023-03-07 15:12:19.171] WARNING searchnode       stderr       SC  3(       256) GetAlloc(    21) GetFree(   648) ExChangeAlloc( 36957) ExChangeFree( 35938) ExactAlloc(     0) Returned(     0) Malloc(    17)
[2023-03-07 15:12:19.171] WARNING searchnode       stderr       SC  4(       512) GetAlloc(    35) GetFree(   648) ExChangeAlloc(  1028) ExChangeFree(    88) ExactAlloc(     0) Returned(     0) Malloc(    30)
[2023-03-07 15:12:19.172] WARNING searchnode       stderr       SC  5(      1024) GetAlloc(    17) GetFree(   648) ExChangeAlloc(   533) ExChangeFree(     8) ExactAlloc(     0) Returned(     0) Malloc(    17)
[2023-03-07 15:12:19.172] WARNING searchnode       stderr       SC  6(      2048) GetAlloc(    21) GetFree(   648) ExChangeAlloc(   594) ExChangeFree(   121) ExactAlloc(     0) Returned(     0) Malloc(    15)
[2023-03-07 15:12:19.172] WARNING searchnode       stderr       SC  7(      4096) GetAlloc(    93) GetFree(   648) ExChangeAlloc(  7769) ExChangeFree(  4905) ExactAlloc(     0) Returned(     0) Malloc(    92)
[2023-03-07 15:12:19.172] WARNING searchnode       stderr       SC  8(      8192) GetAlloc(   156) GetFree(   648) ExChangeAlloc(  4585) ExChangeFree(   242) ExactAlloc(     0) Returned(     0) Malloc(   137)
[2023-03-07 15:12:19.172] WARNING searchnode       stderr       SC  9(     16384) GetAlloc(   550) GetFree(   648) ExChangeAlloc( 11283) ExChangeFree(   323) ExactAlloc(     0) Returned(     0) Malloc(   343)
[2023-03-07 15:12:19.173] WARNING searchnode       stderr       SC 10(     32768) GetAlloc(   652) GetFree(   648) ExChangeAlloc( 14244) ExChangeFree(   691) ExactAlloc(     0) Returned(     0) Malloc(   424)
[2023-03-07 15:12:19.173] WARNING searchnode       stderr       SC 11(     65536) GetAlloc(    14) GetFree(   648) ExChangeAlloc( 10524) ExChangeFree( 10105) ExactAlloc(     0) Returned(     0) Malloc(    14)
[2023-03-07 15:12:19.173] WARNING searchnode       stderr       SC 12(    131072) GetAlloc(  1575) GetFree(   648) ExChangeAlloc(  9725) ExChangeFree(  2368) ExactAlloc(     0) Returned(     0) Malloc(   461)
[2023-03-07 15:12:19.173] WARNING searchnode       stderr       SC 13(    262144) GetAlloc(  7938) GetFree(   648) ExChangeAlloc( 40135) ExChangeFree(  5292) ExactAlloc(     0) Returned(     0) Malloc(  4356)
[2023-03-07 15:12:19.173] WARNING searchnode       stderr       SC 14(    524288) GetAlloc(    42) GetFree(   648) ExChangeAlloc(   746) ExChangeFree(   604) ExactAlloc(     0) Returned(     0) Malloc(    37)
[2023-03-07 15:12:19.174] WARNING searchnode       stderr       SC 15(   1048576) GetAlloc(     3) GetFree(   648) ExChangeAlloc(   521) ExChangeFree(   519) ExactAlloc(     0) Returned(     0) Malloc(     3)
[2023-03-07 15:12:19.174] WARNING searchnode       stderr       SC 16(   2097152) GetAlloc(     0) GetFree(   648) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(  3695) Returned(    39) Malloc(     0)
[2023-03-07 15:12:19.174] WARNING searchnode       stderr       SC 17(   4194304) GetAlloc(     0) GetFree(   648) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   156) Returned(   156) Malloc(     0)
[2023-03-07 15:12:19.174] WARNING searchnode       stderr       #324 active threads. Peak threads #323. 1145 threads created in total.
[2023-03-07 15:12:19.174] WARNING searchnode       stderr       SignalHandler 27 done
[vespa@content-5 /]$ vespa-proton-cmd --local getState
Wanted 262144 as limit for open files but cannot exceed current hard limit: 65536
Wanted 409600 as limit for max user processes but cannot exceed current hard limit: 65536
[2023-03-07 15:14:51.264368] 1820/64947 (vespa-proton-cmd.slobrok.mirror) info: successfully connected to location broker tcp/config-1.gs.com:19099 (mirror initialized with 108 service names)
looking for RTCs matching 'tcp/content-5.gs.com:' (length 26)
found local RTC 'lexdoc/search/cluster.lexdoc/5/realtimecontroller' with connection spec tcp/content-5.gs.com:19103
"online", "true"
"onlineState", "online"
"delayedConfigs", ""
"onlineDocs", "0"
gencnt=0
[vespa@content-5 /]$ vespa-proton-cmd --local getProtonStatus
Wanted 262144 as limit for open files but cannot exceed current hard limit: 65536
Wanted 409600 as limit for max user processes but cannot exceed current hard limit: 65536
[2023-03-07 15:16:11.532904] 1837/52962 (vespa-proton-cmd.slobrok.mirror) info: successfully connected to location broker tcp/config-0.gs.com:19099 (mirror initialized with 108 service names)
looking for RTCs matching 'tcp/content-5.gs.com:' (length 26)
found local RTC 'lexdoc/search/cluster.lexdoc/5/realtimecontroller' with connection spec tcp/content-5.gs.com:19103
"matchengine","OK","state=ONLINE",""
"documentdb:DocumentType1","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType2","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType3","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType4","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType5","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType6","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType7","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType8","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType9","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType10","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType11","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType12","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType13","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType14","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType15","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType16","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType17","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType18","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType19","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType20","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType21","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType22","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType23","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType24","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType25","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType26","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType27","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType28","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType29","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType30","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType31","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType32","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType33","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType34","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType35","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType36","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType37","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType38","OK","state=ONLINE configstate=OK",""
"documentdb:DocumentType39","OK","state=ONLINE configstate=OK",""

Having 40 document types does explain a bit.
I also guess that you have quite a few fields in each document type.
There is a fixed cost associated with every document type and every field.
For all usages I have seen this has been a very small and ignorable quantity. The data would normally be 100x that of the fixed memory cost.
Also note that there is reserved memory for a minimal set of documents in every field. I wonder if that is 1k.
You should be able to track this memory usage looking.
You can inspect this by looking at the metrics of the searchnode.
When I just started a simple test and checked the metrics
content.proton.documentdb.memory_usage.used_bytes and
content.proton.documentdb.memory_usage.allocated_bytes
I saw that it had preallocated 8M for a single document type while it had used only 7k.
You have similar metrics for each field too, so you can drill down and see.
If you gather these metrics and provide some numbers here we can see if there is something that can be done to reduce this cost.
You have 39 document types.
How many fields do they have, and of which type. What is the allocated_bytes for each unique type ?
a unique type defined by:

type of field
fast-search yes/no.

      {
        "name": "content.proton.documentdb.memory_usage.allocated_bytes",
        "description": "The number of allocated bytes",
        "values": {
          "average": 8139960.0,
          "sum": 97679520.0,
          "count": 12,
          "rate": 0.2,
          "min": 8139960,
          "max": 8139960,
          "last": 8139960
        },
        "dimensions": {
          "documenttype": "test"
        }
      },
      {
        "name": "content.proton.documentdb.memory_usage.used_bytes",
        "description": "The number of used bytes (<= allocated_bytes)",
        "values": {
          "average": 7124.0,
          "sum": 85488.0,
          "count": 12,
          "rate": 0.2,
          "min": 7124,
          "max": 7124,
          "last": 7124
        },
        "dimensions": {
          "documenttype": "test"
        }
      },

There is also a rest api where you can drill down and inspect usage.
See https://docs.vespa.ai/en/proton.html#custom-component-state-api
memory usage of the attributes for a document type can be inspected at
/state/v1/custom/component/documentdb//subdb/ready/attribute

... or maybe deploy to Vespa Cloud and use the Memory Visualizer

sum of content.proton.documentdb.memory_usage.allocated_bytes for all 39 document type is 1,405,979,608 bytes which is far off from 20GB. Note this number is exactly equal to content_proton_documentdb_memory_usage_allocated_bytes_last from /prometheus/v1/values.
I also drilled down memory used by individual attributes, documentstore, index using state/v1/custom/component/documentdb/{doctype}/subdb/ API and they all roll up to content.proton.documentdb.memory_usage.allocated_bytes.
So I suspect it's something else that is taking up the memory apart from initial fixed cost.

Just one observation - 20GB is very close 1.4GB X 14 nodes that I have in content cluster. May be it's over allocating memory on each node?
reference data:
content.proton.documentdb.memory_usage.allocated_bytes = 1.4GB per node
total nodes in content cluster = 14

Yes, and no. There is definitely something unaccounted for, but it is still fixed cost.
I have identified quite a bit of memory that set up when constructing a field, but accounted when reporting memory.
But I would like to know the number of fields you have configured, categorized as specified in my previous comment.
I need that in order to if it adds up.
I do not need it broken down on document type, just the aggregated numbers over all of them.

The numbers adds up. more accurate reporting will be part of vespa-8.136 that will released next week.
An effort to reduce memory usage has been started and will happen this month.

Any estimate how much it would be for above schemas?

after reporting fix
after memory reduction effort

Once reporting is more accurate there wil be no change in memory usage. The only change is that reported usage will be closer to  actual usage.
Can not give an estimate as I do not yet know how much can be cut. I guess is that consumption will be 1/4 afterwards.

vespa-8.136.37 contains the changes for more accurate reporting of static memory usage.
Now the accounted and reported memory usage should be closer to actual usage. Not perfect, but better.
@mohsin36 I would be good if you could install this version and see how close you get.

vespa-8.137.25, not yet released, contains a major improvement in static memory usage. If release pipeline and staging goes well it should be available in a day or 2.
@mohsin36 for your use case I expect you to drop from 20G, down below 10G, hopefully even further.

With vespa-8.136.37 it content_proton_documentdb_memory_usage_allocated_bytes_last has changed to 12,413,714,988 bytes from pervious value of 1,405,979,608 bytes.
At a high level, our goal is to optimize schemas and attribute fields to reduce this initial fixed cost further. Can you explain what exactly constitute this initial fixed cost? I know that it is related to number of schemas and number/type of attribute fields. Can we roughly quantify this relation with simple math?

Is the memory footprint still 20G. If so there are still 7-8G that are not properly accounted for, not good.
If you install latest which is either 8.137.25, or 8.139.1 that is in the press, you should see a significantly lower memory footprint. content_proton_documentdb_memory_usage_allocated_bytes_last should drop from 12.5G to around 5G.
However I am mostly interested in finding out where the unaccounted memory is used. So once you have upgraded I would like to see the log after another kill -SIGPROF.
The fixed cost are structures for

efficient memory usage for large document sets.
lock free instant updates.
efficient search

Roughly the static cost for a field can be split in these groups.
1 - Single value numeric attribute     = 10K
2 - Single value string attribute = 300K
3 - Single value numeric attribute with fast-search = 350K
4 - Single value string attribute with fast-search = 400K
5 - Multi value numeric attribute     = 550K
6 - Multi value string attribute = 800K
7 - Multi value numeric attribute with fast-search = 850K
8 - Multi value string attribute with fast-search = 900K
9 - Index fields = 400K

With 8.137.25 image we see memory consumption has dropped to 5,932,843,008 bytes. content_proton_documentdb_memory_usage_allocated_bytes_last is reporting 4,289,615,004 bytes. It's much better then earlier, however still off by 1.53 GB
Stats:
[2023-03-16 14:08:46.329] WARNING searchnode       stderr       SignalHandler 27 caught
[2023-03-16 14:08:46.329] WARNING searchnode       stderr       DataSegment at 0x7f9a206492d8(8389832), AllocPool at 0x7f9a20e497a0(2912), ThreadList at 0x7f9a20e4a380(9568288)
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       Start at 0x10000000000, End at 0x10161a00000(0x18be000) size(5932843008) partialExtension(0) NextLogLimit(11000000000) logLevel(2)
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       Free #  0 block # 2825 chainlength     1 size    2097152
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       Free #  1 block # 2827 chainlength     2 size    4194304
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       SizeClass -4(UNMAPPED) has     3 blocks with    6291456 bytes
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       SizeClass -1(SYSTEM) has    80 blocks with  167772160 bytes
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       SizeClass  0 has   125 blocks with  262144000 bytes
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       SizeClass  1 has    12 blocks with   25165824 bytes
[2023-03-16 14:08:46.330] WARNING searchnode       stderr       SizeClass  2 has    96 blocks with  201326592 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  3 has    32 blocks with   67108864 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  4 has    31 blocks with   65011712 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  5 has    16 blocks with   33554432 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  6 has    14 blocks with   29360128 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  7 has   450 blocks with  943718400 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  8 has   135 blocks with  283115520 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass  9 has    72 blocks with  150994944 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass 10 has   966 blocks with 2025848832 bytes
[2023-03-16 14:08:46.331] WARNING searchnode       stderr       SizeClass 11 has    37 blocks with   77594624 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SizeClass 12 has   233 blocks with  488636416 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SizeClass 13 has   486 blocks with 1019215872 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SizeClass 14 has    37 blocks with   77594624 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SizeClass 15 has     3 blocks with    6291456 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SizeClass 16 has     1 blocks with    2097152 bytes
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       Usage: Total=5932843008(100%), admin=174063616(2%), unused=0(0%), allocated=5758779392(97%)
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       GlobalPool getChunks(17120, 160193) allocChunksList(20):
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SC  0(        32) GetAlloc(   665) GetFree(   646) ExChangeAlloc( 70169) ExChangeFree(  5538) ExactAlloc(     0) Returned(     0) Malloc(   125)
[2023-03-16 14:08:46.332] WARNING searchnode       stderr       SC  1(        64) GetAlloc(    12) GetFree(   646) ExChangeAlloc( 44816) ExChangeFree( 42108) ExactAlloc(     0) Returned(     0) Malloc(    12)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  2(       128) GetAlloc(    98) GetFree(   646) ExChangeAlloc( 90903) ExChangeFree( 78617) ExactAlloc(     0) Returned(     0) Malloc(    96)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  3(       256) GetAlloc(    32) GetFree(   646) ExChangeAlloc( 25495) ExChangeFree( 23598) ExactAlloc(     0) Returned(     0) Malloc(    32)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  4(       512) GetAlloc(    49) GetFree(   646) ExChangeAlloc(  1055) ExChangeFree(    83) ExactAlloc(     0) Returned(     0) Malloc(    31)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  5(      1024) GetAlloc(    16) GetFree(   646) ExChangeAlloc(   532) ExChangeFree(    25) ExactAlloc(     0) Returned(     0) Malloc(    16)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  6(      2048) GetAlloc(    31) GetFree(   646) ExChangeAlloc(   599) ExChangeFree(   155) ExactAlloc(     0) Returned(     0) Malloc(    14)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  7(      4096) GetAlloc(   804) GetFree(   646) ExChangeAlloc( 14657) ExChangeFree(   286) ExactAlloc(     0) Returned(     0) Malloc(   450)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  8(      8192) GetAlloc(   152) GetFree(   646) ExChangeAlloc(  4451) ExChangeFree(   141) ExactAlloc(     0) Returned(     0) Malloc(   135)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC  9(     16384) GetAlloc(   121) GetFree(   646) ExChangeAlloc(  2595) ExChangeFree(   305) ExactAlloc(     0) Returned(     0) Malloc(    72)
[2023-03-16 14:08:46.333] WARNING searchnode       stderr       SC 10(     32768) GetAlloc(  1855) GetFree(   646) ExChangeAlloc( 31723) ExChangeFree(   853) ExactAlloc(     0) Returned(     0) Malloc(   966)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 11(     65536) GetAlloc(    53) GetFree(   646) ExChangeAlloc( 20538) ExChangeFree( 19410) ExactAlloc(     0) Returned(     0) Malloc(    37)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 12(    131072) GetAlloc(   363) GetFree(   646) ExChangeAlloc(  5802) ExChangeFree(  2100) ExactAlloc(     0) Returned(     0) Malloc(   233)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 13(    262144) GetAlloc(   851) GetFree(   646) ExChangeAlloc(  5225) ExChangeFree(  1347) ExactAlloc(     0) Returned(     0) Malloc(   486)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 14(    524288) GetAlloc(    39) GetFree(   646) ExChangeAlloc(   607) ExChangeFree(   465) ExactAlloc(     0) Returned(     0) Malloc(    37)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 15(   1048576) GetAlloc(     3) GetFree(   646) ExChangeAlloc(   382) ExChangeFree(   380) ExactAlloc(     0) Returned(     0) Malloc(     3)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 16(   2097152) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   289) Returned(   288) Malloc(     0)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SC 17(   4194304) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   327) Returned(   327) Malloc(     0)
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       #323 active threads. Peak threads #322. 364 threads created in total.
[2023-03-16 14:08:46.334] WARNING searchnode       stderr       SignalHandler 27 done

Is the memory footprint still 20G. If so there are still 7-8G that are not properly accounted for, not good.
If you install latest which is either 8.137.25, or 8.139.1 that is in the press, you should see a significantly lower memory footprint. content_proton_documentdb_memory_usage_allocated_bytes_last should drop from 12.5G to around 5G. However I am mostly interested in finding out where the unaccounted memory is used. So once you have upgraded I would like to see the log after another kill -SIGPROF.
The fixed cost are structures for

efficient memory usage for large document sets.
lock free instant updates.
efficient search

Roughly the static cost for a field can be split in these groups. 1 - Single value numeric attribute = 10K 2 - Single value string attribute = 300K 3 - Single value numeric attribute with fast-search = 350K 4 - Single value string attribute with fast-search = 400K 5 - Multi value numeric attribute = 550K 6 - Multi value string attribute = 800K 7 - Multi value numeric attribute with fast-search = 850K 8 - Multi value string attribute with fast-search = 900K 9 - Index fields = 400K

How are below memory consumption related to number of schemas and number/type of attribute field? Can we quantify it?

efficient memory usage for large document sets.
lock free instant updates.
efficient search

On static cost for field, can you please explain how are these numbers determined? E.g. How do we derive 800K for multi value string attribute or 400K for index fields?

Those numbers are just a fixed cost when you add field of that type. It is as simple as summing the cost for all of your fields together. These are just the same number as you get when using the custom component rest api.

Good that accounted memory is now at 4.2G out of 5.9G. Better accounting has been added so 8.139.1 will be even closer.
The final improvements have been done and will be part of 8.140.13.
In 2-3 weeks time there will roll out a change that brings the memory consumption even further down.

8.140.13 is out. Please update with how that looks.

Stats for 8.140.13:
actual usage per content node - 5,746,196,480 bytes
content_proton_documentdb_memory_usage_allocated_bytes_last per content node = 3,956,474,008 bytes
Logs:
[2023-03-22 17:10:47.304] WARNING searchnode       stderr       SignalHandler 27 caught
[2023-03-22 17:10:47.304] WARNING searchnode       stderr       DataSegment at 0x7efed9c062d8(8389832), AllocPool at 0x7efeda4067a0(2912), ThreadList at 0x7efeda407380(9568288)
[2023-03-22 17:10:47.304] WARNING searchnode       stderr       Start at 0x10000000000, End at 0x10156800000(0x22d5000) size(5746196480) partialExtension(0) NextLogLimit(11000000000) logLevel(2)
[2023-03-22 17:10:47.304] WARNING searchnode       stderr       Free #  0 block # 2729 chainlength    11 size   23068672
[2023-03-22 17:10:47.304] WARNING searchnode       stderr       SizeClass -4(UNMAPPED) has    11 blocks with   23068672 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass -1(SYSTEM) has    80 blocks with  167772160 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  0 has   125 blocks with  262144000 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  1 has    12 blocks with   25165824 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  2 has    96 blocks with  201326592 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  3 has    32 blocks with   67108864 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  4 has    30 blocks with   62914560 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  5 has    16 blocks with   33554432 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  6 has    14 blocks with   29360128 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  7 has   451 blocks with  945815552 bytes
[2023-03-22 17:10:47.305] WARNING searchnode       stderr       SizeClass  8 has   131 blocks with  274726912 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass  9 has    43 blocks with   90177536 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 10 has   966 blocks with 2025848832 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 11 has    19 blocks with   39845888 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 12 has   232 blocks with  486539264 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 13 has   471 blocks with  987758592 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 14 has     7 blocks with   14680064 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 15 has     3 blocks with    6291456 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       SizeClass 16 has     1 blocks with    2097152 bytes
[2023-03-22 17:10:47.306] WARNING searchnode       stderr       Usage: Total=5746196480(100%), admin=190840832(3%), unused=0(0%), allocated=5555355648(96%)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       GlobalPool getChunks(16263, 157544) allocChunksList(20):
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  0(        32) GetAlloc(   659) GetFree(   646) ExChangeAlloc( 67815) ExChangeFree(  3283) ExactAlloc(     0) Returned(     0) Malloc(   125)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  1(        64) GetAlloc(    12) GetFree(   646) ExChangeAlloc( 62521) ExChangeFree( 59822) ExactAlloc(     0) Returned(     0) Malloc(    12)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  2(       128) GetAlloc(    96) GetFree(   646) ExChangeAlloc( 70271) ExChangeFree( 58112) ExactAlloc(     0) Returned(     0) Malloc(    96)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  3(       256) GetAlloc(    32) GetFree(   646) ExChangeAlloc( 15706) ExChangeFree( 13856) ExactAlloc(     0) Returned(     0) Malloc(    32)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  4(       512) GetAlloc(    46) GetFree(   646) ExChangeAlloc(  1022) ExChangeFree(    61) ExactAlloc(     0) Returned(     0) Malloc(    30)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  5(      1024) GetAlloc(    17) GetFree(   646) ExChangeAlloc(   523) ExChangeFree(    19) ExactAlloc(     0) Returned(     0) Malloc(    16)
[2023-03-22 17:10:47.307] WARNING searchnode       stderr       SC  6(      2048) GetAlloc(    34) GetFree(   646) ExChangeAlloc(   585) ExChangeFree(   143) ExactAlloc(     0) Returned(     0) Malloc(    14)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC  7(      4096) GetAlloc(   648) GetFree(   646) ExChangeAlloc( 19215) ExChangeFree(  4847) ExactAlloc(     0) Returned(     0) Malloc(   451)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC  8(      8192) GetAlloc(   146) GetFree(   646) ExChangeAlloc(  4312) ExChangeFree(   141) ExactAlloc(     0) Returned(     0) Malloc(   131)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC  9(     16384) GetAlloc(    50) GetFree(   646) ExChangeAlloc(  1657) ExChangeFree(   287) ExactAlloc(     0) Returned(     0) Malloc(    43)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 10(     32768) GetAlloc(  1987) GetFree(   646) ExChangeAlloc( 31712) ExChangeFree(   850) ExactAlloc(     0) Returned(     0) Malloc(   966)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 11(     65536) GetAlloc(    29) GetFree(   646) ExChangeAlloc( 10204) ExChangeFree(  9664) ExactAlloc(     0) Returned(     0) Malloc(    19)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 12(    131072) GetAlloc(   347) GetFree(   646) ExChangeAlloc(  5736) ExChangeFree(  2035) ExactAlloc(     0) Returned(     0) Malloc(   232)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 13(    262144) GetAlloc(   763) GetFree(   646) ExChangeAlloc(  4875) ExChangeFree(  1115) ExactAlloc(     0) Returned(     0) Malloc(   471)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 14(    524288) GetAlloc(     7) GetFree(   646) ExChangeAlloc(   420) ExChangeFree(   395) ExactAlloc(     0) Returned(     0) Malloc(     7)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 15(   1048576) GetAlloc(     3) GetFree(   646) ExChangeAlloc(   311) ExChangeFree(   309) ExactAlloc(     0) Returned(     0) Malloc(     3)
[2023-03-22 17:10:47.308] WARNING searchnode       stderr       SC 16(   2097152) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(    68) Returned(    67) Malloc(     0)
[2023-03-22 17:10:47.309] WARNING searchnode       stderr       SC 17(   4194304) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   184) Returned(   184) Malloc(     0)
[2023-03-22 17:10:47.309] WARNING searchnode       stderr       SC 18(   8388608) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   112) Returned(   112) Malloc(     0)
[2023-03-22 17:10:47.309] WARNING searchnode       stderr       #323 active threads. Peak threads #322. 364 threads created in total.
[2023-03-22 17:10:47.309] WARNING searchnode       stderr       SignalHandler 27 done

Good that accounted memory is now at 4.2G out of 5.9G. Better accounting has been added so 8.139.1 will be even closer. The final improvements have been done and will be part of 8.140.13.
In 2-3 weeks time there will roll out a change that brings the memory consumption even further down.

@baldersheim - is there any update on further fix for memory consumption? Which version will include this fix?

Also noticing difference between reported memory and actual memory after feeding documents.
Number of documents on content node =  8,725,736
actual usage per content node 34,574,486,733 bytes (32.2GB) - fetched using "top" on content node: vespa-proton-bi
content_proton_documentdb_memory_usage_allocated_bytes_last = 25,645,086,932 bytes (24GB) - fetch using prometheus
Logs:
[2023-03-29 20:34:56.913] WARNING searchnode       stderrSignalHandler 27 caught
[2023-03-29 20:34:56.913] WARNING searchnode       stderrDataSegment at 0x7f861f5ca2d8(8389832), AllocPool at 0x7f861fdca7a0(2912), ThreadList at 0x7f861fdcb380(9568288)
[2023-03-29 20:34:56.914] WARNING searchnode       stderrStart at 0x10000000000, End at 0x10499200000(0x10ff000) size(19748880384) partialExtension(0) NextLogLimit(11000000000) logLevel(2)
[2023-03-29 20:34:56.914] WARNING searchnode       stderrFree #  0 block # 9212 chainlength     1 size    2097152
[2023-03-29 20:34:56.914] WARNING searchnode       stderrFree #  1 block # 9216 chainlength     2 size    4194304
[2023-03-29 20:34:56.914] WARNING searchnode       stderrFree #  2 block # 9299 chainlength     3 size    6291456
[2023-03-29 20:34:56.915] WARNING searchnode       stderrFree #  3 block # 9335 chainlength    45 size   94371840
[2023-03-29 20:34:56.915] WARNING searchnode       stderrFree #  4 block # 9383 chainlength     7 size   14680064
[2023-03-29 20:34:56.915] WARNING searchnode       stderrFree #  5 block # 9397 chainlength     3 size    6291456
[2023-03-29 20:34:56.915] WARNING searchnode       stderrFree #  6 block # 9401 chainlength     2 size    4194304
[2023-03-29 20:34:56.916] WARNING searchnode       stderrFree #  7 block # 9408 chainlength     1 size    2097152
[2023-03-29 20:34:56.916] WARNING searchnode       stderrSizeClass -4(UNMAPPED) has    64 blocks with  134217728 bytes
[2023-03-29 20:34:56.916] WARNING searchnode       stderrSizeClass -1(SYSTEM) has   108 blocks with  226492416 bytes
[2023-03-29 20:34:56.916] WARNING searchnode       stderrSizeClass  0 has   126 blocks with  264241152 bytes
[2023-03-29 20:34:56.917] WARNING searchnode       stderrSizeClass  1 has    18 blocks with   37748736 bytes
[2023-03-29 20:34:56.917] WARNING searchnode       stderrSizeClass  2 has   119 blocks with  249561088 bytes
[2023-03-29 20:34:56.917] WARNING searchnode       stderrSizeClass  3 has    38 blocks with   79691776 bytes
[2023-03-29 20:34:56.917] WARNING searchnode       stderrSizeClass  4 has   217 blocks with  455081984 bytes
[2023-03-29 20:34:56.917] WARNING searchnode       stderrSizeClass  5 has    92 blocks with  192937984 bytes
[2023-03-29 20:34:56.918] WARNING searchnode       stderrSizeClass  6 has    43 blocks with   90177536 bytes
[2023-03-29 20:34:56.918] WARNING searchnode       stderrSizeClass  7 has   505 blocks with 1059061760 bytes
[2023-03-29 20:34:56.918] WARNING searchnode       stderrSizeClass  8 has   210 blocks with  440401920 bytes
[2023-03-29 20:34:56.918] WARNING searchnode       stderrSizeClass  9 has   127 blocks with  266338304 bytes
[2023-03-29 20:34:56.918] WARNING searchnode       stderrSizeClass 10 has  1219 blocks with 2556428288 bytes
[2023-03-29 20:34:56.919] WARNING searchnode       stderrSizeClass 11 has   384 blocks with  805306368 bytes
[2023-03-29 20:34:56.919] WARNING searchnode       stderrSizeClass 12 has   738 blocks with 1547698176 bytes
[2023-03-29 20:34:56.919] WARNING searchnode       stderrSizeClass 13 has   894 blocks with 1874853888 bytes
[2023-03-29 20:34:56.920] WARNING searchnode       stderrSizeClass 14 has   519 blocks with 1088421888 bytes
[2023-03-29 20:34:56.920] WARNING searchnode       stderrSizeClass 15 has  3891 blocks with 8160018432 bytes
[2023-03-29 20:34:56.920] WARNING searchnode       stderrSizeClass 16 has    63 blocks with  132120576 bytes
[2023-03-29 20:34:56.920] WARNING searchnode       stderrSizeClass 17 has    38 blocks with   79691776 bytes
[2023-03-29 20:34:56.920] WARNING searchnode       stderrSizeClass 18 has     4 blocks with    8388608 bytes
[2023-03-29 20:34:56.921] WARNING searchnode       stderrUsage: Total=19748880384(100%), admin=360710144(1%), unused=0(0%), allocated=19388170240(98%)
[2023-03-29 20:34:56.921] WARNING searchnode       stderrGlobalPool getChunks(22779, 220619) allocChunksList(27):
[2023-03-29 20:34:56.921] WARNING searchnode       stderrSC  0(        32) GetAlloc(   643) GetFree(   646) ExChangeAlloc(1235648) ExChangeFree(1170386) ExactAlloc(     0) Returned(     0) Malloc(   126)
[2023-03-29 20:34:56.921] WARNING searchnode       stderrSC  1(        64) GetAlloc(    18) GetFree(   646) ExChangeAlloc(3938270) ExChangeFree(3935338) ExactAlloc(     0) Returned(     0) Malloc(    18)
[2023-03-29 20:34:56.922] WARNING searchnode       stderrSC  2(       128) GetAlloc(   119) GetFree(   646) ExChangeAlloc(27452097) ExChangeFree(27439183) ExactAlloc(     0) Returned(     0) Malloc(   119)
[2023-03-29 20:34:56.922] WARNING searchnode       stderrSC  3(       256) GetAlloc(    38) GetFree(   646) ExChangeAlloc(1123578) ExChangeFree(1121249) ExactAlloc(     0) Returned(     0) Malloc(    38)
[2023-03-29 20:34:56.922] WARNING searchnode       stderrSC  4(       512) GetAlloc(   231) GetFree(   646) ExChangeAlloc(604449) ExChangeFree(597721) ExactAlloc(     0) Returned(     0) Malloc(   217)
[2023-03-29 20:34:56.922] WARNING searchnode       stderrSC  5(      1024) GetAlloc(    93) GetFree(   646) ExChangeAlloc(234529) ExChangeFree(232712) ExactAlloc(     0) Returned(     0) Malloc(    92)
[2023-03-29 20:34:56.923] WARNING searchnode       stderrSC  6(      2048) GetAlloc(    61) GetFree(   646) ExChangeAlloc(122590) ExChangeFree(121686) ExactAlloc(     0) Returned(     0) Malloc(    43)
[2023-03-29 20:34:56.923] WARNING searchnode       stderrSC  7(      4096) GetAlloc(   695) GetFree(   646) ExChangeAlloc(653763) ExChangeFree(638109) ExactAlloc(     0) Returned(     0) Malloc(   505)
[2023-03-29 20:34:56.923] WARNING searchnode       stderrSC  8(      8192) GetAlloc(   231) GetFree(   646) ExChangeAlloc(425899) ExChangeFree(419668) ExactAlloc(     0) Returned(     0) Malloc(   210)
[2023-03-29 20:34:56.923] WARNING searchnode       stderrSC  9(     16384) GetAlloc(   147) GetFree(   646) ExChangeAlloc(1659471) ExChangeFree(1655550) ExactAlloc(     0) Returned(     0) Malloc(   127)
[2023-03-29 20:34:56.923] WARNING searchnode       stderrSC 10(     32768) GetAlloc(  2301) GetFree(   646) ExChangeAlloc(1520855) ExChangeFree(1483886) ExactAlloc(     0) Returned(     0) Malloc(  1219)
[2023-03-29 20:34:56.924] WARNING searchnode       stderrSC 11(     65536) GetAlloc(   521) GetFree(   646) ExChangeAlloc(17912313) ExChangeFree(17902128) ExactAlloc(     0) Returned(     0) Malloc(   384)
[2023-03-29 20:34:56.924] WARNING searchnode       stderrSC 12(    131072) GetAlloc(   859) GetFree(   646) ExChangeAlloc(9502229) ExChangeFree(9494411) ExactAlloc(     0) Returned(     0) Malloc(   738)
[2023-03-29 20:34:56.924] WARNING searchnode       stderrSC 13(    262144) GetAlloc(  1213) GetFree(   646) ExChangeAlloc(12812398) ExChangeFree(12805895) ExactAlloc(     0) Returned(     0) Malloc(   894)
[2023-03-29 20:34:56.924] WARNING searchnode       stderrSC 14(    524288) GetAlloc(   581) GetFree(   646) ExChangeAlloc(3760011) ExChangeFree(3758296) ExactAlloc(     0) Returned(     0) Malloc(   519)
[2023-03-29 20:34:56.925] WARNING searchnode       stderrSC 15(   1048576) GetAlloc(  6099) GetFree(   646) ExChangeAlloc(2713645) ExChangeFree(2712656) ExactAlloc(     0) Returned(     0) Malloc(  3891)
[2023-03-29 20:34:56.925] WARNING searchnode       stderrSC 16(   2097152) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc( 96729) Returned( 96666) Malloc(     0)
[2023-03-29 20:34:56.925] WARNING searchnode       stderrSC 17(   4194304) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc( 23751) Returned( 23732) Malloc(     0)
[2023-03-29 20:34:56.925] WARNING searchnode       stderrSC 18(   8388608) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc( 14136) Returned( 14135) Malloc(     0)
[2023-03-29 20:34:56.926] WARNING searchnode       stderrSC 19(  16777216) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(  6825) Returned(  6825) Malloc(     0)
[2023-03-29 20:34:56.926] WARNING searchnode       stderrSC 20(  33554432) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   416) Returned(   416) Malloc(     0)
[2023-03-29 20:34:56.926] WARNING searchnode       stderrSC 21(  67108864) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(    37) Returned(    37) Malloc(     0)
[2023-03-29 20:34:56.926] WARNING searchnode       stderrSC 22( 134217728) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(     2) Returned(     2) Malloc(     0)
[2023-03-29 20:34:56.926] WARNING searchnode       stderr#323 active threads. Peak threads #322. 364 threads created in total.
[2023-03-29 20:34:56.926] WARNING searchnode       stderrSignalHandler 27 done
After restarting the content node:
Number of documents on content node =  8,725,736
actual usage per content node: 27,165,668,147 bytes (25.3GB) - fetched using "top" on content node: vespa-proton-bi
content_proton_documentdb_memory_usage_allocated_bytes_last: 24,135,852,992 bytes (22.5GB) - fetch using prometheus
Logs
[2023-03-29 21:22:43.130] WARNING searchnode       stderrSignalHandler 27 caught
[2023-03-29 21:22:43.131] WARNING searchnode       stderrDataSegment at 0x7f929c5d62d8(8389832), AllocPool at 0x7f929cdd67a0(2912), ThreadList at 0x7f929cdd7380(9568288)
[2023-03-29 21:22:43.132] WARNING searchnode       stderrStart at 0x10000000000, End at 0x102e8e00000(0x2451000) size(12496928768) partialExtension(0) NextLogLimit(11000000000) logLevel(2)
[2023-03-29 21:22:43.132] WARNING searchnode       stderrFree #  0 block # 5835 chainlength     1 size    2097152
[2023-03-29 21:22:43.132] WARNING searchnode       stderrFree #  1 block # 5864 chainlength    11 size   23068672
[2023-03-29 21:22:43.132] WARNING searchnode       stderrFree #  2 block # 5878 chainlength     4 size    8388608
[2023-03-29 21:22:43.132] WARNING searchnode       stderrFree #  3 block # 5883 chainlength    76 size  159383552
[2023-03-29 21:22:43.133] WARNING searchnode       stderrSizeClass -4(UNMAPPED) has    92 blocks with  192937984 bytes
[2023-03-29 21:22:43.133] WARNING searchnode       stderrSizeClass -1(SYSTEM) has   100 blocks with  209715200 bytes
[2023-03-29 21:22:43.133] WARNING searchnode       stderrSizeClass  0 has   126 blocks with  264241152 bytes
[2023-03-29 21:22:43.133] WARNING searchnode       stderrSizeClass  1 has    15 blocks with   31457280 bytes
[2023-03-29 21:22:43.134] WARNING searchnode       stderrSizeClass  2 has   101 blocks with  211812352 bytes
[2023-03-29 21:22:43.134] WARNING searchnode       stderrSizeClass  3 has    39 blocks with   81788928 bytes
[2023-03-29 21:22:43.134] WARNING searchnode       stderrSizeClass  4 has   218 blocks with  457179136 bytes
[2023-03-29 21:22:43.134] WARNING searchnode       stderrSizeClass  5 has    47 blocks with   98566144 bytes
[2023-03-29 21:22:43.134] WARNING searchnode       stderrSizeClass  6 has    30 blocks with   62914560 bytes
[2023-03-29 21:22:43.135] WARNING searchnode       stderrSizeClass  7 has   494 blocks with 1035993088 bytes
[2023-03-29 21:22:43.135] WARNING searchnode       stderrSizeClass  8 has   192 blocks with  402653184 bytes
[2023-03-29 21:22:43.135] WARNING searchnode       stderrSizeClass  9 has   128 blocks with  268435456 bytes
[2023-03-29 21:22:43.135] WARNING searchnode       stderrSizeClass 10 has  1111 blocks with 2329935872 bytes
[2023-03-29 21:22:43.136] WARNING searchnode       stderrSizeClass 11 has   339 blocks with  710934528 bytes
[2023-03-29 21:22:43.136] WARNING searchnode       stderrSizeClass 12 has   524 blocks with 1098907648 bytes
[2023-03-29 21:22:43.136] WARNING searchnode       stderrSizeClass 13 has   795 blocks with 1667235840 bytes
[2023-03-29 21:22:43.136] WARNING searchnode       stderrSizeClass 14 has   436 blocks with  914358272 bytes
[2023-03-29 21:22:43.137] WARNING searchnode       stderrSizeClass 15 has  1070 blocks with 2243952640 bytes
[2023-03-29 21:22:43.137] WARNING searchnode       stderrSizeClass 16 has    61 blocks with  127926272 bytes
[2023-03-29 21:22:43.137] WARNING searchnode       stderrSizeClass 17 has    34 blocks with   71303168 bytes
[2023-03-29 21:22:43.137] WARNING searchnode       stderrSizeClass 18 has     7 blocks with   14680064 bytes
[2023-03-29 21:22:43.137] WARNING searchnode       stderrUsage: Total=12496928768(100%), admin=402653184(3%), unused=0(0%), allocated=12094275584(96%)
[2023-03-29 21:22:43.138] WARNING searchnode       stderrGlobalPool getChunks(19444, 199879) allocChunksList(25):
[2023-03-29 21:22:43.138] WARNING searchnode       stderrSC  0(        32) GetAlloc(   463) GetFree(   646) ExChangeAlloc(217100) ExChangeFree(151925) ExactAlloc(     0) Returned(     0) Malloc(   126)
[2023-03-29 21:22:43.138] WARNING searchnode       stderrSC  1(        64) GetAlloc(    15) GetFree(   646) ExChangeAlloc(333739) ExChangeFree(330805) ExactAlloc(     0) Returned(     0) Malloc(    15)
[2023-03-29 21:22:43.138] WARNING searchnode       stderrSC  2(       128) GetAlloc(   121) GetFree(   646) ExChangeAlloc(644573) ExChangeFree(631796) ExactAlloc(     0) Returned(     0) Malloc(   101)
[2023-03-29 21:22:43.138] WARNING searchnode       stderrSC  3(       256) GetAlloc(    64) GetFree(   646) ExChangeAlloc( 32876) ExChangeFree( 30542) ExactAlloc(     0) Returned(     0) Malloc(    39)
[2023-03-29 21:22:43.139] WARNING searchnode       stderrSC  4(       512) GetAlloc(   230) GetFree(   646) ExChangeAlloc( 20505) ExChangeFree( 13720) ExactAlloc(     0) Returned(     0) Malloc(   218)
[2023-03-29 21:22:43.139] WARNING searchnode       stderrSC  5(      1024) GetAlloc(    47) GetFree(   646) ExChangeAlloc( 31679) ExChangeFree( 30229) ExactAlloc(     0) Returned(     0) Malloc(    47)
[2023-03-29 21:22:43.139] WARNING searchnode       stderrSC  6(      2048) GetAlloc(    37) GetFree(   646) ExChangeAlloc( 17682) ExChangeFree( 16738) ExactAlloc(     0) Returned(     0) Malloc(    30)
[2023-03-29 21:22:43.139] WARNING searchnode       stderrSC  7(      4096) GetAlloc(  1215) GetFree(   646) ExChangeAlloc( 97750) ExChangeFree( 82322) ExactAlloc(     0) Returned(     0) Malloc(   494)
[2023-03-29 21:22:43.139] WARNING searchnode       stderrSC  8(      8192) GetAlloc(   278) GetFree(   646) ExChangeAlloc( 27992) ExChangeFree( 21855) ExactAlloc(     0) Returned(     0) Malloc(   192)
[2023-03-29 21:22:43.140] WARNING searchnode       stderrSC  9(     16384) GetAlloc(   229) GetFree(   646) ExChangeAlloc( 27087) ExChangeFree( 23336) ExactAlloc(     0) Returned(     0) Malloc(   128)
[2023-03-29 21:22:43.140] WARNING searchnode       stderrSC 10(     32768) GetAlloc(  2750) GetFree(   646) ExChangeAlloc( 68772) ExChangeFree( 33261) ExactAlloc(     0) Returned(     0) Malloc(  1111)
[2023-03-29 21:22:43.140] WARNING searchnode       stderrSC 11(     65536) GetAlloc(   886) GetFree(   646) ExChangeAlloc(818249) ExChangeFree(808187) ExactAlloc(     0) Returned(     0) Malloc(   339)
[2023-03-29 21:22:43.140] WARNING searchnode       stderrSC 12(    131072) GetAlloc(   967) GetFree(   646) ExChangeAlloc(553063) ExChangeFree(545224) ExactAlloc(     0) Returned(     0) Malloc(   524)
[2023-03-29 21:22:43.141] WARNING searchnode       stderrSC 13(    262144) GetAlloc(  1568) GetFree(   646) ExChangeAlloc(729523) ExChangeFree(723253) ExactAlloc(     0) Returned(     0) Malloc(   795)
[2023-03-29 21:22:43.141] WARNING searchnode       stderrSC 14(    524288) GetAlloc(   601) GetFree(   646) ExChangeAlloc(713757) ExChangeFree(712090) ExactAlloc(     0) Returned(     0) Malloc(   436)
[2023-03-29 21:22:43.141] WARNING searchnode       stderrSC 15(   1048576) GetAlloc(  1885) GetFree(   646) ExChangeAlloc(234746) ExChangeFree(233824) ExactAlloc(     0) Returned(     0) Malloc(  1070)
[2023-03-29 21:22:43.141] WARNING searchnode       stderrSC 16(   2097152) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(  1348) Returned(  1287) Malloc(     0)
[2023-03-29 21:22:43.141] WARNING searchnode       stderrSC 17(   4194304) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   523) Returned(   506) Malloc(     0)
[2023-03-29 21:22:43.142] WARNING searchnode       stderrSC 18(   8388608) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   311) Returned(   309) Malloc(     0)
[2023-03-29 21:22:43.142] WARNING searchnode       stderrSC 19(  16777216) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   218) Returned(   218) Malloc(     0)
[2023-03-29 21:22:43.142] WARNING searchnode       stderrSC 20(  33554432) GetAlloc(     0) GetFree(   646) ExChangeAlloc(     0) ExChangeFree(     0) ExactAlloc(   202) Returned(   202) Malloc(     0)
[2023-03-29 21:22:43.142] WARNING searchnode       stderr#323 active threads. Peak threads #322. 364 threads created in total.
[2023-03-29 21:22:43.142] WARNING searchnode       stderrSignalHandler 27 done

The further improvements did not make it before easter. Currently there are quite a few features inflight, so I can not give an estimate. The gains from that will not be significant when you add documents  to your index.
The numbers you see now makes sense. We only try to track memory used by permanent data structures. There will always be memory used temporarily by background jobs/ feeding/querying that does not belong to the data.
The numbers above seems to be sane and within what can be expected.

I have added issue #26640 to track the remaining known improvement. Closing this one now.

In Vespa 8.181.15 the static memory usage of multivalue attributes is reduced by up to 40%. See https://github.com/vespa-engine/vespa/issues/26640 for details.

authors: baldersheim geirst kkraune mohsin36

repo: vespa-engine/vespa


text:
rename job

with this, we can display a useful job name in the badge on the overview page

16:43:07 $ echo 'Error: Cache is set for non-existing job: main'; exit 1
16:43:07 Error: Cache is set for non-existing job: main

@aressem I think the build failure now is something else, OK to merge?

source: gharchive/pull-request

authors: aressem kkraune

license_type: permissive

license_source: github-api


text:
Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs

Tried your example from:

https://raw.githubusercontent.com/vfarcic/docker-flow-proxy/master/docker-compose-stack.yml

This used to work but since your recent commit, fe6ead0f1b4be332efcc4fdde95864e47d846f71, the service no longer starts. Here is the error:
ubuntu@node1:/vagrant$ docker stack deploy --compose-file docker-compose.yml proxy
Creating network proxy_proxy
Creating service proxy_proxy
Creating service proxy_swarm-listener
ubuntu@node1:/vagrant$ docker service logs -f proxy_proxy
proxy_proxy.1.w0pxdpc0yl8b@node1    | 2017/11/22 00:26:11 Starting HAProxy
proxy_proxy.2.kv9p3u9ef47f@node1    | 2017/11/22 00:26:11 Starting HAProxy
proxy_proxy.2.kv9p3u9ef47f@node1    | 2017/11/22 00:26:11 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.1.w0pxdpc0yl8b@node1    | 2017/11/22 00:26:11 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.1.f5v2px9prplg@node1    | 2017/11/22 00:26:35 Starting HAProxy
proxy_proxy.1.f5v2px9prplg@node1    | 2017/11/22 00:26:35 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.2.l40r5pwx8xk1@node1    | 2017/11/22 00:26:35 Starting HAProxy
proxy_proxy.2.l40r5pwx8xk1@node1    | 2017/11/22 00:26:35 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
╚
ubuntu@node1:/vagrant$
ubuntu@node1:/vagrant$ docker stack ps proxy
ID                  NAME                     IMAGE                                       NODE                DESIRED STATE       CURRENT STATE            ERROR
 PORTS
a1fxb9og94fg        proxy_proxy.1            vfarcic/docker-flow-proxy:latest            node1               Running             Starting 3 seconds ago

w9gacnlf3g3q         \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest            node1               Shutdown            Failed 55 seconds ago    "task: non-zero exit (143): do…"

wqavf48d2b0i         \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest            node1               Shutdown            Failed 4 minutes ago     "task: non-zero exit (143): do…"

wpdbtwpsht2z         \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest            node1               Shutdown            Failed 5 minutes ago     "task: non-zero exit (143): do…"

However, vfarcic/docker-flow-proxy:17.11.11-56 works fine and gives me the following result:
ubuntu@node1:/vagrant$ docker stack services proxy
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
oh8yser8y0c7        proxy_proxy            replicated          2/2                 vfarcic/docker-flow-proxy:17.11.11-56       *:80->80/tcp,*:443->443/tcp
qzcyitcs561s        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest

I'm having the same issue. There seems to be some problem to get the certs:
proxy_proxy.0.hkarreyuha0f@merov2    | 2017/11/27 11:30:31 Starting HAProxy
proxy_proxy.0.hkarreyuha0f@merov2    | 2017/11/27 11:30:31 Getting certs from http://104.239.213.7:8080/v1/docker-flow-proxy/certs

In my case the problem was the DNS (again). I was using 4.2.2.1 and that DNS answers a default IP for non-existing names:
$ dig @4.2.2.1 flkdjfkdf.com
; <<>> DiG 9.11.1-P3-RedHat-9.11.1-3.P3.fc26 <<>> @4.2.2.1 flkdjfkdf.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57495
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;flkdjfkdf.com.			IN	A

;; ANSWER SECTION:
flkdjfkdf.com.		10	IN	A	104.239.213.7
flkdjfkdf.com.		10	IN	A	198.105.244.11

;; Query time: 175 msec
;; SERVER: 4.2.2.1#53(4.2.2.1)
;; WHEN: Tue Nov 28 08:56:22 -02 2017
;; MSG SIZE  rcvd: 63

I switched to Google's 8.8.8.8 and everything went back to normal.

@shazChaudhry Can you try the latest vfarcic/docker-flow-proxy tag? It has a bug fix that might have solved your issue. If it still does not work, please send me any additional info you might have that might help me reproduce the problem.
P.S. Sorry for not responding earlier. Too much work...

Still no joy with the latest tags: vfarcic/docker-flow-proxy:17.11.29-68
Still getting the same error as before. I am curious what this link is:- http://81.200.64.50:8080/v1/docker-flow-proxy/certs
My Environment:
windows 10; vagrant box = "ubuntu/xenial64";
ubuntu@node1:/vagrant$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.11.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: k5nzklcxue9omtyngvuxk2ilz
 Is Manager: true
 ClusterID: 1ltr6u1385r2xjoha0ljtqarr
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.99.101
 Manager Addresses:
  192.168.99.101:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 992280e8e265f491f7a624ab82f3e238be086e49
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-98-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.765GiB
Name: node1
ID: BQNO:W7VH:CZOF:HTIJ:FVMM:QSAW:K4WY:B3FM:DTBW:RCHV:OZWJ:F6H2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

I have tried vfarcic/docker-flow-proxy:17.11.29-68 in Docker4AWS which works as expected.
However, this very latest tag still does not work in my local environment.

@shazChaudhry I think we found the bug. Can you please try again with vfarcic/docker-flow-proxy:17.12.02-72?

@vfarcic Still same issue with vfarcic/docker-flow-proxy:17.12.02-72 in my local environment
If it is of any help, here is my Vagrantfile: https://raw.githubusercontent.com/shazChaudhry/docker-jenkins/master/Vagrantfile
ubuntu@node1:/vagrant$ docker stack deploy --compose-file docker-compose.yml proxy
Creating network proxy_proxy
Creating service proxy_swarm-listener
Creating service proxy_proxy

ubuntu@node1:~$ docker stack services proxy
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
21cm0ketw3ww        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
hpqrusocck0f        proxy_proxy            replicated          0/2                 vfarcic/docker-flow-proxy:17.12.02-72       *:80->80/tcp,*:443->443/tcp

ubuntu@node1:/vagrant$ docker service logs -f proxy_proxy
proxy_proxy.1.z2mc2rhye46p@node1    | 2017/12/03 11:07:32 Starting HAProxy
proxy_proxy.1.zkwybev4i9ie@node1    | 2017/12/03 11:06:45 Starting HAProxy
proxy_proxy.1.y5ikex3lwpr2@node1    | 2017/12/03 11:09:51 Starting HAProxy
proxy_proxy.1.y7ivl3b9nrjt@node1    | 2017/12/03 11:04:49 Starting HAProxy
proxy_proxy.2.wdikv3lo12fg@node1    | 2017/12/03 11:02:29 Starting HAProxy
proxy_proxy.2.wdikv3lo12fg@node1    | 2017/12/03 11:02:29 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.1.z2mc2rhye46p@node1    | 2017/12/03 11:07:32 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.1.oncbfpjzdj77@node1    | 2017/12/03 11:11:00 Starting HAProxy
proxy_proxy.1.zkwybev4i9ie@node1    | 2017/12/03 11:06:45 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs
proxy_proxy.1.y5ikex3lwpr2@node1    | 2017/12/03 11:09:51 Getting certs from http://81.200.64.50:8080/v1/docker-flow-proxy/certs

ubuntu@node1:/vagrant$ docker stack ps --no-trunc proxy
ID                          NAME                     IMAGE                                                                                NODE                DESIRED STATE       CURRENT STATE               ERROR                                                          PORTS
oncbfpjzdj77yqf754o3hat40   proxy_proxy.1            vfarcic/docker-flow-proxy:17.12.02-72@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb       node1               Running             Starting 13 seconds ago
y5ikex3lwpr2dctiipeqbb0pc    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:17.12.02-72@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb       node1               Shutdown            Failed about a minute ago   "task: non-zero exit (143): dockerexec: unhealthy container"
z2mc2rhye46pyd80950k21s4h    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:17.12.02-72@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb       node1               Shutdown            Failed 3 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
zkwybev4i9ielqmhqmks4g7lv    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:17.12.02-72@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb       node1               Shutdown            Failed 4 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
y7ivl3b9nrjtei18rbew37aep    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:17.12.02-72@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb       node1               Shutdown            Failed 6 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
wmsjelsxoeuz89akxt6ueo7hj   proxy_swarm-listener.1   vfarcic/docker-flow-swarm-listener:latest@sha256:ac27bcfba99963d118734f4e74885decf57f08eb92b7fba78164bcdf0466a209   node1               Running             Running 10 minutes ago

I just executed the commands that follow.
curl -o Vagrantfile \
    https://raw.githubusercontent.com/shazChaudhry/docker-jenkins/master/Vagrantfile

vagrant up

vagrant ssh node1

curl -o proxy.yml \
    https://raw.githubusercontent.com/vfarcic/docker-flow-proxy/master/docker-compose-stack.yml

docker network create -d overlay proxy

docker stack deploy -c proxy.yml proxy

docker stack services proxy

It worked and the output of the last command is as follows.
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
gqy3hr0xdhqs        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
kz6t04clpaui        proxy_proxy            replicated          2/2                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tc

Can you please confirm that you executed the same commands? If you did, the failure is most likely related to networking. Is there a corporate proxy/VPN/firewall that might interfere with networking? Can you create a new service that is also attached to the proxy network and examine whether it can reach the proxy?
Something like the following commands should do.
docker service create --name util \
    --mode global --network proxy \
    alpine sleep 1000000

ID=$(docker container ls -q \
    -f "label=com.docker.swarm.service.name=util")

docker container exec -it $ID apk add -U drill

docker container exec -it $ID drill proxy

docker container exec -it $ID drill tasks.proxy

docker container exec -it $ID drill swarm-listener

docker container exec -it $ID drill tasks.swarm-listener

Please post the output of the last four commands. That should give us a clue whether there's a problem with your networking.

Thank you very much for the instructions. I have followed the instructions to the letter and please see the output below:

curl -o Vagrantfile https://raw.githubusercontent.com/shazChaudhry/docker-jenkins/master/Vagrantfile
vagrant up
vagrant ssh node1
curl -o proxy.yml https://raw.githubusercontent.com/vfarcic/docker-flow-proxy/master/docker-compose-stack.yml

ubuntu@node1:~$ docker network create -d overlay proxy
ipwe3mnxcui6rcf9awmolpcjz

ubuntu@node1:~$ docker stack deploy -c proxy.yml proxy
Creating service proxy_proxy
Creating service proxy_swarm-listener

ubuntu@node1:~$ docker stack services proxy
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
088by5ns5m3n        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
fzfrutf7xuz6        proxy_proxy            replicated          0/2                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tcp

ubuntu@node1:~$ docker service create --name util --mode global --network proxy alpine sleep 1000000
kd1v7fry84vcmbe8jgl6yf85t
overall progress: 1 out of 1 tasks
4r52c13xltaq: running   [==================================================>]
verify: Service converged

ubuntu@node1:~$ ID=$(docker container ls -q -f "label=com.docker.swarm.service.name=util")

ubuntu@node1:~$ docker container exec -it $ID apk add -U drill
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ldns (1.6.17-r6)
(2/2) Installing drill (1.6.17-r6)
Executing busybox-1.27.2-r6.trigger
OK: 5 MiB in 13 packages

ubuntu@node1:~$ docker container exec -it $ID drill proxy
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 5420
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; proxy.       IN      A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 2256 msec
;; SERVER: 127.0.0.11
;; WHEN: Sun Dec  3 15:36:23 2017
;; MSG SIZE  rcvd: 23

ubuntu@node1:~$ docker container exec -it $ID drill tasks.proxy
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 47209
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; tasks.proxy. IN      A

;; ANSWER SECTION:
tasks.proxy.    3600    IN      A       81.200.64.50

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 16 msec
;; SERVER: 127.0.0.11
;; WHEN: Sun Dec  3 15:36:46 2017
;; MSG SIZE  rcvd: 45

ubuntu@node1:~$ docker container exec -it $ID drill swarm-listener
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2450
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; swarm-listener.      IN      A

;; ANSWER SECTION:
swarm-listener. 600     IN      A       10.0.0.6

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Sun Dec  3 15:37:05 2017
;; MSG SIZE  rcvd: 62

 docker container exec -it $ID drill tasks.swarm-listener
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 40380
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; tasks.swarm-listener.        IN      A

;; ANSWER SECTION:
tasks.swarm-listener.   600     IN      A       10.0.0.7

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Sun Dec  3 15:37:23 2017
;; MSG SIZE  rcvd: 74

ubuntu@node1:~$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
fzfrutf7xuz6        proxy_proxy            replicated          0/2                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tcp
088by5ns5m3n        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
kd1v7fry84vc        util                   global              1/1                 alpine:latest

ubuntu@node1:~$ docker stack ps --no-trunc proxy
ID                          NAME                     IMAGE                                                                                NODE                DESIRED STATE       CURRENT STATE               ERROR                                                          PORTS
ptz4xtkbkftvhhssbayywu0lc   proxy_proxy.1            vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Running             Starting 4 seconds ago
wh3c6f0spk2cnvgurg9l8gdkz    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed 34 seconds ago       "task: non-zero exit (143): dockerexec: unhealthy container"
xps2km1ud1poyix3ew86hnq62    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed 3 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
yb3p8y4rjf5n60mo8wpgwvcow    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed 4 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
wtcmx0isbeb7fn6skdds2i5vp    \_ proxy_proxy.1        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed 5 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
mhlhisf5uoy5euqmldpq0ylhw   proxy_swarm-listener.1   vfarcic/docker-flow-swarm-listener:latest@sha256:ac27bcfba99963d118734f4e74885decf57f08eb92b7fba78164bcdf0466a209   node1               Running             Running 7 minutes ago
co09lywvudy2hkoz1a231zbne   proxy_proxy.2            vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Running             Starting 5 seconds ago
trj5hmosvrresi45x65fgld8n    \_ proxy_proxy.2        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed about a minute ago   "task: non-zero exit (143): dockerexec: unhealthy container"
ssgido0u30fq49mruzmjtcni9    \_ proxy_proxy.2        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed about a minute ago   "task: non-zero exit (143): dockerexec: unhealthy container"
v6un52g2u7ixwwscryo2c5mzv    \_ proxy_proxy.2        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1               Shutdown            Failed 2 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"
tamypw7xv2q0k1hmtofdfjw55    \_ proxy_proxy.2        vfarcic/docker-flow-proxy:latest@sha256:4c4425fa21ac6440edd762cf7b9f0d4d494afc75e052616c6eb013abdf920abb            node1            Shutdown            Failed 4 minutes ago        "task: non-zero exit (143): dockerexec: unhealthy container"

I can confirm that I do not have a corporate proxy, VPN or firewall that might interfere with networking.
I have always used the same laptop, network and Vagrantfile. I think I started experiencing this issue since your commit fe6ead0f1b4be332efcc4fdde95864e47d846f71
as you can see below, vfarcic/docker-flow-proxy:17.11.11-56 is still working just fine in my local environment:
ubuntu@node1:~$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                       PORTS
v2ygsmqzx8qy        proxy_proxy            replicated          2/2                 vfarcic/docker-flow-proxy:17.11.11-56       *:80->80/tcp,*:443->443/tcp
bhsluc36ua8h        proxy_swarm-listener   replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest

Same issue
I fellow this tour  https://technologyconversations.com/2016/08/01/integrating-proxy-with-docker-swarm-tour-around-docker-1-12-series/
and found proxy keeping restart
ID                  NAME                IMAGE                              NODE                DESIRED STATE       CURRENT STATE            ERROR                              P
gt1vsu5bnjaa        proxy.1             vfarcic/docker-flow-proxy:latest   node-2              Running             Starting 4 seconds ago
yqzraa4blu1g         \_ proxy.1         vfarcic/docker-flow-proxy:latest   node-3              Shutdown            Failed 3 minutes ago     "starting container failed: No…"
zc1jf9sn2e4v         \_ proxy.1         vfarcic/docker-flow-proxy:latest   node-2              Shutdown            Failed 5 minutes ago     "starting container failed: No…"
zfgp49gtubwh         \_ proxy.1         vfarcic/docker-flow-proxy:latest   node-2              Shutdown            Failed 6 minutes ago     "starting container failed: No…"
zdra6mlo909b         \_ proxy.1         vfarcic/docker-flow-proxy:latest   node-2              Shutdown            Failed 13 minutes ago    "task: non-zero exit (143): do…"

➜  docker service logs -f proxy
proxy.1.gt1vsu5bnjaa@node-2    | 2017/12/05 07:27:42 Starting HAProxy
proxy.1.zdra6mlo909b@node-2    | 2017/12/05 07:13:32 Starting HAProxy
proxy.1.zdra6mlo909b@node-2    | 2017/12/05 07:13:32 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
proxy.1.gt1vsu5bnjaa@node-2    | 2017/12/05 07:27:42 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
proxy.1.7y4t91z4te5z@node-1    | 2017/12/05 07:28:10 Starting HAProxy
proxy.1.7y4t91z4te5z@node-1    | 2017/12/05 07:28:10 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
proxy.1.51jckzmy3bzz@node-2    | 2017/12/05 07:28:33 Starting HAProxy
proxy.1.51jckzmy3bzz@node-2    | 2017/12/05 07:28:33 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
proxy.1.triq8b7bx7v3@node-3    | 2017/12/05 07:28:56 Starting HAProxy
proxy.1.triq8b7bx7v3@node-3    | 2017/12/05 07:28:56 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
proxy.1.0bdyvrgtl777@node-1    | 2017/12/05 07:29:35 Starting HAProxy
proxy.1.0bdyvrgtl777@node-1    | 2017/12/05 07:29:35 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
^[proxy.1.qydef86rb0yc@node-3    | 2017/12/05 07:29:58 Starting HAProxy
proxy.1.qydef86rb0yc@node-3    | 2017/12/05 07:29:58 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs

Found the problem... It'll be fixed later today.

@shazChaudhry The problem is (probably) in the name of the service. Something in your network is hijicking requests to the address proxy. You can see that from the drills. Internal IPs of the other services start with 10. tasks.proxy, on the other hand, got 81.200.64.50 which is not the range of Overlay's internal addresses. I experienced the same in my hotel room an hour ago. As soon as I moved to a different WiFi, it worked.
Can you please try the following.

Proxy service: Change the name of the proxy service to, for example, dfp.
Proxy service: Add env. var. SERVICE_NAME=dfp.
Listener service: Change env. vars. DF_NOTIFY_CREATE_SERVICE_URL and DF_NOTIFY_REMOVE_SERVICE_URL to use dfp instead proxyas the address.

@eromoe In your case, point 3 does not apply since that article does not use the listener.
If that fixes the problem, I'll work on rewriting the docs so that they use a name different than proxy.

@vfarcic
Follow http://proxy.dockerflow.com/swarm-mode-auto/  , still same problem .
go-demo and proxy  both can not start.
ID                  NAME                MODE                REPLICAS            IMAGE                                       PORTS
i2ibaklzfylx        go-demo             replicated          0/1                 vfarcic/go-demo:latest
8n7vjyuvnmw7        go-demo-db          replicated          1/1                 mongo:latest
1oaiqtr1pbht        proxy               replicated          0/1                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tcp
xjzr8oyq7bhk        swarm-listener      replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest

@vfarcic
Follow http://proxy.dockerflow.com/swarm-mode-auto/
change proxy name to dfp, and env
docker service create --name swarm-listener \
    --network proxy \
    --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
    -e DF_NOTIFY_CREATE_SERVICE_URL=http://dfp:8080/v1/docker-flow-proxy/reconfigure \
    -e DF_NOTIFY_REMOVE_SERVICE_URL=http://dfp:8080/v1/docker-flow-proxy/remove \
    --constraint 'node.role==manager' \
    vfarcic/docker-flow-swarm-listener

docker service create --name dfp \
    -p 80:80 \
    -p 443:443 \
    --network proxy \
    -e LISTENER_ADDRESS=swarm-listener \
    -e SERVICE_NAME=dfp \
    vfarcic/docker-flow-proxy

dfp still same problem ,  go-demo and dfp both can not start:
ID                  NAME                MODE                REPLICAS            IMAGE                                       PORTS
e0fpaookmub8        dfp                 replicated          0/1                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tcp
i2ibaklzfylx        go-demo             replicated          0/1                 vfarcic/go-demo:latest
8n7vjyuvnmw7        go-demo-db          replicated          1/1                 mongo:latest
0h9730t67xiz        swarm-listener      replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest

@eromoe In that case, the problem is not with DFP but with your networking in general. go-demo is failing because it cannot connect to go-demo-db (without any proxy). The other two services are running because they are not depending on communication to other services through the Overlay networking. Since you're using Docker Machine to create nodes (assuming you're following that article as-is), the ports between the machines are open. Is it a company laptop you're using? Any VPN/proxy/firewall?

It is company real machine.  I didn't using  proxy/VPN , but I would check that again tomorrow :)

Ok, if the service name is not an issue I will then try a different WiFi tomorrow

@vfarcic Looks like you are right, this issue appears to be WiFi specifiic. Moving on to a different WiFi, I could no longer reproduce the issue.
So, it means my home WiFi does not like proxy version > vfarcic/docker-flow-proxy:17.11.11-56
:(

@vfarcic
Check again, no vpn/proxy running.
➜  docker-flow-proxy git:(master) ✗ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                       PORTS
qxl46r2hsl1z        dfp                 replicated          0/1                 vfarcic/docker-flow-proxy:latest            *:80->80/tcp,*:443->443/tcp
z0jrr80t50uo        go-demo             replicated          0/1                 vfarcic/go-demo:latest
101zwf7peciu        go-demo-db          replicated          1/1                 mongo:latest
vzmeydbo1qnv        swarm-listener      replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
➜  docker-flow-proxy git:(master) ✗ curl -i $(docker-machine ip node-1)/demo/hello
curl: (7) Failed connect to 192.168.99.100:80; Connection refused

the network is
vboxnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.99.1  netmask 255.255.255.0  broadcast 192.168.99.255
        inet6 fe80::800:27ff:fe00:0  prefixlen 64  scopeid 0x20<link>
        ether 0a:00:27:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 126294  bytes 7811670 (7.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

@eromoe The problem is not with DFP. If it was, go-demo would work. Can you send the logs of go-demo and dfp?

@vfarcic Sorry for late reply .
dfp:
➜  portal docker service logs -f dfp
dfp.1.gwdlannf7t9v@node-1    | 2017/12/07 05:14:34 Starting HAProxy
dfp.1.z3sbgskhmui4@node-1    | 2017/12/07 02:21:03 Starting HAProxy
dfp.1.gwdlannf7t9v@node-1    | 2017/12/07 05:14:34 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
dfp.1.zyzxifnief92@node-3    | 2017/12/07 00:43:49 Starting HAProxy
dfp.1.zyzxifnief92@node-3    | 2017/12/07 00:43:49 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
dfp.1.z3sbgskhmui4@node-1    | 2017/12/07 02:21:03 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs
dfp.1.xvt4z822gt6e@node-1    | 2017/12/07 02:30:04 Starting HAProxy
dfp.1.xvt4z822gt6e@node-1    | 2017/12/07 02:30:04 Getting certs from http://42.123.125.237:8080/v1/docker-flow-proxy/certs

go-demo:
➜  portal docker service logs -f go-demo
go-demo.1.wfqmcn8e8zbv@node-1    | panic: no reachable servers
go-demo.1.zzvlgzpimtpp@node-1    | panic: no reachable servers
go-demo.1.zzvlgzpimtpp@node-1    |
go-demo.1.wfqmcn8e8zbv@node-1    |
go-demo.1.wfqmcn8e8zbv@node-1    | goroutine 1 [running]:
go-demo.1.zzvlgzpimtpp@node-1    | goroutine 1 [running]:
go-demo.1.zzvlgzpimtpp@node-1    | panic(0x710f60, 0xc4200e0470)
go-demo.1.wfqmcn8e8zbv@node-1    | panic(0x710f60, 0xc4200d8420)
go-demo.1.wfqmcn8e8zbv@node-1    |      /usr/local/go/src/runtime/panic.go:500 +0x1a1
go-demo.1.zzvlgzpimtpp@node-1    |      /usr/local/go/src/runtime/panic.go:500 +0x1a1
go-demo.1.zzvlgzpimtpp@node-1    | main.setupDb()
go-demo.1.wfqmcn8e8zbv@node-1    | main.setupDb()
go-demo.1.wfqmcn8e8zbv@node-1    |      /src/main.go:63 +0x28d
go-demo.1.zzvlgzpimtpp@node-1    |      /src/main.go:63 +0x28d
go-demo.1.zzvlgzpimtpp@node-1    | main.main()
go-demo.1.wfqmcn8e8zbv@node-1    | main.main()
go-demo.1.wfqmcn8e8zbv@node-1    |      /src/main.go:46 +0x49
go-demo.1.zzvlgzpimtpp@node-1    |      /src/main.go:46 +0x49
go-demo.1.zlxlybpbn30i@node-1    | panic: no reachable servers

@eromoe From DFP logs you can see that the IP is wrong. If it would be using Overlay network, it would start with 10.. If that would be the only manifestation of the problem, I'd say that your network does not like address proxy. However, the go-demo logs indicate that it also cannot connect to, in this case,  db. So, there's definitely something wrong with your Overlay network, not DFP. I assume that those are "real" servers, not Docker Machine running on your laptop. Since you already said that there is no VPN/firewall/proxy, the only thing that comes to my mind is that ports between nodes are not opened. Can you confirm that the ports required by Swarm are opened?

I'm having the same issue.  Anyone else on Virgin Media?  I know they send you to their search page when a DNS name can't be resolved.

@rooch84 Did you try using SERVICE_NAME and other env. vars. I described above?

I am on Virgin Media, UK, and have never had this issue until a version > vfarcic/docker-flow-proxy:17.11.11-56. However, this might be a wider issue than Virgin Media WiFi as other people have reported this issue while on a hotel WiFi and in other countries.
Would be really good to have this issue resolved. Otherwise I might never be able to use the latest version of docker-flow-proxy from home again :(

@vfarcic
service ls
➜  /root docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                       PORTS
dok84ulshs2d        go-demo             replicated          0/1                 vfarcic/go-demo:latest
101zwf7peciu        go-demo-db          replicated          1/1                 mongo:latest
f4ucg31ugds5        n1                  replicated          1/1                 nginx:latest                                *:5566->80/tcp
xc9136rf60ho        n2                  replicated          1/1                 nginx:latest                                *:5577->80/tcp
wiezzp1kxfwi        n3                  replicated          1/1                 nginx:latest                                *:5588->80/tcp
bsg5gq79gtxf        proxy               replicated          0/1                 vfarcic/docker-flow-proxy:latest            *:5580->80/tcp,*:55443->443/tcp
kb1071jkvxot        swarm-listener      replicated          1/1                 vfarcic/docker-flow-swarm-listener:latest
j6jo9h8h6oh3        test                replicated          1/1                 nginx:latest
r04cn4q3afuv        u                   replicated          1/1                 phusion/baseimage:latest

u, n1, n2, n3 are on overlay network proxy,  test are not on network proxy.
u, n1, n2 are on node-1 (manager node)
n3 is on node-2

node-1 docker ps
➜  /root docker ps
0aaf5e056958        nginx:latest                                "nginx -g 'daemon ..."   About an hour ago   Up About an hour       80/tcp              test.1.ya99q60h1gnx07jdjg6ze9c11
28ffa26b998a        phusion/baseimage:latest                    "/sbin/my_init"          About an hour ago   Up About an hour                           u.1.fck51ccx16dc0kg3r5j9utrcz
c69cfc85113d        nginx:latest                                "nginx -g 'daemon ..."   About an hour ago   Up About an hour       80/tcp              n2.1.oile1a2vvvwscfly2d2wfr0pd
4725fbdd14ed        nginx:latest                                "nginx -g 'daemon ..."   About an hour ago   Up About an hour       80/tcp              n1.1.87q7lnggj0cl1zyhfoh4l9tgb
a9edba2a10d2        vfarcic/docker-flow-swarm-listener:latest   "docker-flow-swarm..."   2 hours ago         Up 2 hours (healthy)   8080/tcp            swarm-listener.1.7akd26oov9h5mc1pb7bp8bwbi
9995ab7449f7        mongo:latest                                "docker-entrypoint..."   22 hours ago        Up 22 hours            27017/tcp           go-demo-db.1.igryex3mwvbdexyyivfkppxjj

node-2 docker ps
➜  scripts git:(master) ✗ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                             PORTS                       NAMES
95c759684a1e        nginx:latest                       "nginx -g 'daemon ..."   15 seconds ago      Up 14 seconds                      80/tcp                      n3.1.k5zg3baj9ngx44x4ye0aj5go9
c3ae9fc03fd3        vfarcic/docker-flow-proxy:latest   "/sbin/tini -- doc..."   18 seconds ago      Up 12 seconds (health: starting)   80/tcp, 443/tcp, 8080/tcp   proxy.1.r5csmqs08h0yr0dy3uyf0tnb2

enter container u, and do test
docker exec -it u bash

Can access n1,n2,n3 by name:
curl n1.1.87q7lnggj0cl1zyhfoh4l9tgb:80
curl n2.1.oile1a2vvvwscfly2d2wfr0pd:80
curl n3.1.k5zg3baj9ngx44x4ye0aj5go9:80

But I can't access:
test.1.ya99q60h1gnx07jdjg6ze9c11

So, does it mean the DNS problem ?

@shazChaudhry I agree that there a greater problem here that needs to be solved.  I'm using Docker for other purposes but landed on this page because I searched for the unusual IP.  I've also been using Docker from home for several years and not had an issue, which suggests a recent change to Docker itself.
In the mean team, I stumbled across this, which appears to solve it for we Virgin Media users: https://my.virginmedia.com/advancederrorsearch/

@rooch84 - Thank you very much for the suggestion. My issue was indeed down to my Virgin Media WiFi router. Turning off "advanced network error search" has resolved the issue for me.

@vfarcic I will leave it up to you whether you would like to close this issue. I would like to thank you very much for responding quickly and trying to help.
I really appreciate it.

I think you could have solved the problem by changing the name of the proxy service. Still, that option should not exist in your router so you're probably better of maintaining it disabled.
Closing the issue...

I tried to create an other network overlay. Then try  run again with new network. And it works for me. I don't know is it a bug of docker-flow-proxy.

source: gharchive/issue

authors: dmelo eromoe nguyenngoctranvu rooch84 shazChaudhry vfarcic

repo: vfarcic/docker-flow-proxy

license: mit

license_source: bigquery


text:
Improve ITCH parser and writer

See if we can speedup parser.

Potential ideas for speedup:

Avoid converting to string for comparison in long if...
Reorder message in if based on frequency.

New improved output formats:

As (rich) markdown
As JSON
As arrow-ready format (common for all messages, to store in parquet file)

The markdown output is for interactive work and for the CLI
The JSON output is to simplify development and debugging.
The arrow format is to be able to store historical files in parquet format for easy searching and extraction. That way, when we want to look at a subset of stocks on a given day, we can easily query the messages related to those symbol/days and process them.

repo: vgreg/MeatPy


text:
make -C accesshttp 编译失败是什么问题？

chenke@ubuntu:~/viabtc_exchange_server$ sudo make -C accesshttp
make: Entering directory '/home/chenke/viabtc_exchange_server/accesshttp'
gcc -Wall -Wno-strict-aliasing -Wno-uninitialized -g -rdynamic -std=gnu99 -c -o ah_config.o ah_config.c -I ../network -I ../utils -I ../depends
gcc -Wall -Wno-strict-aliasing -Wno-uninitialized -g -rdynamic -std=gnu99 -c -o ah_main.o ah_main.c -I ../network -I ../utils -I ../depends
gcc -Wall -Wno-strict-aliasing -Wno-uninitialized -g -rdynamic -std=gnu99 -c -o ah_listener.o ah_listener.c -I ../network -I ../utils -I ../depends
gcc -Wall -Wno-strict-aliasing -Wno-uninitialized -g -rdynamic -std=gnu99 -c -o ah_server.o ah_server.c -I ../network -I ../utils -I ../depends
gcc -g -rdynamic -o accesshttp.exe  ah_config.o  ah_main.o  ah_listener.o  ah_server.o -L ../utils -lutils -L ../network -lnetwork -L ../depends/hiredis -Wl,-Bstatic -lev -ljansson -lmpdec -lrdkafka -lz -lssl -lcrypto -lhiredis -lcurl -Wl,-Bdynamic -lm -lpthread -ldl -lssl -lldap -llber -lgss -lgnutls -lidn -lnettle -lrtmp -lsasl2 -lmysqlclient
/usr/bin/x86_64-linux-gnu-ld: /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libev.a(ev.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
../makefile.inc:16: recipe for target 'accesshttp.exe' failed
make: *** [accesshttp.exe] Error 1
make: Leaving directory '/home/chenke/viabtc_exchange_server/accesshttp'
chenke@ubuntu:~/viabtc_exchange_server$

建了一个交易所撮合交流群： 862266237

Same issue

I had the same problem

@xiaoxiaoleo @klpeng
Try adding "-no-pie" option to makefile.inc

已解决

authors: chencore klpeng objectt xiaoxiaoleo xtuyaowu


text:
Ability to disable the radio even if its supported?

Is there any way to disable the radio section even if its supported by the Server?
Would be awesome if it would be possible to disable it for certain cases.

Hello, thank you for reaching out. These types of customizations are planned to be included in the settings page, but I can't say exactly when that will happen. However, it's definitely part of the plan.

repo: victoralvesf/aonsoku

license: MIT


text:
Create Matrix.py

how do i convert strings to float?

Dear Kali123
Thank you for preparing your first pull request.  Integration of new code in the project usually follows exchange of email which makes clear (1) which problem is there, (2) how your patch intends to solve  the problem, and (3) some sort of proof that it happenned.  Finally (4) each project is written in its unique style and new code should try to follow it.  In its current state your patch unacceptable.
Victor

repo: victorlei/smop


text:
Visual asset has not tidied up

I was playing around with this generator and found that for one of the asset was probably cropped a little off, so there is a red margin at the right bottom corner.

yeah, I noticed it to. I'll try to fix it later

fixed https://github.com/victorqribeiro/isocity/commit/ccb6523289dd5e65a0a3971e72cd2b0dc282d8c7

authors: JerryWaaaaaaaa victorqribeiro

repo: victorqribeiro/isocity


text:
browserSync proxy and HMR ... is it possible?

Let me start off by saying thanks for all of the hard work that has been poured into these build tools.
Over the weekend I had been working on a project and while it works wonderful on localhost. Eventually I'd love to switch over to a proxy setup with browserSync. I primarily utilize this method for local WP or Craft development.
I've looked throughout browserSync.js and noticed there were proxy calls. And while I was able to pass in a proxy address in, everything setup wise works. It's just that HMR will not trigger.
My guess is that is has something to do with the public path. I've also been researching online and come across discussions that webpack-hot-module doesn't support proxy setups.
https://github.com/webpack/webpack-dev-middleware/issues/67
http://code.fitness/post/2016/02/webpack-public-path-and-hot-reload.html
I ended up changing the proxyConfig to var proxyConfig = "l.local-url" || null; to work. And in the config
"browserSync": {
  "server": false
}

Any help would be helpful.

For me, HMR with BrowserSync proxying is actually working out of the box – the only thing that tripped me up initially was that I've set the root.dest property in config.json to a sub folder ("./public/assets", in my case), which means that BrowserSync will use the wrong path for HMR because in the vanilla setup the public path is equal to the config.tasks.js.dest path.
Here's what I did to make it work:

I added a new property to the browserSync task in config.json, called path:

"browserSync": {
    "proxy": "example.dev",
    "path": "assets"
},

I added the browserSync.path property to the public path in /tasks/browserSync.js:

server.middleware = [
    require('webpack-dev-middleware')(compiler, {
      stats: 'errors-only',
      publicPath: pathToUrl('/', config.tasks.browserSync.path, webpackConfig.output.publicPath)
    }),
    require('webpack-hot-middleware')(compiler)
  ]

That's it; works like a charm.

@mmikkel @besimhu I similarly went through this exact process once I got into developing the templates in Wordpress.
Essentially my folder structure ended up with:
-wp-content
--themes
---<our-theme>
----gulp starter is in here with the standard theme files

Borrowing from @mmikkel above I ended up with:
{
  "root": {
    "src": "./src",
    "dest": "./assets"
  },

  "tasks": {
    "browserSync": {
      "proxy": "http://<our-theme>.dev",
      "path": "wp-content/themes/<our-theme>/assets"
    },

Which sort of got me thinking and wondering if there shouldn't be a npm run proxy that watches CMS files (in this case php) and builds assets to a different destination than the static site to save clogging up the assets folder with the *.html files we're not currently focusing on?
Typically in these cases I find most people will just start to kick out their JS and CSS with a copy task and call it a day, but if we want BrowserSync and Webpack still firing on all cylinders I think it makes sense to have a npm run cms-start and npm run start doing separate things?
What do you guys think?

@mmikkel where did you define pathToUrl in your solution?

@dillonbailey you can pass a files option to the browserSync config to tell it what files to watch and reload on (CMS files). I usually point it at the templates folder for whatever platform i'm on (Rails, Craft, Django, etc).
https://browsersync.io/docs/options#option-files

@mmikkel @dillonbailey just wanted to throw out a very late thank you. I had forgotten to check on this until recently. I ended up switching to a browserify approach. But yes, I too ended up going a similar approach later.
The path option makes sense. Here's an example of how I have it working with blendit:
// task-config.js
browserSync: {
  path: 'wp-content/themes/theme-name/assets',
  server: false,
  proxy: 'proxy-url.dev',
  open: true,
  notify: false,
  reloadOnRestart: true,
  files: [
    './**/*.php',
    '!node_modules/'
  ]
},

... and in gulpfile.js/tasks/browserSync.js replace line 40 with
publicPath: pathToUrl('/', TASK_CONFIG.browserSync.path, webpackConfig.output.publicPath)
All my files are built in the /assets/ folder with the exception of styles. Which is sent to the theme root.
  "src": "stylesheets",
  "dest": "./../"
},```

@besimhu with the latest blendid I don't think the adjustment to gulpfile.js/tasks/browserSync.js is necessary but I haven't tested with the latest.
Glad you got it working!

@dillonbailey I wasn't able to get it working without the adjustment ... unless of course I handled something differently.

authors: besimhu dillonbailey greypants mmikkel stevieg83

repo: vigetlabs/blendid