text:
Failure in pixel sampler when using depth-nerfacto

Attempting to use depth-nerfacto with the following transforms.json:
{
    "frames": [
        {
            "camera_angle_x": 1.6206331253051758,
            "camera_angle_y": 1.0682151317596436,
            "cx": 1922.9263916015625,
            "cy": 1095.8277587890625,
            "depth_file_path": "sensor0\\depth\\000001.png",
            "file_path": "sensor0\\rgb\\000001.png",
            "fl_x": 1826.620849609375,
            "fl_y": 1826.0267333984375,
            "h": 2160,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    0.024848394095897675,
                    -0.3371071219444275,
                    0.9411383271217346,
                    1.4621940851211548
                ],
                [
                    0.9996904134750366,
                    0.009575003758072853,
                    -0.022964637726545334,
                    -0.03897222876548767
                ],
                [
                    -0.0012698600767180324,
                    0.9414176344871521,
                    0.33724066615104675,
                    1.6487826108932495
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 3840,
            "x_fov": 92.85543823242188,
            "y_fov": 61.204219818115234
        },
        {
            "camera_angle_x": 1.6187852621078491,
            "camera_angle_y": 1.066586971282959,
            "cx": 1282.184814453125,
            "cy": 726.3068237304688,
            "depth_file_path": "sensor1\\depth\\000001.png",
            "file_path": "sensor1\\rgb\\000001.png",
            "fl_x": 1220.0023193359375,
            "fl_y": 1219.6160888671875,
            "h": 1440,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    0.7412542700767517,
                    -0.059115394949913025,
                    0.6686161160469055,
                    1.484533429145813
                ],
                [
                    0.6712237000465393,
                    0.06387709826231003,
                    -0.7384974360466003,
                    -1.4292242527008057
                ],
                [
                    0.0009473099489696324,
                    0.9962053298950195,
                    0.08702882379293442,
                    1.1107122898101807
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 2560,
            "x_fov": 92.74956512451172,
            "y_fov": 61.110931396484375
        },
        {
            "camera_angle_x": 1.6254974603652954,
            "camera_angle_y": 1.0724430084228516,
            "cx": 1279.8321533203125,
            "cy": 728.6819458007813,
            "depth_file_path": "sensor2\\depth\\000001.png",
            "file_path": "sensor2\\rgb\\000001.png",
            "fl_x": 1211.830078125,
            "fl_y": 1211.4990234375,
            "h": 1440,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    0.675960123538971,
                    0.0888102799654007,
                    -0.7315672636032104,
                    -1.3041361570358276
                ],
                [
                    -0.7369378805160522,
                    0.08242889493703842,
                    -0.6709158420562744,
                    -1.35117769241333
                ],
                [
                    0.0007180602988228202,
                    0.9926319718360901,
                    0.12116630375385284,
                    1.1181082725524902
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 2560,
            "x_fov": 93.13414764404297,
            "y_fov": 61.44646072387695
        },
        {
            "camera_angle_x": 1.6216586828231812,
            "camera_angle_y": 1.0688626766204834,
            "cx": 1280.57275390625,
            "cy": 728.6508178710938,
            "depth_file_path": "sensor3\\depth\\000001.png",
            "file_path": "sensor3\\rgb\\000001.png",
            "fl_x": 1216.4974365234375,
            "fl_y": 1216.4522705078125,
            "h": 1440,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    -0.7251179814338684,
                    0.0811152532696724,
                    -0.6838305592536926,
                    -1.235578179359436
                ],
                [
                    -0.688602864742279,
                    -0.09331951290369034,
                    0.7191088795661926,
                    1.4263954162597656
                ],
                [
                    -0.005484039429575205,
                    0.9923264384269714,
                    0.12352384626865387,
                    1.127084493637085
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 2560,
            "x_fov": 92.91419982910156,
            "y_fov": 61.2413215637207
        },
        {
            "camera_angle_x": 1.6239713430404663,
            "camera_angle_y": 1.0713775157928467,
            "cx": 1281.7987060546875,
            "cy": 730.7208251953125,
            "depth_file_path": "sensor4\\depth\\000001.png",
            "file_path": "sensor4\\rgb\\000001.png",
            "fl_x": 1213.68359375,
            "fl_y": 1212.969970703125,
            "h": 1440,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    -0.704884946346283,
                    -0.06748456507921219,
                    0.7061041593551636,
                    1.5576385259628296
                ],
                [
                    0.7092986106872559,
                    -0.0590321384370327,
                    0.7024319767951965,
                    1.3530181646347046
                ],
                [
                    -0.005720482673496008,
                    0.9959723949432373,
                    0.08947757631540298,
                    1.1262444257736206
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 2560,
            "x_fov": 93.04670715332031,
            "y_fov": 61.38541030883789
        },
        {
            "camera_angle_x": 1.6218918561935425,
            "camera_angle_y": 1.069124698638916,
            "cx": 1278.5623779296875,
            "cy": 736.3524780273438,
            "depth_file_path": "sensor5\\depth\\000001.png",
            "file_path": "sensor5\\rgb\\000001.png",
            "fl_x": 1216.21337890625,
            "fl_y": 1216.088623046875,
            "h": 1440,
            "k1": 0,
            "k2": 0,
            "k3": 0,
            "k4": 0,
            "p1": 0,
            "p2": 0,
            "transform_matrix": [
                [
                    0.026518484577536583,
                    -0.9629524946212769,
                    0.2683640122413635,
                    0.6008144021034241
                ],
                [
                    0.9993513822555542,
                    0.032080307602882385,
                    0.016360348090529442,
                    0.005734376609325409
                ],
                [
                    -0.024363437667489052,
                    0.26775607466697693,
                    0.9631786346435547,
                    2.4697155952453613
                ],
                [
                    0.0,
                    -0.0,
                    0.0,
                    1.0
                ]
            ],
            "w": 2560,
            "x_fov": 92.92755889892578,
            "y_fov": 61.25633239746094
        }
    ]
}

PS C:\Users\tim\repos\nerfstudio> ns-train depth-nerfacto --data .\data\custom\MISENBERG_T04_06_03_12_11_06_Export_08_28_15_12_37\
C:\Users\tim\repos\nerfstudio\nerfstudio\utils\misc.py:184: RuntimeWarning: Windows does not yet support torch.compile and the performance will be affected.
  warnings.warn(
[13:23:19] Using --data alias for --data.pipeline.datamanager.data                                          train.py:231
──────────────────────────────────────────────────────── Config ────────────────────────────────────────────────────────
TrainerConfig(
    _target=<class 'nerfstudio.engine.trainer.Trainer'>,
    output_dir=WindowsPath('outputs'),
    method_name='depth-nerfacto',
    experiment_name=None,
    project_name='nerfstudio-project',
    timestamp='2023-08-30_132319',
    machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
    logging=LoggingConfig(
        relative_log_dir=WindowsPath('.'),
        steps_per_log=10,
        max_buffer_size=20,
        local_writer=LocalWriterConfig(
            _target=<class 'nerfstudio.utils.writer.LocalWriter'>,
            enable=True,
            stats_to_track=(
                <EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
                <EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
                <EventName.CURR_TEST_PSNR: 'Test PSNR'>,
                <EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
                <EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
                <EventName.ETA: 'ETA (time)'>
            ),
            max_log_size=10
        ),
        profiler='basic'
    ),
    viewer=ViewerConfig(
        relative_log_filename='viewer_log_filename.txt',
        websocket_port=None,
        websocket_port_default=7007,
        websocket_host='0.0.0.0',
        num_rays_per_chunk=32768,
        max_num_display_images=512,
        quit_on_train_completion=False,
        image_format='jpeg',
        jpeg_quality=90
    ),
    pipeline=VanillaPipelineConfig(
        _target=<class 'nerfstudio.pipelines.base_pipeline.VanillaPipeline'>,
        datamanager=VanillaDataManagerConfig(
            _target=nerfstudio.data.datamanagers.base_datamanager.VanillaDataManager[nerfstudio.data.datasets.depth_data
set.DepthDataset],
            data=WindowsPath('data/custom/MISENBERG_T04_06_03_12_11_06_Export_08_28_15_12_37'),
            camera_optimizer=CameraOptimizerConfig(
                _target=<class 'nerfstudio.cameras.camera_optimizers.CameraOptimizer'>,
                mode='SO3xR3',
                position_noise_std=0.0,
                orientation_noise_std=0.0,
                optimizer=AdamOptimizerConfig(
                    _target=<class 'torch.optim.adam.Adam'>,
                    lr=0.0006,
                    eps=1e-08,
                    max_norm=None,
                    weight_decay=0.01
                ),
                scheduler=ExponentialDecaySchedulerConfig(
                    _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                    lr_pre_warmup=1e-08,
                    lr_final=None,
                    warmup_steps=0,
                    max_steps=10000,
                    ramp='cosine'
                ),
                param_group='camera_opt'
            ),
            masks_on_gpu=False,
            images_on_gpu=False,
            dataparser=NerfstudioDataParserConfig(
                _target=<class 'nerfstudio.data.dataparsers.nerfstudio_dataparser.Nerfstudio'>,
                data=WindowsPath('.'),
                scale_factor=1.0,
                downscale_factor=None,
                scene_scale=1.0,
                orientation_method='up',
                center_method='poses',
                auto_scale_poses=True,
                eval_mode='fraction',
                train_split_fraction=0.9,
                eval_interval=8,
                depth_unit_scale_factor=0.001
            ),
            train_num_rays_per_batch=4096,
            train_num_images_to_sample_from=-1,
            train_num_times_to_repeat_images=-1,
            eval_num_rays_per_batch=4096,
            eval_num_images_to_sample_from=-1,
            eval_num_times_to_repeat_images=-1,
            eval_image_indices=(0,),
            collate_fn=<function nerfstudio_collate at 0x000001966C488430>,
            camera_res_scale_factor=1.0,
            patch_size=1,
            pixel_sampler=PairPixelSamplerConfig(
                _target=<class 'nerfstudio.data.pixel_samplers.PairPixelSampler'>,
                num_rays_per_batch=4096,
                keep_full_image=False,
                is_equirectangular=False,
                radius=2
            )
        ),
        model=DepthNerfactoModelConfig(
            _target=<class 'nerfstudio.models.depth_nerfacto.DepthNerfactoModel'>,
            enable_collider=True,
            collider_params={'near_plane': 2.0, 'far_plane': 6.0},
            loss_coefficients={'rgb_loss_coarse': 1.0, 'rgb_loss_fine': 1.0},
            eval_num_rays_per_chunk=32768,
            prompt=None,
            near_plane=0.05,
            far_plane=1000.0,
            background_color='last_sample',
            hidden_dim=64,
            hidden_dim_color=64,
            hidden_dim_transient=64,
            num_levels=16,
            base_res=16,
            max_res=2048,
            log2_hashmap_size=19,
            features_per_level=2,
            num_proposal_samples_per_ray=(256, 96),
            num_nerf_samples_per_ray=48,
            proposal_update_every=5,
            proposal_warmup=5000,
            num_proposal_iterations=2,
            use_same_proposal_network=False,
            proposal_net_args_list=[
                {'hidden_dim': 16, 'log2_hashmap_size': 17, 'num_levels': 5, 'max_res': 128, 'use_linear': False},
                {'hidden_dim': 16, 'log2_hashmap_size': 17, 'num_levels': 5, 'max_res': 256, 'use_linear': False}
            ],
            proposal_initial_sampler='piecewise',
            interlevel_loss_mult=1.0,
            distortion_loss_mult=0.002,
            orientation_loss_mult=0.0001,
            pred_normal_loss_mult=0.001,
            use_proposal_weight_anneal=True,
            use_average_appearance_embedding=True,
            proposal_weights_anneal_slope=10.0,
            proposal_weights_anneal_max_num_iters=1000,
            use_single_jitter=True,
            predict_normals=False,
            disable_scene_contraction=False,
            use_gradient_scaling=False,
            implementation='tcnn',
            appearance_embed_dim=32,
            depth_loss_mult=0.001,
            is_euclidean_depth=False,
            depth_sigma=0.01,
            should_decay_sigma=False,
            starting_depth_sigma=0.2,
            sigma_decay_rate=0.99985,
            depth_loss_type=<DepthLossType.DS_NERF: 1>
        )
    ),
    optimizers={
        'proposal_networks': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.01,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'fields': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.01,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        }
    },
    vis='viewer',
    data=WindowsPath('data/custom/MISENBERG_T04_06_03_12_11_06_Export_08_28_15_12_37'),
    prompt=None,
    relative_model_dir=WindowsPath('nerfstudio_models'),
    load_scheduler=True,
    steps_per_save=2000,
    steps_per_eval_batch=500,
    steps_per_eval_image=500,
    steps_per_eval_all_images=25000,
    max_num_iterations=30000,
    mixed_precision=True,
    use_grad_scaler=False,
    save_only_latest_checkpoint=True,
    load_dir=None,
    load_step=None,
    load_config=None,
    load_checkpoint=None,
    log_gradients=False,
    gradient_accumulation_steps=1
)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
           Saving config to:                                                                    experiment_config.py:136
           outputs\MISENBERG_T04_06_03_12_11_06_Export_08_28_15_12_37\depth-nerfacto\2023-08-30
           _132319\config.yml
           Saving checkpoints to:                                                                         trainer.py:135
           outputs\MISENBERG_T04_06_03_12_11_06_Export_08_28_15_12_37\depth-nerfacto\2023-08-30_132319\ne
           rfstudio_models
           Auto image downscale factor of 1                                                 nerfstudio_dataparser.py:349
Variable resolution, using variable_res_collate
Setting up training dataset...
Caching all 6 images.
╭─────────────────────────────────────────── Viewer ───────────────────────────────────────────╮
│        ╷                                                                                │
│   HTTP │ https://viewer.nerf.studio/versions/23-05-15-1/?websocket_url=ws://localhost:7007   │
│        ╵                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
[NOTE] Not running eval iterations since only viewer is enabled.
Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval.
No Nerfstudio checkpoint to load, so training from scratch.
Disabled tensorboard/wandb event writers
Printing profiling stats, from longest to shortest duration in seconds
VanillaPipeline.get_train_loss_dict: 0.0010
Trainer.train_iteration: 0.0010
Traceback (most recent call last):
  File "c:\users\tim\.pyenv\pyenv-win\versions\3.8.10\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\tim\.pyenv\pyenv-win\versions\3.8.10\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\tim\.pyenv\pyenv-win\versions\3.8.10\Scripts\ns-train.exe\__main__.py", line 7, in <module>
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\scripts\train.py", line 261, in entrypoint
    main(
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\scripts\train.py", line 246, in main
    launch(
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\scripts\train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\scripts\train.py", line 100, in train_loop
    trainer.train()
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\engine\trainer.py", line 255, in train
    loss, loss_dict, metrics_dict = self.train_iteration(step)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\utils\profiler.py", line 127, in inner
    out = func(*args, **kwargs)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\engine\trainer.py", line 474, in train_iteration
    _, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\utils\profiler.py", line 127, in inner
    out = func(*args, **kwargs)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\pipelines\base_pipeline.py", line 298, in get_train_loss_dict
    ray_bundle, batch = self.datamanager.next_train(step)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\data\datamanagers\base_datamanager.py", line 540, in next_train
    batch = self.train_pixel_sampler.sample(image_batch)
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\data\pixel_samplers.py", line 270, in sample
    pixel_batch = self.collate_image_dataset_batch_list(
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\data\pixel_samplers.py", line 243, in collate_image_dataset_batch_list
    collated_batch = {
  File "C:\Users\tim\repos\nerfstudio\nerfstudio\data\pixel_samplers.py", line 244, in <dictcomp>
    key: value[c, y, x]
TypeError: list indices must be integers or slices, not tuple

I'm also getting this error using the main branch.

authors: MartinEthier tim-depthkit

repo: nerfstudio-project/nerfstudio


text:
Multiple serializers for one model

Hi all!
As I already suggested in #59, we can use tag option for splitting serializers. I propose this API:
class HumanSerializer < Surrealist::Serializer
  json_schema do
    { name: String, name_length: Integer }
  end

  def name_length
    name.length
  end
end

class SmallHumanSerializer < Surrealist::Serializer
  json_schema do
    { name: String }
  end
end

class Human
  include Surrealist

  attr_reader :name

  surrealize_with HumanSerializer # default
  surrealize_with SmallHumanSerializer, tag: :small

  def initialize(name)
    @name = name
  end
end

# default behaviour
Human.new('John').surrealize(camelize: true)
# => '{"name": "John", "nameLength": 4}'

# using tags
Human.new('John').surrealize(camelize: true, tag: :small)
# => '{"name": "John"}'

I can realize this functionality and I know that it will be very useful feature.

I like this option and the PR but I think we can do better than tag.
I think as works for both setting and using.
We could differentiate using by making the option with.
@nesaulov ?
Just a minor nitpick.

@AlessandroMinali I don't think with will do, it just looks bad (two withs):
class Something
  include Surrealist

  surrealize_with SomethingSerializer, with: :api
end

as seems okay, but honestly I don't consider it to be better than tag, I think tag is more semantically descriptive in our case.
User.find(34).surrealize(as: :api)
User.find(34).surrealize(tag: :api)

source: gharchive/issue

authors: AlessandroMinali nesaulov nulldef

repo: nesaulov/surrealist

license: mit

license_type: permissive

license_source: bigquery


text:
Import Texture2D from a bundle to a asset

Hello! i need/want import a Texture2D from an asset, i want do this without decode the texture via 'WriteToByteArray' but only return this:

could you tell me how I can do it? thanks

So it is mandatory to transform it into png and then import It again vía image data?

okey, let me see.

Yes, that worked, thank you.

license: MIT

license_source: github-api


text:
Performance Improvement: use Map for storage in ThrottlerStorageService

Is there an existing issue that is already proposing this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe it
The current implementation of the ThrottlerStorageService  uses a plain JavaScript object Record<string, ThrottlerStorageOptions> like key value storage.  Replacing the plain object with a Map could yield significant performance improvements due to the optimized key-value operations provided by the Map data structure.
Describe the solution you'd like
use Map  like key value storage Instead of plain JS object in this place https://github.com/nestjs/throttler/blob/master/src/throttler.service.ts#L11
Teachability, documentation, adoption, migration strategy
proofs: https://www.zhenghao.io/posts/object-vs-map and https://stackoverflow.com/questions/18541940/map-vs-object-in-javascript
What is the motivation / use case for changing the behavior?
because this is a highload service and speeding up read and write operation to storage would be a nice  performance improvement

I agree with the change. Feel free to open a PR
Just keep in mind that the performance depends on how your app is using it. I saw several benchmarks where access to a plain object were faster. Sometimes Map was faster. Maybe this could change in future releases of v8 idk

I agree with the change. But since ThrottlerStorageService is a public API, we can't change the return type of get storage for now otherwise we'll introduce a breaking change
Just keep in mind that the performance depends on how your app is using it. I saw several benchmarks where access to a plain object were faster. Sometimes Map was faster. Maybe this could change in future releases of v8 idk

Since Map is also an object, it can be written and accessed via the [] operator. The only problem is that the keys written inside ThrottlerStorageService will not be accessible from the outside

Would you like to create a PR to address it?

authors: melonges micalevisk

repo: nestjs/throttler


text:
Support for comments and sections

Usual format of google spreadsheet:
keys, english, german, state_column, comments
csv2strings

[x] put comments above translation
[x] macro, comment should go to macro NSLocalizedStringFromTable(@"#{key}",@"#{table}",@"{comment}")

strings2csv

[x] extract comment to new column

ignore sections for now, don't really need it since keys are formatted with domains

repo: netbe/Babelish


text:
试了里面中文名字的语音，女：4519、6865、7143，男：7556、964这几个尚且能用

其它英文名的没去试，试过几个都是洋人说中文的口音。

感谢分享

大家有试出感觉还可以用的也可以分享一下。说实在的，情感这一块还得加强。而且普遍还不够自然（对比微软的），难道得自己训练吗？

效果太差，不好用

authors: WeiminLee leeeex zhy844694805

repo: netease-youdao/EmotiVoice


text:
Stop building the old dashboard

Description
This PR removes the old dashboard from the build matrix, as (i) we are already using a pinned version, (ii) it does no longer build correctly.

/merge

source: gharchive/pull-request

repo: netgroup-polito/CrownLabs


text:
Netlify dev server hangs if function returns 404

This is a bug report.
Netlify dev server hangs when returning status code 404. This occurs when used with Eleventy dev server, though I am not sure if this is related.
I suspect this is somehow related to proxy behaviour - maybe there is some kind of conflict between servers handling 404.

Set up Elventy + Netlify dev as instructed here https://github.com/philhawksworth/eleventyone

Create a following function bug404.js

exports.handler = async (event, context) => {

    return {
        statusCode: 404,
        body: "this hangs"
    }
};

Start server

npx netlify dev

(starts eleventy serve on the background)

Call function using curl

curl --request POST --data "xxx" "http://localhost:8888/api/bug404"

What is the expected behaviour?
The dev server returns the response immediately.
What is the actual behaviour?
The server never finishes processing the response and the curl waits out for it forever. You need to abort with CTRL+C.
The behaviour is the same under actual browsers. AJAX POST, e.g. initiated by jQuery, never returns when status code is set to 404.
However, if you change the status code e.g. to 403, or 200, it works:
exports.handler = async (event, context) => {

    return {
        statusCode: 403,
        body: "this does not hang"
    }
};

curl call returns immediately.
Local Environment Information
  System:
    OS: macOS Mojave 10.14
    CPU: (4) x64 Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  Binaries:
    Node: 11.0.0 - ~/.nvm/versions/node/v11.0.0/bin/node
    Yarn: 1.21.1 - ~/code/capitalgram/node_modules/.bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v11.0.0/bin/npm
  Browsers:
    Chrome: 79.0.3945.130
    Firefox: 72.0.2
    Safari: 12.0
  npmGlobalPackages:
    netlify: 3.0.0

I'm having the same problem. I have a function that I expect to return a 404 in certain cases, and if I send a request with postman, it hangs, and no response is returned. Any other statusCode resolves successfully without modifying any of the other code.
If I then cancel the request in Postman, netlify dev hits an exception and stops (Socket hang up or something similar).
Looks like it's easily reproducible, but if more info is needed I can provide some, or try and set up a codesandbox to demo the issue. I hope this gets worked on!

Actually, to provide an update, this only seems to be happen with nested functions (ie. functions inside subfolders within my functions directory). I copied the pattern implemented in the netlify workshop REST API example (with the API methods being called from a subfolder using a switch statement in a file in the root of the functions directory). When I return a 404 from the API method in the subfolder, the 404 seems to be returned, but then netlify starts looking for html files of the same name. Maybe this is a redirects issue?
Here's a sample of a the logs from a request that returns a 404:
Request from ::1: GET /.netlify/functions/songs/kanye-west-jesus-walkss
api/read
  api/read called
  db-connect
    Starting attempt to connect to db.
    Setting Mongo URI to dev DB
  Error occurred running read: Could not find matching song
Response with status 404 in 1960 ms.
Request from ::1: GET /.netlify/functions/songs/kanye-west-jesus-walkss.html
api/read
  api/read called
  db-connect
    Starting attempt to connect to db.
    Using cached database instance
  Error occurred running read: Could not find matching song
Response with status 404 in 77 ms.
Request from ::1: GET /.netlify/functions/songs/kanye-west-jesus-walkss.htm
api/read
  api/read called
  db-connect
    Starting attempt to connect to db.
    Using cached database instance
  Error occurred running read: Could not find matching song
Response with status 404 in 77 ms.
Request from ::1: GET /.netlify/functions/songs/kanye-west-jesus-walkss/index.html
segments length: 2
Response with status 500 in 1 ms.

I did create a test function called 404test (file: 404test.js) in the root of my functions dir, and it does successfully return 404 to postman when called. Super strange.

I haven't been able to reproduce this while trying in both root functions directory and a subdirectory function. Could you please share a minimal reproducible example!

@miohtama @miohtama:

I haven't been able to reproduce this while trying in both root functions directory and a subdirectory function. Could you please share a minimal reproducible example!

@miohtama @nonissue:

I haven't been able to reproduce this while trying in both root functions directory and a subdirectory function. Could you please share a minimal reproducible example!

Oops, missed your response. Will try and put one together tomorrow!

Running into the same issue. The issue doesn't exist for GET requests, but it does exist for POST requests and possibly other requests.
Here is a minimal project demo with example curl commands and output:
https://github.com/richard-flosi/netlify-dev-404

Same issue here.

I'm having the same exact issue.

The same thing is happening to me. I'm POSTing data to a lambda and when it returns a 404, it hangs. I'm using Postman to hit the API. I'm not sure if it hands in the browser or not (I haven't gotten that far yet).

aeesBhatti closed this in #965 23 hours ago

This is awesome news. When will this fix hit npm?

@donavon netilfy-cli@2.55.0 has just been published that includes a fix for this bug. Please try it out and let us know!

Hello,
I'm using cli version 2.68.4 and i'm still getting the issue.
The server doesn't hang but test extra string on top of the given input. If I manually remove the .html it tries for a .htm and then for a id/index.html, ect ...
here is the output:

Hello,
I'm using cli version 2.68.4 and i'm still getting the issue.
The server doesn't hang but test extra strings (html, htm, index.html) on top of the given input. If I manually remove these strings with a regex it ends up with a BadRequest: invalid expression error (400):

Thanks @terazus for reporting. This seems like a different issue from the one reported. Can you open a new one for it?

Hi @erezrokah
I don't think i need to open a new issue, there's already #1442 which seeems to be related to my issues (im also using netlify dev)
Let me kow if i need to open a new issue anyway :)

Thanks for finding that! This looks like what you're experiencing. Going to cross link it

authors: RaeesBhatti cjinghong donavon erezrokah fpagnoux miohtama nonissue richard-flosi terazus

repo: netlify/cli


text:
Gatsby Site Starter Landing Page Error

Describe the bug
After deploying the gatsby site starter to netlify, the landing page preview shows this error:
ReferenceError: Can't find variable: ___loader
To Reproduce

Go to https://www.netlifycms.org/docs/start-with-a-template/
Click on Deploy to Netlify under Gatsby Site Starter
Follow instructions to create/build site
Go to <your-site>.netlify.com/admin
Click on pages in the left panel, then landing page

Expected behavior
The preview loads and there is no error
Screenshots

Applicable Versions:

netlify-cms-app 2.9.6
netlify-cms-core 2.12.1
Git provider: GitHub
OS: MacOS Mojave
Browser version Safari 12.0.3

Additional context
All files are default, I haven't altered any code nor saved any changes in the cms editor, just followed the instructions to get a gatsby site up

This seems to be localized to safari, the actual landing page of the site loads fine, it's only in the admin preview of the page.
I just tried firefox and chrome and it works as expected. I did hard refresh, empty caches, etc. on safari and the error still shows

It isn't Safari, worked for me (in 12.1.1) following the steps, but the error is present in the console - looks to be gatsby-link, which it seems we're erroneously importing into the landing page preview template. Porting this to the starter repo.

authors: cotyhamilton erquhart

repo: netlify/netlify-cms


text:
Netlify CMS via npm: modals for invitation, email change etc disappearing

Describe the bug
All Modals are disappearing, when opened before the CMS init happened...
This includes the login modal, and any of the modals opened from an email template.
The NPM version results in a very big bundle size (even after webpack), so it takes literally 5-10 seconds for the scripts to load when visiting /admin/.
After the CMS script loaded and the log in screen becomes visible, there's another delay of 1-3 seconds before the CMS init happens.
If the log in button is clicked before that, it is disabled and cannot be clicked again.
This results in various problems for normal users, and also when inviting users, sending password change email etc.

Normal Users (writers and so forth) have to be told to wait 10+ seconds before clicking the login button
Any of the modals opened from an email template, are opened immediately, before the CMS init happens. So these disappear immediately after opening, so it is virtually impossible to enter anything.

Screenshots
Here's a video of the issue:
I pasted the link from my email invitation into the url bar and hit return after the recording starts.
As you can see, it loads forever because the bundle is so huge, the modal appears and at the end when the CMS init happens, it closes on it's own and there's no way to get it back and the login button is also disabled.
Now in the video, there was some time, if you're fast, to enter a new password. But this isn't always the case and most of the time it closes really fast after it opened.
https://user-images.githubusercontent.com/3166991/127622607-909b2036-c692-438d-8448-73441492d3c6.mov
Applicable Versions:
I'm running the following:
netlify-cms-app 2.15.30
netlify-cms-core 2.46.0
Github provider
Node v16.0.0
NPM@7 (legacy peer deps)
CMS configuration
import landingPages from "./collections/landing-pages";
import reviews from "./collections/reviews";
import articles from "./collections/articles";
import authors from "./collections/authors";
import pages from "./collections/pages";
import affiliateLinks from "./collections/affiliate-links";
import CMS from "netlify-cms-app";
import cloudinary from "netlify-cms-media-library-cloudinary";

CMS.registerPreviewStyle("/assets/css/main.css");
CMS.registerMediaLibrary(cloudinary);

// Initialize the CMS object
CMS.init({
  config: {
    load_config_file: false,
    backend: {
      name: window.CMS_BACKEND_NAME,
      repo: window.CMS_REPOSITORY,
      branch: window.CMS_BRANCH,
      proxy_url: window.CMS_PROXY_URL ?? null,
      local_backend: window.CMS_LOCAL_BACKEND ?? false,
    },
    locale: window.CMS_LOCALE,
    site_url: window.CMS_SITE_URL,
    display_url: window.CMS_DISPLAY_URL,
    publish_mode: window.CMS_PUBLISH_MODE,
    logo_url: window.CMS_LOGO,
    media_library: {
      name: "cloudinary",
      config: {
        cloud_name: "XXXXXXXXX",
        api_key: "XXXXXXXXXXX",
        default_transformations: [
          [
            {
              quality: "auto",
              fetch_format: "auto",
            },
          ],
        ],
      },
    },
    collections: [
      ...
    ],
  },
});

index.html"
---
permalink: /admin/index.html
eleventyExcludeFromCollections: true
hide_from_sitemap: true
---
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>{{ site.name | title }} Content Manager</title>

    <script>
      // This global flag enables manual initialization.
      window.CMS_MANUAL_INIT = true;
      window.CMS_SITE_URL = "{{ site.url }}";
      window.CMS_DISPLAY_URL = "{{ site.url }}";
      window.CMS_PUBLISH_MODE = "{{ site.cms.publish_mode }}";
      window.CMS_BRANCH = "{{ site.cms.branch }}";
      window.CMS_LOGO = "{{ site.url }}{{ site.logo }}";

      {% if helpers.environment === "development" %}
      window.CMS_BACKEND_NAME = "proxy";
      window.CMS_LOCAL_BACKEND = true;
      window.CMS_PROXY_URL = "{{ helpers.netlify_proxy_url }}";
      {% else %}
      window.CMS_BACKEND_NAME = "git-gateway";
      window.CMS_REPOSITORY = "{{ site.cms.repository }}";
      {% endif %}
    </script>

    <!-- Include the Netlify Identity platform to authenticate CMS access -->
    <script src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
  </head>
  <body>

    <!-- Include the script that builds the page and powers Netlify CMS -->
    <script src="/assets/js/cms.js" async></script>

  </body>
</html>

Additional context
Is there anything I am doing completely wrong here? Or is that really unexpected behavior?

Ok I understand now why this was happening.
Because I was using the script tag implementation of Netlify Identity, it was finished loading before the entire heavy React netlify cms app, and initialized before the CMS...
I solved this by swapping out ID to the npm version, and then manually initialize it after the CMS, like so:
const netlifyIdentity = require("netlify-identity-widget");

cms.init()...

// init identity after the CMS, otherwise
// there will be issues with login/email modals closing.
netlifyIdentity.init();

PS: the entire login is much cooler with the custom NPM package, maybe you should make that design the default :)

PS: @erezrokah I also just realized, that using the NPM package has a very neat side-effect.
I'm using the MacOS integrated password manager, before this change it did not save the sites login details under the actual domain.
It could not detect it through how the modal works.
Now that I'm using the npm version, and there's a real form on the site my password manager immediately saved everything correctly.
Might be worth to actually really make this the default also for the script tag version.

Hi @madsem, thanks for the detailed issue and following up on it.
I still not sure what's exactly happening, but I'll share some more information.
When you use <script src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script> the Identity widget implicitly calls init for you:
https://github.com/netlify/netlify-identity-widget/blob/e0072e56f8ee94d46b8b1c74c26b62be7122f1d8/src/index.js#L11
When you use const netlifyIdentity = require("netlify-identity-widget"); you need to initialize the widget manually:
https://github.com/netlify/netlify-identity-widget/blob/e0072e56f8ee94d46b8b1c74c26b62be7122f1d8/package.json#L63 points to this file:
https://github.com/netlify/netlify-identity-widget/blob/e0072e56f8ee94d46b8b1c74c26b62be7122f1d8/src/netlify-identity.js#L24
Regardless the CMS expects the Identity widget to be fully loaded and initialized.
We've seen multiple cases where users are using const netlifyIdentity = require("netlify-identity-widget"), but not calling init for the widget, so the CMS has a safety check for that:
https://github.com/netlify/netlify-cms/blob/edaf9804eb51c90bfedb666b1747caf33f12b048/packages/netlify-cms-backend-git-gateway/src/implementation.ts#L104
In all cases it's best to make sure the Widget is loaded and initialized before the CMS is imported.

Hey @erezrokah , thanks for the explanation.
Maybe I didn't explain it clear enough. But the initial problem was because I was using <script src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script> in my index file.
And I was able to fix it by switching to the manual init method you are describing with the NPM version of netlify identity.
The problem was that I am using the NPM/React CMS app along with the script tag, the script tag initialized long before the CMS and caused the modals to close.
After I switched to the NPM version, and manually initialized the identity widget (AFTER the CMS), everything works as expected.
If the first method should have worked as well, then there must be a bug somewhere I suppose

authors: erezrokah madsem


text:
Blog post example doesn't work without meta charset utf-8

I stumbled upon your blog post on HN and after going over your first steps It didn't work, I was getting:
Uncaught SyntaxError: Invalid regular expression: /[Ã€-ÃŸ][Â€-Â¿]|[Ã -Ã¯][Â€-Â¿]{2}|[Ã°-Ã·][Â€-Â¿]{3}/: Range out of order in character class
    at new RegExp (<anonymous>)
    at base64.js:89
    at Object.<anonymous> (base64.js:194)
    at t (bootstrap 4f21d447dcc951a32fbe:19)
    at Object.<anonymous> (_hasUnicode.js:23)
    at t (bootstrap 4f21d447dcc951a32fbe:19)
    at Object.<anonymous> (netlify_logo.svg:1)
    at t (bootstrap 4f21d447dcc951a32fbe:19)
    at Object.<anonymous> (_has.js:3)
    at t (bootstrap 4f21d447dcc951a32fbe:19)

on the console.
I then went here and added your quickstart steps and I got it running, but I think the blog post needs fixing.

Hmm interesting - how can I reproduce?

@erquhart I've ran into this issue before as well. If you just copy the index.html code from A Complete CMS with No Server and 18 Lines of Code, it doesn't work until you add a <meta charset="utf-8"> line to the head.

Very interesting, I ran that code at the time (originally written in like June), maybe I added the meta and didn't think about it.

Still can't reproduce, used the exact code from the blog post.

So I just took this code:
<head>
  <link href="https://unpkg.com/netlify-cms/dist/cms.css" rel="stylesheet"/>
</head>
<body>
  <script src="https://unpkg.com/netlify-cms/dist/cms.js"></script>
</body>

put it in an HTML file, opened the file in Chrome, and I get the error.

@erquhart I wasn't using the node server either, just opening the file in chrome.

@erquhart I think I found the issue: If a server is used (like the http-server in your blog post) that sets the header content-type:text/html; charset=UTF-8. If the file is just opened directly in the browser, no charset is set, unless the file defines it via a <meta> tag.

Oh okay. The post specifies that a server must be used, I'd say won't fix.

@erquhart I'm guessing this could happen on some servers as well -- not all may set UTF-8 by default. I don't know if it would be worth adding a note under the "other servers" section to the effect that it must serve as UTF-8. Most people are going to figure this out, though, so it's probably not a big deal.

Good point, I'll update the post.

If you are going to modify, rather than a note, I would change the initial snippet to a full index.html including the meta charset. It's much quicker entry point for anyone quickly scanning through it.

authors: erquhart hanoii tech4him1


text:
Custom Widgets Folder

It would be nice to have a folder where the community can contribute to custom widgets...or does this exists already, i could not find anything in this repo..

I suggest the first widget to add should be that from the blog https://www.netlify.com/blog/2017/06/20/extending-netlify-cms-part-one-custom-widgets/
What do you think?

@verythorough @erquhart What do you think of this?

I think there are some cases where it would make sense to add a widget to the default widgets in the repo (like the multiselect widget in development now).
In other cases where the widget is less generic, we could add a "community-created widgets" section of the widgets doc, where people can link gists, repos, and/or tutorials for custom widgets they have created. I figure this makes for discoverability in a central location, while still allowing widget creators to store and maintain their code in a way that works for them.

Agreed, I'd love to see major improvements in discoverability. Featuring community extensions is a big part of the current extensibility initiative.

Yes, what really matters is  discoverability..
First, what for me would feel 'natural' is doing a search like this> https://yarnpkg.com/en/packages?q=netlify widget&p=1
So suggesting a default name convention to widget developers would help. like It should start by ...
Second, A linking section would also help for non-default named widgets..
sorry if my english is not that good I am not a native english speaker..

Closing in favor of #1286

authors: erquhart martinjagodic pungggi tech4him1 verythorough


text:
Bump FLP for direct-flp

fix for latest breaking changes in FLP

@msherif1234 I'm force-merging it to unblock PR https://github.com/netobserv/network-observability-operator/pull/502

repo: netobserv/netobserv-ebpf-agent


text:
Sentinel object Magazine.MAGAZINE_FREED is possible be replaced without be restored back

Expected behavior
If I understood it right, the sentinel object Magazine.MAGAZINE_FREED should not be replaced without being restored back.
Actual behavior
https://github.com/netty/netty/blob/afa465a49a3ce7379d64f7632aaf26f8767af6ab/buffer/src/main/java/io/netty/buffer/AdaptivePoolingAllocator.java#L665-L684
The line:671 Chunk nextChunk = NEXT_IN_LINE.get(this); if nextChunk get value MAGAZINE_FREED, which is possible when magazines table is doing resizing, then the line:673 NEXT_IN_LINE.compareAndSet(this, nextChunk, current) will replace the MAGAZINE_FREED object by current, and without restored the MAGAZINE_FREED back.
Possible solution:
Modified line:672 to:
if (nextChunk != null && nextChunk != MAGAZINE_FREED
&& current.remainingCapacity() > nextChunk.remainingCapacity()) {
...
}

Netty version
4.1.115.Final

@laosijikaichele sounds about right ... please do a PR

OK, will do it soon.

authors: laosijikaichele normanmaurer

repo: netty/netty


text:
Discussion of import external library

Hello
Since using netty I did not need to import external libraries.
But since I want to use HTTP2.0 I found some import like this
in : netty/handler/src/main/java/io/netty/handler/ssl/OpenSsl.java
import org.apache.tomcat.jni.Library;
import org.apache.tomcat.jni.Pool;
import org.apache.tomcat.jni.SSL;
import org.apache.tomcat.jni.SSLContext;'

Is'it a new way for netty ?
although it is noted in the main page
Ease of use
Well-documented Javadoc, user guide and examples
No additional dependencies, JDK 5 (Netty 3.x) or 6 (Netty 4.x) is enough
Thanks

It's an optional dependency.  When the tcnatice is unavailable, Netty will not use it.

authors: christophe-blettry trustin


text:
StringDecoder cannot read fragmented packets containing UTF-8 characters

StringDecoder cannot handle fragmented packets if the boundary of a bytebuf falls on the bytes from a UTF-8 character.
For example: when StringDecoder recieves 0xE2 0x98 0xA2 as 1 ByteBufs, it correctly decodes this to the "\u2622" character, but when StringDecoder recieves this as 3 seperate ByteBufs, it will convert it to 3 invalid code points.
Tested on 5.0.0-Alpha2 & 4.1.0.Beta8 & 4.0.33.Final, all failing.
JUnit test:
@Test
public void fragmentedStringTest() {
	String tmp, out;
	EmbeddedChannel channel = new EmbeddedChannel(
			new StringDecoder(StandardCharsets.UTF_8));
	byte[] data = new byte[]{(byte)0xE2,(byte)0x98,(byte)0xA2};

	channel.writeInbound(Unpooled.wrappedBuffer(data, 0, 3));
	out = "";
	while((tmp = (String)channel.readInbound()) != null) {
		out += tmp;
	}
	Assume.assumeThat("Cannot read data unfragmented", out, is("\u2622"));

	channel.writeInbound(Unpooled.wrappedBuffer(data, 0, 1));
	channel.writeInbound(Unpooled.wrappedBuffer(data, 1, 1));
	channel.writeInbound(Unpooled.wrappedBuffer(data, 2, 1));
	out = "";
	while((tmp = (String)channel.readInbound()) != null) {
		out += tmp;
	}
	Assert.assertThat("Cannot read data fragmented", out, is("\u2622"));
}

Test fails with:
java.lang.AssertionError: Cannot read data fragmented
Expected: is "?"
but: was "???"
A solution to this problem would changing StringDecoder to use ByteToMessageDecoder, so it could skip invalid sequences and continue on, but this would break code depending on StringDecoder @Shareable status. Another solution would be duplicating the logic to StringDecoder, but using AttributeKey and Attribute's as alternate storage

@ferrybig actually I would say this is expected as it will just convert anything in the ByteBuf to a String. If you need to handle fragmented data you will need another handler in front of it that takes care of this.

@normanmaurer If this was expected, I would have it throw a exception then, instead of silently corrupting data

@ferrybig maybe we could add some javadoc that is will only "convert" byte[] to "String" without any state-keeping. Like I said if you need something more complex you will need to implement by yourself, as this would also be a lot different depending on the actual charset.

authors: ferrybig normanmaurer


text:
FixedChannelPool out of memory

The following link show out of memory info from hprof:
hprof summary
hprof original file
Steps to reproduce
10 client Pools, 350 connection per pool.
Minimal yet complete reproducer code (or URL to code)
  //client send code, need use any pool class to replace Unpooled.copiedBuffer ?
    ByteBuf byteBuf = Unpooled.copiedBuffer(content, StandardCharsets.UTF_8);
    FullHttpRequest msg = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uri.getRawPath() + "?" + uri.getRawQuery(), Unpooled.wrappedBuffer(byteBuf));
    msg.headers().set(HttpHeaderNames.HOST, uri.getHost());
    msg.headers().set(HttpHeaderNames.CACHE_CONTROL, false);
    msg.headers().set(HttpHeaderNames.ACCESS_CONTROL_ALLOW_HEADERS, false);
    msg.headers().set(HttpHeaderNames.CONTENT_TYPE, new AsciiString("application/json"));
    msg.headers().set(HttpHeaderNames.CONTENT_LENGTH, byteBuf.readableBytes());
   ClientPool.sendMsg(msg);

//ClientPool  sendMsg code
final FixedChannelPool pool = getInstance().poolMap.get(address);
    Future<Channel> future = pool.acquire();

    future.addListener(new FutureListener<Channel>() {

        public void operationComplete(Future<Channel> f) {

            if (f.isSuccess()) {
                Channel ch = null;
                try {
                    ch = f.getNow();
                    ReferenceCountUtil.retain(msg); //need?
                    ch.writeAndFlush(msg);
                } catch (Throwable a) {
                    a.printStackTrace();
                } finally {
                    ReferenceCountUtil.release(msg);  //need?
                }

when received response from server,
ClientPool.release(ctx.channel());
Netty version
nettyall 4.1.16 Final
JVM version (e.g. java -version)
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
OS version (e.g. uname -a)
Linux ip-172-31-9-182.ec2.internal 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

can you share the hprof file ?

Is the above hrof html file enough? or do you need original hrof file?

@sparkfax interesting... the original file would be helpful

@normanmaurer I have added original hrof file.

So I looked at your heap-dump and yes this is the cause of the out of memory. All the DefaultPromise in the heap dump are failed because of too many acquired calls (you can see this by inspect the Throwable that failed it). The problem here is the the Listener holds a reference to the ClientPool that also holds a reference to the DefaultFullHttpRequest which you seem to never release. So yep this will result in a leak for sure.
Here you can see a snip of the issue:

authors: normanmaurer sparkfax


text:
Pooled byte buf allocator

Review PooledByteBufAllocator in respect of jemalloc 4.x changes and
update allocate algorithm.
Motivation:
For size from 512 bytes to chunkSize, we use a buddy algorithm. The drawback is that it has a large internal fragmentation.
Modifications:

add SizeClassesMetric and SizeClasses
remove tiny size, now we have small, normal and huge size
rewrite the structure of PoolChunk
rewrite pooled allocate algorithm in PoolChunk
when allocate subpage, using lowest common multiple of pageSize and elemSize instead of pageSize.
add more tests in PooledByteBufAllocatorTest and PoolArenaTest

Result:
Reduce internal fragmentation.
Closed #3910

@yuanrw I would be super interested in see benchmarks / metrics that shows the effect of the changes. As these changes are super-core I want to be careful

There are benchmarks for PooledByteBufAllocator. The main improving is about allocate normal size.  I use 4.1.48 release as a contrast.
normal size
memory usage benchmarks
Following figures summarize results for the memory usage of chunks.
I repeatedly allocate random size (normal size)  of ByteBuf and then release parts of them. After each allocation, I record total used bytes of  chunks.

PoolChunkUsageBenchmark
memory layout benchmarks
In order to show more details about the allocation policy, I wrote a program that prints layout of chunks after each allocation. An example graph is shown below. With this program we are able to observe the effects between two versions. New version uses memory more effectively than in earlier versions.

PoolChunkLayoutBenchmark
small size
Tiny size is merged to small size now. When allocating small size, we use a slab algorithm which is as same as old version. The only change is sizes of subpages.

PoolChunkUsageSmallBenchmark

@yuanrw did you already sign the icla? https://netty.io/s/icla

@yuanrw did you already sign the icla? https://netty.io/s/icla
Yes :-)

Any update on this PR? @normanmaurer

ptal @carl-mastrangelo

I will have a look soon @yuanrw .

@yuanrw did you run any benchmarks with these changes ?

@ejona86 @carl-mastrangelo @trustin can you have a look ?

@normanmaurer scroll up to see benchmarks and graphs :fire:

Fixed. @normanmaurer PTAL

@netty-bot test this please

Fixed failing tests, please test it again :-)

@chrisvest may be interesting to you as well

Any update here? :-)

@njhill could you have a look buddy ?

@normanmaurer sure I'll try to take a closer look soon. At high level the changes look awesome (thanks @yuanrw!). One probably naive question, did you consider using latest jemalloc 5.x as reference instead?
In addition to the usage profile benchmarks (which look great), it would be good to have some jmh-based timing benchmarks to assess performance difference for high frequency alloc/dealloc workloads (both single-threaded and contented), and maybe check indirect impact on some existing benchmarks which involve allocations... given that this is a nontrivial change to something so core/fundamental.

@normanmaurer sure I'll try to take a closer look soon. At high level the changes look awesome (thanks @yuanrw!). One probably naive question, did you consider using latest jemalloc 5.x as reference instead?
In addition to the usage profile benchmarks (which look great), it would be good to have some jmh-based timing benchmarks to assess performance difference for high frequency alloc/dealloc workloads (both single-threaded and contented), and maybe check indirect impact on some existing benchmarks which involve allocations... given that this is a nontrivial change to something so core/fundamental.

Because there are some enormous changes in jemalloc 5.
See Release 5.0.0 · jemalloc/jemalloc · GitHub.
One of the main changes is using extent instead of chunk. It use rtree( a radix tree) to manage extents and use start memory address of each extent as keys. But It’s meaningless to store address of a java object, since it can move because of GC at any time.
Do we need to introduce these changes？If yes how to do it? How these changes will impact the performance? I think it needs more consideration and discussion.

Do we need to introduce these changes？

@yuanrw no, I was just asking out of curiosity and you gave a good answer thanks! Maybe something to look into more in the future but I expect would require a lot of analysis and consideration like you say.

So I plan to cut a release this week and pull in this one after this to give it a bit more time to bake and others more time to review. @yuanrw thanks for your patience

There is the JMH benchmark results, because of time consuming, I only run tests relevant to pooled allocation.
4.1.49
Benchmark                                             (size)  (tokens)  (useThreadCache)  Mode  Cnt    Score   Error  Units
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         0             false  avgt   20   96.287 ± 1.063  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         0              true  avgt   20   96.782 ± 1.474  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         5             false  avgt   20  104.308 ± 0.730  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         5              true  avgt   20  104.502 ± 1.534  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123        10             false  avgt   20  114.701 ± 0.514  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123        10              true  avgt   20  115.396 ± 0.689  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123       100             false  avgt   20  371.561 ± 1.171  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123       100              true  avgt   20  364.388 ± 5.017  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         0             false  avgt   20   99.649 ± 1.525  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         0              true  avgt   20  102.127 ± 2.453  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         5             false  avgt   20  105.411 ± 1.054  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         5              true  avgt   20  107.440 ± 1.287  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234        10             false  avgt   20  120.568 ± 2.221  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234        10              true  avgt   20  117.831 ± 0.638  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234       100             false  avgt   20  376.430 ± 2.307  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234       100              true  avgt   20  375.129 ± 3.603  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         0             false  avgt   20   91.187 ± 1.516  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         0              true  avgt   20   90.386 ± 1.135  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         5             false  avgt   20   97.487 ± 0.975  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         5              true  avgt   20   99.291 ± 2.091  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345        10             false  avgt   20  111.062 ± 1.292  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345        10              true  avgt   20  110.342 ± 1.549  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345       100             false  avgt   20  366.287 ± 1.281  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345       100              true  avgt   20  364.224 ± 1.423  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         0             false  avgt   20  126.948 ± 0.768  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         0              true  avgt   20  128.202 ± 1.354  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         5             false  avgt   20  135.011 ± 1.803  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         5              true  avgt   20  135.354 ± 1.450  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456        10             false  avgt   20  147.532 ± 0.815  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456        10              true  avgt   20  147.276 ± 0.697  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456       100             false  avgt   20  403.956 ± 1.481  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456       100              true  avgt   20  408.041 ± 3.088  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         0             false  avgt   20  100.463 ± 2.357  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         0              true  avgt   20   99.011 ± 1.098  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         5             false  avgt   20  107.849 ± 0.645  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         5              true  avgt   20  107.360 ± 0.763  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567        10             false  avgt   20  119.641 ± 0.795  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567        10              true  avgt   20  123.131 ± 3.710  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567       100             false  avgt   20  380.128 ± 2.513  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567       100              true  avgt   20  382.363 ± 3.170  ns/op

Benchmark                                                  (size)   Mode  Cnt         Score        Error  Units
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   00000  thrpt   20   8469553.312 ±  77917.811  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   00256  thrpt   20   7969835.081 ±  84971.915  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   01024  thrpt   20   7888660.822 ±  84203.521  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   04096  thrpt   20   7157134.976 ±  44114.898  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   16384  thrpt   20   8524820.724 ± 211291.973  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   65536  thrpt   20   3981280.177 ± 104591.130  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     00000  thrpt   20   8628716.559 ±  87374.273  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     00256  thrpt   20   8551963.878 ± 152934.371  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     01024  thrpt   20   8151001.296 ± 111903.271  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     04096  thrpt   20   7321143.749 ± 169923.659  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     16384  thrpt   20   8555042.078 ± 180845.098  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     65536  thrpt   20   3812659.169 ± 176643.395  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          00000  thrpt   20  10032182.938 ± 104518.019  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          00256  thrpt   20   8618518.773 ± 133628.175  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          01024  thrpt   20   8161732.259 ± 102032.766  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          04096  thrpt   20   6591819.881 ± 572394.518  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          16384  thrpt   20   3535383.735 ±  25941.101  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          65536  thrpt   20   3635397.000 ± 171258.216  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            00000  thrpt   20   9928210.684 ± 158603.498  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            00256  thrpt   20   8525573.003 ± 262929.012  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            01024  thrpt   20   8052403.731 ±  60828.160  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            04096  thrpt   20   5925514.363 ±  33421.839  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            16384  thrpt   20   3638240.901 ±  52665.285  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            65536  thrpt   20   4236426.381 ±  23992.245  ops/s

new
Benchmark                                             (size)  (tokens)  (useThreadCache)  Mode  Cnt    Score    Error  Units
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         0             false  avgt   20   90.361 ±  0.747  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         0              true  avgt   20   90.052 ±  1.052  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         5             false  avgt   20   95.414 ±  3.683  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123         5              true  avgt   20   98.843 ±  1.001  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123        10             false  avgt   20  111.096 ±  6.073  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123        10              true  avgt   20  108.443 ±  1.080  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123       100             false  avgt   20  364.398 ±  1.897  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease      123       100              true  avgt   20  366.795 ±  2.974  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         0             false  avgt   20   91.288 ±  1.760  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         0              true  avgt   20   90.954 ±  1.430  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         5             false  avgt   20   98.771 ±  3.762  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234         5              true  avgt   20   97.892 ±  1.717  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234        10             false  avgt   20  109.855 ±  1.912  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234        10              true  avgt   20  105.101 ±  5.289  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234       100             false  avgt   20  367.235 ±  4.389  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease     1234       100              true  avgt   20  362.622 ±  5.287  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         0             false  avgt   20   92.655 ±  0.958  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         0              true  avgt   20   93.910 ±  1.210  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         5             false  avgt   20  100.928 ±  1.710  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345         5              true  avgt   20  100.981 ±  1.979  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345        10             false  avgt   20  110.934 ±  0.366  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345        10              true  avgt   20  111.953 ±  1.160  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345       100             false  avgt   20  366.731 ±  0.688  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease    12345       100              true  avgt   20  359.103 ±  1.211  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         0             false  avgt   20  322.232 ± 15.611  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         0              true  avgt   20  299.949 ± 13.385  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         5             false  avgt   20  336.126 ±  5.720  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456         5              true  avgt   20  340.586 ±  8.926  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456        10             false  avgt   20  333.419 ±  4.703  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456        10              true  avgt   20  323.919 ±  9.608  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456       100             false  avgt   20  607.050 ± 10.934  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease   123456       100              true  avgt   20  599.322 ±  8.235  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         0             false  avgt   20  314.720 ±  7.189  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         0              true  avgt   20  313.224 ±  7.532  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         5             false  avgt   20  344.692 ±  9.428  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567         5              true  avgt   20  338.428 ±  6.583  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567        10             false  avgt   20  351.835 ±  7.291  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567        10              true  avgt   20  338.317 ± 13.549  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567       100             false  avgt   20  594.185 ±  7.803  ns/op
SimpleByteBufPooledAllocatorBenchmark.getAndRelease  1234567       100              true  avgt   20  613.271 ± 12.206  ns/op

Benchmark                                                  (size)   Mode  Cnt         Score        Error  Units
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   00000  thrpt   20   9396724.688 ± 275275.695  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   00256  thrpt   20   8696819.170 ± 106191.476  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   01024  thrpt   20   8356052.763 ± 106296.771  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   04096  thrpt   20   7810052.887 ± 113403.700  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   16384  thrpt   20   7327999.792 ±  46276.901  ops/s
ByteBufAllocatorBenchmark.defaultPooledDirectAllocAndFree   65536  thrpt   20   4042436.167 ±  18824.463  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     00000  thrpt   20   8754775.971 ± 599149.421  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     00256  thrpt   20   8851510.602 ±  83863.499  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     01024  thrpt   20   8730025.543 ± 242618.215  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     04096  thrpt   20   8493517.235 ±  93658.613  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     16384  thrpt   20   7370346.414 ± 515359.734  ops/s
ByteBufAllocatorBenchmark.defaultPooledHeapAllocAndFree     65536  thrpt   20   3187131.313 ±  30264.696  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          00000  thrpt   20   9066269.095 ±  55265.798  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          00256  thrpt   20   8614155.422 ± 142923.097  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          01024  thrpt   20   8762735.419 ±  94535.330  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          04096  thrpt   20   7218544.347 ± 836320.638  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          16384  thrpt   20   7821193.832 ±  33100.660  ops/s
ByteBufAllocatorBenchmark.pooledDirectAllocAndFree          65536  thrpt   20   3911207.252 ± 104541.091  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            00000  thrpt   20   8995692.383 ±  71609.778  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            00256  thrpt   20   8885720.365 ± 109888.162  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            01024  thrpt   20   8828678.633 ±  67066.615  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            04096  thrpt   20   8010103.921 ± 113250.203  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            16384  thrpt   20   7668189.105 ±  89951.683  ops/s
ByteBufAllocatorBenchmark.pooledHeapAllocAndFree            65536  thrpt   20   3484858.916 ± 286941.691  ops/s

There are 2 notable things.

ByteBufAllocatorBenchmark.pooledDirectAllocAndFree ByteBufAllocatorBenchmark.pooledHeapAllocAndFree param:16384 in new version are much better than old version, because in new code 16384 is small class size and the last one subpage will not be freed.
SimpleByteBufPooledAllocatorBenchmark.getAndRelease param:123456 1234567 in new version is slow, the reason is that we always allocate and release one bytebuf, so in each allocation we have to split and collapse run. However, In ByteBufAllocatorBenchmark.pooledDirectAllocAndFree ByteBufAllocatorBenchmark.pooledHeapAllocAndFree param:65536,
we hold thousands of bytebuf in each iteration, so the result is not much different between two versions.

See whole output in file.
4.1.49-output.txt
new-output.txt

@netty-bot test this please

@yaroot seems there was a leak detected ... please check with ./mvnw -Pleak ...

@normanmaurer I think you pinged the wrong user.

ups... @yuanrw  ^^

@netty-bot test this please

@yuanrw sorry for the slow turnaround... Awesome work!

authors: johnou njhill normanmaurer yuanrw


text:
ReferenceCountedOpenSslEngine unwrap handshake complete status fix

Motivation:
ReferenceCountedOpenSslEngine may unwrap data and complete the handshake
in a single unwrap() call. However it may return HanshakeStatus of
HandshakeStatus of NEED_UNWRAP instead of FINISHED. This may result in
the SslHandler sending the unwrapped data up the pipeline before
notifying that the handshake has completed, and result in out-of-order
events.
Modifications:

if ReferenceCountedOpenSslEngine handshake status is NEED_UNWRAP and
produced data, or NEED_WRAP and consumed some data, we should call
handshake() to get the current state.

Result:
ReferenceCountedOpenSslEngine correctly indicates when the handshake has
finished if at the same time data was produced or consumed.

build failure seems transient, re-kicking build...
2021-04-29T00:53:59.1212316Z [INFO] ---------------< io.netty:netty-testsuite-native-image >----------------
2021-04-29T00:53:59.1213731Z [INFO] Building Netty/Testsuite/NativeImage 4.1.64.Final-SNAPSHOT       [39/44]
2021-04-29T00:53:59.1214891Z [INFO] --------------------------------[ jar ]---------------------------------
2021-04-29T00:53:59.1221965Z Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.6.0/exec-maven-plugin-1.6.0.pom
2021-04-29T01:09:45.5660537Z [INFO] ------------------------------------------------------------------------

2021-04-29T01:09:45.5803695Z [INFO] Netty/Testsuite/NativeImage ........................ FAILURE [15:46 min]

2021-04-29T01:09:45.5836981Z [ERROR] Plugin org.codehaus.mojo:exec-maven-plugin:1.6.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.codehaus.mojo:exec-maven-plugin:jar:1.6.0: Could not transfer artifact org.codehaus.mojo:exec-maven-plugin:pom:1.6.0 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.6.0/exec-maven-plugin-1.6.0.pom: Connection timed out (Read failed) -> [Help 1]


text:
Only flush on channelReadComplete(...) in http2 hello world examples.

Motivation:
In our http1 hello world example we only flush on channelReadComplete(...) to make better use of gathering writes. We should do the same in http2.
Modifications:
Only flush in channelReadComplete(...)
Result:
Better performance and more consistent examples.

also @ejona86 @buchgr  @carl-mastrangelo  FYI

Does FallbackRequestHandler need to be updated too?

@carl-mastrangelo FallbackRequestHandler already does this :)

Perhaps I don't understand, your other change was to move flushing after channelReadComplete for http2 usage.  FallbackRequestHandler  is an http2 class correct?  Does that mean the channelReadComplete method on it should be removed since it is now covered by Http2ConnectionHandler ?

@carl-mastrangelo this is true when the Http2ConnectionHandler is in the pipeline but I not wanted to depend on this for FallbackRequestHandler as it is not really http2 related and so may give the user the wrong impression.

In that case, why does HelloWorldHttp2Handler flush then?  That seems to be http2 specific.

@carl-mastrangelo I think you talk about the one for the frame and multiple codec. Because the user is not aware that Http2ConnectionHandler is in the pipeline and its an implementation detail IMHO.

let me pull this in... we can do a followup if needed.

Cherry-picked into 4.1 (580ac8cd41a426c30aeee363a9bf3dab2f9e303f)

authors: carl-mastrangelo normanmaurer


text:
Remove OIO transport (and transports that depend on it).

Motivation:
This transport is unique because it uses Java's blocking IO (java.io / java.net) under the hood. However it is not clear if this transport is actually useful so it should be removed.
Modifications:

Remove OIO transport and RXTX transport which depend on it.
Remove OioSctp implementations
Remove PerThreadEventLoop* which was only used by OIO transport.

Result:
Fixes https://github.com/netty/netty/issues/8510.

This is the netty 5 counter-part of https://github.com/netty/netty/pull/8579

rebased...


text:
Need to allow server to be embedded

I need to run the actual server from a system test program. Ideally the com.networknt.server.Server would allows doing this by moving the main methods content into an init() method that could be called from my test. I have created a trivial pull request https://github.com/networknt/light-4j/pull/311 that addresses this need.

@farrukhnajmi I have reviewed your PR and there is no issue. I am a little curious about your use case. Could you please explain what is your test harness and why you don't want to use the built-in TestServer for your unit and integration test. This might help us to learn new testing approach. Thanks.

Cool. I remember other people asked to run a light-4j instance in another Java process before and I think this resolves that issue. Thanks a lot. I have assigned the PR for other team members to review. It should be in the next release.

@farrukhnajmi I have reviewed your PR and there is no issue. I am a little curious about your use case. Could you please explain what is your test harness and why you don't want to use the built-in TestServer for your unit and integration test. This might help us to learn new testing approach. Thanks.

I am writing a system test using Spock framework. The test sends a message to a remote service and then uses an actual light-4j server (not test server) to listen for events from the remote service. This protocol is fixed and I cannot change that. My test is an end-to-end system test with no mocks. When events are received from remote service, the test compares them to expected events.

This has been released in 1.5.22. Close the issue. Thank you.

authors: farrukhnajmi stevehu


text:
made a new contribution to networkx

improved test coverage for algorithms in richclub.py

Thanks @chimaobi-okite!

authors: MridulS chimaobi-okite

repo: networkx/outreachy


text:
initial version of IDE setup

SP23 version. Need to update to requirements of new codebase.

In the installation of nvm on mac, if we follow through curl command its working fine. With wget, we expect it has be installed before executing this command "wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash".
Please correct me if my observation is wrong.

Should we include the screenshots and references to run, execute and debug the application using IDE in this tutorial .

I've followed the set of steps mentioned in the document and successfully able to setup IDE and application.
Tested on:

Mac
Windows


text:
zfc supports Mac

Currently we uses Microsoft.CodeAnalysis.Workspaces.Desktop.dll but it run only .NET Framework 4.6 on Windows.
(It includes \Microsoft.CodeAnalysis.Workspaces.Common with net46 package).
Microsoft.Build (MSBuild) can run on .NET Core.
https://github.com/Microsoft/msbuild
If collect from msbuild and create AdhocWorkspace, maybe we can support macintosh.
Sample:
https://github.com/itn3000/dotnet-1.1.0-test/blob/master/Program.cs

@neuecc
MessagePack ran on Mac.(https://github.com/neuecc/MessagePack-CSharp)
Has ZeroFormatter changed anything?

repo: neuecc/ZeroFormatter


text:
Question about D2T datasets

Hi
Why do the D2T datasets (BAGEL, SFHOT, SFRES) contain multiple same reference sentences in ref_summs?

Typically, for a generation task, multiple answers could meet the requirement. Therefore, providing multiple references can also help us to better and more comprehensively assess the quality of the generated text.

Sorry, my question was not clear.
My question is about the inclusion of multiple same references, as in the following example.
# BAGEL, id:0
'ref_summs': ['I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'I am sorry but there are no venues near X in the city centre .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .',
   'There are no places you are looking for near X in the centre of town .']}

The original dataset can be found here: https://github.com/jeknov/EMNLP_17_submission. We take all the references for each mr (meaning representation). Since we take the max when combining the multi-reference results, multiple same references will not affect the final result.

Thank you!

authors: taku-ito yyy-Apple

repo: neulab/BARTScore


text:
Incorrect results can be returned by embedding search when Content storage enabled

There seems to be a correctness issue that manifests if the following conditions are met:

Content storage ENABLED for the Embeddings
Documents being indexed are python dicts
One or more of the documents being indexed has an empty '"text" field in the dict (i.e. {"text":""})

If all of these conditions are met, then it seems as though search results for all documents AFTER the first empty "text" field will be mis-aligned by the number of documents with empty text fields.
See below code to reproduce:
from txtai.embeddings import Embeddings

dict_docs = [(1, {"text":""}, None), (2, {"text":"New York"}, None), (3, {"text":"California"}, None)]
text_docs = [(1, "", None), (2, "New York", None), (3, "California", None)]

dict_content_embed = Embeddings({"content":True,"path":"sentence-transformers/multi-qa-mpnet-base-dot-v1"})
dict_content_embed.index(dict_docs)

dict_no_content_embed = Embeddings({"content":False,"path":"sentence-transformers/multi-qa-mpnet-base-dot-v1"})
dict_no_content_embed.index(dict_docs)

text_content_embed = Embeddings({"content":True,"path":"sentence-transformers/multi-qa-mpnet-base-dot-v1"})
text_content_embed.index(text_docs)

text_no_content_embed = Embeddings({"content":False,"path":"sentence-transformers/multi-qa-mpnet-base-dot-v1"})
text_no_content_embed.index(text_docs)

print(f"Content+Dict Results:\n{dict_content_embed.search('New York')}\n")
print(f"No Content+Dict Results:\n{dict_no_content_embed.search('New York')}\n")
print(f"Content+Text Results:\n{text_content_embed.search('New York')}\n")
print(f"No Content+Text Results:\n{text_no_content_embed.search('New York')}\n")

The code will print:
Content+Dict Results:
[{'id': '3', 'text': 'California', 'score': 0.9999998807907104}, {'id': '2', 'text': 'New York', 'score': 0.2981969118118286}]

No Content+Dict Results:
[(2, 0.9999998807907104), (3, 0.5046365857124329), (1, 0.2981969118118286)]

Content+Text Results:
[{'id': '2', 'text': 'New York', 'score': 0.9999998807907104}, {'id': '3', 'text': 'California', 'score': 0.5046365857124329}, {'id': '1', 'text': '', 'score': 0.2981969118118286}]

No Content+Text Results:
[(2, 0.9999998807907104), (3, 0.5046365857124329), (1, 0.2981969118118286)]

As you can see, the embedding search incorrectly returns document 3 ("California") as the top match when the documents are python dicts and content storage is enabled.

Thank you for the detailed report on this!
I would ensure that None or empty string aren't passed as the text to index as it's not going to produce useful results. With that being said, the behavior should be the same regardless of whether content is enabled. I just checked in a fix for this, thank you again for finding this bug.

authors: davidmezzetti moore-ryan

repo: neuml/txtai


text:
Add basic trading calendar to exchanges

Right now only opening and closing times are considered. Should be extended to at least weekends and possible holidays,

SimpleTradingCalendar excluded weekends

repo: neurallayer/roboquant


text:
Update datasets to include download, dataset metaclass, dataset objects

Connected to #280: we previously discussed adding torchvision-style dataset files that include the ability to download data for sample problems, e.g. Navier-Stokes or Darcy flow. This PR adds this functionality as well as:

creates a base PTDataset class
turns existing sample datasets into objects that inherit from PTDataset
point to data storage hosted publicly on Zenodo and allow users to download data

The Zenodo data is currently stored on archives owned by the NeuralOperator Zenodo organization, linked here: https://zenodo.org/communities/neuraloperator/records?q=&l=list&p=1&s=10&sort=newest

Also aims to fix #328

Looks great, thanks David!

authors: JeanKossaifi dhpitt

repo: neuraloperator/neuraloperator


text:
[ENH] Implemented logic for node selection

Closes #3

Checklist

[x] PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see https://neurobagel.org/contributing/pull_requests for more info)
[x] PR links to GitHub issue with mention Closes #XXXX
[ ] Tests pass
[ ] Checks pass

For new features:

[ ] Tests have been added

For bug fixes:

[ ] There is at least one test that would fail under the original bug conditions.

:rocket: PR was released in v0.1.0 :rocket:

authors: rmanaem surchs

repo: neurobagel/query-tool


text:
Error in Macos build using homebrew's notcurses library

When following the directions for building on macos, it looks like the build process expects to use notcurses/wrappers.h, which seems specific to neurocyte/notcurses-zig.
My output:
$ brew info notcurses
==> notcurses: stable 3.0.9 (bottled)
Blingful character graphics/TUI library
https://nick-black.com/dankwiki/index.php/Notcurses
Installed
/opt/homebrew/Cellar/notcurses/3.0.9_3 (196 files, 9.4MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-04-18 at 10:25:08
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/n/notcurses.rb
License: Apache-2.0
==> Dependencies
Build: cmake ✘, doctest ✘, pandoc ✘, pkg-config ✔
Required: ffmpeg ✔, libdeflate ✔, libunistring ✔, ncurses ✔
==> Analytics
install: 308 (30 days), 738 (90 days), 2,255 (365 days)
install-on-request: 308 (30 days), 738 (90 days), 2,250 (365 days)
build-error: 14 (30 days)

$ rm -r zig-cache ~/.cache/zig

$ ./zig build -Duse_system_notcurses=true --search-prefix /usr/local
install
└─ install flow
   └─ zig build-exe flow Debug native
      └─ zig build-lib wrappers ReleaseFast native 1 errors
/Users/jose/.cache/zig/p/12200a3fa3e5d76f1ff96d1ca5fab64b61ff0e3f28f953ac8464f43986ca7cf4f7eb/include/notcurses/wrappers.h:1:10: error: 'notcurses/notcurses.h' file not found
#include <notcurses/notcurses.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~
/Users/jose/.cache/zig/p/12200a3fa3e5d76f1ff96d1ca5fab64b61ff0e3f28f953ac8464f43986ca7cf4f7eb/src/wrappers.c:1:10: note: in file included from /Users/jose/.cache/zig/p/12200a3fa3e5d76f1ff96d1ca5fab64b61ff0e3f28f953ac8464f43986ca7cf4f7eb/src/wrappers.c:1:
#include <notcurses/wrappers.h>
         ^
error: the following command failed with 1 compilation errors:
/Users/jose/Code/flow/.cache/zig/zig-macos-aarch64-0.12.0/zig build-lib -cflags -DPUBLIC -D_XOPEN_SOURCE=700 -DPRIVATE -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Wno-deprecated-pragma -Wno-deprecated-declarations -Wno-bitwise-instead-of-logical -Wno-unused-but-set-variable -Werror -fno-sanitize=undefined -D_DARWIN_C_SOURCE -- /Users/jose/.cache/zig/p/12200a3fa3e5d76f1ff96d1ca5fab64b61ff0e3f28f953ac8464f43986ca7cf4f7eb/src/wrappers.c -OReleaseFast -I /Users/jose/.cache/zig/p/12200a3fa3e5d76f1ff96d1ca5fab64b61ff0e3f28f953ac8464f43986ca7cf4f7eb/include -Mroot -lc --cache-dir /Users/jose/Code/flow/zig-cache --global-cache-dir /Users/jose/.cache/zig --name wrappers -static -L /usr/local/lib --listen=-
Build Summary: 8/12 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install flow transitive failure
   └─ zig build-exe flow Debug native transitive failure
      └─ zig build-lib wrappers ReleaseFast native 1 errors
error: the following build command failed with exit code 1:
/Users/jose/Code/flow/zig-cache/o/620f3fd59abe3865b3b0b2a807fd28bb/build /Users/jose/Code/flow/.cache/zig/zig-macos-aarch64-0.12.0/zig /Users/jose/Code/flow /Users/jose/Code/flow/zig-cache /Users/jose/.cache/zig --seed 0x7d3caffc -Z016119b1079513ac -Duse_system_notcurses=true --search-prefix /usr/local

Additionally, I tried running ./zig build -Duse_system_notcurses=true --search-prefix /opt/homebrew/Cellar with same output.

This error implies that the build of the the notcurses zig bindings (notcurses-zig) is not able to find the brew install of notcurses. notcurses/wrappers.h is not the file that cannot be found. It's notcurses/notcurses.h that is missing.
You need to set --search-prefix to a directory that contains include/notcurses/notcurses.h
This is usually your brew prefix ($HOMEBREW_PREFIX). If that variable is not set, check the output of brew config.

authors: josei neurocyte

repo: neurocyte/flow


text:
ImageJ as an alternative to FSLeyes for ADS plugin

ImageJ is a well-maintained ecosystem, is dedicated to microscopy 2D data and accepts plugins. So that could be a viable alternative to FSLeyes.

also see: https://imagej.net/Trainable_Weka_Segmentation

and this: https://deeplearning.cms.waikato.ac.nz/

I think we're pretty cemented into sticking with FSLeyes now that it's working, so I'll close this issue.

authors: jcohenadad mathieuboudreau

repo: neuropoly/axondeepseg


text:
Test replacement for onnxruntime==1.4.0 version pinning by installing libomp

Potentially a better fix for #2930. See https://github.com/microsoft/onnxruntime/issues/5344#issuecomment-702124166.

Libomp installed here: https://travis-ci.org/github/neuropoly/spinalcordtoolbox/jobs/731934950#L93
No error! Huzzah.

No error! Huzzah.

hopefully it will still the case for our extensive testing (ie across more platforms) when merging to master

No error! Huzzah.

Nice! We should add the fix for the other 2 osx builds as well, they all seemed to fail with the same issue (https://travis-ci.org/github/neuropoly/spinalcordtoolbox/builds/731682995)

authors: Drulex jcohenadad joshuacwnewton

repo: neuropoly/spinalcordtoolbox


text:
Small Login screen

When I bootup Ubuntu 18.04 LTS the xrdp login screen that comes up is very small as well as the fonts.   I know you can customize the size of the dialog, but the fonts etc are still very small in high res.  Is is possible to customize this screen more???

It is possible right now but if someone implement it, login screen can be scaled.

Sorry, not possible. But we accept contributions.

authors: metalefty ssonnenwald

repo: neutrinolabs/xrdp


text:
Cppcheck 2.8 removed the dependency on z3

Cppcheck removed the z3 dependency in https://github.com/danmar/cppcheck/commit/8f386e15fdedff37486c683d933ccc9a1e307388 so only perform the related steps if necessary.

Nice spot @firewave - thanks.


text:
NVSHAS-8711 Risk page - only return the filtered assets while advance…

…d filter was applied

it looks good

repo: neuvector/neuvector


text:
ttf download broken -> script exits

The current browser identifier for TTF results in a .woff file -> the script then exists with error code 1
possible solution: send an empty browser identifier -> you get a ttf

Thanks for the notice, fixed.

authors: hetzner neverpanic

repo: neverpanic/google-font-download


text:
oauth

adding oauth

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

authors: CLAassistant haihongren


text:
[Gatsby Cloud]  DNS Changes and Documentation Updates: Dev Site

Related Issue
https://github.com/newrelic/developer-website/issues/1270
Acceptance Criteria

[ ] Leave Amplify connected the repository
[ ] move the DNS entry over to Gatsby Cloud --> need to work with ELB
[ ] also move the SSL cert --> need to work with ELB
[ ] Publish any documentation changes (troubleshooting, architecture, etc)

I feel like the harbinger of bad news for this effort.
Fastly limits apex domains to one account. For example:
Account 1 has foo.example.com and example.com. Account 2 attempts to use bar.example.com and is unable to do so. This is because Account 1 owns *.example.com.
We have 2 possible solutions:

Ask if there is a possible work around - JIRA / ELB-1602
Integrate Gatsby Cloud with the existing Fastly account

blocked on the issue with sub-domains on Fastly which Gatsby is working on for June. Ticket TBD.

CNAME transfer ticket ELB-1682

authors: aswanson-nr jpvajda

repo: newrelic/developer-website


text:
[newrelic-logging] Allow securityContext editing in order to resolve fluent bug involving kernel limit issues

Is your feature request related to a problem? Please describe.
There is a bug in Fluentbit currently that results in logs not being processed when under heavy load, as  system limits imposed by the kernel are reached, more info here:
https://github.com/fluent/fluent-bit/issues/2661
An overview of the behaviour can be seen in the above thread, and also in the NewRelic Forum issue linked in Additional Context
Describe the solution you'd like
In order to workaround this isssue, I would like the ability to configure the pod securityPolicy via Helm installation.
Describe alternatives you've considered
We could maintain our own version of this chart, but would prefer to have the feature available
Additional context
Security Context Manifests: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
NewRelic Forum Issue: https://discuss.newrelic.com/t/new-relic-logging-bundle-fluentbit-failing-to-send-logs/180245
The way Helm usually handles this is by a large block in values. When generating a new template, we can see the following:
$ helm create demo
Creating demo

$ cat demo/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
...
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
...

$ cat demo/values.yaml
# Default values for demo.
...
securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000
...

I'll add another happy result from this issue being resolved is it would allow for Orgs like mine that use SELinux on our clusters to assign a Label to the pod to allow it the ability to read logs from the host

repo: newrelic/helm-charts


text:
[nri-bundle]: bump several dependencies

Is this a new chart
What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes # https://github.com/newrelic/helm-charts/issues/932

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[X] Chart Version bumped
[ ] Variables are documented in the README.md
[X] Title of the PR starts with chart name (e.g. [mychartname])

Hi Marc!
Thanks for taking care of this but this PR is opened by @renovate-bot automatically e.g.: PR #930
I am closing it because we have dependencies of nri-kubernetes and nri-kube-events that also need an update and renovate is going to take care of all of them :)

authors: kang-makes marcsanmi


text:
Inventory and host entity not updated when account/license_key is changed

We've seen some issue where inventory information is outdated or host is not visible in the Entity Explorer UI.
We suspect that changing the license_key is affecting both behaviours.
Steps to Reproduce

Have an agent running using license: AAA, ensure host is visible in UI, host inventory is collected, etc.
Stop the agent and change license to BBB
Restart the agent and verify inventory and entity explorer UI
Current behaviour: it seems only inventory deltas as submitted and host is not visible in UI

Your Environment
Reproduced on linux (Ubuntu 20 and RHEL 8)
Alternative
In order to fix this scenario, removing the data folder (var/db/newrelic-infra/data on linux) and restarting the agent should solve the issue.

This behavior also occurs on Windows (tested on Windows Server 2019) and requires deleting C:\ProgramData\New Relic\newrelic-infra\data.

authors: bpeckNR josemore

repo: newrelic/infrastructure-agent