text:
Equalize .log( new Error() ) and .log( { error: new Error() } )

These two are similar except the latter doesn't use the error level.
If error: is provided and there is no tag: property provided, use 'error' for tags.
Move adding error to the tags from transformArgs() to isLevelEnabled()

Implemented in 2.0.0

source: gharchive/issue

repo: good-ware/js-log

license_type: permissive

license_source: github-api


text:
Problem with freetype? Still unable to run coldtype demo

Thanks so much for looking into #27 @stenson! I really appreciate all your help.
On a follow up on that:
I could now get a few steps further, however I had to upgrade pip after it threw a ton of new errors at me. These are for later reference, perhaps this could be added to the install instructions.
either this pip install --upgrade pip setuptools wheel
or into the venv: {...}/venv/bin/python3.9 -m pip install --upgrade pip not sure, I used my absolute path with was proposed by the terminal error. It seems that the PIP inside of the venv always needs to be updated. I tried again after I updated my local pip, and it threw the error about an outdated pip again.
then I do the pip install "coldtype[viewer]"  and can’t see any errors in the log.
However, if trying to continue with coldtype demo
I get a bunch of errors again
Traceback (most recent call last):
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/freetype/raw.py", line 49, in <module>
   _lib = ctypes.CDLL(filename)
 File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 374, in __init__
   self._handle = _dlopen(self._name, mode)
OSError: dlopen(libfreetype.dylib, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/{MY_PATH}/Coldtype/venv/bin/coldtype", line 5, in <module>
   from coldtype.renderer import main
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/__init__.py", line 4, in <module>
   from coldtype.text import *
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/text/__init__.py", line 3, in <module>
   import coldtype.text.composer
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/text/composer.py", line 7, in <module>
   from coldtype.text.reader import Style, StyledString, FittableMixin, Font, normalize_font_path, SegmentedString,  _PenClass, _PensClass
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/text/reader.py", line 42, in <module>
   from coldtype.fontgoggles.font.otfFont import OTFFont
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/fontgoggles/font/otfFont.py", line 7, in <module>
   from ..misc.ftFont import FTFont
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/coldtype/fontgoggles/misc/ftFont.py", line 5, in <module>
   import freetype
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/freetype/__init__.py", line 24, in <module>
   from freetype.raw import *
 File "/{MY_PATH}/Coldtype/venv/lib/python3.9/site-packages/freetype/raw.py", line 52, in <module>
   raise RuntimeError('Freetype library not found')
RuntimeError: Freetype library not found

freetype is installed, running pip install freetype-py claims Requirement already satisfied: freetype-py in ./venv/lib/python3.9/site-packages (2.2.0). It also shows in the list pip list -l as freetype-py 2.2.0
Originally posted by @Mark2Mark in https://github.com/goodhertz/coldtype/issues/27#issuecomment-945078202

Here’s how it’s going.
I reinstalled freetype with brew, and located it in
/opt/homebrew/Cellar/freetype/2.11.0/lib/libfreetype.6.dylib
Then I copied that file to the venv .../venv/lib/python3.9/site-packages/freetype/libfreetype.dylib and renamed it to libfreetype.dylib
Now I don’t get any errors anymore on coldtype demo
But now it is stuck on ... watching venv/lib/python3.9/site-packages/coldtype/demo/demo.py for changes ... while no window appears as expected. No python app in the Dock.

Hi Mark — sorry for the all the trouble here! Unfortunately I haven't seen any of these errors before and I don't have access to an M1 computer, although I know of at least one M1 computer that has successfully run an older version of coldtype, so I'm not sure what the issue there would be. Could definitely be M1, though, since the prebuilt binaries that some python libraries come with might not be built properly for M1, which is what the freetype-py issue sounds like (libfreetype.dylib: mach-o, but wrong architecture) except that I can't find anything online about people freetype-py doesn't work on M1 which seems a little odd.
What OS version are you running?
And if you are comfortable modifying the coldtype source, if you locate the coldtype/renderer/__init__.py file in your venv, could you try modifying the initialize_gui_and_server function to have an additional print at the beginning, so it'd look something like this:
    def initialize_gui_and_server(self):
        self.winmans.add_viewers()

        print("GLSK >>>", self.winmans.glsk)

The glsk there is the viewer (a GLFW window), so if that's None then it's not being created, which could happen if the skia-python or glfw libraries aren't being installed properly.

Hi @stenson Thanks so much for your quick reply.
I think we’re pretty close, as the M1 issue was apparently semi-solved by me dumping the latest dylib in there myself. (I think you could bundle the newest freetype as it ships from here (that’s the one I uploaded as a zip in my comment above). There it’s mentioned to be compatible with Silicon and Big Sur.
I was happy to add your debug print, however it prints GLSK >>> None
Can I somehow install skia-python and or glfw in a way that coldtype can use it?
Thanks so so much for all your help, I am still so excited to get it running. Never saw it on my screen yet :D

Thanks so much.
I couldn’t find the version, but I ran this pip insall and it seems it was already that version:
Requirement already satisfied: glfw==2.3.0 in ./venv/lib/python3.9/site-packages (2.3.0)

Hmm, yeah I guess that should have the fix then. Do you know where glfw is installed to in your homebrew?

I don’t have it installed in homebrew (generally I try to not use homebrew anyway, just got it for a necessary web dev project).
But I did that build as mentioned here

Now I was building a GLFW with the commands as state here and got a folder with a build that might work (I can run the example files in {...}/glfw-3.3.2/build/examples).
which just sits in a folder. If I knew if I could pick a few files out of that and dump it into the venv …

Ah, so sory for that …

I did it now with the Homebrew one and got a tiny bit futher with a next error now.
Then I found the same-named libglfw in the arm build (again see see my quoted part above) and it shows the same error.
But I got the feeling we’re closer
❯ coldtype demo
... watching venv/lib/python3.9/site-packages/coldtype/demo/demo.py for changes ...
GLSK >>> <coldtype.renderer.winman.glfwskia.WinmanGLFWSkia object at 0x110427850>
[1]    62709 segmentation fault  coldtype demo

After the error, an NSAlert pops up, telling me that "Python quit unexpectedly."
Thank you so much for sticking with me, I know how annoying it is to debug such things, especially remotely. I really appreciate all your help!

Thank you all for your help. Unfortunately now I don’t know if it was just this one little line as mentioned in my last answer. I will try to start from scratch tomorrow and see if that already solves it. Otherwise I’ll go through the steps and make a list here of M1 requirements. It could be that the homebrew and/or self-built dylib was also necesary.

@Mark2Mark great to hear it's working! My guess is that pin and False might be a big part of it, although wouldn't solve the needing-to-homebrew-install for freetype & glfw
@jo-lang thanks for chiming in! very helpful to get another m1 perspective in here — did you have rosetta enabled when you built the virtualenv?

Before turning the gathered info about dependencies into a proper list, i just quickly dump more things here:

FFmpeg brew install ffmpeg
...

I've just released version 2.4.0 of the package glfw, along with wheels for macOS with arm64, so the pyglfw half of this issue should be fixed.
I'm using the binaries provided by GLFW3 and do not have an M1 macbook myself, so if there are any issues with the wheels, please let me know!

@FlorianRhiem thanks for letting us know! working great on my m1 laptop with an arm python3.9

authors: FlorianRhiem Mark2Mark stenson

repo: goodhertz/coldtype


text:
install_requirements.sh errors out: fatal error: Python.h

Description
New install of Coral and this project.  Ran bash gstreamer/install_requirements.sh and script errored out with:
 aarch64-linux-gnu-g++: lap/lapmod.cpp
   lap/_lapjv.cpp:4:10: fatal error: Python.h: No such file or directory
    #include "Python.h"
             ^~~~~~~~~~
   compilation terminated.

Source: line 184 et seq.  from gstreamer_install_attemp_1.script.txt (uploaded).
I then performed:
 sudo apt install python3-dev

then reran bash install_requirements.sh which then succeeded; see log at gstreamer_install_attempt_2.script.txt
gstreamer_install_attemp_1.script.txt
gstreamer_install_attempt_2.script.txt
Click to expand!
Issue Type
Build/Install
Operating System
Mendel Linux
Coral Device
Dev Board
Other Devices
No response
Programming Language
No response
Relevant Log Output
aarch64-linux-gnu-g++: lap/lapmod.cpp
    lap/_lapjv.cpp:4:10: fatal error: Python.h: No such file or directory
     #include "Python.h"
              ^~~~~~~~~~
    compilation terminated.

Hi  @jlpoolen Thanks for reporting!  I have updated the install_requestments.sh file.

authors: hjonnala jlpoolen

repo: google-coral/example-object-tracker

license_source: bigquery


text:
kfac_jax error when running H2 example script

Hi, I'm trying to run the example script for the H2 molecule on Colab, but I run into an attribute error on the first iteration of training. Below are the commands I'm using to install the relevant packages:
pip install git+https://github.com/deepmind/ferminet@main
pip install numpy==1.26.0 #Error in importing pyscf with version 1.26.1

the script I'm trying to run:
import sys

from absl import logging
from ferminet.utils import system
from ferminet import base_config
from ferminet import train

# Optional, for also printing training progress to STDOUT.
# If running a script, you can also just use the --alsologtostderr flag.
logging.get_absl_handler().python_handler.stream = sys.stdout
logging.set_verbosity(logging.INFO)

# Define H2 molecule
cfg = base_config.default()
cfg.system.electrons = (1,1)  # (alpha electrons, beta electrons)
cfg.system.molecule = [system.Atom('H', (0, 0, -1)), system.Atom('H', (0, 0, 1))]

# Set training parameters
cfg.batch_size = 256
cfg.pretrain.iterations = 100

train.train(cfg)

as well as the full printout of the error message:
INFO:absl:Starting QMC with 1 XLA devices per host across 1 hosts.
converged SCF energy = -1.05642988216974  <S^2> = -4.4408921e-16  2S+1 = 1
INFO:absl:No checkpoint found. Training new model.
INFO:absl:Pretrain iter 00000: 0.0987188
INFO:absl:Pretrain iter 00001: 0.0468968
INFO:absl:Pretrain iter 00002: 0.021173
INFO:absl:Pretrain iter 00003: 0.012926
INFO:absl:Pretrain iter 00004: 0.012574
INFO:absl:Pretrain iter 00005: 0.0137347
INFO:absl:Pretrain iter 00006: 0.0137845
INFO:absl:Pretrain iter 00007: 0.012732
INFO:absl:Pretrain iter 00008: 0.0110899
INFO:absl:Pretrain iter 00009: 0.00934778
INFO:absl:Pretrain iter 00010: 0.00778915
INFO:absl:Pretrain iter 00011: 0.00656673
INFO:absl:Pretrain iter 00012: 0.00565565
INFO:absl:Pretrain iter 00013: 0.00497928
INFO:absl:Pretrain iter 00014: 0.00446487
INFO:absl:Pretrain iter 00015: 0.00406806
INFO:absl:Pretrain iter 00016: 0.00370217
INFO:absl:Pretrain iter 00017: 0.00335341
INFO:absl:Pretrain iter 00018: 0.00301634
INFO:absl:Pretrain iter 00019: 0.00270557
INFO:absl:Pretrain iter 00020: 0.00244722
INFO:absl:Pretrain iter 00021: 0.00228125
INFO:absl:Pretrain iter 00022: 0.00220615
INFO:absl:Pretrain iter 00023: 0.00218246
INFO:absl:Pretrain iter 00024: 0.00216123
INFO:absl:Pretrain iter 00025: 0.00209267
INFO:absl:Pretrain iter 00026: 0.00195859
INFO:absl:Pretrain iter 00027: 0.0017916
INFO:absl:Pretrain iter 00028: 0.00160594
INFO:absl:Pretrain iter 00029: 0.00143077
INFO:absl:Pretrain iter 00030: 0.0012787
INFO:absl:Pretrain iter 00031: 0.00115612
INFO:absl:Pretrain iter 00032: 0.00107149
INFO:absl:Pretrain iter 00033: 0.00101318
INFO:absl:Pretrain iter 00034: 0.00097588
INFO:absl:Pretrain iter 00035: 0.000957538
INFO:absl:Pretrain iter 00036: 0.000952348
INFO:absl:Pretrain iter 00037: 0.000960992
INFO:absl:Pretrain iter 00038: 0.000957644
INFO:absl:Pretrain iter 00039: 0.00094491
INFO:absl:Pretrain iter 00040: 0.000913838
INFO:absl:Pretrain iter 00041: 0.000867143
INFO:absl:Pretrain iter 00042: 0.000815364
INFO:absl:Pretrain iter 00043: 0.000763372
INFO:absl:Pretrain iter 00044: 0.000719524
INFO:absl:Pretrain iter 00045: 0.00068491
INFO:absl:Pretrain iter 00046: 0.000651115
INFO:absl:Pretrain iter 00047: 0.000626584
INFO:absl:Pretrain iter 00048: 0.000609355
INFO:absl:Pretrain iter 00049: 0.000602466
INFO:absl:Pretrain iter 00050: 0.000594917
INFO:absl:Pretrain iter 00051: 0.000595118
INFO:absl:Pretrain iter 00052: 0.000592292
INFO:absl:Pretrain iter 00053: 0.000585884
INFO:absl:Pretrain iter 00054: 0.000574762
INFO:absl:Pretrain iter 00055: 0.000557475
INFO:absl:Pretrain iter 00056: 0.000534383
INFO:absl:Pretrain iter 00057: 0.000514163
INFO:absl:Pretrain iter 00058: 0.000497474
INFO:absl:Pretrain iter 00059: 0.000483247
INFO:absl:Pretrain iter 00060: 0.000474224
INFO:absl:Pretrain iter 00061: 0.000468507
INFO:absl:Pretrain iter 00062: 0.000463785
INFO:absl:Pretrain iter 00063: 0.000460483
INFO:absl:Pretrain iter 00064: 0.000456877
INFO:absl:Pretrain iter 00065: 0.000450266
INFO:absl:Pretrain iter 00066: 0.000445674
INFO:absl:Pretrain iter 00067: 0.00043909
INFO:absl:Pretrain iter 00068: 0.00043164
INFO:absl:Pretrain iter 00069: 0.000425373
INFO:absl:Pretrain iter 00070: 0.000416835
INFO:absl:Pretrain iter 00071: 0.000408702
INFO:absl:Pretrain iter 00072: 0.000404959
INFO:absl:Pretrain iter 00073: 0.000398033
INFO:absl:Pretrain iter 00074: 0.000392218
INFO:absl:Pretrain iter 00075: 0.000388479
INFO:absl:Pretrain iter 00076: 0.000385467
INFO:absl:Pretrain iter 00077: 0.000381409
INFO:absl:Pretrain iter 00078: 0.000377661
INFO:absl:Pretrain iter 00079: 0.00037244
INFO:absl:Pretrain iter 00080: 0.000368109
INFO:absl:Pretrain iter 00081: 0.000364462
INFO:absl:Pretrain iter 00082: 0.000360276
INFO:absl:Pretrain iter 00083: 0.0003561
INFO:absl:Pretrain iter 00084: 0.000352395
INFO:absl:Pretrain iter 00085: 0.000348523
INFO:absl:Pretrain iter 00086: 0.000344737
INFO:absl:Pretrain iter 00087: 0.000342277
INFO:absl:Pretrain iter 00088: 0.000339568
INFO:absl:Pretrain iter 00089: 0.000336274
INFO:absl:Pretrain iter 00090: 0.000333011
INFO:absl:Pretrain iter 00091: 0.000329609
INFO:absl:Pretrain iter 00092: 0.000326805
INFO:absl:Pretrain iter 00093: 0.000322345
INFO:absl:Pretrain iter 00094: 0.000320427
INFO:absl:Pretrain iter 00095: 0.000316732
INFO:absl:Pretrain iter 00096: 0.000314699
INFO:absl:Pretrain iter 00097: 0.000311589
INFO:absl:Pretrain iter 00098: 0.000308642
INFO:absl:Pretrain iter 00099: 0.000305587
INFO:absl:==================================================
INFO:absl:Graph parameter registrations:
INFO:absl:{'envelope': [{'pi': 'Auto[scale_and_shift_tag_1]',
               'sigma': 'Auto[scale_and_shift_tag_0]'},
              {'pi': 'Auto[scale_and_shift_tag_3]',
               'sigma': 'Auto[scale_and_shift_tag_2]'}],
 'layers': {'input': {},
            'streams': [{'double': {'b': 'Auto[repeated_dense_tag_1]',
                                    'w': 'Auto[repeated_dense_tag_1]'},
                         'single': {'b': 'Auto[repeated_dense_tag_0]',
                                    'w': 'Auto[repeated_dense_tag_0]'}},
                        {'double': {'b': 'Auto[repeated_dense_tag_3]',
                                    'w': 'Auto[repeated_dense_tag_3]'},
                         'single': {'b': 'Auto[repeated_dense_tag_2]',
                                    'w': 'Auto[repeated_dense_tag_2]'}},
                        {'double': {'b': 'Auto[repeated_dense_tag_5]',
                                    'w': 'Auto[repeated_dense_tag_5]'},
                         'single': {'b': 'Auto[repeated_dense_tag_4]',
                                    'w': 'Auto[repeated_dense_tag_4]'}},
                        {'single': {'b': 'Auto[repeated_dense_tag_6]',
                                    'w': 'Auto[repeated_dense_tag_6]'}}]},
 'orbital': [{'w': 'Auto[repeated_dense_tag_7]'},
             {'w': 'Auto[repeated_dense_tag_8]'}]}
INFO:absl:==================================================
INFO:absl:Burning in MCMC chain for 100 steps
INFO:absl:Completed burn-in MCMC steps
INFO:absl:Initial energy: -1.1759 E_h
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-df9aebf4a1f2> in <cell line: 22>()
     20 cfg.pretrain.iterations = 100
     21
---> 22 train.train(cfg)

16 frames
/usr/local/lib/python3.10/dist-packages/ferminet/train.py in train(cfg, writer_manager)
    712     for t in range(t_init, cfg.optim.iterations):
    713       sharded_key, subkeys = kfac_jax.utils.p_split(sharded_key)
--> 714       data, params, opt_state, loss, unused_aux_data, pmove = step(
    715           data,
    716           params,

/usr/local/lib/python3.10/dist-packages/ferminet/train.py in step(data, params, state, key, mcmc_width)
    313
    314     # Optimization step
--> 315     new_params, state, stats = optimizer.step(
    316         params=params,
    317         state=state,

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/optimizer.py in step(self, params, state, rng, data_iterator, batch, func_state, learning_rate, momentum, damping, global_step_int)
   1214       batch = next(data_iterator)
   1215
-> 1216     return self._step(params, state, rng, batch, func_state,
   1217                       learning_rate, momentum, damping)
   1218

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/utils/staging.py in decorated(instance, *args)
    246           pmap_funcs[key] = func
    247
--> 248         outs = func(instance, *args)
    249
    250       else:

    [... skipping hidden 12 frame]

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/utils/misc.py in wrapped(instance, *args, **kwargs)
    283       method_name = method_name[1:]
    284     with jax.named_scope(f"{class_name}_{method_name}"):
--> 285       return method(instance, *args, **kwargs)
    286
    287   return wrapped

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/optimizer.py in _step(self, params, state, rng, batch, func_state, learning_rate, momentum, damping)
   1020
   1021     # Update curvature estimate
-> 1022     state = self._maybe_update_estimator_curvature(
   1023         state,
   1024         func_args,

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/optimizer.py in _maybe_update_estimator_curvature(self, state, func_args, rng, ema_old, ema_new, sync)
    723   ) -> "Optimizer.State":
    724     """Updates the curvature estimates if it is the right iteration."""
--> 725     return self._maybe_update_estimator_state(
    726         state,
    727         self.should_update_estimate_curvature(state),

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/optimizer.py in _maybe_update_estimator_state(self, state, should_update, update_func, **update_func_kwargs)
    678     state = state.copy()
    679
--> 680     state.estimator_state = lax.cond(
    681         should_update,
    682         functools.partial(update_func, **update_func_kwargs),

    [... skipping hidden 13 frame]

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/optimizer.py in _update_estimator_curvature(self, estimator_state, func_args, rng, ema_old, ema_new, sync)
    696   ) -> curvature_estimator.BlockDiagonalCurvature.State:
    697     """Updates the curvature estimator state."""
--> 698     state = self.estimator.update_curvature_matrix_estimate(
    699         state=estimator_state,
    700         ema_old=ema_old,

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/utils/misc.py in wrapped(instance, *args, **kwargs)
    283       method_name = method_name[1:]
    284     with jax.named_scope(f"{class_name}_{method_name}"):
--> 285       return method(instance, *args, **kwargs)
    286
    287   return wrapped

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/curvature_estimator.py in update_curvature_matrix_estimate(self, state, ema_old, ema_new, batch_size, rng, func_args, estimation_mode)
   1239
   1240     # Compute the losses and the VJP function from the function inputs
-> 1241     losses, losses_vjp = self._compute_losses_vjp(func_args)
   1242
   1243     if "fisher" in estimation_mode:

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/utils/misc.py in wrapped(instance, *args, **kwargs)
    283       method_name = method_name[1:]
    284     with jax.named_scope(f"{class_name}_{method_name}"):
--> 285       return method(instance, *args, **kwargs)
    286
    287   return wrapped

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/curvature_estimator.py in _compute_losses_vjp(self, func_args)
   1037   def _compute_losses_vjp(self, func_args: utils.FuncArgs):
   1038     """Computes all model statistics needed for estimating the curvature."""
-> 1039     return self._vjp(func_args)
   1040
   1041   def params_vector_to_blocks_vectors(

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/tracer.py in wrapped_transformation(func_args, return_only_jaxpr, *args)
    379       return jaxpr
    380     else:
--> 381       return f(func_args, *args)
    382
    383   return wrapped_transformation

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/tracer.py in _layer_tag_vjp(processed_jaxpr, primal_func_args)
    781
    782   # First compute the primal values for the inputs to all layer tags
--> 783   layer_input_values = forward()
    784   primals_dict = dict(zip(layer_input_vars, layer_input_values))
    785

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/tracer.py in forward()
    698     for eqn in processed_jaxpr.jaxpr.eqns:
    699
--> 700       write(eqn.outvars, tgm.eval_jaxpr_eqn(eqn, read(eqn.invars)))
    701
    702       if isinstance(eqn.primitive, tags.LossTag):

/usr/local/lib/python3.10/dist-packages/kfac_jax/_src/tag_graph_matcher.py in eval_jaxpr_eqn(eqn, in_values)
     63
     64   if jax_version > (0, 4, 11):
---> 65     user_context = jax_extend.source_info_util.user_context
     66   else:
     67     user_context = jax.core.source_info_util.user_context

AttributeError: module 'jax.extend' has no attribute 'source_info_util'

I cannot reproduce this with jax 0.4.19. What version of jax are you using?
Also pyscf 2.4.0 works fine for me with numpy 1.26.1 -- are you also using an older version of pyscf?
pip freeze is helpful to show the versions of all modules installed.

The version of jax was 0.4.16, I've installed version 0.4.19 and the code runs fine, thanks! As for pyscf, I'm running version 2.4.0 as well, but when I try importing it (with numpy version 1.26.1) I get
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-2-39797ef5fbb5>](https://localhost:8080/#) in <cell line: 1>()
----> 1 import pyscf

10 frames
[/usr/local/lib/python3.10/dist-packages/pyscf/__init__.py](https://localhost:8080/#) in <module>
     97 from pyscf import lib
     98 from pyscf import gto
---> 99 from pyscf import scf
    100 from pyscf import ao2mo
    101

[/usr/local/lib/python3.10/dist-packages/pyscf/scf/__init__.py](https://localhost:8080/#) in <module>
    103
    104 from pyscf import gto
--> 105 from pyscf.scf import hf
    106 rhf = hf
    107 from pyscf.scf import rohf

[/usr/local/lib/python3.10/dist-packages/pyscf/scf/hf.py](https://localhost:8080/#) in <module>
     31 from pyscf import lib
     32 from pyscf.lib import logger
---> 33 from pyscf.scf import diis
     34 from pyscf.scf import _vhf
     35 from pyscf.scf import chkfile

[/usr/local/lib/python3.10/dist-packages/pyscf/scf/diis.py](https://localhost:8080/#) in <module>
     24 import numpy
     25 import scipy.linalg
---> 26 import scipy.optimize
     27 from pyscf import lib
     28 from pyscf.lib import logger

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/__init__.py](https://localhost:8080/#) in <module>
    408
    409 from ._optimize import *
--> 410 from ._minimize import *
    411 from ._root import *
    412 from ._root_scalar import *

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_minimize.py](https://localhost:8080/#) in <module>
     25 from ._trustregion_krylov import _minimize_trust_krylov
     26 from ._trustregion_exact import _minimize_trustregion_exact
---> 27 from ._trustregion_constr import _minimize_trustregion_constr
     28
     29 # constrained minimization

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_trustregion_constr/__init__.py](https://localhost:8080/#) in <module>
      2
      3
----> 4 from .minimize_trustregion_constr import _minimize_trustregion_constr
      5
      6 __all__ = ['_minimize_trustregion_constr']

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py](https://localhost:8080/#) in <module>
      3 from scipy.sparse.linalg import LinearOperator
      4 from .._differentiable_functions import VectorFunction
----> 5 from .._constraints import (
      6     NonlinearConstraint, LinearConstraint, PreparedConstraint, Bounds, strict_bounds)
      7 from .._hessian_update_strategy import BFGS

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_constraints.py](https://localhost:8080/#) in <module>
      6 from ._optimize import OptimizeWarning
      7 from warnings import warn, catch_warnings, simplefilter
----> 8 from numpy.testing import suppress_warnings
      9 from scipy.sparse import issparse
     10

[/usr/local/lib/python3.10/dist-packages/numpy/testing/__init__.py](https://localhost:8080/#) in <module>
      9
     10 from . import _private
---> 11 from ._private.utils import *
     12 from ._private.utils import (_assert_valid_refcount, _gen_alignment_data)
     13 from ._private import extbuild

[/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py](https://localhost:8080/#) in <module>
     55 IS_PYSTON = hasattr(sys, "pyston_version_info")
     56 HAS_REFCOUNT = getattr(sys, 'getrefcount', None) is not None and not IS_PYSTON
---> 57 HAS_LAPACK64 = numpy.linalg._umath_linalg._ilp64
     58
     59 _OLD_PROMOTION = lambda: np._get_promotion_state() == 'legacy'

AttributeError: module 'numpy.linalg._umath_linalg' has no attribute '_ilp64'

However, this is not crucial because using version 1.26.0 of numpy solves the issue for me.

authors: eliasteve jsspencer

repo: google-deepmind/ferminet


text:
Why dont we expose MjStepArgs in the PreUpdate and PostUpdate EventHandler?

Is your feature request related to a problem? Please describe.
Would love to have access to model and data through PreUpdate and PostUpdate handles. I know I can get it using a reference to MjScene but it just makes the code cleaner.
Describe the solution you'd like
https://github.com/google-deepmind/mujoco/blob/c8e839a6662199cf94153906e1b11790dbe3e04d/unity/Runtime/Components/MjScene.cs#L82
https://github.com/google-deepmind/mujoco/blob/c8e839a6662199cf94153906e1b11790dbe3e04d/unity/Runtime/Components/MjScene.cs#L84C3-L84C39
Convert this to a template with MjStepsArgs

I added the EventHandlers originally, and since using them now I agree with all of them using MjStepArgs, and I'm happy to submit a PR for that. Furthermore I'm considering marking the EventHandlers as events explicitly. This means only MjScene is allowed to perform any operation on them (e.g. invoke them, clear them etc), everywhere else only subscribe/unsubscribe is available.
Would it break any functionality for you if I swap them for events?

It wont break things for me as of now. I am using the event handlers with += and -= accesors which are the standard way of doing things anyway! Thanks for the jiffy reply.

@erez-tom This issue can be closed now

authors: Balint-H vidurvij-apptronik

repo: google-deepmind/mujoco


text:
Add optax.tree_utils.tree_random_split.

This exposes the formerly private function _tree_rng_keys_split in optax/tree_utils/_random.py to the public API.
I've found this to be a useful helper function for manipulation of random trees, and intend to use it for future PRs.

Thanks for doing that, it's a good idea, happy to know it can be useful. (sorry for the incremental review, didn't mean to do it like that).

source: gharchive/pull-request

authors: carlosgmartin vroulet

repo: google-deepmind/optax


text:
Create an interactive Dice Roller app: Android Basics with Compose

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-compose-build-a-dice-roller-app?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-compose-unit-2-pathway-2%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-compose-build-a-dice-roller-app#6
In which task and step of the codelab can this issue be found?
7-Build the dice-roll logic
Describe the problem
the code of : var result by remember { mutableStateOf(1) } resulting an error of
Steps to reproduce?

Go to...7-Build the dice-roll logic
Click on...or use the code of : var result by remember { mutableStateOf(1) }
See error...Type 'TypeVariable(T)' has no method 'getValue(Nothing?, KProperty<*>)' and thus it cannot serve as a delegate and Variable expected

Versions
Android Studio version: 2022.2.1 Patch 2
API version of the emulator: API 33
Solutions
i replace the problem code with this
var rollResult = remember {
mutableStateOf(1)
}
it work and the project was able to run as intended
Additional information
Include screenshots if they would be useful in clarifying the problem.

Hi @zamri98
Thank you for writing in with your issue.
In the lesson, directly following this line of code is a "Note" box that explains the imports needed to resolve this issue.
Please follow these steps to see if this resolves it for you.
If you still experiencing it, please do not hesitate to create a new issue.
Hope you are enjoying the course!
Regards.


text:
How to pass in a JSON variable into env_vars

Question
I'm trying to pass in a JSON variable into env_vars but ending up with issues parsing the commas, as parseEnvVars thinks those are separate environment variables and hence looks for = without finding it.

Hi @Doggie52
Thanks for the report, this is much asked for usecase. Hoping to allow this with https://github.com/google-github-actions/deploy-cloud-functions/pull/34 which will allow you to pass in a yaml file and then you should be able to
foo: '{"bar":"baz"}'

Would that use case allow for passing in a yaml file file with a variable set from a secret?
Basically I have a secret that is a JSON object, and I want to pass this JSON object (as a string) into an env variable for my function.

While possible with #34 the general cloud functions best practice is not use env vars for secrets in general. More details here: https://cloud.google.com/functions/docs/env-var#managing_secrets
Closing this issue since we released 0.3.0 which has the env_var_file feature. Feel free to reopen if you run into any issues.

Thanks for the advice, turns out it was an entirely unnecessary secret to pass anyway. Great new feature though regardless!


text:
Support Adding Additional Changelog Types

TL;DR
I need to generate a changelog without PR links for each commit and would like to do so by adding my own changelog-type as described here.
Detailed design
Docs mentioned that we can use ChangelogNotes interface to add additional changelog types; but this seems to not apply to the Github action.
Additional information
No response

See related: https://github.com/googleapis/release-please/issues/2080

It will be nice if they provide some clear and concise instructions about how to do things. As by most google products, the existing documentation is ambiguous and it is not clear if some things are even possible.


text:
JWT solution for react google pay button

Is your feature request related to a problem? Please describe.
Yes, the problem is that the current Google Pay Button implementation in React does not allow the use of the JWT solution provided by the Google Pay API for Web. This solution is crucial for platform partners who manage more than 100 web domains, as it allows the integration of Google Pay without registering individual web domains in the Google Pay & Wallet Console. Specifically, there is no support for passing the authJwt and merchantOrigin parameters in the merchantInfo object, which are necessary for using JWT.
Describe the solution you'd like
Please add support for authJwt and merchantOrigin in the merchantInfo object in the Google Pay Button for React. This would allow developers to use the JWT solution directly in their payment requests.
<GooglePayButton
  environment="TEST"
  paymentRequest={{
    apiVersion: 2,
    apiVersionMinor: 0,
    allowedPaymentMethods: [
      {
        type: 'CARD',
        parameters: {
          allowedAuthMethods: ['PAN_ONLY', 'CRYPTOGRAM_3DS'],
          allowedCardNetworks: ['MASTERCARD', 'VISA'],
        },
        tokenizationSpecification: {
          type: 'PAYMENT_GATEWAY',
          parameters: {
            gateway: 'example',
            gatewayMerchantId: 'exampleGatewayMerchantId',
          },
        },
      },
    ],
    merchantInfo: {
      merchantId: '12345678901234567890',
      merchantName: 'Demo Merchant',
      // Add support for these fields:
      // authJwt: 'aaaaa.bbbbb.ccccc',
      // merchantOrigin: 'sub-merchant.com',
    },
    transactionInfo: {
      totalPriceStatus: 'FINAL',
      totalPriceLabel: 'Total',
      totalPrice: '100.00',
      currencyCode: 'USD',
      countryCode: 'US',
    },
  }}
  onLoadPaymentData={paymentRequest => {
    console.log('load payment data', paymentRequest);
  }}
/>

Describe alternatives you've considered A clear and concise description of any alternative solutions or features
you've considered.

Additional context Add any other context or screenshots about the feature request here.
Adding support for authJwt and merchantOrigin will significantly ease the integration process for platform partners using the JWT solution. This feature is particularly beneficial for developers who manage multiple sub-merchants under one platform, simplifying the onboarding process and reducing the need to register each domain separately.

Thanks for the response @dmengelt really looking forward to the feature, as of now, I'm considering using the Google Pay script and passing the data manually. However, I would prefer to use a React button library for our Google Pay integration.

Yes. For now you would need to integrate using pay.js directly.

can I contribute to the project for this feature @dmengelt ?

@devalensio yes of course! A couple of thoughts:

Make sure to first update the Google Pay types when adding support for authJwt and merchantOrigin
Please update all the different implementation is this repo (not only the React one)

authors: devalensio dmengelt

repo: google-pay/google-pay-button


text:
#Photochat dataset questions:

Hey, I am a student do some researches on VD. I noticed that you've proposed a great dataset called PhotoChat for multi-model  dialog generative system.
However, it seems that both the url provided and id are invalid for me now. Could you give me some tips how can I solve it without download the whole image challenge dataset with is 500G+(too expensive for me).
train/0a5e922c15ffb08d https://c6.staticflickr.com/1/156/400565484_fdfaa551e7_o.jpg
train/048fe12fa832143f https://c8.staticflickr.com/1/29/96515447_4ac8050343_o.jpg
train/080a8a8692a66687 https://farm8.staticflickr.com/6082/6042280582_c7a5d88b51_o.jpg
train/07f372b1aba8b165 https://farm6.staticflickr.com/7416/9782494465_1f1a8a315f_o.jpg
train/a593754ce22e87da https://c3.staticflickr.com/4/3488/3697088360_5b402b5c66_o.jpg
train/0617ddfd18b1343f https://c8.staticflickr.com/6/5565/15132241305_540a42cf78_o.jpg

@XiaoxueZang
Best regards,
Zeyu Li

train/0a5e922c15ffb08d https://c6.staticflickr.com/1/156/400565484_fdfaa551e7_o.jpg
train/048fe12fa832143f https://c8.staticflickr.com/1/29/96515447_4ac8050343_o.jpg
train/080a8a8692a66687 https://farm8.staticflickr.com/6082/6042280582_c7a5d88b51_o.jpg
train/07f372b1aba8b165 https://farm6.staticflickr.com/7416/9782494465_1f1a8a315f_o.jpg
train/a593754ce22e87da https://c3.staticflickr.com/4/3488/3697088360_5b402b5c66_o.jpg
train/0617ddfd18b1343f https://c8.staticflickr.com/6/5565/15132241305_540a42cf78_o.jpg
train/03d53e7254fc863f https://c8.staticflickr.com/4/3920/14957339372_3a2e4335c7_o.jpg
train/a331e848c0e8265c https://c5.staticflickr.com/4/3647/3375325262_bfcf571c08_o.jpg
train/0a6fb3d70086e39a https://c3.staticflickr.com/6/5589/14534135669_401f933315_o.jpg

I will replace the images of these dialogs to 'None'  if necessary.

repo: google-research/google-research


text:
Further improvements of the Strange Stories task using psychology professor feedback

Dear Googlers,
This is an update to the accepted and merged strange_stories_task. We recently received the professor's approval for using her Strange Stories psychology battery in this task. Additionally, she provided us with immensely helpful feedback and edits, which I implemented in this pull request. She has also accepted our invitation to be a co-author of this task.
Specifically, we:

improved grammar and removed typos
rephrased better trick/trap questions
corrected mistakes in grading
corrected some questions' wording for clarity
removed two questions that can be misinterpreted or are ambiguous

The pyTests passed using the command: pytest -s bigbench/api/test_tasks.py --tasks strange_stories_json
Please let me know if I can update the Strange Stories task using this PR before it's evaluated in the paper. Thank you!
Previous PRs are available here: #494 (improvements)  #352 (initial acceptance and merge)  #262 (closed PR)
Co-authors: @lewkowycz and Professor Happé
Reviewer: @r-barnes
Metareviewer: @cdfreeman-google
I am grateful for the support and help from the co-authors and reviewers above.

Thank you for these improvements!!
Not a reviewer, but a quick technical comment: you probably want "example_input_prefix": "Context: ", (with the space after the colon) in both of your json files, for text formatting reasons.

Please let me know if there's anything else. I hope this can be merged in time for the model evaluation :)
Thanks!

I just discussed this with @cdfreeman-google who is supportive of the updated version. Thank you for making these improvements!! Merging now.

Great! Thanks everyone.

authors: Sohl-Dickstein chen-yifu

repo: google/BIG-bench


text:
Ice40 placement

Generate files critpath.png and critpath.log in the build gateware/ directory.   The PNG file contains a color-coded plot of placement color coded by LiteX, CPU, and CFU; the plot is overlayed with the critical path.   The LOG file contains the text list of cells on the critical path, and also the color key for the placement.
To do: generalize to make it easy to adapt for ECP5, Nexus, or any other part that uses Nextpnr.

We might need to do an apt install of matplotlib if not using Conda.

repo: google/CFU-Playground


text:
EarlGrey 2.2.0 | Firebase Test Lab support

Hi all,
I've added EarlGrey to the project using this guide.
Even though all tests run locally just fine, I'm having trouble running them in the Firebase Test Lab. Each time I get an error with the following message:
libswiftPhotos.dylib: Code object is not signed.
My UITests target builds locally on a real device without any code signing issues. All code signing is done manually because Apple introduced 2FA a few months ago, and I’m unable to use AppStoreConnect API Key because enterprise accounts do not support it.
Question: Is EarlGrey 2 compatible with Firebase Test Lab?

I have the same problem. Everything works fine locally, but fails on Firebase Test Lab with a mysterious Failed to add forwardInvocation.
See also:

https://github.com/google/EarlGrey/issues/1043#issuecomment-1339936617

authors: bartekpacia bkunat

repo: google/EarlGrey


text:
remove chrome web store script

What's changed, or what was fixed?

removes chrome web store script

Fixes: N/A
Target Live Date: 2018-08-17

[x] This has been reviewed and approved by @kaycebasques
[x] I have run npm test locally and all tests pass.
[x] I have added the appropriate type-something label.
[ ] I've staged the site and manually verified that my content displays correctly.

CC: @petele

Whoops!
There were 3 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
src/content/en/tools/lighthouse/index.md#L137 - Common typo found: 'node' Should it be 'Node'?
src/content/en/tools/lighthouse/index.md#L138 - Common typo found: 'node' Should it be 'Node'?
src/content/en/tools/lighthouse/index.md#L5 - WF Tag 'wf_updated_on' must be within the last 7 days.

:tada: This has been pushed live to https://developers.google.com/web/

authors: WebFundBot kaycebasques

repo: google/WebFundamentals


text:
add link to github in the "yes" response

What's changed, or what was fixed?

provide opportunity to leave more feedback for the users who click "yes"

Fixes: N/A
Target Live Date: 2018-12-20

[x] This has been reviewed and approved by @kaycebasques
[x] I have run npm test locally and all tests pass.
[x] I have added the appropriate type-something label.
[x] I've staged the site and manually verified that my content displays correctly.

CC: @petele

Whoops!
There were 1 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
src/content/en/_shared/helpful.html#L37 - Consider adding a "was this page helpful?" widget to your page: https://developers.google.com/web/resources/widgets#helpful


text:
adb-sync seems to close standard input?

foo (containing file 1.txt) and bar (containing file 2.txt) are two directories under /sdcard on my phone.
When I run the following script test.sh on my computer:
#!/bin/sh
while read -r dir; do
    ./adb-sync -R /sdcard/"$dir" .
done <<EOF
foo
bar
EOF

The result is that only the first directory foo is synced:
$ ./test.sh
INFO:root:Sync: local b'./foo', remote b'/sdcard/foo'
INFO:root:Scanning and diffing...
INFO:root:Pull: b'./foo'
INFO:root:Pull: b'./foo/1.txt'
[100%] /sdcard/foo/1.txt
INFO:root:Total: 1 KB/s (312 bytes in 0.225s)

The problem is read -r dir fails in the second iteration, perhaps implying stdin in closed.

Interesting issue.
This is probably because adb-sync calls "adb shell", which in turn connects stdin to the phone.
I think I can fix this by making adb-sync not connect stdin to its subprocesses anymore.

I face a similar problem : the directories I want to sync are listed in a CSV file 'file.csv' (2 fields on each line : SOURCE,DEST).
When I run the bash script
#!/bin/bash
while read line; do
./adb-sync $SOURCE $DEST
done < file.csv
only the first line of file.csv is read and consequently, only the first directory is synced.
I've found a workaround. Instead of a 'while... do... done' loop, I use a 'for... do... done' loop.
#!/bin/bash
for line in $(cat file.csv); do
./adb-sync $SOURCE $DEST
done
It works fine, but I couldn't explain why !

Pretty sure it is not adb-sync but adb that eats standard input, so I
suggest filing a bug against adb. We could work around it in adb-sync but
it would probably be preferred to fix adb.
Having said that, as a workaround you can also use
adb-sync ... </dev/null
in your scripts.
On Sun, Jul 18, 2021, 05:23 Jarival @.***> wrote:

I face a similar problem : the directories I want to sync are listed in a
CSV file 'file.csv' (2 fields on each line : SOURCE,DEST).
When I run the bash script
#!/bin/bash
while read line; do
./adb-sync $SOURCE $DEST
done < file.csv
only the first line of file.csv is read and consequently, only the first
directory is synced.
I've found a workaround. Instead of a 'while... do... done' loop, I use a
'for... do... done' loop.
#!/bin/bash
for line in $(cat file.csv); do
./adb-sync $SOURCE $DEST
done
It works fine, but I couldn't explain why !
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/adb-sync/issues/38#issuecomment-882027535, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB5NMF6XBEROXSTYTCPEZ3TYKMRLANCNFSM4G44LTUA
.

authors: Jarival divVerent lolilolicon

repo: google/adb-sync


text:
FAQ had dead links in the "What about Python?" section

Both links referencing examples return 404s.

Thanks, correct links are https://github.com/google/bazel/blob/master/tools/build_rules/py_rules.bzl and https://github.com/google/bazel/tree/master/examples/py.  I'll update the FAQ.

authors: brettcannon kchodorow

repo: google/bazel


text:
Added support for custom include directories in samples

The change allows, among other things, samples to share code between C++ and shaders (e.g. struct definitions or bindings).
Also edited .gitignore to ignore tags files from ctags.

Here is a code snippet illustrating how it'd be used in a sample's CMakeLists.txt.
add_samples_for_all_apis(
    NAME ${PROJECT_NAME}
    SOURCES "main.cpp"
    SHADER_DEPENDENCIES "my_shader"
    ADDITIONAL_INCLUDE_DIRECTORIES "${PPX_DIR}/assets/my_project"
)

Then, in main.cpp, one could include files from ${PPX_DIR}/assets/my_project as below.
#include "shaders/shared.hlsli"

repo: google/bigwheels


text:
Which messages can be deduplicated?

There are many opportunities for messages to be deduplicated as they are serialized into the BinExport2 format.
For example, mnemonics can trivially be deduplicated, and different instructions can refer to the same mnemonic indices. For example, pop eax and pop ebx might both refer to same mnemonic Mnemonic { name="pop" }. In fact, the proto documentation explicitly calls this out:
https://github.com/google/binexport/blob/39f6445c232bb5caf5c4a2a996de91dfa20c48e8/binexport2.proto#L209-L211
Likewise, Expressions and Operands can be deduplicated in a similar way.
My primary question is: can instructions be deduplicated?
The protobuf documentation says so:
https://github.com/google/binexport/blob/39f6445c232bb5caf5c4a2a996de91dfa20c48e8/binexport2.proto#L25-L34
So, perhaps that answers my question. However, I want to confirm that this implies that there is not a 1:1 mapping from instruction index to instruction address, since the same deduplicated instruction could be found in multiple basic blocks. I know the address is not stored within each instruction.
For example, as we process protobuf files with capa, our initial implementation builds an index from instruction index to instruction address, so that we can quickly recover function name -> basic block -> instruction -> address (and the reverse). But, if we cannot rely on a 1:1 mapping, we'll reorganize our index a bit.
(note, I suppose the first instruction of a function/vertex will be at the start of a basic block, so the first instruction will necessarily have the address set explicitly, which is useful. But I'd like to confirm the assumptions (and update the documentation) so BinExport2 consumers don't make mistakes).

Assuming that instructions can be deduplicated, and therefore there's not a 1:1 mapping between instruction index and instruction address, then I think there's an ambiguity in the Reference infrastructure:
https://github.com/google/binexport/blob/main/binexport2.proto#L273-L297
A Reference connects an instruction (and optionally operand/expression) to a string or global address, and it does so via an instruction index. However, not all instances of the same instruction may reference the same string/global address. For example, consider the instruction mov ebx, [eax]. In some scenarios, the binary analysis framework may be able to resolve where eax points to, and in other cases not, or may resolve it to a different address. But because the reference is keyed on the instruction index, rather than instruction address, I don't see a way to disambiguate which instance of the instruction references the global address. Do you have any insight or ideas?

I think there might be a bit of confusion here about what is meant by the de-duplication. True, an instruction can be part of multiple basic blocks and basic blocks and be part of
multiple functions. So in order to avoid a storage/memory blow-up, functions only store a
list of basic block indices and basic blocks only store instruction index ranges.
So in the end, there still is a 1:1 mapping between an instruction and its address in the
binary. However, we only actually store the address if it is

the entry-point of a function,
the first instruction of a basic block (usually indicated by not having "flow")
there exists padding space between the instruction and the next one (as determined by the
disassembler)

Now if you have a Reference to an Instruction via instruction_index, you will need to
compute the instruction's address if that is not stored (using GetInstructionAddress()).
OTOH, we are directly de-duping mnemonics and strings and only store those once.

Thanks for the explanation @cblichmann! I understand that there's an implicit guarantee of a 1:1 mapping between instruction indices and addresses. That makes some things much simpler.
Is it ok for BinExport2 exporters to de-duplicate the storage for Expressions and Operands? I don't think there are address-like details that may leak into this abstraction. Do you know of any BinExport2 consumers that would break on with this assumption?
If its all the same to you, it would be nice to be able to do this. In one I-promise-I-didn't-cherry-pick example, the five most common operands accounted for around 75% of all operands in a substantial program:
https://github.com/mandiant/capa/blob/dc8c7e8861b6d4d6eeef9c03f62b7e1728600de6/scripts/inspect-binexport2.py#L166-L170
Deduplicating the storage of Expressions and Operands would lead to a smaller file size (but so would gzipping the file, so this isn't very compelling). More importantly, tools like capa that process all operands looking for particular values can save a huge amount of time by only considering the unique Expressions and Operands.
Exporters that deduplicate Expressions and Operands could enable better performance for some consumers, while exporters that don't duplicate are just fine and remain compatible, too.
--
I've been casually writing a BinExport2 exporter for my personal Rust-based analysis framework, and its a great learning experience. I appreciate a lot of the design decisions you've made with the format. Thanks for your patience as you answer my questions :-)

[...]
Is it ok for BinExport2 exporters to de-duplicate the storage for Expressions and Operands? I don't think there are address-like details that may leak into this abstraction. Do you know of any BinExport2 consumers that would break on with this assumption?

I think you're right (not 100% sure, though, need to think about this more). But I don't think anyone will break because of this. BinDiff mostly does not care as long as it can render the expressions.

If its all the same to you, it would be nice to be able to do this. In one I-promise-I-didn't-cherry-pick example, the five most common operands accounted for around 75% of all operands in a substantial program:
https://github.com/mandiant/capa/blob/dc8c7e8861b6d4d6eeef9c03f62b7e1728600de6/scripts/inspect-binexport2.py#L166-L170
Deduplicating the storage of Expressions and Operands would lead to a smaller file size (but so would gzipping the file, so this isn't very compelling).

This is not too surprising, actually. This is also the reason we to the instruction histogram before assigning the mnemnonic ids. After all the distribution of instruction mnemonics is similar (hello mov :)).

More importantly, tools like capa that process all operands looking for particular values can save a huge amount of time by only considering the unique Expressions and Operands.
Exporters that deduplicate Expressions and Operands could enable better performance for some consumers, while exporters that don't deduplicate are just fine and remain compatible, too.

If we implement this, we should do it in the C++ binexport2_writer.cc as well in the Ghidra exporter. But maybe we hide it behind a feature flag for now? Care to create a feature request to track this?

[...]
I've been casually writing a BinExport2 exporter for my personal Rust-based binary analysis framework, and its a great learning experience.

Tell me more :)

I appreciate a lot of the design decisions you've made with the format. Thanks for your patience as you answer my questions :-)

That praise should go to Sören. I'll make sure to forward...

Thanks for the detailed explanations @cblichmann . I'm not seeing the following in practice:

So in the end, there still is a 1:1 mapping between an instruction and its address in the binary. However, we only actually store the address if it is
[...]
the first instruction of a basic block (usually indicated by not having "flow")
[...]

e.g. Ghidra's BinExportExporter logic (source):

// Write the full instruction address iff:
// - there is no previous instruction
// - the previous instruction doesn't have code flow into the current one
// - the previous instruction overlaps the current one
// - the current instruction is a function entry point

Specifically, "the previous instruction doesn't have code flow into the current one" would not record e.g. the address of the first instruction of a basic block that follows a conditional jump?
I'm seeing similar behavior in IDA's BinExport files but I couldn't easily verify. Can you please provide further guidance on this?

Specifically, "the previous instruction doesn't have code flow into the current one" would not record e.g. the address of the first instruction of a basic block that follows a conditional jump?

The intend of phrasing it like this was to capture the situation of "new basic block started" as well as "instruction is a jump target" (might even be a jump into the middle of a function). In those cases one would expect BinExport to contain the instruction address.
Calling this "flow" is due to IDA Pro's FF_FLOW flag, which might be confusing.

Specifically, "the previous instruction doesn't have code flow into the current one" would not record e.g. the address of the first instruction of a basic block that follows a conditional jump?

The intend of phrasing it like this was to capture the situation of "new basic block started" as well as "instruction is a jump target" (might even be a jump into the middle of a function). In those cases one would expect BinExport to contain the instruction address. Calling this "flow" is due to IDA Pro's FF_FLOW flag, which might be confusing.

My main question here is whether or not BinExport is expected to record the address of the first instruction of all basic blocks? Depending on this answer, what I am seeing may be considered either a bug or a feature.

Based on what I understand @cblichmann said, I think you've found a bug and we should try to develop a test case. Do you see this behavior in all samples? I wonder why this hasn't been noticed in BinDiff before.
I wonder if consumers of BinExport aren't asserting the presence of the address field and are ending up with the default value.

Based on what I understand @cblichmann said, I think you've found a bug and we should try to develop a test case. Do you see this behavior in all samples? I wonder why this hasn't been noticed in BinDiff before.
I wonder if consumers of BinExport aren't asserting the presence of the address field and are ending up with the default value.

I'm 4 for 4 samples with BinExport files that exhibit this behavior, including BinExport files generated by IDA and Ghidra, so I'd expect this to be an issue for most. I've opened https://github.com/google/binexport/issues/130 and https://github.com/google/binexport/issues/129. Happy to continue discussion there - I've already opened an internal draft for #130 .

authors: cblichmann mike-hunhoff williballenthin

repo: google/binexport


text:
missing properties check not working for boolean field

When a class has a boolean field, and I misspell the name of the field, a test of the form
if (this.myBolean)

does not report an error even when using compiler option --jscomp_error=missingProperties.  Whereas a test of the form
if (this.myBolean == true)

will correctly report the following error:
compile_js.sh src/sims/engine2D/ChainApp.js build/sims/engine2D/ChainApp_en.js true false
src/lab/util/MyClass.js:14: WARNING - Property myBolean never defined on myphysicslab.lab.util.MyClass
if (this.myBolean == true)
         ^

Here is the example class:
goog.provide('myphysicslab.lab.util.MyClass');
/**
@constructor
*/
myphysicslab.lab.util.MyClass = function() {
/** @type {boolean} */
this.myBoolean = true;
};

/**
@return {string}
*/
myphysicslab.lab.util.MyClass.prototype.check = function() {
if (this.myBolean)
  return "yes";
else
  return "no";
};

Assuming this is the old type inference (which is the tied to the missingProperties diagnostic group).  This was an attempt to allow existence checks:
if (a && a.b && a.b.c) ...
I don't personally think it was a good idea

This is happening with new type inference also.

Just ran into this again, it happens with numbers or strings as well, not just boolean:  Assigning to non-existent fields in a class generates no errors with NTI or old compiler.  For example, adding code like this to any class method gives no errors:
this.fooBar_ = 42;
this.baz_ = "frumble";

Here is the set of options being used.  Includes missingProperties.
+ java -jar ../javascript/closure-compiler/build/compiler.jar --new_type_inf
--compilation_level=SIMPLE --closure_entry_point=myphysicslab.sims.engine2D.ChainApp
--js=src --js=../javascript/closure-library --only_closure_dependencies=true
--generate_exports --jscomp_error=checkTypes --jscomp_error=missingProperties
--jscomp_error=missingProvide --jscomp_error=missingRequire
--jscomp_warning=accessControls --jscomp_warning=ambiguousFunctionDecl
--jscomp_warning=checkVars --jscomp_warning=const --jscomp_warning=constantProperty
--jscomp_warning=fileoverviewTags --jscomp_warning=globalThis
--jscomp_warning=invalidCasts --jscomp_warning=misplacedTypeAnnotation
--jscomp_warning=missingReturn --jscomp_warning=newCheckTypes
--jscomp_warning=strictModuleDepCheck --jscomp_warning=suspiciousCode
--jscomp_warning=typeInvalidation --jscomp_warning=undefinedNames
--jscomp_warning=undefinedVars --jscomp_warning=unknownDefines
--jscomp_warning=uselessCode --jscomp_warning=visibility --extra_annotation_name=alias
--warning_level=VERBOSE --define=myphysicslab.lab.util.UtilityCore.ADVANCED=false
--define=goog.DEBUG=true '--define=goog.LOCALE='\''en'\''' --externs=my_externs.js
--language_in=ECMASCRIPT5

Assigning has different behavior than property access, because it creates a property. If you want to prevent accidental property creation by assignment, you should annotate the constructor with @struct

Thanks, I didn't know about using @struct.  Using @struct seems to also fix the original problem reported above.  I guess I'll add @struct to all my classes now.  I can't think of any classes where I add properties after creation.

authors: concavelenz dimvar myphysicslab

repo: google/closure-compiler


text:
Make storage properties non-null

I don't think these can be null.
https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage
https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage

Looks good, I'll take a look at importing this to see if there're any violations in google.

Thanks!

authors: blickly bugreportuser


text:
CSS Syntax-Highlighting won't recognize all keywords

The CSS rules for this code won't be highlighted equally:

    .first_rule {
       font-size: 12px;
    }
    .second_rule:hover {
       font-size: 12px;
    }

`.first_rule` will have a plain black color, but `.second_rule` will be
highlighted in blue because it is interpreted a keyword. It seems as if
`.first_rule` would also become a keyword, as soon as I append a pseudo class
(or anything else beginning with a  colon) to it. This lets the user assume
that there is a difference between the two kinds of identifiers, but indeed
there isn't.

Please have a look here to see this in action:
http://meta.stackoverflow.com/questions/132758/syntax-highlighting-wont-recogniz
e-all-css-keywords

Original issue reported on code.google.com by twiligh7sparkle on 16 May 2012 at 11:29

Live demo to test the issue: https://jsfiddle.net/amroamroamro/dz98x97r/

authors: GoogleCodeExporter amroamroamro

repo: google/code-prettify


text:
Add SPR implementation to atari_100k lab

I'd like to add support for SPR to the Atari 100k lab project.  My implementation is mostly siloed to minimize changes to the existing algorithms.  SPR needs a custom model-based style replay buffer that returns subtrajectories, and a version of noisy nets that allows noise to be toggled on and off within a function, both of which are included.
This implementation performs a bit better than the original version; I found median 0.45 over 100 seeds, compared to 0.395 for the original.
@psc-g @agarwl

@googlebot I signed it!

repo: google/dopamine