text:
referenceFilters and robotsTxt: robots.txt being checked for "external" sites

Dear norconex team,
could you please clarify, why the http collector does check the robots.txt of the "remote" (or "external") sites, although it is configured to "stay-on-site", e.g.
<startURLs stayOnDomain="true" stayOnPort="true" stayOnProtocol="true">
        <url>https://www.intranet.site.com/index.htm</url>
</startURLs>

<referenceFilters>
        <filter
            class="com.norconex.collector.core.filter.impl.ExtensionReferenceFilter"
            onMatch="exclude" >jpg,gif,png,ico,css,js</filter>
</referenceFilters>

<robotsTxt ignore="false" class="com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider"/>

When the collector finds an external link (link to another site/domain), it tries to download the robots.txt from that site. Basically, it not that bad for the intranet sites, but if a link is pointed to an Internet site, the crawler gets run into a connection timeout.
I tried to add an "include" regex (with the domain name) into <referenceFilters>, but it did not help.
The question, is it possible to disable the checking of "robots.txt" not belonging to the sites configured inside <startURLs>
Thanks!

I could not reproduce this. It never tries to download the robots.txt on another site. Did you encounter this with 2.3.0-SNAPSHOT? Can you please attached your config?
If you change the loglevel in classes/log4j.properties to DEBUG, you should see lines like these to indicate external URLs are rejected.  Do you get them?
DEBUG [LinkExtractorStage] URL not in crawl scope: http://externalsite.com/blah
DEBUG [URLCrawlScopeStrategy] Rejected domain for: http://externalsite.com/blah

Just tested again with the latest snapshot (norconex-collector-http-2.3.0-20151017.032546-23.zip).
I cannot reproduce the issue now - no external "robots.txt" are getting fetched.
P.S. the more I work with the http-collector, the more I like it :) You guys have been doing a really good job!

Good feedback like this is why we keep doing it.  Thank you!

Sorry to drop in, but even with the latest snapshot, I experience the same issue.
I realized this behavior when I saw sporadic exceptions in the log (e.g. SSLHandshakeException) during a crawl with the message "Not able to obtain robots.txt at: ..." for external references that should not be crawled but only listed by the crawler event "URLS_EXTRACTED".
The setting of the stayOnDomain attribute doesn't change the behaviour and I also use a RegexReferenceFilter that is applied in code.
Unfortunately, I can't see the LinkExtractorStage or URLCrawlScopeStrategy in the log for the fetched robot.txt (after changing the collectors to DEBUG), but I see it in the stack trace, e.g.:
WARN  [StandardRobotsTxtProvider] Not able to obtain robots.txt at: https://www.openstreetmap.ch/robots.txt
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1446)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:209)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:837)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider.getRobotsTxt(StandardRobotsTxtProvider.java:80)
	at com.norconex.collector.http.pipeline.queue.HttpQueuePipelineContext.<init>(HttpQueuePipelineContext.java:41)
	at com.norconex.collector.http.pipeline.importer.LinkExtractorStage.queueURL(LinkExtractorStage.java:131)
	at com.norconex.collector.http.pipeline.importer.LinkExtractorStage.executeStage(LinkExtractorStage.java:89)
	at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:31)
	at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:24)
	at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
	at com.norconex.collector.http.crawler.HttpCrawler.executeImporterPipeline(HttpCrawler.java:285)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:475)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:376)
	at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:703)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

My config is located at #163. What am I missing?
Thanks for your help!

Re-opening. I will try reproduce with your config.

@comschmid, after approximately how many docs processed do you get this exception?

First, let me explain that the exception is correct, since the SSL certificate is expired. Thanks to this kind of exceptions I realized that robots.txt of external references are fetched, which is not intended, according the current configuration. This exception is just an example and I just realized, that this exception is not thrown for this exact configuration, but an SSLHandshakeException for another external domain is thrown.
Now, let me answer your question:
I don't see that there is a certain limit of docs that need to be processed.
As of now, a complete crawl with this configuration produces 58 documents and I see a SSLHandshakeException after about 20% of the log output (with log level DEBUG for the collector).
I also checked with Wireshark (with filter 'http.connection') and I already see from the beginning a lot of connections to external domains, requesting the robots.txt.

@comschmid, I just crawl "http://www.etools.ch" with your config.  It worked just fine. I got no exception and the site mentioned in the exception your reported was filtered out and no attempt to get a robot for it was made.  The rejection:
Issue Crawler: 2015-10-22 00:23:13 DEBUG - Rejected domain for: https://www.openstreetmap.ch/
Issue Crawler: 2015-10-22 00:23:13 DEBUG - URL not in crawl scope: https://www.openstreetmap.ch/

Do you get that rejection too?
What may be happening is using an old crawl database.  Did  you delete your work directory before trying again (especially the crawldb)? When new features are introduced or when you make particular config changes, you sometimes need to do a fresh re-crawl. The reason being, those external sites may have been previously crawled so the crawler has kept a memory of them and tries them again in case they have changed.
Try again with a fresh crawl and let me know if you still have the problem.

Thanks for your feedback!
Unfortunately I don't get this rejection messages, even by setting all log levels to DEBUG.
And yes, I did a fresh crawl.
FYI: I set the RegexReferenceFilter with the value https?://www\.etools\.ch/.* in code.
There must be something I looked over...

This is not related to why it tries to go outside, but if you want to get rid of the SSL Exception, this flag will probably do it for you:
<httpClientFactory>
  <trustAllSSLCertificates>true</trustAllSSLCertificates>
</httpClientFactory>

Your RegexReferenceFilter suggests I did not use the exact same config as yours.  If you do it in the code, I understand it may be trouble to convert it to a config.  This has not been tried a lot, but you can try to save the config created via code to a file using HttpCrawlerConfig#saveToXML(Writer) and attach it here.

Thanks for the pointer concerning trusting all SSL certificates.
Unfortunately, the creation of the config XML stops somewhere in the middle with an NPE:
Exception in thread "main" java.lang.NullPointerException
	at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteAttr(BaseNsStreamWriter.java:493)
	at com.ctc.wstx.sw.BaseNsStreamWriter.writeAttribute(BaseNsStreamWriter.java:228)
	at com.norconex.commons.lang.xml.EnhancedXMLStreamWriter.writeAttribute(EnhancedXMLStreamWriter.java:208)
	at com.norconex.importer.handler.filter.impl.RegexMetadataFilter.saveFilterToXML(RegexMetadataFilter.java:138)
	at com.norconex.importer.handler.filter.AbstractDocumentFilter.saveHandlerToXML(AbstractDocumentFilter.java:134)
	at com.norconex.importer.handler.AbstractImporterHandler.saveToXML(AbstractImporterHandler.java:224)
	at com.norconex.importer.ImporterConfig.writeObject(ImporterConfig.java:295)
	at com.norconex.importer.ImporterConfig.writeObject(ImporterConfig.java:282)
	at com.norconex.importer.ImporterConfig.writeHandlers(ImporterConfig.java:275)
	at com.norconex.importer.ImporterConfig.saveToXML(ImporterConfig.java:246)
	at com.norconex.collector.core.crawler.AbstractCrawlerConfig.writeObject(AbstractCrawlerConfig.java:443)
	at com.norconex.collector.core.crawler.AbstractCrawlerConfig.writeObject(AbstractCrawlerConfig.java:423)
	at com.norconex.collector.core.crawler.AbstractCrawlerConfig.saveToXML(AbstractCrawlerConfig.java:266)
...

Yeah, saving the config that way is very rarely done.  It does not mean it should not work though :-)  I will look at fixing the issue you just reported.

The latest snapshot should save properly now. I recommend you try again with it and attach the generated config. If you are overwriting an existing install with the snapshot, make sure you do not get Jar conflicts (different versions of the same Jars under lib).

Thanks again for you attention and support!
I updated the norconex jars to the latest snapshot as advised and now I could generate the corresponding XML:-) Please note, that the external robots.txt still gets fetched.
Following the resulting XML, created from HttpCrawlerConfig :
<crawler class="com.norconex.collector.http.crawler.HttpCrawlerConfig" id="www.etools.ch">
	<numThreads>2</numThreads>
	<workDir>D:\data\workspaces\mole\lizard\collector\work</workDir>
	<maxDocuments>-1</maxDocuments>
	<orphansStrategy>IGNORE</orphansStrategy>
	<crawlDataStoreFactory class="com.norconex.collector.core.data.store.impl.mapdb.MapDBCrawlDataStoreFactory" />
	<referenceFilters>
		<filter class="com.norconex.collector.core.filter.impl.RegexReferenceFilter" onMatch="include" caseSensitive="false">https?://www\.etools\.ch/.*</filter>
		<filter class="com.norconex.collector.core.filter.impl.ExtensionReferenceFilter" onMatch="exclude" caseSensitive="false">jpg,jpeg,gif,png,ico,css,js</filter>
		<filter class="com.norconex.collector.core.filter.impl.RegexReferenceFilter" onMatch="exclude" caseSensitive="true">.+\?.*</filter>
	</referenceFilters>
	<metadataFilters>
		<filter class="com.norconex.collector.core.filter.impl.RegexMetadataFilter" onMatch="exclude" caseSensitive="false" field="Content-Type">.*css.*|.*javascript.*</filter>
	</metadataFilters>
	<crawlerListeners>
		<listener class="com.comcepta.lizard.vadian.VadianEventListener" />
	</crawlerListeners>
	<importer>
		<tempDir>D:\data\workspaces\mole\lizard\collector\work\temp</tempDir>
		<parseErrorsSaveDir>D:\data\workspaces\mole\lizard\collector\work\parseerrors</parseErrorsSaveDir>
		<maxFileCacheSize>10485760</maxFileCacheSize>
		<maxFilePoolCacheSize>104857600</maxFilePoolCacheSize>
		<preParseHandlers>
			<tagger class="com.norconex.importer.handler.tagger.impl.DocumentLengthTagger" field="document.size.preparse" overwrite="false" />
		</preParseHandlers>
		<tagger class="com.norconex.importer.parser.GenericDocumentParserFactory" splitEmbedded="false" />
		<postParseHandlers>
			<tagger class="com.norconex.importer.handler.tagger.impl.ConstantTagger" />
			<filter class="com.norconex.importer.handler.filter.impl.RegexMetadataFilter" onMatch="exclude" caseSensitive="false">.*</filter>
			<transformer class="com.norconex.importer.handler.transformer.impl.ReduceConsecutivesTransformer" maxReadSize="65536" caseSensitive="true">
				<reduce>\s</reduce>
				<reduce>\n</reduce>
				<reduce>\r</reduce>
				<reduce>\t</reduce>
				<reduce>\n\r</reduce>
				<reduce>\r\n</reduce>
				<reduce>\s\n</reduce>
				<reduce>\s\r</reduce>
				<reduce>\s\r\n</reduce>
				<reduce>\s\n\r</reduce>
			</transformer>
			<transformer class="com.norconex.importer.handler.transformer.impl.ReplaceTransformer" maxReadSize="65536" caseSensitive="false">
				<replace>
					<fromValue>class=".*?"</fromValue>
					<toValue></toValue>
				</replace>
			</transformer>
			<transformer class="com.norconex.importer.handler.transformer.impl.StripBetweenTransformer" maxReadSize="65536" caseSensitive="false" inclusive="true">
				<stripBetween>
					<start><style.*?></start>
					<end></style></end>
				</stripBetween>
				<stripBetween>
					<start><script.*?></start>
					<end></script></end>
				</stripBetween>
			</transformer>
			<tagger class="com.norconex.importer.handler.tagger.impl.DocumentLengthTagger" field="document.size.postparse" overwrite="false" />
			<filter class="com.norconex.importer.handler.filter.impl.NumericMetadataFilter" onMatch="exclude" field="document.size.postparse">
				<condition operator="lt" number="100.0" />
			</filter>
			<tagger class="com.norconex.importer.handler.tagger.impl.LanguageTagger" shortText="false" keepProbabilities="false" fallbackLanguage="" />
			<tagger class="com.norconex.importer.handler.tagger.impl.CharacterCaseTagger">
				<characterCase fieldName="keywords" type="lower" applyTo="field" />
				<characterCase fieldName="description" type="lower" applyTo="field" />
			</tagger>
			<tagger class="com.norconex.importer.handler.tagger.impl.KeepOnlyTagger">
				<fields>content,title,keywords,description,tags,collector.referrer-reference,collector.depth,document.reference,document.language,document.size.preparse,document.size.postparse</fields>
			</tagger>
		</postParseHandlers>
	</importer>
	<committer class="com.norconex.committer.solr.SolrCommitter">
		<sourceReferenceField keep="false">document.reference</sourceReferenceField>
		<targetReferenceField>id</targetReferenceField>
		<targetContentField>content</targetContentField>
		<queueDir>D:\data\workspaces\mole\lizard\collector\work\commitqueue</queueDir>
		<queueSize>10</queueSize>
		<commitBatchSize>5</commitBatchSize>
		<maxRetries>1</maxRetries>
		<maxRetryWait>5000</maxRetryWait>
		<solrURL>http://localhost:8983/solr/test1</solrURL>
		<solrUpdateURLParams>
			<commitDisabled>false</commitDisabled>
		</solrUpdateURLParams>
	</committer>
	<documentChecksummer class="com.norconex.collector.core.checksum.impl.MD5DocumentChecksummer" keep="false" targetField="collector.checksum-doc" disabled="false">
		<sourceFields>content,title,keywords,description</sourceFields>
	</documentChecksummer>
	<spoiledStateStrategyResolver class="com.norconex.collector.core.spoil.impl.GenericSpoiledReferenceStrategizer" fallbackStrategy="DELETE">
		<mapping state="BAD_STATUS" strategy="GRACE_ONCE" />
		<mapping state="NOT_FOUND" strategy="DELETE" />
		<mapping state="ERROR" strategy="GRACE_ONCE" />
	</spoiledStateStrategyResolver>
	<userAgent>Mozilla/5.0 (compatible; TestCrawler/0.1; +http://www.comcepta.com)</userAgent>
	<maxDepth>5</maxDepth>
	<keepDownloads>false</keepDownloads>
	<startURLs stayOnProtocol="false" stayOnDomain="false" stayOnPort="false">
		<url>http://www.etools.ch/</url>
	</startURLs>
	<urlNormalizer class="com.norconex.collector.http.url.impl.GenericURLNormalizer" disabled="false">
		<normalizations>lowerCaseSchemeHost,upperCaseEscapeSequence,decodeUnreservedCharacters,removeDefaultPort,removeFragment,removeDotSegments,addTrailingSlash,removeDuplicateSlashes,removeSessionIds,upperCaseEscapeSequence</normalizations>
		<replacements>
			<replace>
				<match>&view=print</match>
				<replacement>&view=html</replacement>
			</replace>
		</replacements>
	</urlNormalizer>
	<delay class="com.norconex.collector.http.delay.impl.GenericDelayResolver" default="1000" scope="site" ignoreRobotsCrawlDelay="false" />
	<httpClientFactory class="com.norconex.collector.http.client.impl.GenericHttpClientFactory">
		<cookiesDisabled>false</cookiesDisabled>
		<authPort>-1</authPort>
		<authFormCharset>UTF-8</authFormCharset>
		<proxyPort>0</proxyPort>
	</httpClientFactory>
	<connectionTimeout>30000</connectionTimeout>
	<socketTimeout>30000</socketTimeout>
	<connectionRequestTimeout>30000</connectionRequestTimeout>
	<expectContinueEnabled>false</expectContinueEnabled>
	<maxRedirects>50</maxRedirects>
	<maxConnections>200</maxConnections>
	<trustAllSSLCertificates>false</trustAllSSLCertificates>
	<maxConnectionsPerRoute>20</maxConnectionsPerRoute>
	<maxConnectionIdleTime>10000</maxConnectionIdleTime>
	<maxConnectionInactiveTime>0</maxConnectionInactiveTime>
	<robotsTxt class="com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider" />
	<sitemapResolverFactory class="com.norconex.collector.http.sitemap.impl.StandardSitemapResolverFactory" lenient="false" />
	<canonicalLinkDetector class="com.norconex.collector.http.url.impl.GenericCanonicalLinkDetector">
		<contentTypes>text/html,application/xhtml+xml,vnd.wap.xhtml+xml,x-asp</contentTypes>
	</canonicalLinkDetector>
	<metadataFetcher class="com.norconex.collector.http.fetch.impl.GenericMetadataFetcher">
		<validStatusCodes>200</validStatusCodes>
	</metadataFetcher>
	<metadataChecksummer class="com.norconex.collector.http.checksum.impl.LastModifiedMetadataChecksummer" keep="false" targetField="collector.checksum-metadata" disabled="false" />
	<httpDocumentFetcher class="com.norconex.collector.http.fetch.impl.GenericDocumentFetcher">
		<validStatusCodes>200</validStatusCodes>
		<notFoundStatusCodes>404</notFoundStatusCodes>
		<headersPrefix />
	</httpDocumentFetcher>
	<robotsMeta class="com.norconex.collector.http.robot.impl.StandardRobotsMetaProvider" />
	<linkExtractors>
		<extractor class="com.norconex.collector.http.url.impl.GenericLinkExtractor" maxURLLength="2048" ignoreNofollow="false" keepReferrerData="true">
			<contentTypes>text/html,application/xhtml+xml,vnd.wap.xhtml+xml,x-asp</contentTypes>
			<tags>
				<tag name="a" attribute="href" />
				<tag name="input" attribute="src" />
				<tag name="img" attribute="src" />
				<tag name="form" attribute="action" />
				<tag name="meta" attribute="http-equiv" />
				<tag name="link" attribute="href" />
				<tag name="iframe" attribute="src" />
				<tag name="frame" attribute="src" />
				<tag name="base" attribute="href" />
			</tags>
		</extractor>
	</linkExtractors>
	<preImportProcessors>
		<processor class="com.comcepta.lizard.vadian.VadianEventListener" />
	</preImportProcessors>
</crawler>

Thanks to your config, I was able to reproduce.  Stay tuned for a fix.

Fixed. Let me know how this new snapshot release goes.

Thanks a lot, this snapshot fixed this issue, I also checked it with Wireshark and looks good!
Just a quick question: where can I add additional HTTP request headers, such as 'Accept-Language'?
BTW: when I was looking at the generated XML above, I realized that some of the child-elements of <httpClientFactory>appear after this element, like <connectionTimeout>.

Try to create new issues when you have questions/new issues not related to the original issue you are in. That's much easier to keep track.
Right now there is no configurable feature to add request headers.  The easiest would probably be to extend GenericHttpClientFactory to add this, by overwriting its buildCustomHttpClient method.  That's a pretty easy addition so I will add a new feature request for it.
For the order problem when you save, that should be considered a bug so it should be in its own ticket.  I will create it this time.
Since the original issue is fixed, I am closing.

authors: comschmid essiembre jetnet

repo: Norconex/collector-http


text:
Old NSM interfaces might remain in proxy after OOM kill

Describe the bug
When proxy process gets killed by OOM, there's no chance for the proxy to close them obviously. However, NSM might not be able to clean them up either when the MaxTokenLifetime expires due to rpc error: code = PermissionDenied desc = no sufficient privileges.
The old interfaces do not seem to cause any problem, but they are confusing. (Their IPs are also leaked.)
The connected peer interface in LB also remains. (and can be used to reach the proxy).
To Reproduce
Steps to reproduce the behavior:

Kill a proxy process using KILL signal. (For example from its worker.)
Wait the container to recover and check the NSM interfaces towards LBs.
Wait MaxTokenLifetime (10 minutes by default) and check the interfaces agains. If they are present check the logs of the local nsmgr or forwarder for the Close.

Expected behavior
The interfaces should be event removed (probably by NSM).
(Resolving the leaked IP issue sound much more complex.)
Context

Network Service Mesh: v1.6.2

Meridio: 115ca87fdf6fc6ae4084469976bdfc8414e6fc49
Author: Lionel Jouin lionel.jouin@est.tech
Date:   Fri Mar 24 19:25:56 2023 +0100
Use Helm chart to deploy e2e operator env

Helm charts are now used in the kind-operator test env
Enable Meridio and TAPA versions in the test scope
Set hostPID to true in vpp-forwarder

Logs
Apr 20 16:29:35.434ESC[37m [TRAC] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(2.1)    close={"id":"4967924d-fc4b-42df-b071-20824f7fa158","network_service":"load-balancer.trench-a.default","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532326","name":"load-balan-fb82"}},"context":{"ip_context":{"src_ip_addrs":["172.16.3.3/24","fd00:0:0:3::3/64"],"dst_ip_addrs":["172.16.3.2/24","fd00:0:0:3::2/64"],"excluded_prefixes":["11.0.0.0/22","11.0.4.0/23","11.0.6.0/24","12.0.0.1/32","12.0.27.146/32","12.0.116.178/32","12.0.155.217/32","12.0.184.219/32","12.0.201.243/32","12.0.250.201/32"]},"MTU":1446},"labels":{"nodeName":"vm-002"},"path":{"index":2,"path_segments":[{"name":"proxy-trench-a-cfqvk","id":"fb82d6e2-ecaf-496e-8072-0c90da0ddfd1-proxy-trench-a-cfqvk-load-balancer.trench-a.default-load-balancer-trench-a-7c885b6d9b-xgvx8","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL2RlZmF1bHQiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL25zbWdyLXNhIl0sImV4cCI6MTY4MjAwODE3NH0.0QtsWuA3vEKWj0dEwIkkCgPu5Rf9uBD5hzT50GAqnUcWrQ0P_XfEZyCbX-Z7l-Bzzg_S0tiyAp0pcDVrVg-CFw","expires":{"seconds":1682008174,"nanos":455339069}},{"name":"nsmgr-tr5n7","id":"44a7009f-5a59-4329-a253-38ae6ccd93d5","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL2RlZmF1bHQiXSwiZXhwIjoxNjgyMDA4MTc1fQ.exoviVXnqYh4aYdzm1Qx9JpxVS5mYlMYzDQaPs7RiMjm7ilyy9rJJAQLhY258EHpd1gUa6vlqy5LfD9-bJqc2Q","expires":{"seconds":1682008175,"nanos":429149769}},{"name":"forwarder-vpp-thdt2","id":"4967924d-fc4b-42df-b071-20824f7fa158","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiXSwiZXhwIjoxNjgyMDA4MTc2fQ.kcaJ8HkJmLu5A9Q02R7ACiolZ6JUS00Vcuzxt1MgC5_fq4Vc-ob57E-yQTqDky9suZjjlMrevtJWw9fLX4pwfg","expires":{"seconds":1682008176,"nanos":273042269},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"nsmgr-27j6j","id":"ddfd106c-e964-4f9f-bb54-9b6e1912a0f8","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL2RlZmF1bHQiXSwiZXhwIjoxNjgyMDA4MTc3fQ.8BBI4hhSXXw-yoXJwSsNP3O08x9Q4TOLAuS_oTOnu37ZtjnFo3DbC6nmAJCUpSrfEOQesq8Nk2MhFeLqrKZrWA","expires":{"seconds":1682008177,"nanos":830338082}},{"name":"forwarder-vpp-cp8sp","id":"eebdbac2-5393-4cd1-a148-6aa2fb41544d","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL21lcmlkaW8tYXV0aC10cmVuY2gtYSJdLCJleHAiOjE2ODIwMDgxNzh9.0FEurb6Wm_U1OXlwGNQ7CEHWHwx1uB37m3I_oXYotGckctlmfYihBcNDlPvvX3LKWGKbjzihbf8CQfYvHhfOqw","expires":{"seconds":1682008178,"nanos":249942792},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"load-balancer-trench-a-7c885b6d9b-xgvx8","id":"30958abe-2bde-4f45-83a0-e67fa66d801b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL21lcmlkaW8tYXV0aC10cmVuY2gtYSIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCJdLCJleHAiOjE2ODIwMDgxNzh9.5q5UYPUNFCV5aw-4yPSmG0sjGNW9JhsdXTlq_xO2wjVsV5Bcx1EMSNCevk5PBtLNthIg3O152flg6lLUjOQFZA","expires":{"seconds":1682008178,"nanos":250666242}}]},"network_service_endpoint_name":"load-balancer-trench-a-7c885b6d9b-xgvx8","payload":"ETHERNET"}
Apr 20 16:29:35.434ESC[37m [TRAC] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(3)   ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Close()
Apr 20 16:29:35.434ESC[37m [TRAC] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(3.1)     close-diff={"path":{"path_segments":{"2":{"expires":{"nanos":434310849,"seconds":1682008775},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiXSwiZXhwIjoxNjgyMDA4Nzc1fQ.nbjavxp1tsSZxvPiiKISnXo8WFsVmRWhX9AL7-ZuSLEpPjtfh8Xpddia3SKjjL7hDWS6hAlMOpT1NTXqM8s0Zg"}}}}
Apr 20 16:29:35.434ESC[37m [TRAC] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(4)    ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Close()
Apr 20 16:29:35.435ESC[31m [ERRO] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(4.1)      rpc error: code = PermissionDenied desc = no sufficient privileges;  Error returned from sdk/pkg/networkservice/common/authorize/authorizeServer.Close;      github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.logError;               /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/trace/common.go:206;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Close;              /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/trace/server.go:71;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Close;                /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/trace/server.go:86;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/common/updatetoken/server.go:84;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Close;              /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/trace/server.go:69;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Close;                /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/trace/server.go:86;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Close;             /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/core/next/server.go:65;    github.com/networkservicemesh/sdk/pkg/networkservice/common/begin.(*eventFactoryServer).Close.func1;            /root/go/pkg/mod/github.com/networkservicemesh/sdk@v1.6.2/pkg/networkservice/common/begin/event_factory.go:225; github.com/edwarnicke/serialize.(*Executor).process;            /root/go/pkg/mod/github.com/edwarnicke/serialize@v1.0.7/serialize.go:68;        runtime.goexit;         /go/src/runtime/asm_amd64.s:1571;
Apr 20 16:29:35.435ESC[37m [TRAC] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(3.2)     close-response-diff={"id":"4967924d-fc4b-42df-b071-20824f7fa158","path":{"index":2,"path_segments":{"2":{"expires":{"nanos":434310849,"seconds":1682008775},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiXSwiZXhwIjoxNjgyMDA4Nzc1fQ.nbjavxp1tsSZxvPiiKISnXo8WFsVmRWhX9AL7-ZuSLEpPjtfh8Xpddia3SKjjL7hDWS6hAlMOpT1NTXqM8s0Zg"}}}}
Apr 20 16:29:35.435ESC[31m [ERRO] [id:4967924d-fc4b-42df-b071-20824f7fa158] [type:networkService] ESC[0m(3.3)     Error returned from sdk/pkg/networkservice/common/authorize/authorizeServer.Close: rpc error: code = PermissionDenied desc = no sufficient privileges
Apr 20 16:29:35.854ESC[37m [TRAC] [id:5ae244a3-d784-4ec5-b3b4-0d5951b1b7db] [type:networkService] ESC[0m(2.1)    close={"id":"5ae244a3-d784-4ec5-b3b4-0d5951b1b7db","network_service":"load-balancer.trench-a.default","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532326","name":"load-balan-0515"}},"context":{"ip_context":{"src_ip_addrs":["172.16.3.5/24","fd00:0:0:3::5/64"],"dst_ip_addrs":["172.16.3.4/24","fd00:0:0:3::4/64"],"excluded_prefixes":["11.0.0.0/22","11.0.4.0/23","11.0.6.0/24","12.0.0.1/32","12.0.27.146/32","12.0.116.178/32","12.0.155.217/32","12.0.184.219/32","12.0.201.243/32","12.0.250.201/32"]},"MTU":1500},"labels":{"nodeName":"vm-002"},"path":{"index":2,"path_segments":[{"name":"proxy-trench-a-cfqvk","id":"051579bb-4ced-459c-bd02-e17721370210-proxy-trench-a-cfqvk-load-balancer.trench-a.default-load-balancer-trench-a-7c885b6d9b-k7fnk","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL2RlZmF1bHQiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL25zbWdyLXNhIl0sImV4cCI6MTY4MjAwODE3NH0.uNZ4KPC2NZbqHDIxpXFbLhxxXSWhkZ1VbD3s3k4yVr86mMP0w24Vw7DsWmVyIv3ywDMMqUIHyOvA2tHnIdY7UQ","expires":{"seconds":1682008174,"nanos":472660529}},{"name":"nsmgr-tr5n7","id":"3902abe8-3678-43b9-bcbf-569c716a8685","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL2RlZmF1bHQiXSwiZXhwIjoxNjgyMDA4MTc1fQ.1FuetkXxojB8W_CsYdydm3Bswz_e27-ttZoTvfCL_lRQiHDvCQtvOcBhitrIRq64snn8dfMSWw_p6LoNwGCkAw","expires":{"seconds":1682008175,"nanos":852200649}},{"name":"forwarder-vpp-thdt2","id":"5ae244a3-d784-4ec5-b3b4-0d5951b1b7db","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL21lcmlkaW8tYXV0aC10cmVuY2gtYSJdLCJleHAiOjE2ODIwMDgxNzZ9.LNIU7MDeu7AJtRjU2n5xPRa4Q-zn_uagGjQ5NQTe3oSrbLf-r_Mw6AvnzR_Z_3izq77Yq4FEu3P9MxmMjvnFbg","expires":{"seconds":1682008176,"nanos":643886529},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"load-balancer-trench-a-7c885b6d9b-k7fnk","id":"db2bd643-5505-4c38-9704-7eb194b51f44","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL21lcmlkaW8tYXV0aC10cmVuY2gtYSIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvZGVmYXVsdCJdLCJleHAiOjE2ODIwMDgxNzZ9.m0FIfhJPKfomZ6eXTBBWef7gjuc1xddGLfo959MFBg4zvNapP2Ephv-sYKVSrcunCRzE2uU88cyTdU0CkkXMFw","expires":{"seconds":1682008176,"nanos":644706639}}]},"network_service_endpoint_name":"load-balancer-trench-a-7c885b6d9b-k7fnk","payload":"ETHERNET"}

There were improvements in NSM related to cleaning up connections. Haven't noticed "PersmissionDenied" error for a while.
There were also improvements in Meridio:
-IP garbage collection for Meridio IPAM:
https://github.com/Nordix/Meridio/issues/527
-Proxy can restore previous connections to LBs by learning them from NSMgr using NSM's Monitor Connection feature:
https://github.com/Nordix/Meridio/pull/536
https://github.com/Nordix/Meridio/issues/542
Run verification tests with most recent Meridio and NSM >= v.1.13.2

source: gharchive/issue

repo: Nordix/Meridio

license_type: permissive

license_source: github-api


text:
knife vault create is failing

Hi.
I am running chef-vault 2.6.1 with chefdk 0.7.0. Trying to run knife vault create  command and it keeps blowing up on an error I don't understand.
knife vault create secrets vaultuser1 --config vaultuser1.json --search '*:*' --admins jenkins --mode client
ERROR: CONFIGURATION ERROR:You have invalid ruby syntax in your config file /Users/jcarap/git/data_bags/vault/vaultuser1.json

SyntaxError: /Users/jcarap/git/data_bags/vault/vaultuser1.json:2: syntax error, unexpected ':', expecting =>
 "id": "vaultuser1",
      ^
/Users/jcarap/git/data_bags/vault/vaultuser1.json:2: syntax error, unexpected ',', expecting end-of-input
Relevant file content:
  1: {
  2:  "id": "vaultuser1",
  3:  "password": "imthetoughestcookietocrack",

The strangest thing is that I ran a test with the exact same gem and chef-server a couple of weeks ago with it worked just fine. Is there something broken with my chef-server setup I need to fix?

I love when I figure out my human error right when I press submit. used wrong flag.

repo: Nordstrom/chef-vault

license_source: bigquery


text:
update SVGO and other devDependencies

Alternative to #145
N.B.:

happiness left heavily outdated
svgtopng.js: unmodified sample code from https://github.com/domenic/svg2png/blob/master/README.md
Although current svg2png already has cli support we still need helper buildscript
because builtin cli lacks overwrite option (https://github.com/domenic/svg2png/issues/100)
In our current setup dist\images\toggle.png has to be overwritten during build process

Hi, very nice of you to contribute this! I have a couple of issues with this PR, though.

I don't see the need to update so many dependencies at once when all we want is a better svgo.
Does updating svg2png and rewriting the way it's used get us any improvements?

It seems you forgot to add pn as a dependency. Looking at the Travis CI setup, we don't actually call build, only test, so it might not notice but I think build would fail.

Raising the minimum for Leaflet from 1.0.3 to 1.4 is big breaking change which I won't do without a very compelling reason.

I guess all we really need to do is update svgo and not much else, the way I see it.

I don't see the need to update so many dependencies at once when all we want is a better svgo.

Well, I will try to explain my point.

There are 2 kinds of dependencies.

dependencies in production, where we should update carefully, because if we change our version requirements we can break other projects that use Leaflet-Minimap (as not every project will comply to that new version of our dependency).
devDependencies, where we can use whatever version that we want, and nobody else cares.

We have only 1 dependency of 1st type (and it's incorrectly defined in devDependencies).
Technically leaflet 1.4 is fully compatible within minor version changes, but I agree that we leave it ^1.0.

And main question: why we ever need to update any dependency? I have my answer:

community updates every package day by day making a lot of improvements. Some of them are subtle but still important.
currently dependencies' versions defined in our project.json are extremely outdated. Try npm audit and you see that there are a lot of known vulnerabilities.

Does updating svg2png and rewriting the way it's used get us any improvements?

I believe that we should keep all dependencies in actual state. It's safe.
I've just updated this PR according to my vision.

It seems you forgot to add pn as a dependency.

It is dependency of svg2png, we have not define it explicitly.
Try to build and see.

leaflet moved from devDependencies to peerDependancies as demanded by
PLUGIN-GUIDE.

source: gharchive/pull-request

repo: Norkart/Leaflet-MiniMap


text:
Configure a custom domain for API GW

Contact me with details about the custom domain.

User Story
As a user, I want to go to a custom domain that is easier to remember than API Gateway's url
Time to complete
Estimated: 8 hours
Actual:
Acceptance Criteria

[ ]

Activities Documentation

Date
Activity
Time Spent

03/05/2022
Research create a custom domain using AWS
1 hr

03/06/2022
Research create a custom domain on AWS using Route 53
1 hr

Pull Request

authors: KristinJJ toddysm


text:
Add ability to reset equipped clothing

Our character is currently permanently broken because we have an accessory equipped that doesn't exist. Being able to reset everything we have equipped should fix that for us and anyone else that may have this problem in the future.

I implemented a save fixer which should solve this issue.

repo: NotNite/webfishing-save-editor

license: MIT


text:
Deleted package detected

I'm a Cyber Security researcher and developer of PackjGuard [1] to address open-source software supply chain attacks.
Issue
During my research, I detected a deleted package in this repository.
Details
Specifically, the package pytabtex mentioned in file README at line 16 does not exist on the public PyPI registry. A bad actor can hijack this package to propagate malicious code.
Impact
Not only your apps/services using https://github.com/NourOM02/pytabtex repo code are vulnerable to this attack, but the users of your open-source Github repo could also fall victim.
You could read more about such attacks here: https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
Remediation
Please highlight this in file README and register a placeholder package for pytabtex on public PyPI soon to remediate.
To automatically fix such issues in future, please install PackjGuard Github app [1].
Thanks!

PackjGuard is a Github app that monitors your repos 24x7, detects vulnerable/malicious/risky open-source dependencies, and creates pull requests for auto remediation: https://github.com/marketplace/packjguard

Thanks for keeping me updated. However, I intentionally deleted it from  PyPi because parts of the documentation were messing. I re-upload and everything seems to work fine.


text:
Added text to voice feature

Tried different libraries but because of tkinter continuous loop pygame was used to play the audio files

Working confirmed in local environment. Thanks for the cool enhancement😂

authors: NozomuYoshida Relativiteit


text:
Update scala-library to 2.13.13

About this PR
📦 Updates org.scala-lang:scala-library from 2.13.6 to 2.13.13
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-lang", artifactId = "scala-library" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.scala-lang", artifactId = "scala-library" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:n:2

Superseded by #181.

repo: NthPortal/life-vest


text:
ET for authentication

The ET charter is to find bugs when:

registering an account
signing in
signing out

This is assigned to @aldion, but I did a tiny bit of testing.

Tried to register with usernames that had a beginning space, trailing space, and a middle space. Site didn't allow any of them as appropriate.

Registered for an account. Used an email address with a "+" character in it. Received the email. Clicked on the confirmation link. Was able to login just fine. notes: Email confirmation message doesn't actually include the username. We might want to change that later.

Confirmed an account (as in step #1). Then went back to old email and clicked the confirmation URL again. Got an error message "Could not confirm your email address.". This is expected, but we probably should indicate that the account is already confirmed. However, this is lower priority.

While logged off, clicked the Log On link. Logged in. Took me to the home page

Went to upload packages page. Clicked Log Off. Logged back in. Logged in and redirected back to upload package page

Logged out, clicked Upload Package, logged back in. Logged in and was redirected to upload package page

Logged out, typed in /dbadmin/. Logged in as a non-Admin user. Received a default 403 error message. Would be nice to make this a nicer looking error page. But very low priority.

I verified authentication across all supported broser. I've not been able to Log On using IE10 (external IE10 issue)

authors: Haacked aldion anglicangeek

repo: NuGet/NuGetGallery


text:
Pushing powershell module to NuGetGallery Trouble

So now my next objective is to use our private nuget gallery as a powershell cmdlet repo.  Attempted to push this module to my gallery.

I performed these actions http://docs.myget.org/docs/how-to/publish-a-powershell-module-to-myget
I also did the invoke-webrequest from https://github.com/OneGet/oneget/issues/174
Fiddler's syntax view shows this  message:
<?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-US">The requested resource does not support http method 'PUT'.</m:message></m:error>
Header:

Googling this problem leads me to this, however, I think this fix is above my pay grade.
[((http://stackoverflow.com/questions/12765636/the-requested-resource-does-not-support-http-method-get))]

Push should go to /api/v/2/package (don't see the /package suffix in the above logs)

Thanks.

authors: haingoctu maartenba


text:
[Bug] V2 feed showing unlisted packages for FindPackagesById()

The package Catel.Core 5.0.0-alpha0234 is unlisted.
However it is displayed in the FindPackagesById()?id='catel.core'&$skip=600 query, which is unexpected.
In NuGet.Server for example, unlisted packages are ignored for this particular query, whereas NuGet.org does seem to include unlisted packages. Interestingly, Search() and friends properly exclude unlisted packages from the result data.

The default behavior to include unlisted packages seems by design: unlisted packages still must be able to restore, which, on v2, is done by a call to api/v2/FindPackagesById.
However, an optimization can be done here, because restore should always provide the version parameter too!
The above call does not include a version number, and thus, we should hide unlisted packages by default when the parameter is missing, to avoid discovery of these unlisted packages.

This test seems faulty: https://github.com/NuGet/NuGetGallery/blob/master/tests/NuGetGallery.Facts/Services/FeedServiceFacts.cs#L719
I see no reason to return unlisted packages on v2 FindPackagesById when no version is explicitly provided. The whole point of unlisting packages is to hide them from search / make them no longer discoverable, and this API seems to violate that behavior/design.

This would break the client, unfortunately there is no doc listing all the scenarios but 3.0.0 originally took this approach on the client side and had to roll everything back.
The client uses FindPackagesById to discover all versions, listed and unlisted, and then it prioritizes listed packages first when resolving dependencies. 2.x would use either.
This is important for this scenario:
A (unlisted) -> B (unlisted) with no version range specified.
If only B 1.0.0 exists and it is unlisted it would currently get installed, but if FindPackagesById changes behavior then A would fail to install since B would be missing.
It may also be the case that if all packages unlisted then install-package is suppose to work still. Also, I think there is an issue on the client that install-package doesn't even check for unlisted when finding the latest version.

But FindPackagesById() also does not return a Listed property of some kind, so the consumer of the API has no clue if it is listed or unlisted in this case. Fine with either but I'd like to know about listed / unlisted :-)

Plus does that mean NuGet.Server is faulty then?

But FindPackagesById() also does not return a Listed property of some kind

The published date is used in some scenarios by the client to determine if the package is listed.

Plus does that mean NuGet.Server is faulty then?

I would say so. I don't think NuGet.Server supported unlisting in the past, it did an actual delete.
VSTS feeds were removing unlisted packages from FindPackagesById recently and it was determined that it was incorrect.

Ah well :-) Will see if we can work around it.
Have a good weekend Justin!

To summarize things: the conclusion is that NuGet.Server doesn't should unlisted packages in FindPackagesById() and this is a bug?

Sounds like it, yes.

Thanks for the insights, @emgarten!
I agree package deprecation improvements could alleviate some of these issues.
Besides the NuGet.Server bug, I think the design choice for FindPackagesById should be documented too as it is not obvious looking at the code why some decisions were made.

There does seem to be ab ug here in the gallery. For unlisted packages, normally the published date is set to 1970-..... In this case, the published date is <d:Published m:type="Edm.DateTime">1900-01-01T00:00:00</d:Published> - which is not "unlisted", just jurassic :-)

Two action items here:
In the 2.x release of NuGet.Server, unlisted packages should be included in the FindPackagesById() output and the published date should be set to 1900. The listed property is not observed by NuGet client.
In the 3.x release of NuGet.Server, unlisted packages are already shown in the FindPackagesById() output, but the published date is not set to 1900 as with 2.x. This also needs to be fixed.

Fixed in the following releases:
https://www.nuget.org/packages/NuGet.Server/2.14.2
https://www.nuget.org/packages/NuGet.Server/3.0.1

authors: emgarten joelverhagen maartenba skofman1 xavierdecoster


text:
[BugBash] Manage Organizations UI issues

Scenario 22, per @anangaur
The above 3 issues...

@chenriksson , can you please add more details here?

From the OneNote:

Looks like @anangaur wanted to put the icon next to the Org name instead of a different column, and remove the Org link. Not sure if this is covered by the UI work that was discussed.
We should also look at consistency across site for this change... Manage Packages has the icon in a separate column too:

Another note--due to how we use bootstrap in these tables, making major changes to the column spacing is not entirely possible. We could use CSS directly in the tables to make the spacing changes but that's much more complex than bootstrap.

authors: chenriksson scottbommarito skofman1


text:
Auto-add Microsoft a co-owner for packages pushed by blessed Organizations

Fixes https://github.com/NuGet/Engineering/issues/1419
Opening this PR already for early review and feedback.
Note: this focuses on the package push API side of things. Not covering the web upload scenario, yet.

                        if (!(ConfigurationService.Current.AsynchronousPackageValidationEnabled && ConfigurationService.Current.BlockingAsynchronousPackageValidationEnabled))

on nuget.org async validation is enabled, so this code block will never be called

Refers to: src/NuGetGallery/Controllers/ApiController.cs:533 in 6228911. [](commit_id = 6228911d9196b59402fffc2c12f6350378d1eb02, deletion_comment = False)

                        if (!(ConfigurationService.Current.AsynchronousPackageValidationEnabled && ConfigurationService.Current.BlockingAsynchronousPackageValidationEnabled))

Great point! Plumbing the warnings down to orchestrator would be a mess.
Couldn't we do the warning emails asynchronously for "blessed" owners? This would also mean we could check for existing IDs that don't fall under a namespace.

In reply to: 407812683 [](ancestors = 407812683)

Refers to: src/NuGetGallery/Controllers/ApiController.cs:533 in 6228911. [](commit_id = 6228911d9196b59402fffc2c12f6350378d1eb02, deletion_comment = False)

In today's update:

new MicrosoftTeamSubscription

the new RequirePackageMetadataCompliancePolicy is now generic, and validation values are passed in via policy State (which is configurable in the db)
no more using IEntitiesContext directly in RequirePackageMetadataCompliancePolicy (instead using IUserService and IPackageOwnershipManagementService
a new unit test asserts that the new package security policy calls service APIs using commitChanges:false

Open questions:

if package policy results in warnings, a sync message is currently sent when gallery is configured to use async validation
there's still a todo in the OnSubscribeAsync of MicrosoftTeamSubscription: should that be part of v1?

@skofman1 oops, good catch :) I've pushed an update that addresses that scenario. Notice the subtle difference in email subject (published vs pushed) and email body (no link to email settings when push warning message is sent).
Merging is blocked by @scottbommarito's request for changes, but if all good, feel free to merge and deploy to DEV later today.

@scottbommarito your review is still requesting changes. Can you please check if changes are good, or let me know what needs changed still?
@karann-msft @skofman1 had an offline discussion with @joelverhagen regarding the email with warnings message, and he had an interesting suggestion:

Is it necessary the email is sent immediately?
Why don’t we just create a dedicated job to scan Microsoft owner IDs every day and send an email once on IDs that don’t have reserved namespaces. This way we get the benefit for existing IDs too. Or we can make it a report that PMs see and reach out to teams manually. Since this is sent from the same sender as normal package push and to the same recipient, don’t you think it’s likely this message will go unnoticed? I imagine the package push notification is noise for a lot of users.

I guess it all depends on the answer to the above question, and the amount of effort we want to put into it.

Xavier, as I commented above regrading Joel's suggestion: it's good, but let's not invest the work now, and leave the code as is.
Feel free to merge if you think Scott's comments were addressed.

authors: joelverhagen skofman1 xavierdecoster


text:
Make min-version check run before ResolveAssemblyReferences target

When min version is 16299, other build errors occur before it gets to execute this target. When it resolves the assemblies, and these assemblies require 16299, it fails.
Moving the BeforeTargets from Build;Rebuild to ResolveAssemblyReferences avoids this issue.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/guides/Create-UWP-Controls.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@karann-msft @anangaur Can one of you check this?

authors: dotMorten kraigb saldana


text:
Bytecode caching

What does this PR do?
I finished the bytecode caching
Why was it initiated? Any relevant Issues?
#865
PR Checklist

[x] Correct base branch selected? Should be develop branch.
[ ] All tests still pass. Check the developer manual about Running the Tests.
There are Github Actions tests that cover the most important
things however, and you are welcome to rely on those, but they might not
cover enough.
[ ] Ideally new features or fixed regressions ought to be covered via new tests.
[ ] Ideally new or changed features have documentation updates.

I have fixed the pylint error, but I am not responsible for the Windows error.

Annoyingly I had to click run for PRs now, I will look into the new results, the Windows error looked like the print only, so hopefully it's good. Onefile tests currently occasionally hang, I have stuff for that in the queue on factory, but it's not complete. Sorry about the sad state of you having to wait for me.

Oh, that was just for debugging purposes. I must have forgotten to delete it. I will delete this with the following commits. I still tried to add an option that changes the location of the whole cache, so ccache too. Unfortunately the test failed with an error I can't explain.
Nuitka-Scons:INFO: Backend C compiler: /usr/lib/ccache/gcc-8 (gcc-8).
AttributeError: 'NoneType' object has no attribute 'cache_dir':
  File "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/build/Backend.scons", line 1292:
    assume_yes_for_downloads=assume_yes_for_downloads,
  file "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/build/SconsCaching.py", line 173:
    ccache_dir = os.path.join(getCacheDir(), "ccache")
  File "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/utils/AppDirs.py", line 44:
    if Options.getCacheDir() is not None:
  File "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/Options.py", line 426:
    return options.cache_dir

Maybe you know what the problem is?

@Ben-Faessler this looks strange:

File "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/utils/AppDirs.py", line 44:
if Options.getCacheDir() is not None:

Could this be something you did, never was that like in that modules code, unless I am very mistaken.

As an aside, for development, I recommend to do python setup.py develop which will make Nuitka register something that hooks your git checkout, that avoids a need to install. Alternatively, use the runners in the bin folder, they will set the sys.path to the code near them, but I think develop is the most clever way, esp. if you are in a virtualenv.

@Ben-Faessler this looks strange:

File "/usr/local/lib/python3.7/dist-packages/Nuitka-0.6.15rc5-py3.7.egg/nuitka/utils/AppDirs.py", line 44:
if Options.getCacheDir() is not None:

Could this be something you did, never was that like in that modules code, unless I am very mistaken.

Yes, that comes from me. I need that to manipulate the _cache_dir.

Should I commit the changes for the command line option? Then maybe you can tell what is wrong.

Some parts of the code are used by the scons process and the nuitka process. Only nuitka has a functional nuitka.Options module, scons will not work with that. You need to not use options in there, but more on the outside, in code only used by the frontend, and not from Backend.scons for things to work.

Yes, you can always push what you have got, I can then comment on it, and ask for changes.

I think you are not yet using the cached stuff in the pushed code, or I am missing something. There needs to be some change in the tree building, where it's considered.

In my tests, Nuitka used the cached bytecode. However, only if nothing has changed in the folder where the programme to be compiled is located (deleting or adding files and folders). Meanwhile I have the same modules in the cache 5 times...

I just got an idea. Would it make sense if I move the caching in a separate module. So you could limit the BytecodeDemotion on their actual job, and it would be easier for example. implement a C code caching. As a name suggest "nuitka.utils.Cache"

The way this works now, is that Nuitka in nuitka.tree.Building creates a compiled module and optimizes it and later when optimization is finished, it is replaced with a uncompiled module. Your cache kicks in too late, where it doesn't avoid this.
You need to make use of the cache here: https://github.com/Nuitka/Nuitka/blob/0c88a3c7fa0f79613d09b998d3151ace3601312e/nuitka/tree/Building.py#L992
There, you should if bytecode is the compilation mode decided, for an UncompiledPythonModule rather than a CompiledPythonModule, and then it will be great. Using the cache where you use it in terms of reading back has no point, although it probably demonstrates correctness, but you didn't achieve it to be avoiding the compilation step at all, outside of demoting more things of course, but that's for the option to control that.

Regarding the location of things, I need think a bit. Do whatever feels right to you, it's all changeable later, but I think nuitka.optimization and nuitka.tree are better places, or nuitka.Caching maybe even, as this is probably something that they all are sharing, but not really a utility if you know what I mean, at least not outside of determining version hashes, etc.

Where do I get all the values for the parameters?
Code snippet from Recursion.py:
module_name=module.getFullName(),
filename=module_filename,
bytecode=marshal.dumps(
compile(
source_code, module_filename, "exec", dont_inherit=True
)
),
is_package=module.isCompiledPythonPackage(),
user_provided=True,
technical=False,
)

I guess, you need to cache all required information. The bytecode isn't actually important to cache, you can just compile it again, but it being a package, is contextual knowledge. Can you give a better point, so I can tell why you have to do that? I thought you only have to replace this:
            result = CompiledPythonModule(
                module_name=module_name,
                is_top=is_top,
                mode=decideCompilationMode(is_top, module_name, source_ref),
                future_spec=None,
                source_ref=source_ref,
            )

There this is all available and you just need to change the class to be constructed, depending on the decided compilation mode, making that the condition of an if statement and checking the cache availability.

I thought I had to do something like that.
            mode = decideCompilationMode(is_top, module_name, source_ref)
            if mode == "bytecode":
                result = UncompiledPythonModule(
                    module_name=module_name,
                    filename=source_ref.getFilename(),
                    bytecode=
                    is_package=False,
                    user_provided=False,
                    technical=False,
                    )
            else:
                result = CompiledPythonModule(
                    module_name=module_name,
                    is_top=is_top,
                    mode=mode,
                    future_spec=None,
                    source_ref=source_ref,
                )

There this is all available and you just need to change the class to be constructed, depending on the decided compilation mode, making that the condition of an if statement and checking the cache availability.

Where should I change that?

That code lives in nuitka.tree.Building function decideModuleTree. Do not bother with is_main because the top level module must not be bytecode or else you get into trouble with the linkage. In fact, should is_top be set, you must not do it as well, which will matter for compiling in --module mode.

In buildModuleTree when you got the uncompiled module, you definitely should not use the source code anymore, that code wants to add the nodes with the Python code to the compiled module, which was created empty. Just check if its a isCompiledPythonModule and then and that to checking if there is a source filename. That branch exists e.g. because extension modules are not having source either. Being bytecode module is the same, except you were passed in source filename, but now you know to not use it anymore.

Some parts of the code are used by the scons process and the nuitka process. Only nuitka has a functional nuitka.Options module, scons will not work with that. You need to not use options in there, but more on the outside, in code only used by the frontend, and not from Backend.scons for things to work.

How else can I use it?

Some parts of the code are used by the scons process and the nuitka process. Only nuitka has a functional nuitka.Options module, scons will not work with that. You need to not use options in there, but more on the outside, in code only used by the frontend, and not from Backend.scons for things to work.

How else can I use it?

You need to do it in code not shared between them, the appdata module clearly is too low level. You should only use it where it makes sense, or is it your intention to affect all appdata caching dirs there? Just don't do that I suppose. There are good reasons for using standard directories, and even if not, that's a separate issue.

In buildModuleTree when you got the uncompiled module, you definitely should not use the source code anymore, that code wants to add the nodes with the Python code to the compiled module, which was created empty. Just check if its a isCompiledPythonModule and then and that to checking if there is a source filename. That branch exists e.g. because extension modules are not having source either. Being bytecode module is the same, except you were passed in source filename, but now you know to not use it anymore.

Can you please be more precise?

I am manually merging this right now, rebasing and such, and making it harmless (needs an experimental flag), so it can be included in the next release, and then debug it for a bit, I think it ought to work almost.

I am deciding against caching of the bytecode though, as it would be impacted by modes, e.g. no_doc, no_annotations and the like, for no good reason. Also plugins can interact with this better that way. This is assuming that bytecode compilation itself is fast, which I think is true.
This is only about caching the included modules of said module, to avoid the compilation of Python to figure those out.

This is on factory, how to do proceed with this... I don't want you have this as rejected, when it fact it should be accepted, so I guess, I am going to merge it, and immediately undo it on github develop with a rebase of current factory, when I do the next pre-release.
It's still not working, but disabled by default, the only reason is that dependent modules are not imported when using the cache, so they are missing during optimization. That should be easy to overcome. But I would like to eliminate some duplication in nuitka.Recursion and nuitka.Building related to creating modules, as they both have to similar things.

authors: Ben-Faessler kayhayen

repo: Nuitka/Nuitka


text:
Initial setting

I started the application TicketDesk 2.5 (beta), but I do not know how to log in to the admin panel, it asks for e-mail.... You wrote that there will be a stage of creating a database but this did not happen, I immediately saw the main application window....

I am not sure how that could happen -- source control shouldn't contain a database at all, so it'd have to go to the setup page first to create one (http://whatever/first-run-setup).
I do think there may be an issue with logging in with the current head of the develop branch though -- we merged in some changes recently. I was planning to look into that a bit more to be sure. You might want to stick to the master branch if you have issues.

Actually, you can disregard the issue with logins on develop branch -- turns out that was a local issue with my own browser. Seems to work correctly.

repo: NullDesk/ticketdesk

license: MS-PL


text:
cannot connect after import profile

Cannot connect after import profile to openvpn connect (download from playstore). works on my pc but not on mobile
MI 10T Pro android version 10 OpenVPN Connect – Fast & Safe SSL VPN Client

Some mobile networks filter VPN connections. But if you still have issues please provide the client connection log so we can get a better idea of what is going on.

authors: Nyr myh-st

repo: Nyr/openvpn-install

license: mit


text:
Welcome/Landing Page

Hello,
I have 1 quick question about this project.
Is it possible to edit the client/server configuration file in a way that when you connect to OpenVPN it automatically opens a Landing page?
I need it so that clients that connect with OpenVPN to a network can have an easy intranet page from where they can have links to all the services in the network?
Thank you

Check the documentation for --up parameter, you can run a command when connecting with it.

authors: Nyr Stefan-CPlanet


text:
Update to NASA Policy object

we need to update the relevant part of the Nasa policy object.
Here it says the policy "Expects deposit into PubMed Central" where it SHOULD say "Requires deposit into..."
Also, you might notice how NIHMS are being rendered with weird characters in the paragraph? Could you please change that as weel

This looks like a minor assets/data update.  The culpret is cutting and
pasting smart quotes, which never works right
On Sun, Jun 10, 2018, 11:23 AM Hanh Vu notifications@github.com wrote:

we need to update the relevant part of the Nasa policy object.
Here it says the policy "Expects deposit into PubMed Central" where it
SHOULD say "Requires deposit into..."
[image: screen shot 2018-06-10 at 11 21 09 am]
https://user-images.githubusercontent.com/11050142/41203085-87ab2aa6-6ca0-11e8-9460-85a14061b1a5.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/OA-PASS/pass-ember/issues/499, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfm-YFwcHYB7w87y0W-Q-ySbqjeGq3uks5t7TncgaJpZM4UhwY4
.

@htpvu I think the part that says "Expects deposit..." might be built into the webpage. It's not part of the policy. I can fix the garbled quotes.

done

Verified on rev 7f2a26bd

authors: birkland htpvu karenhanson

repo: OA-PASS/pass-ember


text:
sign up / login button on plugin isn't "our blue"

Could there be a mouseover too?

It is now

authors: JosephMcArthur markmacgillivray

repo: OAButton/backend


text:
Clarification on using null+type:object in composed schemas v3.1.0

Hi team,
In the upcoming v3.1.0 spec we will be allowed to use type: "null" definitions.
Using a discriminator allows us to quickly move through an inheritance tree using the discriminator.propertyName passed in a payload.
So if discriminator.propertyName = "animalType"
and we deserialize an Animal with the payload {"animalType: "Dog"} this works.
But sometimes users want to pass in empty payload values like null or empty dict {}.
Should we allow those types in the oneOf definition when we are in a composed schema?
Question 1: clarification on using type: "null" + object in oneOf w/ discriminator
Should this be valid?
Animal:
oneOf:
- $ref: NullType
- $ref: Pet
- $ref: ObjectType
discriminator:
  propertyName: animalType

NullType:
type: "null"

ObjectType:
type: object

Some context in python is that None and {} are both Falsey values for a variable that holds a dict.
My proposal is that we allow these null and {} empty types because some clients will expect, accept and understand these empty/zero-state values.
This would allow for rapid deserialization through an inheritance tree while also allowing servers to return null/{} empty state values.
Note about the where we would include type: object in the oneOf list:
If we do allow empty object type, then we may want to list it last.
That would be safer for generator that loop through oneOf types and deserialize if the payload matches the schema. That would allow a match on Pet before {}.
This point applies more when one includes type: object without using a discriminator in Animal.
Because order matters for that use case, it would be simplest if we require that the type: object definition be the last one when used in oneOf.
@handrews @philsturgeon @tedepstein @johncmunson
This question came up here
Question 2: clarification on using type: type: "null" + object in anyOf without discriminator
Should this be invalid?
Animal:
allOf:
- $ref: Blah
anyOf:
- type: null
- type: object

I think that null and type: object should be invalid in anyOf definition.
Question 3: clarification on using type: type: "null" + object in allOf without discriminator
What should we do for allOf?
Animal:
allOf:
- type: null
- type: object

Type null should be invalid because it does not apply to object schemas
Type: object. Not sure about how I feel on this one, it adds no new information to the schema.

I believe a lot of this is covered in #2143, where the oneOf doesn't need a discriminator to make it work.

@philsturgeon I read through that issue and didn't see any examples or discussion that addressed these three questions. These questions hinge around mixing types in oneOf/anyOf/allOf definition. Did I miss something? Where specifically did you mean?
Can you tell me what you think we should do in each of the above 3 cases and why?

@spacether TBH I can never remember how discriminator works, except that (from a JSON Schema processing model perspective) it's really problematic.  I think because it implicitly (again, from the point of view of what JSON Schema understands) changes how validation schemas are applied.
I have no idea whether type: "null" would be allowed in OAS 3.1.  I see your point about it being a use case (I personally avoid nulls but that's a stylistic choice related to available typing systems, and sometimes you have to work with what's already there so we should take that into account).
If you ignore whatever discriminator is doing, your oneOf example is perfectly valid JSON Schema aside from the $refs not looking like proper URI references.  I assume that's just shorthand for the example.
The schema:
allOf:
  - type: object
  - type: "null"

will always fail validation.  The type keyword applies to all data types (and constrains which ones are legal).  An instance can never be both null and an object at the same time, so this always fails.

@philsturgeon I read through that issue and didn't see any examples or discussion that addressed these three questions.

@spacether the whole thing is super relevant, from the possibility of it being deprecated entieely, to the examples showing oneOf mixed with discriminator. This issue talks about how a discriminator can be replaced with oneOf usage, so you don't need to use discriminator in the first place.
Here's a polymorphic example I helped a client with. Actions can be one of these two types, and common action shares some common properties.
type: object
properties:
  id:
    type: string
  action:
    oneOf:
      - $ref: '#/definitions/ActionType1'
      - $ref: '#/definitions/ActionType2'
definitions:
  ActionType1:
    allOf:
      - $ref: '#/definitions/CommonAction'
      - type: object
        properties:
          type:
            type: string
            enum: [type1]
          type1SpecificField:
            type: string
            example: someValue
  ActionType2:
    allOf:
      - $ref: '#/definitions/CommonAction'
      - type: object
        properties:
          type:
            type: string
            enum: [type2]
          type2SpecificField:
            type: string
            example: someValue
  CommonAction:
    type: object
    properties:
      commonField:
        type: string
        example: someCommonValue

The enum: [type1] and enum: [type2] is the replacement for discriminator (in OAS3.1 that'll be const: type1 and const: type2.
Beyond that, if discriminator is deprecated, question 2 and 3 seem a bit moot. Question 2 is possibly either valid or invalid with or without a discriminator (depends if a solo type: null works or not), and question 3 is invalid with or without a discriminator because an allOf cannot enforce two totally different types at the same time.
Are you seeking edge cases for your code generator or are you trying to find out how to use something for a project, because maybe that sort of background would provide better context for helping you out.

@philsturgeon in the questions I say whether the discriminator exists, so questions 2 + 3 apply to non-discriminator use cases.
Thanks for sharing that context about the possibility of the discriminator being deprecated. My question 1 is specifically about what we do with a discriminator. When will the spec committee make a decision about whether to deprecate it? The current draft spec includes the discriminator and I asked question 1 because the payloads that I write about deserializing lack the required propertyName key value pair when they are null or {}.
Thank you for letting me know what you think should happen for use cases 2 + 3.
It sounds like mixed types(non-object types) can only be included in the below use cases:

VALID: oneOf w/ non-object types + NO allOF + NO anyOf + NO properties in the composed schema
VALID: anyOf w/ non-object types + NO allOF + NO oneOf + NO properties in the composed schema
INVALID: non-object types in anyOf
Should the spec clarify that?

@spacether the whole thing is super relevant, from the possibility of it being deprecated entieely, to the examples showing oneOf mixed with discriminator. This issue talks about how a discriminator can be replaced with oneOf usage, so you don't need to use discriminator in the first place.

One unanswered question about deprecating the discriminator is the performance impact, as discussed in #2143. Until there is a solid proposal, we need to continue supporting discriminators, and even if that happens, there will still be such OAS documents to support for a long time.

Are you seeking edge cases for your code generator or are you trying to find out how to use something for a project, because maybe that sort of background would provide better context for helping you out.

I'll let Justin respond for his use cases. In our case (Cisco), we are both authors and consumers of OAS documents. We have server-side implementation of OAS v3, clients for v2 and v3 in multiple languages (java, python, go, powershell), and we contribute to OpenAPITools such that as a goal, the tooling can work with any input OAS v3 document, not just ours. We need to be able to consume OAS documents that have been written by 3rd parties, which may or may not have edge cases. Some OAS doc tend to be using bleeding edge features, other are using basic features from OAS v2. We do use discriminators because without it performance is significantly worse (see #2143 for why). We have also started using the "null" type from OAS 3.1.
Related to this, I have created https://github.com/OAI/OpenAPI-Specification/pull/2216 as an attempt to clarify the discriminator.

To summarize: All of the examples in the questions are "valid" in the sense that they will parse and tools can try to use them.  But (like many things that are legal to do in JSON Schema), they may or may not make any sense.

allOf two conflicting types, no matter what types they are, will always fail
anyOf multiple types is valid and has many use cases.  You more often see this with oneOf but it could happen with anyOf if there are other branches that might be true alongside one of the branches with different types
trying to use discriminator with schemas that don't include the type discriminating field falls under "garbage in, garbage out" - tools can do whatever with it.  As an OAD author, you probably want to anyOf the "null" type outside of the discriminator construct - discriminator only makes sense if it is not null.

authors: handrews philsturgeon sebastien-rosset spacether

repo: OAI/OpenAPI-Specification


text:
Missing scores of zero-shot clinical subsitutions

Hi,
The link on GitHub for zero-shot clinical substitution scores (https://marks.hms.harvard.edu/proteingym/zero_shot_clinical_substitution_scores.zip) seems not working.
For the model scores downloaded directly from the webpage, I found that all score files are empty.
Can you please help fix this issue?
Thanks

Hi @tuanqdinh,
Thanks for catching this! Looks like there was an error when we zipped the files originally, and it only got the file structure without the content. I've updated the link now so it should be working correctly.

Thank you  for the prompt update. Can I ask what version of TranceptEVE model (L, M, S) was used for ClinVar (achieving 0.92 AUC)?
And also, I find that the scores of TranceptEVE M and S are not complete (392 and 0 files compared 2525 files of model L).
Can you please double check?
Thanks

Hi Tuan,
The Clinical Scores for TranceptEVE all use TranceptEVE L. The incomplete
scores for the M and S models were included by accident, and I'll remove
those soon (they were from earlier partial scoring runs).
Best,
Daniel Ritter
On Tue, Feb 13, 2024 at 10:00 PM Tuan Q. Dinh @.***>
wrote:

Thank you for the prompt update. Can I ask what version of TranceptEVE
model (L, M, S) was used for ClinVar (achieving 0.92 AUC)?
And also, I find that the scores of TranceptEVE M and S are not complete
(392 and 0 files compared 2525 files of model L).
Can you please double check?
Thanks
—
Reply to this email directly, view it on GitHub
https://github.com/OATML-Markslab/ProteinGym/issues/22#issuecomment-1943012411,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIRYOXHWMHXJFIUDHPVQCNLYTQSGFAVCNFSM6AAAAABDFQYGTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGAYTENBRGE
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

authors: danieldritter tuanqdinh

repo: OATML-Markslab/ProteinGym


text:
Add startAuthorizedRequest optional body parameter to support POST

This PR introduces changes needed to perform POST request using convenience function startAuthorizedRequest with HTTP body parameter.
Before this change self.client.request was performed without optional body parameter.

Looks like there is danger<--TravisCI issue For your GitHub repo, you need to expose: DANGER_GITHUB_API_TOKEN causing build to fail.

repo: OAuthSwift/OAuthSwift


text:
Fix the signature format for PLAINTEXT

I encountered an issue regarding the signature format when using PLAINTEXT as the signature method. And after doing some research and debugging, I found that OAuthSwift was formatting the signature incorrectly.
While this may not be the best solution, I have tested it with the (private) api that uses PLAINTEXT, and that does fix the issue.
Just for reference, the signature format for plaintext is highlighted in the Dropbox documentation.

I think I never implemented it
https://github.com/OAuthSwift/OAuthSwift/blob/77a861bb6bd12a806451301a44d238e65ec568bb/Sources/OAuthSwiftCredential.swift#L110
just make the enum to introduce other method like rsa
thanks

authors: chrishannah phimage


text:
Fix null pointer exceptions on page import

Fixes #172 (and another, similar bug w.r.t. TextLine/@index)

These patches have not been applied during the recent merges. Should I redirect the PR to master?

@maxnth I updated the PR to the upstream changes. I wonder why you have not merged it yet, though.
Meanwhile, I have found another bug: if for some reason, a TextRegion without @type survived import, then the PageXMLWriter must be able to cope with it, too. (Also throws NPE currently.) Do you want me to add a fix for that as well?

@maxnth I updated the PR to the upstream changes. I wonder why you have not merged it yet, though.

I used your fix (thanks for that) in a feature branch that was supposed to contain all other changes which are needed to allow LAREX to work with text regions without a type (UI + PageXMLWriter) as you already mentioned.
Sadly some other things got in the way and I didn't yet come to actually implementing these.
If it helps, I could add this fix to dev directly, otherwise I'd add it when the branch is complete.

I see. Well, there's no such feature branch on the server, and no messages here, and then there was a new conflicting change right in the same section which I had to merge, so I just assumed this got forgotten.
No, it's fine with me. I can keep up my own master as long as necessary. I'll probably wait merging your dev activity until you found time to deal with this.

The proposed fix is now available in the dev branch (see #172).
If the implemented behavior doesn't work as excepted feel free to report back.

authors: bertsky maxnth


text:
ConnectedBracketRef_T missing its ref.

SOme reference pointers were missing it's refType or had optional refType

For some reason, this issue was not committed to the working_draft. The issue is re-opened.


text:
Decla overzicht van wat is goedgekeurd

Handig om te hebben om een jaarverslag te maken enzo.

Ongeveer


text:
dynamic-layers/meta-mender-tegra: add fixed state script

for the cboot+U-Boot case, which adds a mkdir command that
is missing from the upstream copy.
Signed-off-by: Matt Madison matt@madison.systems

This was supposed to go in with yesterday's update, but got missed.

This was supposed to go in with yesterday's update, but got missed.


text:
Refactor DTMF

Saves ~200 bytes

We still have quite a few things to refactor on the original decompiled code...
Everything looks good with the C2x standard and the other optimizations.
Thank you very much for your contribution !

authors: JuantAldea reppad


text:
Fix/v3 multiple data source providers

This just fixes the configurability of additional data source extensions - without it, there isn't a way to define a second data source type without removing the default extension.

Unit test is failing

@sedghi  - as requested, I moved the types into their own directory.  Also fixed the unit test.

I think I have all your issues resolved @sedghi, except that I left the Consumer as that is a well known name - ideally it would be a Consumer on a given type, but on a general class it works fairly well.

repo: OHIF/Viewers


text:
Redux state mutations in reducers

In our application, reducers of ohif-core cause failures of immutability checks as implemented by redux-immutable-state-invariant redux middleware.
So far we found 2 mutations: see PR #22
There could be more. Please consider implementing a systematic review and automatic checking of the reducer code.
Some of the tools that help to avoid state-mutation-related bugs (or to find them early):

object spread (@babel/plugin-proposal-object-rest-spread) -  the golden standard of shallow-copying objects as of right now. It allows to avoid Object.assign(), which is arguably more verbose and error prone.
redux-immutable-state-invariant redux middleware
unit testing reducers and asserting immutability of input state

Additional info:

Why immutability is required in Redux

👋 Hi @ivan-aksamentov, thanks for bringing this to our attention and for the PR. We're still feeling things out, so any suggestions in a similar vein are appreciated. Please don't hesitate to create PRs for anything else out of line with best practices.

authors: dannyrb ivan-aksamentov

repo: OHIF/ohif-core


text:
fix connection reuse

See https://github.com/OJ/gobuster/pull/79 for details. This fixes a nasty bug that prevented connection reuse
CC @OJ

Thanks @FireFart, this was an easy one to land. I appreciate your efforts! I've added another tag/release here for this.
I know that #79 has a bunch of other stuff in it, and I'm working my way through the code at the moment to get my head around it. I just thought landing and releasing this was a good idea. Thanks!

awesome! Thought it would be a quick win for the current installations out there

authors: FireFart OJ

repo: OJ/gobuster


text:
Bioconda package

It would be great to have VD in Bioconda for greater visibility and auto-building of new Conda packages for new releases of VD. Docker and Singularity containers are also created for all Bioconda packages, which is handy for developing containerized workflows.
https://bioconda.github.io/contributor/index.html
It's fairly simple to get a tool into Bioconda: make a PR to bioconda-recipes with a meta.yaml of packages necessary for building/running VD.
There's lots of examples to go based off of, e.g.

cutadapt meta.yaml

Thank you for giving our tool a try and for bringing this issue to our attention. We appreciate your patience and are here to support you through the installation process. To address the dependencies issue you encountered with conda, we recommend trying an alternative package manager, Mamba, which is designed to handle dependencies more efficiently. Here's how you can set it up:
conda create -n vd -y
conda activate vd
mamba install -c bioconda -c conda-forge -c charronp variantdetective

If you still face installation issues, it might be helpful to attempt the installation from source if you haven't yet. You can find detailed instructions for this method at the following link:
https://github.com/OLF-Bioinformatics/VariantDetective#installation-from-source
Please give these steps a try and let us know if you encounter any further difficulties. We're committed to ensuring that you can successfully use VariantDetective.

Thanks Phil,
I tried it once more although conda currently uses mamba by default.
I first had to install mamba inside the created env as I do not have it in my base (installs a LOT of extra packages with itself).
It failed now on openssl (see below)
Could python 3.6 be required for your package to work?
I will not attempt installing from source as my server does not allow it and wait until the conda install process gets revised and works to try again.
Best regards,
Stephane

running conda 24.1.2
ubuntu 22.04.4

(base) u0002316@gbw-s-pacbio01:~ $ conda create -n variant_detective -y
Channels:
 - defaults
 - conda-forge
 - bioconda
 - r
 - anaconda
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/miniconda3/envs/variant_detective

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate variant_detective
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) u0002316@gbw-s-pacbio01:~ $ conda activate variant_detective
(variant_detective) u0002316@gbw-s-pacbio01:~ $ mamba install -c bioconda -c conda-forge -c charronp variantdetective
Command 'mamba' not found, did you mean:
  command 'samba' from deb samba (2:4.15.13+dfsg-0ubuntu1.5)
Try: sudo apt install <deb name>
(variant_detective) u0002316@gbw-s-pacbio01:~ $ conda install mamba
Channels:
 - defaults
 - conda-forge
 - bioconda
 - r
 - anaconda
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/miniconda3/envs/variant_detective

  added / updated specs:
    - mamba

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bzip2-1.0.8                |       h5eee18b_5         262 KB
    requests-2.31.0            |   py39h06a4308_1          96 KB
    tzdata-2024a               |       h04d1e81_0         116 KB
    urllib3-2.1.0              |   py39h06a4308_1         156 KB
    ------------------------------------------------------------
                                           Total:         630 KB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  brotli-python      pkgs/main/linux-64::brotli-python-1.0.9-py39h6a678d5_7
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_5
  c-ares             pkgs/main/linux-64::c-ares-1.19.1-h5eee18b_0
  ca-certificates    pkgs/main/linux-64::ca-certificates-2023.12.12-h06a4308_0
  certifi            pkgs/main/linux-64::certifi-2024.2.2-py39h06a4308_0
  cffi               pkgs/main/linux-64::cffi-1.16.0-py39h5eee18b_0
  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0
  conda              pkgs/main/linux-64::conda-4.12.0-py39h06a4308_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-2.2.0-py39h06a4308_0
  conda-package-str~ pkgs/main/linux-64::conda-package-streaming-0.9.0-py39h06a4308_0
  cryptography       pkgs/main/linux-64::cryptography-41.0.3-py39h130f0dd_0
  icu                pkgs/main/linux-64::icu-58.2-he6710b0_3
  idna               pkgs/main/linux-64::idna-3.4-py39h06a4308_0
  krb5               pkgs/main/linux-64::krb5-1.20.1-h568e23c_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libarchive         pkgs/main/linux-64::libarchive-3.5.2-h5de8990_0
  libcurl            pkgs/main/linux-64::libcurl-8.2.1-h91b91d3_0
  libedit            pkgs/main/linux-64::libedit-3.1.20230828-h5eee18b_0
  libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
  libiconv           pkgs/main/linux-64::libiconv-1.16-h7f8727e_2
  libnghttp2         pkgs/main/linux-64::libnghttp2-1.52.0-ha637b67_1
  libsolv            pkgs/main/linux-64::libsolv-0.7.24-he621ea3_0
  libssh2            pkgs/main/linux-64::libssh2-1.10.0-h37d81fd_2
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
  libxml2            pkgs/main/linux-64::libxml2-2.9.14-h74e7548_0
  lz4-c              pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0
  mamba              conda-forge/linux-64::mamba-0.17.0-py39h951de11_0
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
  openssl            pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0
  pcre2              pkgs/main/linux-64::pcre2-10.42-hebb0a14_0
  pip                pkgs/main/linux-64::pip-23.3.1-py39h06a4308_0
  pycosat            pkgs/main/linux-64::pycosat-0.6.6-py39h5eee18b_0
  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
  pyopenssl          pkgs/main/linux-64::pyopenssl-23.2.0-py39h06a4308_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.18-h7a1cb2a_0
  python_abi         conda-forge/linux-64::python_abi-3.9-2_cp39
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0
  reproc             pkgs/main/linux-64::reproc-14.2.4-h295c915_1
  reproc-cpp         pkgs/main/linux-64::reproc-cpp-14.2.4-h295c915_1
  requests           pkgs/main/linux-64::requests-2.31.0-py39h06a4308_1
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.100-py39h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-68.2.2-py39h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0
  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0
  urllib3            pkgs/main/linux-64::urllib3-2.1.0-py39h06a4308_1
  wheel              pkgs/main/linux-64::wheel-0.41.2-py39h06a4308_0
  xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_0
  yaml               pkgs/main/linux-64::yaml-0.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0
  zstandard          pkgs/main/linux-64::zstandard-0.19.0-py39h5eee18b_0
  zstd               pkgs/main/linux-64::zstd-1.5.5-hc292b87_0

Proceed ([y]/n)?

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(variant_detective) u0002316@gbw-s-pacbio01:~ $ mamba install -c bioconda -c conda-forge -c charronp variantdetective

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.17.0) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████

Looking for: ['variantdetective']

charronp/linux-64        [====================] (00m:00s) Done
r/linux-64               [====================] (00m:00s) Done
anaconda/noarch          [====================] (00m:00s) Done
pkgs/main/noarch         [====================] (00m:00s) Done
bioconda/noarch          [====================] (00m:01s) Done
bioconda/linux-64        [====================] (00m:01s) Done
r/noarch                 [====================] (00m:00s) Done
charronp/noarch          [====================] (00m:00s) Done
anaconda/linux-64        [====================] (00m:00s) Done
pkgs/r/noarch            [====================] (00m:00s) Done
conda-forge/noarch       [====================] (00m:03s) Done
pkgs/r/linux-64          [====================] (00m:00s) Done
pkgs/main/linux-64       [====================] (00m:01s) Done
conda-forge/linux-64     [====================] (00m:07s) Done

Pinned packages:
  - python 3.9.*

Encountered problems while solving:
  - nothing provides openssl >=1.1.1,<1.1.2.0a0 needed by python-3.6.7-h0371630_0

Thanks for trying my suggestions. Here are a couple comments:

I first had to install mamba inside the created env as I do not have it in my base (installs a LOT of extra packages with itself).

From the Mamba documentation, installing Mamba into any other environment than base is not supported. As you mentioned, mamba ends up installing many libraries and these would cause conflicts with the VariantDetective dependencies if installed in the same environment, If we ignore Mamba's recommendations about installing outside of base and you want to use a dedicated mamba environment, I was able to install VariantDetective using the following steps:
# Create a mamba environment
conda create -n mamba mamba
conda activate mamba
# From inside this environment, create a VariantDetective environment
mamba create -n variantdetective -c bioconda -c conda-forge -c charronp variantdetective
conda activate variantdetective

Please try that out and let me know if that solves the issues.

Could python 3.6 be required for your package to work?

Yes, due to some tools needing that specific version of Python, the pipeline requires v3.6 currently.

I will not attempt installing from source as my server does not allow it and wait until the conda install process gets revised and works to try again.

If the server allows installing Python modules via pip, then the source installation is exactly that. However, you would know the restrictions of the server and I recommend following any steps that are allowed by your group.

Or else, if you create a working docker, I will be happy to give it a try.

That is something we can look into for the future, thanks for the recommendation.

authors: peterk87 philcharron-cfia splaisan

repo: OLF-Bioinformatics/VariantDetective


text:
🛑 Staging is down

In 6117aa7, Staging (https://staging.one.org/international) was down:

HTTP code: 403
Response time: 274 ms

Resolved: Staging is back up in ed75773.