text:
Autocomplete snippets generated by emmet-ls

I don't get emmet snippets from https://github.com/aca/emmet-ls while typing in (for example) css file, but when I press Tab on string that should generate snippet menu with emmet option pops up (snippet is not expanded in menu as it is with Vetur). But I am then unable to select and expand it with enter.
Please see here:
https://user-images.githubusercontent.com/69750637/108285825-3eead380-7188-11eb-8b15-f0eb6f708fae.mov
Any idea what might be wrong here? emmet snippets provided by Vetur in Vue file do work as expected.
Configuration:
compe:
local t = function(str)
  return vim.api.nvim_replace_termcodes(str, true, true, true)
end
local map = vim.api.nvim_set_keymap

require "compe".setup {
  preselect = "always",
  source = {
    path = true,
    buffer = true,
    vsnip = true,
    nvim_lsp = true,
    spell = {filetypes = {"mail", "gitcommit"}}
  }
}

-- Various key mappings.
map("i", "<C-e>", "compe#close('<C-e>')", {silent = true, expr = true})
map("i", "<C-f>", "compe#scroll({delta = +4})", {silent = true, expr = true})
map("i", "<C-d>", "compe#scroll({delta = -4})", {silent = true, expr = true})

-- Use (s-)tab to:
--- move to prev/next item in completion menu
--- jump to prev/next snippet field
local check_back_space = function()
  local col = vim.fn.col(".") - 1
  if col == 0 or vim.fn.getline("."):sub(col, col):match("%s") then
    return true
  else
    return false
  end
end

_G.tab_complete = function()
  if vim.fn.pumvisible() == 1 then
    return t "<C-n>"
  elseif vim.fn.call("vsnip#available", {1}) == 1 then
    return t "<Plug>(vsnip-expand-or-jump)"
  elseif check_back_space() then
    return t "<Tab>"
  else
    return vim.fn["compe#complete"]()
  end
end
_G.s_tab_complete = function()
  if vim.fn.pumvisible() == 1 then
    return t "<C-p>"
  elseif vim.fn.call("vsnip#jumpable", {-1}) == 1 then
    return t "<Plug>(vsnip-jump-prev)"
  else
    return t "<S-Tab>"
  end
end

map("i", "<Tab>", "v:lua.tab_complete()", {expr = true})
map("s", "<Tab>", "v:lua.tab_complete()", {expr = true})
map("i", "<S-Tab>", "v:lua.s_tab_complete()", {expr = true})
map("s", "<S-Tab>", "v:lua.s_tab_complete()", {expr = true})

-- Work with nvim-autopairs.
local npairs = require("nvim-autopairs")

OnEnter = function()
  if vim.fn.pumvisible() == 1 then
    return vim.fn["compe#confirm"]()
  else
    return npairs.check_break_line_char()
  end
end

map("i", "<CR>", "v:lua.OnEnter()", {expr = true, noremap = true})

lspconfig:
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities.textDocument.completion.completionItem.snippetSupport = true

local configs = require "lspconfig/configs"

configs.emmet_ls = {
  default_config = {
    cmd = {"emmet-ls", "--stdio"},
    filetypes = {"html", "css", "scss"},
    root_dir = function()
      return vim.loop.cwd()
    end,
    settings = {}
  }
}

lspconfig.emmet_ls.setup {
  capabilities = capabilities
}

Great thanks!

I don't know if emmet-ls supports VSCode behavior or not.
If emmet-ls supports VSCode's completion behavior, I want to support emmet-ls on nvim-compe.

@hrsh7th Thanks for coming back to me. I am not sure what you mean by VSCode behaviour and whether emmet-ls supports it, but codebase seems rather simple and it's based on vscode-languageserver (https://github.com/aca/emmet-ls/blob/a6c80782fc50c884e50cb8449d3d1842db2f0ef1/src/server.ts#L11). How can I help?

The LSP completion is really complex because coc.nvim / VSCode / and other completion engine does not behave as the same.
If emmet-ls can work as expected on VSCode, I want to support it on compe.
But if emmet-ls is specialized in the completion-nvim, I can't support it.

I would like to raise an issue with emmet-ls, just not sure what to ask for exactly.

Okay, I will look in to how to work compe with emmet-ls.

https://github.com/windwp/emmet-ls
this is not a problem of nvim-compe.
@cloggier  you can test my fork . snippet completion is working now but the cursor is not correct and a indent is always space :)
I don't know how to fix a cursor issue because the library emmet on npm is not supported that.

Can I use npm to install your version?
And so many thanks for looking into it. You could have just said it is not an issue with compe and leave it there. Your commitment and support you provide to the community is outstanding and I can't thank you enough!

remove the old global package emmet-ls then in directory of my fork run npm run build  and npm link

Thank you for your work @windwp

@hrsh7th  hi i need a little help about vim-vsnip
if i have a snippet like this

\n\t
${1}
\n\t
${2}
\n\t
${3}
\n\t
${4}
\n

${1} ${2} is a cursor format
i want to use vim-vsnip feature to jump to next cursor is that possible?

It works so much better now, thanks!
Couple of issues that actually might be compe related, so will stick them here.

compe is not always opening completion menu and I have to press tab (compe#complete()) to show, it then works fine for most cases.

compe will not open completion menu at all for this example snippet from documentation: p!+m10e!; opens fine, but not automatically, if I remove last !.

Pressing <backspace> in typed in snippet does not update completion menu item and its preview. So if you type m10+p10 (css snippet) it will correctly show in completion menu and expand in preview window, but then deleting it so it shows only m10 does not update what compe shows. I have to cancel completion and trigger it again to show correctly.

Can't get compe to trigger for ! snippet (not opening completion menu at all).

tr+ is not showing at all (tr is OK).

[not compe problem?] html:4t wants to expand to  instead proper html skeleton (see here https://docs.emmet.io/cheat-sheet/).

It almost works perfectly though!

@windwp Do you have a mapping that vim-vsnip's README.md?
https://github.com/hrsh7th/vim-vsnip#2-setting

Another issue I just discovered is:

that emmet-ls' completion item will overshadow those of css lsp, like shown below:

With emmet:

and without:

@hrsh7th It is a snippet from lsp.

that emmet-ls' completion item will overshadow those of css lsp, like shown below:

I will tacke it maybe (I think it is compe's problem)

@windwp I think this is language-server's problem. I guess the reason is that language server would not set insertTextFormat = 2 to CompletionItem.

You can set insertTextFormat in https://github.com/aca/emmet-ls/blob/a6c80782fc50c884e50cb8449d3d1842db2f0ef1/src/server.ts#L146

@hrsh7th  Thank it is working now. :heart:

I believe few of my concerns listed in few posts above are still relevant. Could check them later.

@windwp fork works 'kind of' for me but it still working really poorly.
I do get snippet completions sometimes.. but mostly not.

My compe setup is
require'compe'.setup {
  enabled = true;
  autocomplete = true;
  debug = false;
  min_length = 1;
  preselect = 'enable';
  throttle_time = 80;
  source_timeout = 200;
  resolve_timeout = 800;
  incomplete_delay = 400;
  max_abbr_width = 100;
  max_kind_width = 100;
  max_menu_width = 100;
  documentation = {
    border = { '', '' ,'', ' ', '', '', '', ' ' }, -- the border option is the same as `|help nvim_open_win|`
    winhighlight = "NormalFloat:CompeDocumentation,FloatBorder:CompeDocumentationBorder",
    max_width = 120,
    min_width = 60,
    max_height = math.floor(vim.o.lines * 0.3),
    min_height = 1,
  };

  source = {
    path = true;
    buffer = true;
    calc = true;
    nvim_lsp = true;
    nvim_lua = true;
    vsnip = true;
    ultisnips = true;
    luasnip = true;
  };
}

and emmet-ls setup
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities.textDocument.completion.completionItem.snippetSupport = true

if not lspconfig.emmet_ls then
  configs.emmet_ls = {
    default_config = {
      cmd = {'emmet-ls', '--stdio'};
      filetypes = {'html', 'css'};
      root_dir = function(fname)
        return vim.loop.cwd()
      end;
      settings = {};
    };
  }
end

lspconfig.emmet_ls.setup{
  capabilities = capabilities,
}

Are others getting similar behavior as me or are there any known fixes?

my fork is already merged so you can install it by npm.
i don't use that server now but if the complete doesn't appear so you can force it appear by ctrl+space

Thank you for your work. I have the same problem visible above. A space is added all the time before and after the tag. What can I do so that these extra spaces do not add up?

authors: JonasRSV cloggier hrsh7th mGuerino windwp


text:
Still get a complete entry from the preview

Describe the bug
I can still complete from the previous preview, after I choose another entry.
To Reproduce
  vim.o.completeopt = "menuone,noselect"
  vim.cmd("set pumheight=15")
  vim.cmd("set shortmess+=c")

  require "compe".setup {
    enabled = true,
    autocomplete = true,
    debug = false,
    min_length = 1,
    preselect = "always",
    throttle_time = 80,
    source_timeout = 200,
    incomplete_delay = 400,
    max_abbr_width = 100,
    max_kind_width = 100,
    max_menu_width = 100,
    documentation = true,
    source = {
      path = true,
      buffer = true,
      calc = true,
      nvim_lsp = true,
      nvim_lua = true,
      vsnip = true
    }
  }

Screenshots (optional)

first key

second key

If I select composer, the complete entry that will appear is from vsnip

Sorry. I can't reproduce it...

It's ok, I downgrade to version 2

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
Mistake in modules slide?

In the following slide, import export module base is probably incorrect.
(I don't find any "import export" on https://en.cppreference.com/w/cpp/language/modules.)

Presumably, the correct syntax is import export base, but needs to be checked.

Yeah, this looks wrong. It should probably be export import base;.
Generally, this example looks very complicated. It may benefit from a little overhaul. I will have a look in a few days.

authors: bernhardmgruber hageboeck

repo: hsf-training/cpluspluscourse


text:
html.tag() should accept multiple sets of attributes, and merge

To give some context, here are a bunch of existing patterns to do with html.tag, i.e. found in a wide variety of content functions' generate code:
html.tag('div',
  {id: 'foo', class: 'bar'},
  html.tag(...one descendant tag...))

html.tag('div', {
  [html.someSpecialAttribute]: 'something or other',
  longAttribute: longAttributeValue,
  longAttribute: longAttributeValue,
  longAttribute: longAttributeValue,
}, html.tag(...one descendant tag...))

html.tag('div', {
  longAttributeLine: blablabla,
  longAttributeLine: blablabla,
  longAttributeLine: blablabla,
}, [
  descendant1,
  descendant2,
  descendant3,
])

These are all mostly fine patterns, so they're basically what most code tries to conform to - more on that in a second. The middle one, though, does have an awkwardness: its descendant tag is badly aligned (three characters instead of two, respective to tab), so further nested tags are basically screwed. Usually its attributes get moved to a separate line, giving the tag a whole lot of not so nice padding / empty space:
html.tag('div',
  {
    [html.someSpecialAttribute]: 'something or other',
    longAttribute: longAttributeValue,
    longAttribute: longAttributeValue,
    longAttribute: longAttributeValue,
  },
  html.tag(...one descendant tag...,
    ...with multiple lines...))

When we have only one (common) attribute, we usually treat it like an inline item, which gives another convenient way to avoid syntax nastiness:
html.tag('div', {id: 'my-important-element'},
  html.tag(...one descendant tag...))

html.tag('div', {class: 'recurring'}, [
  descendant1,
  descendant2,
  descendant3,
])

Good and well. Small attributes work nicely with small or big descendants, and big attributes work nicely with big attributes. Big attributes are okay with really small descendants, though it isn't great when you have a single descendant that's multiline.
Okay, so syntax aside for a moment, here are some common behavioral patterns:
html.tag('a', {
  href: to('localized.bimBam', data.directory, 'bom'),
  class: slots.class,
}, linkContent)

html.tag('foo', {
  ...slots.attributes,
  class: ['my-recurring-class', slots.class],
    /* or WORSE... */
  class: ['my-recurring-class', slots.attributes?.class],
})

Yikes. So these are basically circumstances around combining attributes from multiple different sources.
Both examples have problems. For the first one: This seemingly works well to provide any class to the element, including, for example, multiple classes. But it doesn't handle accepting any other attributes (think: dataset, id, style), and it requires hard-coding a class slot onto the content function - even though class isn't really any more special or innately meaningful than other potentially uesful attributes.
And the second: Now we're accepting any set of attributes, in principle, but we're either 1) using two slots to do it (and ignoring class on the ostensibly generic attributes slot!), or 2) inspecting the live value of a slot, which is always an antipattern.
So what's the proposal?
Accept multiple sets of attributes in html.tag!
This is primarily a solution to the behavior end, so compare:
html.tag('a',
  {href: to('localized.bimBam', data.directory, 'bom')},
  slots.attributes,
  linkContent)

html.tag('foo',
  {class: 'my-recurring-class'},
  slots.attributes)

Wow cool. That is both easier to read and solves all the behavioral problems.
Or does it? Yes, really! The magic is that Attributes will automatically and intelligently combine attributes. Which is to say, attributes that accept a set of simultaneous independent values will be automatically combined - class and style are the ones that come to mind, but maybe there are more.
So in the second example, if slots.attributes includes a class attribute, it'll automatically get combined with 'my-recurring-class' (even though that original attribute value was provided as a string rather than an array). Whereas if the first example's slots.attributes includes href, that'll overwrite the previous value, because there's no such thing as a link that leads to two places (sorry).
That's basically arbitrary behavior, but it applies only for very select attributes, and overriding is generally better than e.g. just doing nothing. We could add an [html.overrideAttributes]: 'always' option - but it's probably better to save that for later, after having using combining attributes a lot more and getting a better sense for it.
How about syntax niceness? Well, it helps with building up attributes gradually, i.e. with variables defined earlier in the generate function:
html.tag('span', {class: 'cool-span'},
  attributesOnSpan,
  html.tag('img', {class: 'cool-img'},
    attributesOnImg))

It allows for passing variously-shaped attributes separate from each other:
html.tag('blockquote', {class: 'everyday-blockquote'},
  {[html.joinChildren]: html.tag('br')},
  {[html.edgeWhitespace]: 'never'},
  {'data-album-directory': data.albumDirectory},
  relations.colorAttributes
    .slots({context: 'content', color: data.color}),
  slots.attributes,
  relations.content)

That's a big deal for git-style line-based composition, and for making syntax less dependant on behavior (or, for goodness sake, vice versa!).
Building on the last one, it allows for grouping of directly related attributes:
html.tag('p', {class: 'foo'},
  ({[html.joinChildren]: html.tag('br'),
    [html.edgeWhitespace]: 'never'}),
  ({'data-album-directory': data.albumDirectory,
    'data-track-directory' data.trackdirectory}),
  relations.content)

It allows for cleaner conditional application of multiple sets of attributes, as well as multiple attributes at once:
html.tag('span',
  slots.mode === 'inline' &&
    {class: 'something-or-other-inline'},
  slots.mode === 'blammo' &&
    {class: 'holy-beans', id: 'blammo-zone'},
  descendants)

// compare before:
html.tag('span', {
  class:
    (slots.mode === 'inline'
      ? 'something-or-other-inline'
   : slots.mode === 'blammo'   /* <- will be duped */
      ? 'holy-beans'
      : null),
    /* inline case-like evaluation is appropriate for some situations,
     * but not for such a simple example where a different framing
     * expresses related attributes near each other AND deduplicates
     * the common condition! */

  id:
    (slots.mode === 'blammo'  /* <- dupe */
      ? 'blammo-zone'
      : null),
    /* or: */
  id: slots.mode === 'blammo' && 'blammo-zone',
   /* ^ dupe */
}, descendants)

Nice.
There are possibly more use cases, but it can be explored more when it's actually put into practice.

See #331: We should take care to implement merging in a generic, top-down manner - basically like the syntax for Object.assign. This should be separate from the main tag/attributes constructor, so it's easier to compose with a separate "only combine certain sets of attributes at once" behavior later on.

repo: hsmusic/hsmusic-wiki


text:
Get rid of the length printed with the AMBF origin axes

Appears by default in 3D view

Fixed in #12


text:
Setting client parameters

Is there a way to set the underlying client parameters?  I am using the Apache client, and I need to set "http.protocol.handle-redirects".  But I'm not sure how I can do that.

Assuming this can be done on an instance of the client, I just added a feature to allow client-specific configuration, but it has not been released yet. If't checked into the development branch if you want to try it out - let us know if this doesn't do what you want it to. We should be cutting a release in a week or two.

That looks brilliant - I was vaguely thinking of something like this late last night while writing horrible reflection code to access private variables!  :-)
I'd like to  ask for one more piece: could you do exactly the same thing in the "verb" blocks?  Specifically, before client.execute(request, new Handler(requestConfig), context(requestConfig)) in ApacheHttpConfig.exec().
The reason is, for my crazy use case (I am testing my OAuth2 flow - for some requests I want to follow redirects, for others I don't), I need to set redirectsEnabled on a request-by-request basis (I reuse the same HttpBuilder instance for the entire test).
So we would end up with code like
ApacheHttpBuilder.configure {
    request.uri = 'blah'
}.get {
    client.clientCustomizer {  HttpRequestBase client ->
        RequestConfig.Builder requestBuilder = RequestConfig.custom()
        requestBuilder.setRedirectsEnabled(false)
        client.setConfig(requestBuilder.build())
    }
}

Hmm... not sure we can add client configuration at the request level without a lot of refactoring since the client is configured at the global level and shared across requests (remember it also needs to be supported across three different client implementations as well).
Could you get the same effect with two different HttpBuilder instances (one following redirects and one not)?
Lastly, if you have any good suggestions for our OAuth support, feel free to add them to that issue - it's been just hanging around since neither of us really do much OAuth right now.

I see what you mean - Client is only available on HttpObjectConfig, not in the superclass.  You'd have to move the inner Client class up to HttpConfig.
But I'm pretty sure I could get it done with two instances as you suggest.
Looking forward to this release!
As for OAuth, I'll take a look at the ticket.  This work originated as I was trying to migrate from http-builder to http-builder-ng, and it was my OAuth test classes from long ago that broke horribly.

I am going to close this - I will be cutting a release (0.17.0) by Monday - keep an eye on our twitter feed for notification.

authors: cjstehno kenwdelong

repo: http-builder-ng/http-builder-ng

license_source: bigquery


text:
Update sbt to 1.5.7

Updates org.scala-sbt:sbt from 1.5.6 to 1.5.7.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Codecov Report

Merging #168 (a7c51df) into master (b4cde5f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #168   +/-   ##
=======================================
  Coverage   98.27%   98.27%
=======================================
  Files           1        1
  Lines          58       58
  Branches        2        2
=======================================
  Hits           57       57
  Misses          1        1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4cde5f...a7c51df. Read the comment docs.

source: gharchive/pull-request

authors: codecov-commenter scala-steward


text:
Random OpenSSL / Headers parsing error

Hi,
We have a private project that sync Shopify e-commerce with another inventory system. We have multiple shops that sync multiple times a day meaning our platform likely does over 100K of calls every day to shopify's api.
We used to use this gem until now because over the year we have been getting these kind of errors randomly

Usage was
HTTP.basic_auth(user: api_key, pass: api_password).get("url", { params: { page: 123 } })

I had no idea how to fix those errors even with hours of googling. I went ahead and replaced the gem with HTTParty as an attempt and to my surprise I am no longer getting random errors.
Here is a diff of my changes that fixed the error in case it helps

I'm leaving this here not really sure if this is of any help but I've been having those errors for at least the past year and I update all my gems monthly.
PS: I would really rather stick to HTTP.rb because of it's API. The developers have done a wonderful job at it
PS 2: The server is a Ubuntu Server 20.04 LTS and the app is running inside a Ruby 3.1.0 Docker image (using Dokku)

A “TLS write client hello” error indicates the remote end is closing the socket before the client has finished sending the very first message in the TLS handshake.
This a low-level error originating in OpenSSL that can have a multitude of causes, everything from network errors to incompatible TLS configurations between client and server.
Is it sporadic? Is there an accompanying error on the server side you can share?

A “TLS write client hello” error indicates the remote end is closing the socket before the client has finished sending the very first message in the TLS handshake.
This a low-level error originating in OpenSSL that can have a multitude of causes, everything from network errors to incompatible TLS configurations between client and server.
Is it sporadic? Is there an accompanying error on the server side you can share?

Yes it is sporadic / random. Both exceptions are. I wish I could provide more information. All I have is the error shown above. There is no log on the Shopify side. All I can tell is that it's mostly while fetching the products (it's paginated so pulling all of them can take a lot of calls) https://shopify.dev/api/admin-rest/2022-07/resources/product#get-products

The first error is happening inside OpenSSL long before any HTTP interaction takes place.
I’m not sure why it would be working with HTTParty, which is just a Net::HTTP wrapper last I checked. Perhaps it has a different OpenSSL configuration?

The first error is happening inside OpenSSL long before any HTTP interaction takes place.
I’m not sure why it would be working with HTTParty, which is just a Net::HTTP wrapper last I checked. Perhaps it has a different OpenSSL configuration?

I admit the comparison between llhttp and Net::HTTP is above my knowledge, I wouldn't know. All I can tell is when it comes to gem usage, I did not customize anything more than you see on the screenshot above. I believe it is safe to say i'm using the default settings when it comes to HTTP.rb and HTTParty

Again, in the case of the first error, it’s occurring inside the Ruby OpenSSL extension, used by both libraries. The HTTP parser is irrelevant in that case.

Again, in the case of the first error, it’s occurring inside the Ruby OpenSSL extension, used by both libraries. The HTTP parser is irrelevant in that case.

Right. Then I wouldn't know because the only change I have done is the gem replacement and the diff is in the first post

If you can ask someone what the server-side TLS configuration is, that might be helpful

I could try asking in their forum https://community.shopify.com/c/shopify-community/ct-p/en
Is there something more specific you want me to ask?

What TLS versions and ciphersuites are supported, and ideally the client-side configuration they would prefer.
As far as I can tell this is a server-side issue, although it may be impacted by the local client-side TLS configuration.

I opened up this https://community.shopify.com/c/technical-q-a/rest-api-random-ssl-errors-and-header-parsing/m-p/1789546#M108546

What TLS versions and ciphersuites are supported, and ideally the client-side configuration they would prefer.
As far as I can tell this is a server-side issue, although it may be impacted by the local client-side TLS configuration.
If you can report this issue upstream somewhere, that'd be great.

We started to get randomly the same error again but this time with HTTParty
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 peeraddr=23.227.38.74:443 state=SSLv3/TLS write client hello
/app/vendor/bundle/ruby/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock'
/app/vendor/bundle/ruby/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `ssl_socket_connect'
/app/vendor/ruby-3.1.0/lib/ruby/3.1.0/net/http.rb:1048:in `connect'
/app/vendor/ruby-3.1.0/lib/ruby/3.1.0/net/http.rb:976:in `do_start'
/app/vendor/ruby-3.1.0/lib/ruby/3.1.0/net/http.rb:965:in `start'
/app/vendor/ruby-3.1.0/lib/ruby/3.1.0/net/http.rb:1530:in `request'
/app/vendor/bundle/ruby/3.1.0/gems/httparty-0.20.0/lib/httparty/request.rb:152:in `perform'
/app/vendor/bundle/ruby/3.1.0/gems/httparty-0.20.0/lib/httparty.rb:613:in `perform_request'
/app/vendor/bundle/ruby/3.1.0/gems/httparty-0.20.0/lib/httparty.rb:527:in `get'
/app/vendor/bundle/ruby/3.1.0/gems/httparty-0.20.0/lib/httparty.rb:646:in `get'
/app/lib/shopify/api.rb:65:in `next_page'
/app/lib/shopify/api.rb:59:in `get_paginated'
/app/lib/shopify/product.rb:8:in `list'

I guess we can close the issue since it ain't gem specific. However I wish there was something we could have done

Yeah, as I said earlier this seems like a low-level OpenSSL error.
It's kind of surprising you're encountering this with Shopify though, since they're heavy Ruby users.
I'd suggest continuing to pursue it with them, especially if you can reproduce it across multiple HTTP clients.

I can confirm we are seeing the same thing sporadically with the Shopify API.

authors: james-em schappim tarcieri

repo: httprb/http

license: mit


text:
Update javascript assert example

Add js directive

Thanks for your quick updates!


text:
Updating stat_nova module and added some new misc functions

Updating stat_nova module and added some new misc functions

@basepi @fossam FYI... Bugs have been opened at https://git.corp.adobe.com/spartans/hubble/issues to track all discussed issues.

@anuragpaliwal80 Any reason you're opening issues in your fork instead of in the main hubble repo?

@mew1033 I love to open tickets in main hubble repo but I thought it might not be good to open security related issues public ally or opened on our internal site.
And accidentally opened all issues of this PR internally, few of them could be opened on githuub as well. Will keep that in mind.

@anuragpaliwal80 I just realized that the tickets you opened were in git.corp, not public github. Sorry about the confusion!

repo: hubblestack/hubble


text:
consumer testcases or examples needed!

please add some consumer testcases or examples. I try to test it but failed to get message from kafka!

Well, I managed to try it out. Here is the working example.
KafkaManualConsumer.java

Thank you for the feedback and test case we will add it as soon as possible

it does work with kafka 0.10.0.1

@edwardsayer when you are still using this, can you update your example and add it to the README.md?

authors: edwardsayer eemmiirr marcust piyushwadhwani

repo: hubrick/vertx-kafka-service


text:
OnGuideChangedListener 会回调两次

OnGuideChangedListener 会回调两次

在activity  中不会 在 fragment 会出现

repo: huburt-Hu/NewbieGuide


text:
🛑 Akkoma is down

In cf9c0c0, Akkoma (https://akkoma.chez.chat) was down:

HTTP code: 404
Response time: 157 ms

Resolved: Akkoma is back up in 48c4951 after 6 minutes.

repo: hufman/upptime


text:
🛑 VGMdb API is down

In 1adda53, VGMdb API (https://vgmdb.info) was down:

HTTP code: 0
Response time: 0 ms

Resolved: VGMdb API is back up in 540bad6 after 4 minutes.


text:
🛑 Akkoma is down

In 50726dc, Akkoma (https://akkoma.chez.chat) was down:

HTTP code: 404
Response time: 71 ms

Resolved: Akkoma is back up in a7a0a6e after 12 minutes.


text:
add checkstyle plugin

Add checkstyle plugin

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.yuanbingze seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

I had signed CLA

I have read the CLA Document and I hereby sign the CLA

hugegraph-comman与dolphinscheduler中的checkstyle对比一、插件版本  hugegraph-common         3.1.0dolphinscheduler              3.1.2建议使用3.1.2。两个版本配置上稍微有点区别，hugegraph-common中的checkstyle.xml不能直接在高版本中使用。需要调整部分model的依赖级别二、注释方面：hugegraph-common  采用了大量中文注释，虽然便于国内社区人员理解，但是会影响hugegraph在全球传播。dolphinscheduler 注释误解较多，应该是某位同学大量复制并未仔细检查造成的。建议不使用注释，或者注释全部参考官方注释，保证无歧义  https://checkstyle.sourceforge.io/三、具体参数对比

父模块名称
子模块名称
hugegraph版本
dolphinscher版本
说明
选用标准

ROOT
FileTabCharacter
eachLine:true
eachLine:true

一致

不同版本中，父模块不同
LineLength
最大长度：80ignorePattern：^package.|^import.|a href|href|http://|https://|ftp://
最大长度：200ignorePattern：^ ** *[^ ]+$

dolphinscheduler

ROOT
RegexpSingleline

format:System.out.println
根据正则表达式检查单行是否有不良操作

TreeWalker
PackageName
format:^[a-z]+(.[a-z][a-z0-9])$
format:^[a-z]+(.[a-z][a-z0-9])
检查包名称是否遵守命名规约
一致

TreeWalker
ImportOrder

staticGroups:org.apache.dolphinscheduler,org.apache,java,javax,org,comseparatedStaticGroups:truegroups:org.apache.dolphinscheduler,org.apache,java,javax,org,comordered:trueseparated:trueoption:topsortStaticImportsAlphabetically:true
定义导入包的顺序

TreeWalker
AvoidStarImport

检查没有import语句使用*号
一致

TreeWalker
RedundantImport

检查是否存在多余的import语句，比如重复的，java自带的包，相同包下的其他类
一致

TreeWalker
UnusedImports

检查是否存在没有使用的import语句
一致

TreeWalker
EmptyLineSeparator

allowMultipleEmptyLines:falseallowMultipleEmptyLinesInsideClassMembers:falsetokens:PACKAGE_DEF, IMPORT, STATIC_IMPORT, CLASS_DEF,INTERFACE_DEF, ENUM_DEF, STATIC_INIT, INSTANCE_INIT, METHOD_DEF,CTOR_DEF
检查空行是否符合要求。案例https://checkstyle.sourceforge.io/config_whitespace.html#EmptyLineSeparator

TreeWalker
ModifierOrder

检查修饰符是否符合Java建议，顺序是：public、protected、private、abstract、default、static、final、transient、volatile、synchronized、native、strictfp
一致

TreeWalker
TypeName

检查类名称是否符合规范

TreeWalker
ClassTypeParameterName

format:(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)
检查泛型类类型参数是否符合规范

TreeWalker
ConstantName
format:^[A-Z][A-Z0-9](_[A-Z0-9]+)$

检查常量（用static final修饰的字段）的名称是否遵守命名规约

TreeWalker
LocalVariableName
format:^[a-z][a-zA-Z0-9_]*$
tokens:VARIABLE_DEFformat:^a-z?$
检查局部变量的名称是否遵守命名规约
dolphinscheduler

TreeWalker
MemberName
format:^[a-z][a-zA-Z0-9]*$
format:^[a-z][a-zA-Z0-9]*$
检查成员变量（非静态字段）的名称是否遵守命名规约
一致

TreeWalker
ParameterName
format:^[a-z][a-zA-Z0-9]*$
format:^a-z?$
检查参数名称是否遵守命名规约
一致

TreeWalker
CatchParameterName

format:^a-z?$
检查catch参数名是否符合要求

TreeWalker
ArrayTypeStyle
javaStyle:true

检查数组是否属于java风格，方括号放在类型后面，而不是变量后面，比如：int[] nums（合法），int nums[]（不合法）
一致

TreeWalker
MethodTypeParameterName
默认  "^[A-Z]$"
format:(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)
检查方法类型是否遵守命名规约
dolphinscheduler

TreeWalker
MethodName
format:^[a-z][a-zA-Z0-9]*$
format:^[a-z][a-z0-9][a-zA-Z0-9_]*$
检查方法名称是否遵守命名规约
hugegraph

TreeWalker
InterfaceTypeParameterName

format:(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)

dolphinscheduler

TreeWalker
EqualsHashCode

覆盖equals()方法的类必须也覆盖了hashCode()方法
hugegraph

TreeWalker
MissingSwitchDefault

switch语句必须含有default子句
hugegraph

TreeWalker
DefaultComesLast

switch语句的default必须放在所有的case分支之后
hugegraph

TreeWalker
NoFinalizer

验证类中是否定义了finalize()方法。
dolphinscheduler

TreeWalker
GenericWhitespace

检查范型标记<>的周围的空格是否遵守标准规约https://checkstyle.sourceforge.io/config_whitespace.html#GenericWhitespace
dolphinscheduler

TreeWalker
Indentation

basicOffset:4braceAdjustment:0caseIndent:4throwsIndent:2lineWrappingIndentation:4arrayInitIndent:4
检查Java代码的缩进是否正确。
dolphinscheduler

TreeWalker
UpperEll

long类型的字面量如果要以"L"结尾，必须是大写的"L"，而非小写的"l"
一致

TreeWalker
IllegalImport

regexp:trueillegalPkgs:^com.google.api.client.repackaged,                      ^avro.shaded, ^org.apache.hadoop.hbase.shaded,                      ^org.apache.hadoop.shaded,                      ^javax.ws.rs.ext,                      ^cc.concurrent.mango.util.concurrent,                      ^org.apache.curator-test.shaded,                      ^com.sun.istack,                      ^org.jetbrains.annotations,                      ^jline.internal,                      ^com.cronutils.utils,                      ^javax.ws.rs.ext,                      ^org.jboss.netty.util.internal,                      ^com.sun.javafx,                      ^io.reactivex.annotations,                      ^org.codehaus.jacksonillegalClasses:^java.util.logging.Logging,                      ^sun.misc.BASE64Encoder,                      ^sun.misc.BASE64Decoder,                      ^jdk.internal.jline.internal.Nullable
检查是否有非法导入的包与类
dolphinscheduler

TreeWalker
NoWhitespaceBefore

tokens:COMMA, SEMI, POST_INC, POST_DEC, DOT, ELLIPSIS, METHOD_REFallowLineBreaks:true
检查指定标记之前没有空格。
dolphinscheduler

TreeWalker
ParenPad

左圆括号之后和右圆括号之前是否需要有一个空格，不需要
一致

TreeWalker
OperatorWrap
option:eol
option:NLtokens:BAND, BOR, BSR, BXOR, DIV, EQUAL, GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD, NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR, METHOD_REF
代码换行时，运算符位置，比如：+、&&、?、: 等dolphinscheduler要求在下一行行首hugegraph 要求在本行末尾
dolphinscheduler

TreeWalker
EmptyStatement

severity:error
代码中不允许有空语句，也就是单独的;符号
hugegraph

TreeWalker
AnnotationLocation

allowSamelineMultipleAnnotations:falseallowSamelineSingleParameterlessAnnotation:falseallowSamelineParameterizedAnnotation:truetokens:METHOD_DEF, CTOR_DEF
检查注解位置是否符合规范
dolphinscheduler

TreeWalker
EmptyCatchBlock

exceptionVariableName:expected
检查空的catch块。
dolphinscheduler

TreeWalker
CommentsIndentation

检查缩进与代码是否一致
dolphinscheduler

TreeWalker
NoLineWrap

severity:error
检查（默认import & package）是否换行
dolphinscheduler去除（severity:error）

TreeWalker
NeedBraces

severity:error
检查是否需要大括号，在if lese等场景使用
dolphinscheduler去除（severity:error）

TreeWalker
LeftCurly
option:eol
severity:error
检查{和左边的代码是否在同行
dolphinscheduler去除（severity:error）

TreeWalker
WhitespaceAround

allowEmptyConstructors:trueallowEmptyMethods:trueallowEmptyTypes:trueallowEmptyLoops:true
检查指定标记的周围是否有空格，比如：if、for、while、synchoronized 等
dolphinscheduler

TreeWalker
AvoidNestedBlocks

allowInSwitchCase:true
检查是否有嵌套块 (允许switch语句中使用)
dolphinscheduler

TreeWalker
IllegalTokenText

tokens:STRING_LITERAL, CHAR_LITERALformat:\u00(09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\(0(10|11|12|14|15|42|47)|134)
检查文本内容是否非法字符
dolphinscheduler

TreeWalker
AvoidEscapedUnicodeCharacters

allowEscapesForControlCharacters:trueallowByTailComment:trueallowNonPrintableEscapes:true
限制使用Unicode escapes
dolphinscheduler

TreeWalker
EmptyBlock

option:TEXTtokens:LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH
检查空块
dolphinscheduler

TreeWalker
OneStatementPerLine

确保每行只有一条语句
dolphinscheduler

TreeWalker
MultipleVariableDeclarations

检查代码段和行中是否有多次变量声明
dolphinscheduler

TreeWalker
MissingSwitchDefault

检测switch语句里是否有默认部分
dolphinscheduler

TreeWalker
FallThrough

检查switch是否缺少break、continue、return、throw
dolphinscheduler

TreeWalker
JavadocStyle

endOfSentenceFormat:
自定义检查以验证javadoc
dolphinscheduler

TreeWalker
JavadocType

scope:protectedallowMissingParamTags:true
检查javadoc的类型
dolphinscheduler

@seagle-yuan Thank you very much, your sorting is very clear.
There are 2 conflicts I think can keep the existing style:

LineLength: Prefer to keep line length 80 chars, because too long code can lead to poor readability. Maybe it can be relaxed to 100 characters later if necessary.
OperatorWrap: The operator location is placed at the end of the line when wrapping line, we consider that when we read the code, it means "it's not over yet, please continue to the next line".

IMO the other 5 conflicts can follow the style of dolphinscheduler.

I have read the CLA Document and I hereby sign the CLA

I have read the CLA Document and I hereby sign the CLA

recheck

I have read the CLA Document and I hereby sign the CLA

authors: CLAassistant javeme seagle-yuan yuanbingze

repo: hugegraph/hugegraph


text:
example/cv_example.py fp8 error

System Info
Hopper GPU

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
RuntimeError: Tensors must be CUDA and dense
Expected behavior
RuntimeError: Tensors must be CUDA and dense

Traceback (most recent call last):
File "./cv_example.py", line 198, in
main()
File "./cv_example.py", line 194, in main
training_function(config, args)
File "./cv_example.py", line 161, in training_function
outputs = model(inputs).to(accelerator.device)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1533, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1129, in forward
self._sync_buffers()
File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1727, in _sync_buffers
self._sync_module_buffers(authoritative_rank)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1731, in _sync_module_buffers
self._default_broadcast_coalesced(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1754, in _default_broadcast_coalesced
self._distributed_broadcast_coalesced(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1668, in _distributed_broadcast_coalesced
dist._broadcast_coalesced(
RuntimeError: Tensors must be CUDA and dense

repo: huggingface/accelerate


text:
Size of the saved model checkpoint after extending pretraining using FSDP is larger than that of the base model.

I am do extended pretraining of a 20B model using FSDP. I observe that the size of model saved after extended pretraining is larger than the base model. This is how I am saving the model.
       unwrapped_model = accelerator.unwrap_model(model)
        state_dict = None
        if os.environ.get("ACCELERATE_USE_FSDP") == "true":
            logger.info(f'using ACCELERATE plugin')
            full_state_dict_config = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
            with FSDP.state_dict_type(unwrapped_model,
                                      StateDictType.FULL_STATE_DICT, full_state_dict_config):
                state_dict = accelerator.get_state_dict(unwrapped_model)
        if accelerator.is_main_process:
            unwrapped_model.save_pretrained(
                output_dir,
                is_main_process=accelerator.is_main_process,
                save_function=accelerator.save,
                state_dict=state_dict,
                safe_serialization=FLAGS.safe_serialization
            )

I am using safetensors for saving the model.
Size of base model on disk:

Size of extended pretrained model on disk:

If you see size of last shard has increased from 1359201312 to 2567185552
Any reason of this behavior ?
Following is my FSDP configuration:
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: NO_PREFETCH
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: GPTBigCodeBlock
machine_rank: 0
main_process_ip: 172.168.12.19
main_process_port: 55930
main_training_function: main
mixed_precision: 'bf16'
num_machines: 10
num_processes: 80
use_cpu: false

I am using accelerate==0.20.3. My transformers and torch version are following:
transformers              4.29.0                   pypi_0    pypi
torch                     2.0.1                    pypi_0    pypi

cc @pacman100

Hello @dineshkh, could you please try the latest released version of Accelerate and let us know if the issue still persists.

@pacman100 I have tried with accelerate==0.22.0 and but still saved model checkpoint size is larger compared to the base model.
Also, with accelerate==0.23.0 my code is running for 2 steps and giving OOM error. The same code is working when I was using accelerate==0.20.3 or accelerate==0.22.0. I am using gradient checkpointing and flash attention v2 via monkey patch provided by you.
    if FLAGS.gradient_checkpointing:
        model.gradient_checkpointing_enable()

@pacman100, @LysandreJik  any update on this ?

Hello, I did the following:

Code issue_1958.py:

from accelerate import Accelerator
import torch
from transformers import AutoModelForCausalLM, get_linear_schedule_with_warmup
from torch.distributed.fsdp import (
    FullyShardedDataParallel as FSDP,
    StateDictType,
    FullStateDictConfig,
)

def main():
    model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
    accelerator = Accelerator()
    model.save_pretrained("delete_me_base", is_main_process=accelerator.is_main_process)
    model = accelerator.prepare(model)
    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)
    lr_scheduler = get_linear_schedule_with_warmup(
        optimizer=optimizer,
        num_warmup_steps=0,
        num_training_steps=1000,
    )
    optimizer, lr_scheduler = accelerator.prepare(optimizer, lr_scheduler)

    full_state_dict_config = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
    with FSDP.state_dict_type(
        model, StateDictType.FULL_STATE_DICT, full_state_dict_config
    ):
        state_dict = accelerator.get_state_dict(model, unwrap=False)

    unwrapped_model = accelerator.unwrap_model(model)
    unwrapped_model.save_pretrained(
        "delete_me",
        is_main_process=accelerator.is_main_process,
        save_function=accelerator.save,
        state_dict=state_dict,
    )
    print("Done saving model")

if __name__ == "__main__":
    main()

Accelerate config accelerate config --config_file issue_1958.yaml

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

run accelerate launch --config_file issue_1958.yaml issue_1958.py :

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.53s/it]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.53s/it]
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.90s/it]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.90s/it]
Done saving model
Done saving model
Done saving model
Done saving model

Inspecting base model weights ls -lh delete_me_base/:

total 26G
-rw-rw-r-- 1 sourab sourab  694 nov.  21 08:52 config.json
-rw-rw-r-- 1 sourab sourab  188 nov.  21 08:52 generation_config.json
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:52 model-00001-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:52 model-00002-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:52 model-00003-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:52 model-00004-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:52 model-00005-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 2,6G nov.  21 08:53 model-00006-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab  24K nov.  21 08:53 model.safetensors.index.json

Inspecting fsdp saved model ls -lh delete_me:

total 26G
-rw-rw-r-- 1 sourab sourab  694 nov.  21 08:53 config.json
-rw-rw-r-- 1 sourab sourab  188 nov.  21 08:53 generation_config.json
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:53 model-00001-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:53 model-00002-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:53 model-00003-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:53 model-00004-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 4,6G nov.  21 08:53 model-00005-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab 2,6G nov.  21 08:54 model-00006-of-00006.safetensors
-rw-rw-r-- 1 sourab sourab   48 nov.  21 08:53 model.safetensors
-rw-rw-r-- 1 sourab sourab  24K nov.  21 08:54 model.safetensors.index.json

Therefore, the sizes of the checkpoints are same. It isn't increasing due to FSDP.

authors: LysandreJik dineshkh pacman100


text:
[BUG]{'image_encoder', 'feature_extractor'} was not found in config. Values will be initialized to default values. Dreambooth LORA

Prerequisites

[X] I have read the documentation.
[X] I have checked other issues for similar problems.

Backend
Local
Interface Used
CLI
CLI Command
Fetching 17 files: 100% 17/17 [00:03<00:00,  5.44it/s]
{'image_encoder', 'feature_extractor'} was not found in config. Values will be initialized to default values.
Loading pipeline components...:   0% 0/7 [00:00<?, ?it/s]{'attention_type', 'reverse_transformer_layers_per_block', 'dropout'} was not found in config. Values will be initialized to default values.
UI Screenshots & Parameters
No response
Error Logs
Fetching 17 files: 100% 17/17 [00:03<00:00,  5.44it/s]
{'image_encoder', 'feature_extractor'} was not found in config. Values will be initialized to default values.
Loading pipeline components...:   0% 0/7 [00:00<?, ?it/s]{'attention_type', 'reverse_transformer_layers_per_block', 'dropout'} was not found in config. Values will be initialized to default values.
Additional Information
Default all values except: dreambooth LORA SDXL 1.0 to complete and data

does it cause results to be bad?

Sorry I should have been more specific. It gets stuck there and doesn't progress for me. I have had success reverting back to version 1.7.5 & 1.7.2 installs.

With the current version, I observed the same issue for ‘autotrain.trainers.dreambooth’ with "model":"stabilityai/stable-diffusion-xl-base-1.0".
The weights are calculated. Subsequently, the following messages where displayed:
Steps: 100% 500/500 [56:02<00:00,  7.01s/it, loss=0.0476, lr=0.0003]
{'latents_std', 'latents_mean', 'force_upcast'} was not found in config. Values will be initialized to default values.
…
{'feature_extractor', 'image_encoder'} was not found in config. Values will be initialized to default values.
…
{'dropout', 'reverse_transformer_layers_per_block', 'attention_type'} was not found in config. Values will be initialized to default values.

The SDXL LoRA from the current version does not work at all. Reverting back to version 0.7.1 resolved the issue used with the same dataset and the same parameters.
Please note that the messages above where not observed with version 0.7.1.

dreambooth seems to be working fine. please update to 0.7.36 or later

I can partially confirm that. It seems that all versions work fine, but the most recent ones no longer work on computers with low RAM (<= 12GB). In contrast, version 0.7.1 continues to function on 'smaller' computers. With the current version(s), the conversion to the pytorch_lora_weights_kohya.safetensors file appears to be aborted due to a lack of memory, and the created file pytorch_lora_weights.safetensors seems not to be usable without this conversion.

ohh okay. so conversion is the only issue for low ram machines?

From my perspective, it looks like this could it be.

you can ignore:
{'latents_std', 'latents_mean', 'force_upcast'} was not found in config. Values will be initialized to default values.
…
{'feature_extractor', 'image_encoder'} was not found in config. Values will be initialized to default values.
…
{'dropout', 'reverse_transformer_layers_per_block', 'attention_type'} was not found in config. Values will be initialized to default values.

I get the same errors and my loras dont work. I tried twice and used a paid Colab account, is that considered a low vram machine?

what are same errors? i dont see any errors mentioned in this issue. what do you mean when you say lora doesnt work? how are you doing the inferencce? unfortunately, its not possible to help without any details :)

How can I reverting back to version 0.7.1 - 1.7.5 - 1.7.2 installs?

repo: huggingface/autotrain-advanced


text:
Viewer works on dataset landing page but not on full viewer page: UnexpectedApiError

The viewer works on the dataset landing page; see for example: https://huggingface.co/datasets/hyperdemocracy/us-congress-bills

However it raises UnexpectedApiError  on the full viewer page; see: https://huggingface.co/datasets/hyperdemocracy/us-congress-bills/viewer/default/train

Fix Hub issue: https://huggingface.co/datasets/hyperdemocracy/us-congress-bills/discussions/1
CC: @galtay @lhoestq

This is because the row group size of the parquet export of the dataset is too big for the API.
So we need to either modify maxRowGroupByteSizeForCopy (currently 500_000_000) and/or maxArrowDataInMemory (currently 300_000_000) to make them compatible.
I think it's fine to lower maxRowGroupByteSizeForCopy to 300_000_000 and re-run the config-parquet-and-info job for this dataset.
I can take care of this next week when I'm back, or feel free to do it @albertvillanova

Also reported in: https://huggingface.co/datasets/RiddleHe/commonsense-baseline/discussions/1

I can take care of the small modification you proposed, @lhoestq.

authors: albertvillanova lhoestq

repo: huggingface/datasets-server


text:
Fix flaky test of the temporary directory used by load_from_disk

The test is flaky, here is an example of random CI failure:
https://github.com/huggingface/datasets/commit/73ed6615b4b3eb74d5311684f7b9e05cdb76c989
I fixed that by not checking the content of the random part of the temporary directory name

CI failed because of a server error - merging

repo: huggingface/datasets


text:
Add support for categorical/dictionary types

Arrow has a very useful dictionary/categorical type (https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html). This data type has significant speed, memory and disk benefits over pa.string() when there are only a few unique text strings in a column.
Unfortunately, huggingface datasets currently does not support this type. So huggingface datasets cannot natively read many parquet files that use this datatype .This PR adds support for Huggingface Datasets to read categorical/dictionary data.
Note: This PR functions by simply converting those dictionary/categorical types to strings. This means that huggingface datasets cannot take advantage of the compute benefits of categoricals, but it significantly simplifies logic. At this time, I do not think it makes sense to optimize categorical support within huggingface datasets and that we should only try to optimize later, if necessary.
Closes #5706

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq  Thanks a ton for helping this get merged!

authors: EthanSteinberg HuggingFaceDocBuilderDev


text:
Fix python version check

I'm using poetry to manage my environment and using python 3.9 on Ubuntu. On my system, sys.version lists my python version as "3.9.0+", which was causing an error in the file I modified. This should solve the problem.

Examples
Search for any integer number in the string
re.findall(r'\d+', "3.8.3rc1")
# ['3', '8', '3', '1']

Limit to the first 3 elements
re.findall(r'\d+', "3.8.3rc1")[:3]
# ['3', '8', '3']

Does it really work for any integer?
re.findall(r'\d+', "323.82.334rc1")[:3]
# ['323', '82', '334']

re.findall(r'\d+', "3.9.0+")
# ['3', '9', '0']

see https://github.com/huggingface/huggingface_hub/pull/82

@ericricky, could you share your pyproject.toml file that yields 3.9.0+ as a python version when calling sys.version? I'm not managing to obtain the +, and as @ulf1 mentioned this isn't PEP compliant so thinking if we should have an exception for this or not.

This should have been adressed by https://github.com/huggingface/huggingface_hub/pull/82

authors: LysandreJik ericricky ulf1


text:
Arch Linux: llm-ls binary is not downloaded by default

Hi there, very happy about and eager to use this plugin.
A minor issue:
On my system (Arch-Linux, Intel® Core™ i7 Processor) the llm-ls binary is not downloaded automatically at first start as claimed in the Readme.
In the settings menu it seems to expect llm-ls-amd64-unknown-linux-gnu-0.4.0 which does not exist on the release page.
Manually downloading and using the x86_64-unknown-linux version from https://github.com/huggingface/llm-ls/releases works.

Hi @lienharc, thanks for letting me know. I think there are still some improvements to make regarding how to handle the multi platform setup (lots of edge cases, as shown here 😅).
I'll take a look at how I can improve this when I have more time, for now manually downloading the binary will have to be the way to go.

authors: McPatate lienharc

repo: huggingface/llm-intellij


text:
remove some print in snli file

This PR removes some print statements in snli.py

I guess you can just rebase from master to fix the CI

authors: lhoestq mariamabarham

repo: huggingface/nlp


text:
updated langchain text-generation pipeline to work with latest release 0.2.5

Updated langchain APIs to work with langchain 0.2.5.
Update langchain installation instructions in README and also added a max input length in the example otherwise the prompt with the context didn't fit and answers were wrong.

@regisss please help review.Thx!

@regisss  could you please review? Thx!

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

authors: HuggingFaceDocBuilderDev rbrugaro

repo: huggingface/optimum-habana


text:
Enable_gptj

What does this PR do?
as title

FP32 outputs:

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

@regisss gptj is here, have a look if you are free

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@ZhaiFeiyue FYI, I tried bf16 and DeepSpeed-inference and it's the same as OPT: bf16 worked well and same error with DS.


text:
Symbol use in optimum: fix misprint

What does this PR do?
This PR fixes a misprint critical for optimum symbol use. Without it, the feature couldn't work properly.
@echarlaix, @eaidova
Before submitting

[N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[N/A] Did you make sure to update the documentation with your changes?
[N/A] Did you write any new necessary tests?

@echarlaix  @IlyasMoutawwakil could you please trigger precommit, thanks

@echarlaix  @IlyasMoutawwakil  could you please help to retrigger precommit?

there seem to be some failures are they related to this change ?

I might be mistaken but this one seems related no ?
FAILED tests/openvino/test_modeling.py::OVModelForCausalLMIntegrationTest::test_compare_to_transformers_35_glm4 - RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:245:
Exception from src/bindings/python/src/pyopenvino/core/infer_request.hpp:54:
Caught exception: Exception from src/plugins/intel_cpu/src/node.cpp:593:
[CPU] Concat node with name 'aten::cat/Concat' Check 'TRShape::merge_into(output_shape, in_copy)' failed at src/core/shape_inference/include/concat_shape_inference.hpp:43:
While validating node 'opset1::Concat aten::cat/Concat (opset1::Reshape aten::flatten/Reshape[0]:f32[?,4,?,4], opset1::VariadicSplit VariadicSplit_18980331[1]:f32[?,4,?,4]) -> (f32[?,4,?,8])' with friendly_name 'aten::cat/Concat':
Shape inference input shapes {{6,4,11,4},{6,4,1,4}}
Argument shapes are inconsistent; they must have the same rank, and must have equal dimension everywhere except on the concatenation axis (axis 3).

I might be mistaken but this one seems related no ?
FAILED tests/openvino/test_modeling.py::OVModelForCausalLMIntegrationTest::test_compare_to_transformers_35_glm4 - RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:245:
Exception from src/bindings/python/src/pyopenvino/core/infer_request.hpp:54:
Caught exception: Exception from src/plugins/intel_cpu/src/node.cpp:593:
[CPU] Concat node with name 'aten::cat/Concat' Check 'TRShape::merge_into(output_shape, in_copy)' failed at src/core/shape_inference/include/concat_shape_inference.hpp:43:
While validating node 'opset1::Concat aten::cat/Concat (opset1::Reshape aten::flatten/Reshape[0]:f32[?,4,?,4], opset1::VariadicSplit VariadicSplit_18980331[1]:f32[?,4,?,4]) -> (f32[?,4,?,8])' with friendly_name 'aten::cat/Concat':
Shape inference input shapes {{6,4,11,4},{6,4,1,4}}
Argument shapes are inconsistent; they must have the same rank, and must have equal dimension everywhere except on the concatenation axis (axis 3).

@IlyasMoutawwakil  that is not related to @jane-intel changes, but her provided improvement highlighted an issue with postion_ids processing in OVModelForCausalLM for models from glm models family. I fixed that as part of this PR.

@eaidova thank you for fixing tests/openvino/test_modeling.py::OVModelForCausalLMIntegrationTest::test_compare_to_transformers_35_glm4
Other failures are unrelated to this PR. @IlyasMoutawwakil, @echarlaix please preceded with merge

authors: IlyasMoutawwakil eaidova jane-intel

repo: huggingface/optimum-intel


text:
Load Time data errors in benchmark_gptq

System Info
- Platform: Windows-10
- Python version: 3.10.13
- PyTorch version: 2.1.2+cu118
- Huggingface_hub version: 0.20.1
- Transformers version: 4.36.2
- Accelerate version: 0.25.0
- optimum version: 1.16.1
- auto_gptq version: 0.5.1+cu118

Who can help?
No response
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)
python benchmark_gptq.py --model TheBloke/llava-v1.5-13B-GPTQ  --num-batches 4 --batch-size 1 --gptq --task text-generation --use-exllama --exllama-version 2 --generate

Expected behavior
I tried to test the model many times, but the load time data I get varies a lot.

It's a 6.7GB sized model and was placed on a hard drive, so I think 67 seconds of data is correct.

Thank you @kirayomato, I remember noticing large variability there as well. The relevant code is here: https://github.com/huggingface/optimum/blob/0f0a66303425b476bd5e209c076419a404238bb3/tests/benchmark/benchmark_gptq.py#L306-L329 https://github.com/huggingface/optimum/blob/0f0a66303425b476bd5e209c076419a404238bb3/tests/benchmark/benchmark_gptq.py#L354-L355
Does something look wrong to you there?

authors: fxmarty kirayomato

repo: huggingface/optimum


text:
Adds Vera (Vector Based Random Matrix Adaption) #2

Continuation of #1039.
Should now be 95% on par with that PR, with some minor changes on my part + resolving merge conflicts.
Examples and docs have not been included yet.
~TODOS:~

[x] Add documentation
[x] Add examples

https://arxiv.org/abs/2310.11454
Notable changes vis-à-vis 1039:

Some refactors around how the initialization of VeraModel proceeds, should be more straightforward now.
Add tests around saving and loading, which needs some special considerations for VeRA.
Fixed some issues with multiple adapters, requires more strictness (e.g. not allowing multiple different prng keys on the same model).
projection_prng_key now has a valid default value (0) in the config.
Removed support for Embedding to reduce complexity: Supporting Embedding layers with VeRA makes very little sense because its shape is always different from the linear layers' shapes. Therefore, they cannot share the vera_A and vera_B matrices, resulting in an error. The only conceivable way to support Embedding layers would be to only target that layer (and possibly the output layer if it shares the weight), but that more or less defeats the purpose of using VeRA. We may revisit support for Embeddings in the future, maybe if we can enable vera_A and vera_B to be of different shapes. Until then, let's support the most common use cases and simplify our lives.

To ensure that the vera_A and vera_B weights are shared (but not other tensors), I added some tests that check their corresponding data_ptr()s.
Moreover, I wrote a small script to check the amount of memory taken by the model. For this, I used a very high rank of 10000, so that vera_A and vera_B should be quite large. Then I compared the GPU memory taken for a model with a single layer having a VeRA adapter vs a model with many layers having a VeRA adapter. We should expect that both should take roughly the same memory, since most parameters are shared.
Here is the script:
from transformers import AutoModelForCausalLM
from peft import get_peft_model, VeraConfig, LoraConfig
from peft.tuners.vera import VeraLayer
from peft.tuners.lora import LoraLayer
import gc
import torch

RANK = 10000
model_id = "facebook/opt-125m"

config_cls = VeraConfig
layer_cls = VeraLayer

def get_gpu_memory():
    torch.cuda.synchronize()  # Wait for all kernels to finish
    gpu_info = {
        'allocated': f"{torch.cuda.memory_allocated(0) / 2**30:.4f}GB",
        'reserved': f"{torch.cuda.memory_reserved(0) / 2**30:.4f}GB",
    }
    print(gpu_info)

print("before loading the base model")
get_gpu_memory()

model = AutoModelForCausalLM.from_pretrained(model_id).cuda()
print("after loading the model")
get_gpu_memory()

config = config_cls(task_type="CAUSAL_LM", target_modules=["model.decoder.layers.0.self_attn.k_proj"], r=RANK)
model = get_peft_model(model, config)
num_vera_layers = len([m for m in model.modules() if isinstance(m, layer_cls)])
print(f"after adding {num_vera_layers} adapted layers with rank {RANK}")
get_gpu_memory()

del model
torch.cuda.empty_cache()
gc.collect()

print("after resetting")
get_gpu_memory()

model = AutoModelForCausalLM.from_pretrained(model_id).cuda()
print("after loading the base model")
get_gpu_memory()

config = config_cls(task_type="CAUSAL_LM", target_modules=["v_proj", "q_proj"], r=10000)
model = get_peft_model(model, config)
num_vera_layers = len([m for m in model.modules() if isinstance(m, layer_cls)])
print(f"after adding {num_vera_layers} adapted layers with rank {RANK}")
get_gpu_memory()

For VeRA, the results are:
before loading the base model
{'allocated': '0.0000GB', 'reserved': '0.0000GB'}
after loading the model
{'allocated': '0.4677GB', 'reserved': '0.5176GB'}
after adding 1 adapted layers with rank 10000
{'allocated': '0.5264GB', 'reserved': '0.5762GB'}
after resetting
{'allocated': '0.0000GB', 'reserved': '0.0000GB'}
after loading the base model
{'allocated': '0.4677GB', 'reserved': '0.5176GB'}
after adding 24 adapted layers with rank 10000
{'allocated': '0.5273GB', 'reserved': '0.5762GB'}

As we can see, when adapting 24 layers vs 1 layer, the memory used is almost identical. We expect a small increase because vera_lambda_b and vera_lambda_d are not shared, so this is in line with our expectations.
As a sanity check, if we do the same with LoRA instead of VeRA, we see a big increase in memory used:
before loading the base model
{'allocated': '0.0000GB', 'reserved': '0.0000GB'}
after loading the model
{'allocated': '0.4677GB', 'reserved': '0.5176GB'}
after adding 1 adapted layers with rank 10000
{'allocated': '0.5263GB', 'reserved': '0.5762GB'}
after resetting
{'allocated': '0.0000GB', 'reserved': '0.0000GB'}
after loading the base model
{'allocated': '0.4677GB', 'reserved': '0.5176GB'}
after adding 24 adapted layers with rank 10000
{'allocated': '1.8740GB', 'reserved': '1.9238GB'}

All if this is a strong indicator to me that the memory sharing actually works. If anyone has ideas for more tests, let me know.

@dkopi @vvvm23 I think I'm pretty much finished with the implementation itself, docs and examples are yet to come. Still, if you have time, I'd be happy with a review or if you can run some tests to see if the implementation performs as expected. The changes compared to the original PR are documented above, the core VeRA computation hasn't been changed, though.

Hi @BenjaminBossan, I can do a review some time this week.

@vvvm23 Did you have time to take a look?

@pacman100 Thanks for the feedback, indeed I hadn't checked the docstring example. It is now changed to a working model.
Your comment also prompted me to take a look at the models that are pre-configured in TRANSFORMERS_MODELS_TO_VERA_TARGET_MODULES_MAPPING. This was just a copy of the the LoRA settings. Unfortunately, not all models work, I had to exclude some popular ones like Mistral, Mixtral, Phi, and gemma. The issue is again that the shapes of the target layer weights can differ. Hopefully, we can add the feature of supporting multiple different weight shapes in the future.

It would be great to add @vvvm23 and @dkopi as co-authors for all their guidance and work!

Yes, that was indeed my plan. @vvvm23 @dkopi could you please let me know how you want to be added as co-authors?

Likewise, you can add
 Co-authored-by: Alex McKinney <alex.f.mckinney@gmail.com>

Done :tada:
Thanks again so much @vvvm23 for doing the majority of the work and @dkopi for your constant feedback.
Let's hope that VeRA gains traction in the community. For the future, I'll add this list of improvements for VeRA that have yet to be implemented (contributions are welcome):

Make VeRA work with different weight shapes. This is IMO the biggest limitation right now. Most straightforward way would be to have one pair of fixed vera_A/vera_B weights per target weight shape.
There are cases where the shapes are the same when transposed, so (4096, 1024) vs (1024, 4096) (down and up projections for instance) - I wonder if we can use the same weights here and just transpose them.
If this is done, implement VeRA for more layer types than just Linear. Right now, supporting, for instance, Embedding, makes little sense, as it almost always has a different shape than Linear.
Support quantized weights, most notably bnb.
Support DoRA ("DVoRA")

Thanks again @BenjaminBossan ! please tag me in issues and PRs related to improvements :)

repo: huggingface/peft


text:
FIX: Ensure Device Compatibility for BOFT Forward/Merging

Fixes #2219
Description
This pull request resolves above issue regarding BOFT forward/merging with CUDA by ensuring that all relevant tensors and models are moved to the correct device. This change is necessary to prevent issues such as zero matrices and test failures when using CUDA.
Changes

Ensuring that all operations will be performed on the same device:

Added torch_device = infer_device() to infer and set the appropriate device (CPU or CUDA) at the beginning of the MultipleActiveAdaptersTester class.
Modified prepare_inputs_for_testing() to move input tensors to the inferred device using .to(self.torch_device).
Updated model initialization in test_merge_layers_multi to move models to the inferred device with .to(self.torch_device).

Due to floating point precision on CUDA, for test_multiple_active_adapters_merge_and_unmerge the absolute tolerance had to be decreased from atol=1e-5 to atol=1e-4

Testing

Verified that tests in tests/test_custom_models.py pass successfully with these changes, on both Linux (Ubuntu) and Windows
Ensured compatibility across different environments, including CPU and CUDA-enabled setups.

Thanks a lot for opening this PR to fix the issue with the BOFT tests. It appears that your fork was outdated when opening the PR, as it contains unrelated changes and merge conflicts. Could you please try merging with/rebasing on the current main branch? If this creates new conflicts, opening a new PR from an up-to-date fork might be the easier solution.

Just merged main into my PR's branch (I was expecting that as I was implementing the changes from v0.13.2. I just waited for a confirmation to merge from your side)

@BenjaminBossan Can you please run the failed test again? There were too many requests: `429 Client Error: Too Many Requests for url: https://huggingface.co/api/models/hf-internal-testing/tiny-stable-diffusion-torch

@d-kleine Not sure what's going on with the CI right now, I'll rerun it next week, hopefully it solves itself.

Alright, thanks!

I just have also fixed the fbd_cuda deprecation warning: https://github.com/huggingface/peft/issues/2219#issuecomment-2501253966
As the GitHub Actions workflows seem to run only on CPU, please ensure when reviewing to run the tests on a CUDA-enabled device and also empty the cache to see if the fbd_cuda warnings are gone now.

Great, thanks! 👍🏻

Okay, so we haven't heard back in a week, I'll just go ahead and merge this now. Again, thanks for this fix @d-kleine.

Thanks for merging! 🙂

authors: BenjaminBossan d-kleine


text:
Implementation of 15% words masking in pretraining

In the BERT paper, they randomly mask 15% words for pretraining, and that's exactly what they do in the TF version.
https://github.com/google-research/bert/blob/0fce551b55caabcfba52c61e18f34b541aef186a/create_pretraining_data.py#L342
However, the implementation here is a little bit different, instead of randomly select 15% tokens, it assigns a probability of 15% to each token, that is, each token has a probability of 15% to be masked. That means, each time we might have less than or more than 15% tokens masked.
So, is it correct to mask tokens with the expectation of 0.15 rather than fixed 15%?
https://github.com/huggingface/pytorch-pretrained-BERT/blob/f9cde97b313c3218e1b29ea73a42414dfefadb40/examples/lm_finetuning/simple_lm_finetuning.py#L276-L301

It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.

It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.

I found the same problem that the implementation is different from tensorflow. But the key point is not the fixed 15% prob of all the token. If  we use the implementation of pytorch will produce two extreme case especially for short sentences like article title,usually 10-20 characters.
case 1. sentence with too much '[MASK]'
case 2. sentence with none '[MASK]'
both case1 and case2 would cause the drop of performance. case 1 make the model difficult to predict and case2 would not produce the loss.
Given a corpus with an average sentence length of 10. The implementation of tensorflow would generate 1 '[MASK]' for the sentences, but the implementation of pytorch would have :
0.85^10                      = 0.19 to generate 0 '[MASK]'
0.15 * 0.85^9 * 10      =0.34 to generate 1 '[MASK]'
0.15^2 * 0.85^8  * 45 =0.27 to generate 2 '[MASK]'
0.15^2 * 0.85^7 * 120 =0.12 to generate 3 '[MAKS]'
...
If we roughly consider the sentence with 15% '[MASK]' is  appropriate, only 1/2 '[MASK]' is useful for training models. So only 0.34 + 0.27  = 0.61 training case is useful.
And we found it is this is a very serious problem for short text.

authors: thomwolf tsaikenyoo zhangsh950618

repo: huggingface/pytorch-pretrained-BERT


text:
Adding XLNet model

Current status:

[x] model with commented code and pretrained loading logic
[x] tokenizer
[x] tests for model and tokenizer
[x] checking standard deviation of hidden states with TF model is ok (max dev btw 1e-4 & 1e-5 until last layer, last layer 1e-3, should be ok)
[x] converting and uploading model to S3

Model/tokenizer are usable, now just need to

[ ] check the model behave well under various conditions and in a few corner cases
[ ] add a few classes variants (for sequence and token fine-tuning)
[ ] add a finetuning example with results close to TF
[ ] add models in README
[ ] add models on torch.hub

@thomwolf I could get the XLNetLMHeadModel running, but I have some issues with the "normal" XLNetModel implementation:
import torch
from pytorch_pretrained_bert import XLNetTokenizer, XLNetModel

import logging
logging.basicConfig(level=logging.INFO)

tokenizer = XLNetTokenizer.from_pretrained("xlnet-large-cased")

text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.encode(text)

indexed_tokens = torch.tensor([tokenized_text])

model = XLNetModel.from_pretrained("xlnet-large-cased")
model.eval()

with torch.no_grad():
    hidden_states, mems = model(indexed_tokens)
    print(hidden_states)

-> is currently not working, hidden_states = model(indexed_tokens) always returns nans for the embeddings.
Thanks :heart:

Indeed the from_pretrained() was not loading weights in XLNetModel (did you see that in the logs?). Should be fixed now. This is all very WIP so beware @stefan-it!

Codecov Report

Merging #711 into master will decrease coverage by 0.08%.
The diff coverage is 61.71%.

@@            Coverage Diff             @@
##           master     #711      +/-   ##
==========================================
- Coverage   62.27%   62.18%   -0.09%
==========================================
  Files          18       22       +4
  Lines        3979     4742     +763
==========================================
+ Hits         2478     2949     +471
- Misses       1501     1793     +292

Impacted Files
Coverage Δ

pytorch_pretrained_bert/modeling_transfo_xl.py
52.93% <ø> (ø)
:arrow_up:

pytorch_pretrained_bert/modeling_openai.py
68.11% <0%> (ø)
:arrow_up:

...ytorch_pretrained_bert/modeling_xlnet_utilities.py
0% <0%> (ø)

pytorch_pretrained_bert/modeling.py
79.49% <0%> (ø)
:arrow_up:

pytorch_pretrained_bert/modeling_gpt2.py
69.79% <0%> (ø)
:arrow_up:

...rained_bert/convert_xlnet_checkpoint_to_pytorch.py
0% <0%> (ø)

pytorch_pretrained_bert/__init__.py
100% <100%> (ø)
:arrow_up:

pytorch_pretrained_bert/modeling_xlnet.py
65.22% <65.22%> (ø)

pytorch_pretrained_bert/tokenization_xlnet.py
72.94% <72.94%> (ø)

pytorch_pretrained_bert/tokenization_gpt2.py
82.44% <0%> (-1.07%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c304593...c946bb5. Read the comment docs.

Thanks for the fix - it's working now :)
Sorry, if that's not the right place here:
I've implemented a new embedding class in flair. But when the complete model is going to be saved (via torch.save), the following error message is shown:
Traceback (most recent call last):
  File "train_xlnet.py", line 36, in <module>
    max_epochs=500)
  File "/mnt/flair/flair/trainers/trainer.py", line 341, in train
    self.model.save(base_path / "best-model.pt", pickle_module=self.pickle_module)
  File "/mnt/flair/flair/nn.py", line 86, in save
    self.save_torch_model(model_state, str(model_file), pickle_module)
  File "/mnt/flair/flair/nn.py", line 76, in save_torch_model
    torch.save(model_state, str(model_file), pickle_protocol=pickle_protocol)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 149, in _with_file_like
    return body(f)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 297, in _save
    pickler.dump(obj)
TypeError: can't pickle SwigPyObject objects

This error message does not appear, when e.g. using the GPT1 embeddings. Do you have any hint, what's going wrong here 🤔

I made a finetuning model with this. The training loss seems to be reducing implying that model is learning, however, during inference all values are the same.

Am I correct that model.eval() does not work with this yet? It seems all predictions generated are the same.
This is how im doing it:
config = XLNetConfig('config.json')
model = XLNetForSequenceClassification(config, num_labels=3)
model.load_state_dict(torch.load("xlnet_pytorch.bin"))
model.to(device)
for param in model.parameters():
    param.requires_grad = False
model.eval()

@thomwolf I think this is branch still WIP yes? Because the run_xlnet_squad.py still refers to BertForQuestionAnswering. SHould I just change that to XLnetForQuestionAnswering and it should work? or maybe I should wait more?

Yeah, I'm still on it, finishing the tests of XLNetForSequenceClassification so you should wait more (or help me code the XLNetForQuestionAnswering, haha).
I'll make the description of the PR more explicit that only the base model is up now and XLNetForSequenceClassification and XLNetForQuestionAnswering are not ready yet.

I think I found the root cause of the serialization problem (described in one of the previous comments here):
The sentencepiece processor object cannot be correctly serialized. I found a similar issue in the xnmt library:
https://github.com/neulab/xnmt/pull/351

Yeah, I'm still on it, finishing the tests of XLNetForSequenceClassification so you should wait more (or help me code the XLNetForQuestionAnswering, haha).
I've started to work on it with whatever time I've :(
So far I'm getting an error in the modeling_xlnet.py in line 752:
    attention_mask = attention_mask.transpose(0, 1).contiguous() if attention_mask is not None else None

Whoever finishes first should let here know..But appreciate your work here @thomwolf.

@stefan-it do you know if there is a workaround to the sentencepiece serialization issue?

@thomwolf I think this can be fixed with:
    def __getstate__(self):
        state = self.__dict__.copy()
        state["sp_model"] = None
        return state

    def __setstate__(self, d):
        self.__dict__ = d
        try:
            import sentencepiece as spm
        except ImportError:
            logger.warning("You need to install SentencePiece to use XLNetTokenizer: https://github.com/google/sentencepiece"
                           "pip install sentencepiece")
        self.sp_model = spm.SentencePieceProcessor()
        self.sp_model.Load(self.vocab_file)

In the XLNetTokenizer class. A nice test case would be:
import pickle
from pytorch_pretrained_bert import XLNetTokenizer

tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')

text = "Munich and Berlin are nice cities"
filename = "tokenizer.bin"

subwords = tokenizer.tokenize(text)

pickle.dump(tokenizer, open(filename, "wb"))

tokenizer_new = pickle.load(open(filename, "rb"))
subwords_loaded = tokenizer_new.tokenize(text)

assert subwords == subwords_loaded

I'm hitting a bug where the head mask created here:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/xlnet/pytorch_pretrained_bert/modeling_xlnet.py#L854
is a list of None values instead of just None, which eventually results in an error:
  File "../pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling_xlnet.py", line 397, in rel_attn_core
    attn_prob = attn_prob * head_mask
TypeError: mul(): argument 'other' (position 1) must be Tensor, not list

I think the fix is to either make the mask a single None value, or to index it with the layer number when it's used.

@stefan-it Great, thanks!

@nikitakit Yeah I'm refactoring the API among models to make it more consistent/simpler to switch among models. Will be finished soon.

While modifying run_xlnet_squad.py I looked through the run_squad.py of the xlnet repo. It seems he made a bunch of changes to convert_examples_to_features (adding lcs etc.) Also, I didn't see that he put the cls at the end; rather p before than q (I think; hope I'm not wrong). Have you started to work on it @thomwolf ? It's interesting :)

One quick update in case someone else is also working on the Squad XLnet fine tuning. I got this error and working on it seems that the end logits in the QA layer is buggy.
Traceback (most recent call last):   File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 478, in <module>     main()   File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 470, in main     result = evaluate(args, model, tokenizer, prefix=global_step)   File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 219, in evaluate     args.null_score_diff_threshold)   File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/utils_squad.py", line 463, in write_predictions     feature_null_score = result.start_logits[0] + result.end_logits[0] TypeError: unsupported operand type(s) for +: 'float' and 'list'`

Is there any rough estimation as to when "a finetuning example with results close to TF" will be available?

Thanks for the fix - it's working now :)
Sorry, if that's not the right place here:
I've implemented a new embedding class in flair. But when the complete model is going to be saved (via torch.save), the following error message is shown:
Traceback (most recent call last):
  File "train_xlnet.py", line 36, in <module>
    max_epochs=500)
  File "/mnt/flair/flair/trainers/trainer.py", line 341, in train
    self.model.save(base_path / "best-model.pt", pickle_module=self.pickle_module)
  File "/mnt/flair/flair/nn.py", line 86, in save
    self.save_torch_model(model_state, str(model_file), pickle_module)
  File "/mnt/flair/flair/nn.py", line 76, in save_torch_model
    torch.save(model_state, str(model_file), pickle_protocol=pickle_protocol)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 149, in _with_file_like
    return body(f)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 297, in _save
    pickler.dump(obj)
TypeError: can't pickle SwigPyObject objects

This error message does not appear, when e.g. using the GPT1 embeddings. Do you have any hint, what's going wrong here 🤔

Hi, have you solved this issue? I'm trying to implement XLMRoberta in my model based on flair.

authors: abhishekkrthakur avisil codecov-io lukgarn nikitakit stefan-it thomwolf wangxinyu0922


text:
Is pipeline parallelism supported?

Feature request
pipeline parallelism
Motivation
To support running model on multiple nodes.
Your contribution
No.

No and it's probably never going to land. Multi node inference increases the latency by too much.
Quantization should be the type of methods to reach for if a given model exceeds the hardware you have.

inference

Thank you!

repo: huggingface/text-generation-inference


text:
unable to install .whl directly

hi,
my system is ubuntu 16.04 64bit, i tried both pip and pi3, tried both to ln -s python to python2/python3 under /usr/bin
$ pip install --upgrade tensorflow-0.11.0rc0-py3-none-any.whl
tensorflow-0.11.0rc0-py3-none-any.whl is not a supported wheel on this platform.
i'm new to python, not sure if it is my env issue relative to python, do you happen to know to debug/fix it? thanks.

i must did something wrong that day.
i just tried again and it works now

Cool :-)

authors: guoyejun hughperkins

repo: hughperkins/tensorflow-cl


text:
Whether some value should only be extracted by cookie？And how to get the cookie by PostAgent?

I got a strange thing when parsing this site. Before log-inning I couldn't get the value enc_uin  on the comment part but it could be got on the article part and other values on the comment part were also well caught. After log-inning I got another new value uin by using cookie.
comment part:
  "expected_update_period_in_days": "2",
  "url": "https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v2?bid=10247&pid=1112558-1528855165_10247_&num=10&start=0",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "user": {
      "path": "$.result.comments[*].user.nick_name"
    },
    "gender": {
      "path": "$.result.comments[*].user.gender"
    },
    "enc_uin": {
      "path": "$.result.comments[*].user.enc_uin"
    }
  },
  "headers": {
    "referer": "https://buluo.qq.com/p/detail.html?bid=10247&pid=1112558-1528855165_10247_"
  }
}

result:
[00:00:00] INFO -- : Fetching https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v2?bid=10247&pid=1112558-1528855165_10247_&num=10&start=0
[00:00:01] INFO -- : Extracting json at $.result.comments[*].user.nick_name
[00:00:01] INFO -- : Values extracted: ["还有谁！！！", "还有谁！！！", "千寻", "垃圾", "垃圾", "垃圾", "千寻", "如初🌸", "如初🌸", "夜雨声烦"]
[00:00:01] INFO -- : Extracting json at $.result.comments[*].user.gender
[00:00:01] INFO -- : Values extracted: [1, 1, 255, 2, 2, 2, 255, 2, 2, 2]
[00:00:01] INFO -- : Extracting json at $.result.comments[*].user.enc_uin
[00:00:01] INFO -- : Values extracted: []
[00:00:01] ERROR -- : Error when fetching url: Got an uneven number of matches for : {"user"=>{"path"=>"$.result.comments[*].user.nick_name"}, "gender"=>{"path"=>"$.result.comments[*].user.gender"}, "enc_uin"=>{"path"=>"$.result.comments[*].user.enc_uin"}}

article part:
  "expected_update_period_in_days": "2",
  "url": "https://buluo.qq.com/cgi-bin/bar/post/content?bid=10247&pid=1112558-1528855165_10247_",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "enc_uin": {
      "path": "$.result.user_info.enc_uin"
    }
  },
  "headers": {
    "referer": "https://buluo.qq.com/p/detail.html?bid=10247&pid=1112558-1528855165_10247_"
  }
}

result:
[
  {
    "enc_uin": "8FT2Rpo3ZKZk6ZwoEqtbSA"
  }
]

By using cookie:
{
  "expected_update_period_in_days": "2",
  "url": "https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v2?bid=10247&pid=1112558-1528855165_10247_&num=10&start=0",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "uin": {
      "path": "$.result.comments[*].user.uin"
    },
    "enc_uin": {
      "path": "$.result.comments[*].enc_uin"
    }
  },
  "headers": {
    "referer": "https://buluo.qq.com/p/detail.html?bid=144106&pid=5794662-1516500479",
    "cookie": "{% credential Q-login cookie %}"
  }
}

result:
[00:00:00] INFO -- : Dry Run started
[00:00:00] INFO -- : Fetching https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v2?bid=10247&pid=1112558-1528855165_10247_&num=10&start=0
[00:00:01] INFO -- : Extracting json at $.result.comments[*].user.uin
[00:00:01] INFO -- : Values extracted: [2102980263, 2102980263, 1581112400, 3205752411, 3205752411, 3205752411, 1581112400, 3354429359, 3354429359, 1960106343]
[00:00:01] INFO -- : Extracting json at $.result.comments[*].enc_uin
[00:00:01] INFO -- : Values extracted: []
[00:00:01] ERROR -- : Error when fetching url: Got an uneven number of matches for : {"uin"=>{"path"=>"$.result.comments[*].user.uin"}, "enc_uin"=>{"path"=>"$.result.comments[*].enc_uin"}}

But the cookie was copied from the page of chrome develop tool after I loginned in😹. I can't get the cookie by PostAgent.
{
  "post_url": "https://ui.ptlogin2.qq.com/cgi-bin/login?style=9&appid=1006102&daid=371&s_url=https%3A%2F%2Fbuluo.qq.com%2Fmobile%2Fbarindex.html%3F_wv%3D1027%26_bid%3D128%26from%3Dpc%26bid%3D10247&low_login=0&hln_css=https%3A%2F%2Fpub.idqqimg.com%2Fqqun%2Fxiaoqu%2Fmobile%2Fimg%2Flogin-logo.png",
  "expected_receive_period_in_days": "1",
  "content_type": "form",
  "method": "post",
  "payload": {
    "username": "{% credential Q-user %}",
    "password": "{% credential Q-pass %}"
  },
  "headers": {
    "Cookie": "{{cookie}}"
  },
  "emit_events": "true",
  "no_merge": "false",
  "output_mode": "clean",
  "disable_redirect_follow": "true",
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1"
}

result:
  {
    "body": "",
    "headers": {
      "Date": "Tue, 10 Jul 2018 14:12:07 GMT",
      "Content-Type": "text/html",
      "Content-Length": "0",
      "Connection": "keep-alive",
      "Server": "QZHTTP-2.38.41",
      "Strict-Transport-Security": "max-age=0"
    },
    "status": 501
  }

If I change the method to get it outputs the following, but it's different from the one I copied after log-inning and is useless to get the uin.
 "headers": {
      "Date": "Tue, 10 Jul 2018 14:12:36 GMT",
      "Content-Type": "text/html",
      "Content-Length": 17751,
      "Connection": "keep-alive",
      "Server": "QZHTTP-2.38.41",
      "P3p": "CP=\"CAO PSA OUR\"",
      "Cache-Control": "max-age=86400",
      "Set-Cookie": "pt_user_id=1795827860676532641; EXPIRES=Fri, 07-Jul-2028 14:12:37 GMT; PATH=/; DOMAIN=ui.ptlogin2.qq.com;, pt_login_sig=wUlgTn3fxJGbOYUb69Rar7IvZ0TkFBjiuLPivsPiYiJUG84LlHVNBB9wE3vhU7JD; PATH=/; DOMAIN=ptlogin2.qq.com;, pt_clientip=f65a3692e0997d30; PATH=/; DOMAIN=ptlogin2.qq.com;, pt_serverip=3fe10af17164f9b9; PATH=/; DOMAIN=ptlogin2.qq.com;, login_param=appid%3D1006102%26daid%3D371%26hln_css%3Dhttps%253A%252F%252Fpub.idqqimg.com%252Fqqun%252Fxiaoqu%252Fmobile%252Fimg%252Flogin-logo.png%26low_login%3D0%26password%3D7259070tuqq%26s_url%3Dhttps%253A%252F%252Fbuluo.qq.com%252Fmobile%252Fbarindex.html%253F_wv%253D1027%2526_bid%253D128%2526from%253Dpc%2526bid%253D10247%26style%3D9%26username%3D3339857065; PATH=/; DOMAIN=ui.ptlogin2.qq.com;, uikey=eaf42e2d594c8d54b357af3a4833dfbf7f07e42791b40f6b37c016054f4e37f5; PATH=/; DOMAIN=ptlogin2.qq.com;, ptui_identifier=000DF39348B3DBA5DCAB47B46F95FA0C8418C7FB656010BD5823BE3D; PATH=/; DOMAIN=ui.ptlogin2.qq.com;",
      "Last-Modified": "Thu, 08 Mar 2012 02:04:00 GMT",
      "Strict-Transport-Security": "max-age=0"
    },
    "status": 200

In fact what I want is to get the unique ID such as enc_uin or uin. Currently I can't get either of them.
My first question is how can I extract the enc_uin if there is no cookie. If it's impossible the only way is to get the cookie by PostAgent to extract the uin . But I failed😔. By the way, I find someone solved this by PhantomJs on the site but how to achieve this by huginn?

comment part:
"expected_update_period_in_days": "2",
"url": "https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v2?bid=10247&pid=1112558-1528855165_10247_&num=10&start=0",
....
result:

The JSON the server is responding with is not including the value you are looking for try using $.result.comments as the path (and the only extraction option to avoid the "uneven amount of matches error") to get everything in that JSON array.

"status": 501

HTTP status 501 most likely means something went wrong with your request.

If I change the method to get it outputs the following, but it's different from the one I copied after log-inning and is useless to get the uin.

The cookie you get on different systems will never be the same, so that is normal 😄

By the way, I find someone solved this by PhantomJs on the site but how to achieve this by huginn?

Maybe https://phantomjscloud.com/ will work. If the site is requiring JavaScript to log in.

@dsander Ha，so nice to see u again!😸

The JSON the server is responding with is not including the value you are looking for try using $.result.comments as the path (and the only extraction option to avoid the "uneven amount of matches error") to get everything in that JSON array.

I tried $.result.commentsbut the output value of uin was always 0 and had no enc_uin key even I could see it on the page of chrome develop tool. Only after log-inning in and using cookie the key uin had the right output. I just applied another QQ account for you and please help me to have a look at that👻. If login needs verification code I'll send you here.
{
        "replay": 0,
        "index": 1,
        "op": 0,
        "liketotal": 0,
        "ispostor": 0,
        "uin": 0,
        "pid": "1112558-1528855165_10247_",
        "tlist_index": 1,
        "like": 0,
        "cid": "2980473-1528884829",
        "time": 1528884829,
        "isdel": 0,
        "postsrc": 1,
        "is_black": 0
}

Maybe https://phantomjscloud.com/ will work. If the site is requiring JavaScript to log in.

From now on what I konw about phantomjscloud is to make the page static state to get the content but how to get cookie by that? This site has the code which have got the cookie by phantomjs. If phantomjscloud can't get the cookie how to get it by huginn with those codes?

I am not sure if the login can be automated with Huginn, it seems to be doing a couple of requests, which seem to be trigged from Javascript.
After I got the cookie requesting the JSON worked.
{
  "expected_update_period_in_days": "2",
  "url": "https://buluo.qq.com/cgi-bin/bar/post/get_comment_by_page_v3?bid=10247&pid=1759938-1530848707_10247_&num=20&start=0&barlevel=1&r=0.29110315122991737&bkn=589647426",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "user": {
      "path": "$.result.comments[*].user.nick_name"
    },
    "gender": {
      "path": "$.result.comments[*].user.gender"
    },
    "enc_uin": {
      "path": "$.result.comments[*].user.enc_uin"
    }
  },
  "headers": {
    "referer": "https://buluo.qq.com/p/detail.html?bid=10247&pid=1759938-1530848707_10247_",
    "cookie": "pgv_pvi=4567318528; pgv_si=s4156928000; ptisp=os; pt2gguin=o1041428077; RK=dJJdIq1kbS; ptcz=598c663c909548010cc97a085b19a4983802208d25d72411724fb30288b78d99; enc_uin=9qUrVQdjUiIgl3NM4uBaNg; uin=o1041428077; skey=@792PRBo2v; p_uin=o1041428077; pt4_token=froTGuFqfzDxnFBYF9a4yRdhVMZ8PC1-nKZ35jTGTO8_; p_skey=R*rBtZKgBHSfcIhpbiw50qqNmt3r980GJuMUCVOd1YA_"
  }
}

I copied the "cookie" header from the chrome development tools.

Copying and pasting the cookie is a feasible way to fetch the uin value but the cookie has a time limit. Almost one or two days later when the cookie becomes invalid all of the uin value will return to 0. You can test and verify it  tomorrow or later😹. So getting the fresh cookie with huginn maybe slove the problem but I don't know how.

So getting the fresh cookie with huginn maybe slove the problem but I don't know how.

The problem with that is that they are using JavaScript to somehow sign/encrypt the password. I think you best bet is to use the phantomjs code you linked on your local computer and update the cookie on Huginn using a WebhookAgent.

Dose that mean I must open my computer one or two days one time to update the cookie? But when the cookie becomes invalid is hard to know exactly. Is there anyway to fetch the enc_uin which needn't to login and depend on the cookie. I think enc_uin should be fetched because it's there on the page of chrome develop tool. Maybe getting cookie is too much of trouble.

think enc_uin should be fetched because it's there on the page of chrome develop tool.

Fetched from where? The JSON data is not including enc_uin when the request is made without a cookie.

Maybe that is my always misunderstanding about JSON that the key seen on the chrome develop tool page should all be included in the data and could be fetched🙈. It's hard to implement that one or two days a time to open computert to get the cookie. Sigh!

@dsander I think I may have solved this problem after changing v2 to v3 in the url and then I fetched the enc_uin smoothly. Let's forget about the cookie😹. Thank u very much and enjoy the weekend~

Nice!That is way easier 😄

authors: dsander tortoo

repo: huginn/huginn