Coveralls badge showing coverage unknown

[Risto-Stevcev]

https://coveralls.io/github/Risto-Stevcev/bastet

[kdondziak]

@afinetooth Any updates about this? I have experienced the same problem. https://coveralls.io/github/benetech/ServiceNetGateway

Update: I've managed to solve this. In my case build was only main for feature branch and not for the main branch to which badge was connected.

[afinetooth]

Sorry for the delay @Risto-Stevcev. Thanks @kdondziak, yes, that was going to be my first question, @Risto-Stevcev, whether you had submitted at least one build against your default branch.

In your case, it looks like you have, although I'm seeing something I haven't seen before, which is that we can't identify the branches for any of your builds. Note your recent builds table:

Also not seeing a CI service (under VIA).

Can you please explain how you've set up your project to post coverage results to coveralls? I see a .travis.yml file in your project, but I'm not seeing any coveralls-related details / commands. Thus we're not receiving the builds from Travis and are not able to link back to those builds for debug output.

My first recommendation would be to post to coveralls from CI, and call coveralls with the verbose flag (-v or --verbose) which will add debug output to your CI output. That will let us follow the build link back to Travis to see exactly what you're sending to coveralls.

Thanks,
James

[Risto-Stevcev]

Hi @afinetooth ,
Thanks for the response. I haven't set up a CI build for the project, I'm just running coveralls manually. Is it possible to use the badge without having a CI set up?

[afinetooth]

Hi @Risto-Stevcev,

It should be generally possible to post coverage results from your local machine and to use the badge that way. However, the README for your integration library, Bisect_ppx (our community-created OCaml integration library) only describes use cases for Travis CI, CircleCI and Github Actions. I don't see why the same procedures should not work from your local development environment, but it may be a matter of changing the file structure or making different calls from the command line.

Have you inquired in that project about how to call coveralls from your local dev env?

The issue is making sure your POSTs to our API are correct, in that we're receiving all required info and it's correct.

Are you able to send me the curl commands, or otherwise some log of the full request you're sending to https://coveralls.io/api/v1/jobs?

Since we expect to receive POSTs to /jobs from CI, this makes me think you'll need to provide the git hash that's otherwise listed as optional here:
https://docs.coveralls.io/api-reference#endpoint

If you prefer to keep your info private, email us at support@coveralls.io and just reference this issue.

[Risto-Stevcev]

It was a while back, but from what I remember the reason why I didn't want to set up CI for coveralls was due to the permissions it asks for Github -- it was either the ability to write to the repo or to write to all of my repos. I wasn't sure why it would even be needed this since I was posting the coveralls data manually from my local without providing any write permissions. I'm not sure if this was for Coveralls or Travis

This is what it's sending:
https://github.com/Risto-Stevcev/bastet/blob/master/Makefile#L82

I'll post a bit later today what entries in the json it's sending specifically

[Risto-Stevcev]

This is structure of the coverage.json that I'm sending:

{
  "repo_token": "foo",
  "source_files": [
    {
      "name": "bastet/src/Dual.re",
      "source_digest": "5aae0a60ef729d60b812ad3687e21efb",
      "coverage":      [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,null,null,null,0,0,null,null,null,null,0,null,null,null,null,null,32,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,null,0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,0,null,null,null,0,null,0,null,null,null,null,null,null,null]
    },
    ...
  ]
}

[afinetooth]

Commands look correct. Only one more required value missing from that POST body, which is service_name, which can be anything besides the values for known CI's.

Can you try sending?:

{
  "repo_token": "foo",
  "service_name": "local",
  "source_files": [
    {
      "name": "bastet/src/Dual.re",
      "source_digest": "5aae0a60ef729d60b812ad3687e21efb",
      "coverage":      [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,null,null,null,0,0,null,null,null,null,0,null,null,null,null,null,32,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,null,0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,0,null,null,null,0,null,0,null,null,null,null,null,null,null]
    },
    ...
  ]
}

Also, try calling coveralls in verbose mode.

That will add some debug output to stdout and give us insight into exactly what you're sending and how the API is responding.

Looks like it can be set with bisect_ppx here:
https://github.com/aantron/bisect_ppx/blob/6f294d1691726e7bb03cbee1daeb61985103ae27/src/report/report.ml#L144

[Risto-Stevcev]

How do I run verbose mode? I'm just using the API:
https://docs.coveralls.io/api-reference

This is the output with the "service_name": "local",:

curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs
{"message":"Couldn't find a repository matching this job.","error":true}

[afinetooth]

@Risto-Stevcev based on that API response you will need to make sure your repo_token is part of the JSON in your coverage.json file. You can grab that from the settings page for your repo.

Apologies, verbose mode is not available for cURL, only for specific integrations. It's the integrations that display the debug output on what they're packaging up to POST to our API.

[Risto-Stevcev]

This is what I'm getting with curl's verbose mode:

$ curl -v -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs                                                                                                                                                                                  *   Trying 172.67.72.111:443...                                     
* Connected to coveralls.io (172.67.72.111) port 443 (#0)                              
* ALPN, offering h2                                                                    
* ALPN, offering http/1.1                                                                                                                                                                                                                                                         * successfully set certificate verify locations:                                                                                         
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt                                           
  CApath: none                                                                         
* TLSv1.3 (OUT), TLS handshake, Client hello (1):                                      
* TLSv1.3 (IN), TLS handshake, Server hello (2):                                       
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):                               
* TLSv1.3 (IN), TLS handshake, Certificate (11):                                                
* TLSv1.3 (IN), TLS handshake, CERT verify (15):                                                           
* TLSv1.3 (IN), TLS handshake, Finished (20):                                                                                             
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):                                                                               * TLSv1.3 (OUT), TLS handshake, Finished (20):                                                                                            
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384                                                   
* ALPN, server accepted to use h2                                                                         
* Server certificate:                                                                                     
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com                    
*  start date: Aug  9 00:00:00 2020 GMT                                                                   
*  expire date: Aug  9 12:00:00 2021 GMT                                                                  
*  subjectAltName: host "coveralls.io" matched cert's "coveralls.io"                                                                     
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3                                                                          
*  SSL certificate verify ok.                                       
* Using HTTP2, server supports multi-use                            
* Connection state changed (HTTP/2 confirmed)                                                                                            
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0                                                         
* Using Stream ID: 1 (easy handle 0x5622e3f45e80)                                                                                        
> POST /api/v1/jobs HTTP/2                                          
> Host: coveralls.io                                                
> user-agent: curl/7.71.1                                           
> accept: */*                                                       
> content-length: 46216                                             
> content-type: multipart/form-data; boundary=------------------------5e401e11a77ba95e                                                   
>                                                                   
* We are completely uploaded and fine                               
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):                                                                                    
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):                                                                                    
* old SSL session ID is stale, removing                             
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!                                                                              < HTTP/2 200                                                        
< date: Mon, 15 Feb 2021 00:29:03 GMT                               
< content-type: application/json; charset=utf-8                                                                                          
< content-length: 66                                                
< set-cookie: __cfduid=d3e80e06ede8293c4d5646cd7fafd5c271613348942; expires=Wed, 17-Mar-21 00:29:02 GMT; path=/; domain=.coveralls.io; HttpOnly; SameSite=Lax
< cache-control: max-age=0, private, must-revalidate                                                                                     
< etag: W/"832537077610c81f8e833fd39f5eed79"                                                                                             
< status: 200 OK                                                    
< strict-transport-security: max-age=15552000                                                                                            
< x-content-type-options: nosniff                                   
< x-frame-options: SAMEORIGIN                                       
< x-powered-by: Phusion Passenger Enterprise 6.0.5                                                                                       
< x-powered-by: cloud66                                             
< x-request-id: c4453af7-ab54-4dc0-b031-dcfbb2d2a7cd                                                                                     
< x-runtime: 0.374400                                               
< x-xss-protection: 1; mode=block                                   
< cf-cache-status: DYNAMIC                                          
< cf-request-id: 0844b04b9200001ca06739a000000001                                                                                        
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"                                     
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=zjeWMxJf%2BeopkwOVKmmwPBkbywdwtSzHb0PdgNPvCaLlf2Yfa5ZoLkn%2FdnMxJQisQwzgGK5wxNAQ8Mlsgl%2FvKb1ZCCa5A4ZgzesTInc%3D"}]}
< nel: {"report_to":"cf-nel","max_age":604800}                                                                                           
< server: cloudflare                                                
< cf-ray: 621ae98c181e1ca0-SOF                                      
<                                                                   
* Connection #0 to host coveralls.io left intact                                                                                         
{"message":"Job ##5.1","url":"https://coveralls.io/jobs/75101086"}

[afinetooth]

@Risto-Stevcev that response indicates success in that it includes a Job URL matching the (new) job you just created (cURL status: 200 OK):

{"message":"Job ##5.1","url":"https://coveralls.io/jobs/75101086"}

You can see the report for the job at:
https://coveralls.io/jobs/75101086

And for the associated build at:
https://coveralls.io/builds/37122123

The coverage badge still shows unknown at this point because, in order to get badge coverage, which is the main coverage stat for your repo, you need at least one build against the repo's default branch, which in most cases is master.

Note that you are not passing branch data with your builds (the BRANCH field reads (?) for each build here):
https://coveralls.io/repos/294312/builds

At this point, you'll want to make sure you're supplying info for the git parameter, per the API Reference here:
https://docs.coveralls.io/api-reference

Example:

"git": {
  "head": {
    "id": "b31f08d07ae564b08237e5a336e478b24ccc4a65",
    "author_name": "Nick Merwin",
    "author_email": "...",
    "committer_name": "Nick Merwin",
    "committer_email": "...",
    "message": "version bump"
  },
 "branch": "master",
 "remotes": [
  {
    "name": "origin",
    "url": "git@github.com:lemurheavy/coveralls-ruby.git"
  }]}

That git parameter is labeled "optional" for standard usage because standard usage is to build your project in CI, and CI will automatically pass along the data in that git hash.

[Risto-Stevcev]

Thanks that worked. It sounds like this is expected behavior -- Is it possible to add a note in the API docs about this?
Something like:
Note: The Coveralls badge will show "Coverage: unknown" if you don't supply the git properties in coverage.json when sending via API

I'll try to get travis fully setup but it's a bit of a headache with the current setup

[afinetooth]

@Risto-Stevcev happy to hear it. That is the expected behavior.

Good suggestion on the doc update. I added the following note to the API Reference:

NOTE:
While OPTIONAL for seeing coverage reports at Coveralls, git (or commit_sha) is REQUIRED to receive coverage reports at your git host (ie. Github). This includes coverage badges, pull request checks and pull request comments. Coverage badges pertain only to the default branch of your repo and therefore require branch data. Coveralls requires git/commit data to send checks and comments to the right pull request.

Submitted a PR. It should appear here in the next 24-48 hrs.

Closing this issue for now. We can re-open if necessary.