We got any Python 'experts' here on TA?

U

User.191

Guest
Looking for help with getting Python to be able to validate a SSL cert - signed by Digi for a wildcard but I'm getting the following error when I try to connect to one of our servers that's protected with it:

Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

I can bypass it by telling the connection to not verify the cert, but I'd really like to not do that if I can help it.
 
D

Deleted member 199

Guest
I only dabble in python but it looks like there's an intermediate cert missing.

Most CA's don't issue certs from their root, they have 1, 2 or 3 intermediaries, forming a chain.

You probably need have the server use a cert bundle, with the intermediaries from the CA.
 
U

User.191

Guest
I only dabble in python but it looks like there's an intermediate cert missing.

Most CA's don't issue certs from their root, they have 1, 2 or 3 intermediaries, forming a chain.

You probably need have the server use a cert bundle, with the intermediaries from the CA.
Yeah, I figured that and I got the bundle downloaded. I've zero clue as to where these actually need to live though.

Best I've found is something like a CABundle but am unsure where that would be.

Also guessing it doesn't need the PEM - just the intermediate cert. We've solved this issue on a Tomcat server by using

Code:
openssl pkcs7 -inform PEM -outform DER -in bundle.p7b -print_certs | sed -n '/subject=\/C=US\/O=DigiCert Inc\/CN=DigiCert/,/END C/p'|tail -n +3 > Intermediate.ce

Then adding that to the jks file.

Dunno what the analog would be with Python though.
 
D

Deleted member 199

Guest
That is some crazy voodoo right there to extract certificate chunks but ok.

So is python doing the TLS termination itself? Do you know what app server it's using? Or is it behind something like haproxy or nginx or something, and then proxying over plain http, wsgi or similar to the actual python service?

Typically you either specify an all-in-one bundle (i.e. cert + any intermediaries, the last of which is signed by the root cert that the client already has) - HAProxy expects this type of cert if you're using SNI to load a directory of certs; or you specify a cert (just the cert) + a 'chain' that is typically a bundle of 1 or more intermediates (apache works this way via the `SSLCertificateChainFile` directive).
 
U

User.191

Guest
That is some crazy voodoo right there to extract certificate chunks but ok.

So is python doing the TLS termination itself? Do you know what app server it's using? Or is it behind something like haproxy or nginx or something, and then proxying over plain http, wsgi or similar to the actual python service?

Typically you either specify an all-in-one bundle (i.e. cert + any intermediaries, the last of which is signed by the root cert that the client already has) - HAProxy expects this type of cert if you're using SNI to load a directory of certs; or you specify a cert (just the cert) + a 'chain' that is typically a bundle of 1 or more intermediates (apache works this way via the `SSLCertificateChainFile` directive).
Here's what I'm doing:

Python:
import datetime
import tableauserverclient as TableauServer

HYPER = '\\\\Host.test.local\\Hypers$\\Test.hyper'
TOKEN_ID = 'TestUser'
TOKEN = '***SuperSecretSquirrelToken***'
HOST = 'https://tableau-test.<hiddendomain>.com'
SITE_ID = 'TableauTest'
PROJECT_ID = '9c36314f-44ee-3e4c-cc99-d2f1299944e8'

_tableauServer = TableauServer.Server(HOST, use_server_version=False)
_tableauServer.add_http_options({'verify': False})
auth = TableauServer.PersonalAccessTokenAuth(TOKEN_ID, TOKEN, SITE_ID)
publish_mode = TableauServer.Server.PublishMode.Overwrite
with _tableauServer.auth.sign_in(auth):
    new_datasource = TableauServer.DatasourceItem(PROJECT_ID)

There's nothing I can see in the library that allows me to specify where the bundle lies, so I'm left looking for where Python thinks it might be.

Cue obligatory XKCD:

1626206588022.png
 
D

Deleted member 199

Guest
that's the 'client' right? You need to have the server set to include the intermediates in it's response. Edit: whatever is running `https://tableau-test.<hiddendomain>.com` in this example.

Is the 'server' end also python?
 
U

User.191

Guest
Ok I think this page will help, assuming you or some one who can think and breath at the same time, has administrator access to the server.

Nope, it's not that. That's all working just jim-dandy.

We've Java based clients that work just fine because I knew were to stick the pieces from the bundle.
 
D

Deleted member 199

Guest
… it’s very weird to be putting the *bundle* client side.

the only way I can think you’d do that for python is to install the extra intermediates in the system ssl dir as individual certs if it’s more than one in the chain, then run openssl -rehash.

But that’s still a very weird solution IMO.
 
U

User.191

Guest
… it’s very weird to be putting the *bundle* client side.

the only way I can think you’d do that for python is to install the extra intermediates in the system ssl dir as individual certs if it’s more than one in the chain, then run openssl -rehash.

But that’s still a very weird solution IMO.

Yeah - this has been the story throughout - and I think it may be somewhat particular to DigiCert as well.

I didn't think about running openssl rehash though - I'm gonna give that a try.
 
U

User.191

Guest
… it’s very weird to be putting the *bundle* client side.

the only way I can think you’d do that for python is to install the extra intermediates in the system ssl dir as individual certs if it’s more than one in the chain, then run openssl -rehash.

But that’s still a very weird solution IMO.
The thick has plottened. Since I needed this to work mainly on WIndows, I installed PyCharm onto one of my Windows servers and tested it there, getting the same errors.

So I then started to try all manner of things - mainly in the Certificate stores for both CurrentUser and LocalMachine. Still nothing. I played around with various files as well. Played around with a lot in fact, each attempt yielding failure.

Right up until I came across the python-certifi-win32 module. And finally it worked!

Sadly, I can't replicate this on another server!!!! I've the same code running the same version of Python etc. and I still can't replicate my success....

And since I made so many changes earlier on I'm lost as to which was the one that made this work. I know it's server related because on of my team was able to execute the same code on that server with success - so whatever the change was it's not local to my user.

So now I'm going to dig into the source to that module to see just what its doing which might help me find "The Answer".
 

thekev

Elite Member
Posts
1,102
Reaction score
1,661
The thick has plottened. Since I needed this to work mainly on WIndows, I installed PyCharm onto one of my Windows servers and tested it there, getting the same errors.

So I then started to try all manner of things - mainly in the Certificate stores for both CurrentUser and LocalMachine. Still nothing. I played around with various files as well. Played around with a lot in fact, each attempt yielding failure.

Right up until I came across the python-certifi-win32 module. And finally it worked!

Sadly, I can't replicate this on another server!!!! I've the same code running the same version of Python etc. and I still can't replicate my success....

And since I made so many changes earlier on I'm lost as to which was the one that made this work. I know it's server related because on of my team was able to execute the same code on that server with success - so whatever the change was it's not local to my user.

So now I'm going to dig into the source to that module to see just what its doing which might help me find "The Answer".

Think of me as an armchair expert.

First, I sincerely hope you're using Python 3.6 or above.

This is largely an SSL/Tableau issue rather than a Python one in spite of the Java mentions.
The exception it's raising isn't built in at all. It's presumably derived from RuntimeError or similar by whatever wraps the SSL libraries here.

You didn't include it, but I assume that exception is being triggered by the context manager, which signs you in/out there. Ignoring same Python version, do their relevant environment variables and OpenSSL (or other SSL library) versions match?

A lot of the Linux stuff is a bit tenuous on Windows, ever since Windows diverged from Linux on the bitwidth of "long" types.
 
U

User.191

Guest
Think of me as an armchair expert.

First, I sincerely hope you're using Python 3.6 or above.

This is largely an SSL/Tableau issue rather than a Python one in spite of the Java mentions.
The exception it's raising isn't built in at all. It's presumably derived from RuntimeError or similar by whatever wraps the SSL libraries here.

You didn't include it, but I assume that exception is being triggered by the context manager, which signs you in/out there. Ignoring same Python version, do their relevant environment variables and OpenSSL (or other SSL library) versions match?

A lot of the Linux stuff is a bit tenuous on Windows, ever since Windows diverged from Linux on the bitwidth of "long" types.
3.9

Right now I'm trying to work out a missing SSL library - I ripped all the environment stuff from the second server and recreated my project. Right now I'm getting a "Can't connect to HTTPS URL because the SSL module is not available." error...
 
U

User.191

Guest
3.9

Right now I'm trying to work out a missing SSL library - I ripped all the environment stuff from the second server and recreated my project. Right now I'm getting a "Can't connect to HTTPS URL because the SSL module is not available." error...
OK, got past that - had a bad configuration.

However I'm back to code not validating the cert. If only I knew what the one change I had made was...
 

thekev

Elite Member
Posts
1,102
Reaction score
1,661
3.9

Right now I'm trying to work out a missing SSL library - I ripped all the environment stuff from the second server and recreated my project. Right now I'm getting a "Can't connect to HTTPS URL because the SSL module is not available." error...

That is a very explicit error. Are you missing the appropriate package from pip or conda? If not, you may just have an environment variable missing somewhere. They're indexed using plain text keys in os.environ, but it sounds like it's not installed in that environment. If you're using Pycharm, it may install its own environment, which differs from your system Python.

You can always tell what Python a terminal will load using "which python" or "which python3" in a unix-like terminal or "where ..." in a cmd like one. This is the kind of thing I could probably fix in person, but it's difficult to do it just by reading your error messages. I've never come across these exact ones before, as I rarely have to deal with SSL issues.

As a last resort, you can trace through this using pdb. It's quite informative for stuff like this at times, just time consuming.
 
U

User.191

Guest
That is a very explicit error. Are you missing the appropriate package from pip or conda? If not, you may just have an environment variable missing somewhere. They're indexed using plain text keys in os.environ, but it sounds like it's not installed in that environment. If you're using Pycharm, it may install its own environment, which differs from your system Python.

You can always tell what Python a terminal will load using "which python" or "which python3" in a unix-like terminal or "where ..." in a cmd like one. This is the kind of thing I could probably fix in person, but it's difficult to do it just by reading your error messages. I've never come across these exact ones before, as I rarely have to deal with SSL issues.
Like I say, that was a farked up environment (somehow - it was brand spanking new!).
 

thekev

Elite Member
Posts
1,102
Reaction score
1,661
Like I say, that was a farked up environment (somehow - it was brand spanking new!).

So others have come across similar but non-identical issues on stackoverflow. They get around it by manually loading certificates, although this may require you to check how Tableau's interface is loading them, which you probably can. It seems somewhat unnecessarily complicated. If I had to do something like this, I would probably make my own context wrapper on top of theirs to hide the ugliness. You can do this with contextlib.ContextDecorator or similar.



 
U

User.191

Guest
So others have come across similar but non-identical issues on stackoverflow. They get around it by manually loading certificates, although this may require you to check how Tableau's interface is loading them, which you probably can. It seems somewhat unnecessarily complicated. If I had to do something like this, I would probably make my own context wrapper on top of theirs to hide the ugliness. You can do this with contextlib.ContextDecorator or similar.




Trust me, Tableau IS ugly - horribly ugly. How ugly you ask? Ugly enough for it to take up to THIRTY FUCKING MINUTES just to restart it in out test environment.

The only reason we're even doing this is because the API that they normally give you to do what we're trying to do is totally shit as well. How shitty is it you ask? Shitty enough for them to NOT USE IT themselves!

Right now I'm left trying to see what the one change I made on Server X1 this morning that I haven't made on server X2. I know it's down to the intermediate certificate and it looks like Tableau won't send the chain correctly, hence the need to do what we're doing.
 
U

User.191

Guest
FINALLY!

Sussed it - the reason this was so hard to find is that when I tried to find out what I'd done on server X1 I went into the cert store and ripped out all the certificates I could find relating to this wildcard, and yet STILL the app ran fine.

The reason being that what that python-certifi-win32 module does is finds any certificates it feels are part of the chain and drops them into the %LOCALAPPDATA%\.certifi\cacert.pem file.

It never cleans them up though if they're deleted.

So the answer was just just clone that file and now it's working...

Holy krapballz
 
Top Bottom