You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Oct 12, 2023. It is now read-only.
Description of the environment:
We have a microservices running in Azure Kubernetes Services (AKS), which communicates with Azure Key Vault to obtain certain secrets from there. We use Azure Spring plugin to reach Azure Key Vault. For authentication we use an Azure service principal. Everything was work smoothly. Description of the issue:
Recently we added Istio Service Mesh and implemented a Control Egress Traffic Istio for Azure Key Vault:
_apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: azurekeyvault
spec:
hosts:
azurevaultxxxxx.vault.azure.net
ports:
number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
Now, we are getting an exception ("Failed to do authentication") and here is the log: java.lang.IllegalStateException: Failed to configure KeyVault property source
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:72) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:22) ~[azure-spring-boot-2.0.6.jar!/:na]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:184) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:170) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:358) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at foa3k.configserver.ConfigServerApplication.main(ConfigServerApplication.java:12) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]
Caused by: java.lang.IllegalStateException: Failed to do authentication.
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:48) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.getAuthenticationCredentials(KeyVaultCredentials.java:113) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.access$100(KeyVaultCredentials.java:27) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$2.authenticate(KeyVaultCredentials.java:81) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.followUpRequest(RetryAndFollowUpInterceptor.java:284) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:152) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$1.intercept(KeyVaultCredentials.java:59) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.execute(RealCall.java:69) ~[okhttp-3.8.1.jar!/:na]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[retrofit-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40) ~[adapter-rxjava-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[adapter-rxjava-2.4.0.jar!/:na]
at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10247) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10214) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.single(BlockingObservable.java:341) ~[rxjava-1.2.0.jar!/:1.2.0]
at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecrets(KeyVaultClientImpl.java:2951) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.KeyVaultClient.listSecrets(KeyVaultClient.java:911) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.createOrUpdateHashMap(KeyVaultOperation.java:88) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.(KeyVaultOperation.java:43) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:62) ~[azure-spring-boot-2.0.6.jar!/:na]
... 22 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_181]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_181]
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:45) ~[azure-spring-boot-2.0.6.jar!/:na]
... 68 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[na:1.8.0_181]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[na:1.8.0_181]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) ~[na:1.8.0_181]
at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:148) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:86) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:85) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:880) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:68) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:176) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:165) ~[adal4j-1.3.0.jar!/:1.3.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Based on our analysis of the source code available from GitHub, it appears that very likely failure happens during attempt to authenticate HTTP Istio proxy (see RetryAndFollowUpInterceptor.java:284), not microservice itself.
So, we are really trying to get clarity why Istio proxy is not properly handled here.
Also, how Azure Key Vault treats proxies and how they should be configured when client behind them uses an Azure service principal for authentication.
For clarity we tried other non Azure Key Vault related http calls and they are going smoothly (without any problems). We have problem only when our microservice tries to reach Azure Key Vault by using Azure Spring plugin.
Description of the environment:
We have a microservices running in Azure Kubernetes Services (AKS), which communicates with Azure Key Vault to obtain certain secrets from there. We use Azure Spring plugin to reach Azure Key Vault. For authentication we use an Azure service principal. Everything was work smoothly.
Description of the issue:
Recently we added Istio Service Mesh and implemented a Control Egress Traffic Istio for Azure Key Vault:
_apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: azurekeyvault
spec:
hosts:
ports:
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: azurekeyvault
spec:
hosts:
tls:
sni_hosts:
route:
host: azurevaultxxxxx.vault.azure.net
port:
number: 443
weight: 100_
Now, we are getting an exception ("Failed to do authentication") and here is the log:
java.lang.IllegalStateException: Failed to configure KeyVault property source
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:72) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:22) ~[azure-spring-boot-2.0.6.jar!/:na]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:184) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:170) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:358) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at foa3k.configserver.ConfigServerApplication.main(ConfigServerApplication.java:12) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]
Caused by: java.lang.IllegalStateException: Failed to do authentication.
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:48) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.getAuthenticationCredentials(KeyVaultCredentials.java:113) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.access$100(KeyVaultCredentials.java:27) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$2.authenticate(KeyVaultCredentials.java:81) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.followUpRequest(RetryAndFollowUpInterceptor.java:284) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:152) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$1.intercept(KeyVaultCredentials.java:59) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.execute(RealCall.java:69) ~[okhttp-3.8.1.jar!/:na]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[retrofit-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40) ~[adapter-rxjava-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[adapter-rxjava-2.4.0.jar!/:na]
at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10247) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10214) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.single(BlockingObservable.java:341) ~[rxjava-1.2.0.jar!/:1.2.0]
at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecrets(KeyVaultClientImpl.java:2951) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.KeyVaultClient.listSecrets(KeyVaultClient.java:911) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.createOrUpdateHashMap(KeyVaultOperation.java:88) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.(KeyVaultOperation.java:43) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:62) ~[azure-spring-boot-2.0.6.jar!/:na]
... 22 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_181]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_181]
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:45) ~[azure-spring-boot-2.0.6.jar!/:na]
... 68 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[na:1.8.0_181]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[na:1.8.0_181]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) ~[na:1.8.0_181]
at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:148) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:86) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:85) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:880) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:68) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:176) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:165) ~[adal4j-1.3.0.jar!/:1.3.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Based on our analysis of the source code available from GitHub, it appears that very likely failure happens during attempt to authenticate HTTP Istio proxy (see RetryAndFollowUpInterceptor.java:284), not microservice itself.
So, we are really trying to get clarity why Istio proxy is not properly handled here.
Also, how Azure Key Vault treats proxies and how they should be configured when client behind them uses an Azure service principal for authentication.
For clarity we tried other non Azure Key Vault related http calls and they are going smoothly (without any problems). We have problem only when our microservice tries to reach Azure Key Vault by using Azure Spring plugin.