Red Gate forums :: View topic - Setting up Deployment Manager
Return to www.red-gate.com RSS Feed Available

Search  | Usergroups |  Profile |  Messages |  Log in  Register 
Go to product documentation
Deployment Manager
Deployment Manager forum

Setting up Deployment Manager

Search in Deployment Manager forum
Post new topic   Reply to topic
Jump to:  
Author Message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Fri Oct 04, 2013 9:26 pm    Post subject: Setting up Deployment Manager Reply with quote

We are having all sorts of problems getting Deployment Manager (DM) setup in our environment.

Initially we tried to use the Windows Authentication and the DM portal would never come up; it would generate a 500 internal server error.

I found that if I didn't check the use Windows Authentication, the portal would come up.

Now connecting target machines is a problem. The agent that is installed on the same machine as the manager appears to come up. Sometimes it doesn't thought.

The bigger problem is the database server target doesn't come up. It throws an error:

Failed: http://eerepagedevdb1:10301/

2013-10-04 16:21:58 -04:00 ERROR System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryGetChannel()
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel)
at System.ServiceModel.Channels.ClientReliableChannelBinder`1.Request(Message message, TimeSpan timeout, MaskingMode maskingMode)
at System.ServiceModel.Channels.RequestReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
at System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableRequestSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at RedGate.Deploy.Shared.Contracts.IHealthService.CheckHealth()
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at CallSite.Target(Closure , CallSite , Object )
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberCallSite(Object target, String_OR_InvokeMemberName name, Object[] args, String[] tArgNames, Type tContext, Boolean tStaticContext, CallSite& callSite)
at ImpromptuInterface.Dynamic.ImpromptuForwarder.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at RedGate.Deploy.Server.Proxies.WcfSecurityWrapper`1.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object )
at ActLike_IHealthService_a341aaf64f0449348fe6942ca27b2141.CheckHealth()
at RedGate.Deploy.Server.Tasks.Health.CheckAgentHealthActivity.<Execute>b__0()

My systems guy disabled the firewalls on both manager and agent machines with no success.

It did appear to work once. I asked the systems guy and he said he just ran the agent tool on the DB server as an administrator... We rebooted both machines to see if we could pinpoint what was happening, now the DB1 agent won't connect.

Very frustrating...

What log files is the error referring to? I looked in the Red Gate logs directory, but you can't look at the those files while the server is up.

Man, haven't even really done anything yet and having all these problems. I'd hate to see what problems we'd have when we try and connect to the production servers in a DMZ.

Help Sad
Back to top
View user's profile Send private message
robin



Joined: 16 Nov 2007
Posts: 15

PostPosted: Mon Oct 07, 2013 10:57 am    Post subject: Reply with quote

Hi Steve,

Messages from the deployment agents will be logged to the windows application log, and can be examined with the windows event viewer.

Please could you look in the services control panel and make sure that the service 'Red Gate Deployment Agent' is started on both machines that you wish to connect to. If its status does not show as started, could you try to start it and then try to connect using deployment manager again. If it doesn't work can you send us any messages that you see in the event log.

Is it possible that you have another service running on those machines that is also trying to bind to port 10301?

Regarding the problems you experienced with windows authentication, if you turn the feature back on and check the portal again, then you should get a message in the event log from asp.net. Please could you also let us have those details.
_________________
Robin Hellen
Test Engineer
Deployment Manager
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Mon Oct 07, 2013 2:25 pm    Post subject: Reply with quote

Thanks for the reply. I'll check those items. It's interesting that when I came in and looked at the status screen both targets machines had a green check mark. When I manually performed a help check, the connection to our DB server failed again. I looked at the history and it appears as if sometimes it connects and sometimes it doesn't. Here is the activity log for today and output from a successful help check:

Failed Manual health check Steve Monday, October 07, 2013 9:19 AM -04:00
Successful Check Agent health [System] Monday, October 07, 2013 8:59 AM -04:00
Successful Check for updated version [System] Monday, October 07, 2013 8:54 AM -04:00
Failed Check Agent health [System] Monday, October 07, 2013 8:29 AM -04:00
Successful Scheduled database backup [System] Monday, October 07, 2013 8:02 AM -04:00
Failed Check Agent health [System] Monday, October 07, 2013 7:59 AM -04:00
Failed Check Agent health [System] Monday, October 07, 2013 7:29 AM -04:00
Successful Check Agent health [System] Monday, October 07, 2013 6:59 AM -04:00
Successful Check for updated version [System] Monday, October 07, 2013 6:54 AM -04:00
Failed Check Agent health [System] Monday, October 07, 2013 6:29 AM -04:00
Failed Check Agent health [System] Monday, October 07, 2013 5:59 AM -04:00
Successful Check for updated version [System] Monday, October 07, 2013 4:54 AM -04:00
Successful Scheduled database backup [System] Monday, October 07, 2013 4:01 AM -04:00
Successful Check for updated version [System] Monday, October 07, 2013 2:54 AM -04:00
Successful Check for updated version [System] Monday, October 07, 2013 12:54 AM -04:00
Successful Scheduled database backup [System] Monday, October 07, 2013 12:04 AM -04:00

---------------------------------------------------------------------

Check Agent health

2013-10-07 08:59:23 -04:00 INFO The following Agents will be checked:
2013-10-07 08:59:23 -04:00 INFO - http://eerepagedevweb1:10301/
2013-10-07 08:59:23 -04:00 INFO - http://eerepagedevdb1:10301/
2013-10-07 08:59:23 -04:00 INFO Begin health checks
2013-10-07 08:59:24 -04:00 INFO
2013-10-07 08:59:24 -04:00 INFO Health results:
2013-10-07 08:59:24 -04:00 INFO - ONLINE: http://eerepagedevweb1:10301/, running version 2.2.16.1 on machine EEREPAGEDEVWEB1 as SYSTEM
2013-10-07 08:59:24 -04:00 INFO - ONLINE: http://eerepagedevdb1:10301/, running version 2.2.16.1 on machine EEREPAGEDEVDB1 as SYSTEM
2013-10-07 08:59:24 -04:00 INFO Health check complete!

Success: http://eerepagedevweb1:10301/

2013-10-07 08:59:23 -04:00 INFO Health check successful. Running version: 2.2.16.1

Success: http://eerepagedevdb1:10301/

2013-10-07 08:59:24 -04:00 INFO Health check successful. Running version: 2.2.16.1
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Mon Oct 07, 2013 3:38 pm    Post subject: Reply with quote

Messages from the deployment agents will be logged to the windows application log, and can be examined with the windows event viewer.

I looked in the Event Viewer --> Windows Logs --> Application for the EEREPAGEDEVDB1 server where the agent is failing and there were no log entries after I tried a Health Check and it failed.

I did see entries on the EEREPAGEDEVWEB1 server where the deployment manager runs. The entries are from the Deployment Manager and are an error:

2013-10-07 09:29:19,898 [40] ERROR RedGate.Deploy.Server.Tasks.TaskRunner [(null)] - System.AggregateException: One or more errors occurred. ---> RedGate.Deploy.Server.Tasks.ActivityFailedException: One or more Agents were not online. Please see the output log for details.
at RedGate.Deploy.Server.Tasks.Health.HealthControllerActivity.<Execute>d__6.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at RedGate.Deploy.Server.Tasks.TaskRunner.<>c__DisplayClass1.<Execute>b__0()
---> (Inner Exception #0) RedGate.Deploy.Server.Tasks.ActivityFailedException: One or more Agents were not online. Please see the output log for details.
at RedGate.Deploy.Server.Tasks.Health.HealthControllerActivity.<Execute>d__6.MoveNext()<---

Followed by a warning:

2013-10-07 09:29:20,008 [40] WARN RedGate.Deploy.Server.Tasks.TaskRunner [(null)] - Task tasks-466 exited with error One or more Agents were not online. Please see the output log for details.

Please could you look in the services control panel and make sure that the service 'Red Gate Deployment Agent' is started on both machines that you wish to connect to.

EEREPAGEDEVDB1

Red Gate Deployment Agent is running

EEREPAGEDEVWEB1

Red Gate Deployment Agent is running
Red Gate Deployment Manager is running

Here is some additional information:

From EEREPAGEDEVWEB1

portqry -n EEREPAGEDEVDB1 -e 10301

192.168.5.179

TCP port 10301 (unknown service): LISTENING

portqry -n EEREPAGEDEVWEB1 -e 10301

192.168.5.174

TCP port 10301 (unknown service): LISTENING

netstat -ano | findstr 10301

TCP 0.0.0.0:10301 0.0.0.0:0 LISTENING 4
TCP [::]:10301 [::]:0 LISTENING 4

PID 4: System

From EEREPAGEDEVDB1

portqry and netstat same as EEREPAGEDEVWEB1 results
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1113
Location: My desk.

PostPosted: Mon Oct 07, 2013 8:40 pm    Post subject: Reply with quote

Thanks Steve.

I'm assuming both the machines are just on the same network segment based on the IP addresses- is there anything odd like multiple network cards in one or other machine?

Does the healthcheck to the local agent instance on the DM server suffer the same issue, or does that always work?

Another thing to check - though unlikely - is that you've disabled the power-saving feature on the network card in Device Manager in case it's going to sleep during times of inactivity and isn't reactivating quickly enough for things to work?
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Mon Oct 07, 2013 9:46 pm    Post subject: Reply with quote

Both machines are on an internal network segment.

The health check appears to work fine on EEREPAGEDEVWEB1. Both the agent and the manager are on this machine.

Both machines are awake when the errors occur.

These machines are setup on as virtual servers.


Last edited by SteveGTR on Tue Oct 08, 2013 3:14 pm; edited 1 time in total
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Mon Oct 07, 2013 9:54 pm    Post subject: Reply with quote

I tried using each of the DB1 IP addresses as Agent URL on the Edit Target Machine dialog and each failed with the same message:

System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party.

I tried to regenerate the server keys and resetting both the environment setting and the agent setting on each server. The DB server is still offline.
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Mon Oct 07, 2013 10:24 pm    Post subject: Reply with quote

I disabled the two VMware interfaces on the DB1 server and removed all but the 1 IP address on WEB1, flushed the DNS cache on WEB1, and retried and it still fails on DB1.

At a loss...
Back to top
View user's profile Send private message
robin



Joined: 16 Nov 2007
Posts: 15

PostPosted: Tue Oct 08, 2013 10:07 am    Post subject: Reply with quote

Hi Steve,

We have seen this issue before when the clocks on the agent and the server machine are out of sync. (see our troubleshooting page)
_________________
Robin Hellen
Test Engineer
Deployment Manager
Back to top
View user's profile Send private message
SteveGTR



Joined: 11 Jan 2012
Posts: 66

PostPosted: Tue Oct 08, 2013 3:07 pm    Post subject: Reply with quote

Eureka! That was it. The DB server's time was 45 minutes ahead of the actual time. Resetting the time on the DB server solved the problem --- whew.

Thanks for all your help. I can now move on to the important stuff Smile
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic All times are GMT + 1 Hour
Page 1 of 1

 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group