Red Gate forums :: View topic - How to search all the deployment logs at once?
Return to www.red-gate.com RSS Feed Available

Search  | Usergroups |  Profile |  Messages |  Log in  Register 
Go to product documentation
Deployment Manager
Deployment Manager forum

How to search all the deployment logs at once?

Search in Deployment Manager forum
Post new topic   Reply to topic
Jump to:  
Author Message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Thu Dec 19, 2013 7:49 pm    Post subject: How to search all the deployment logs at once? Reply with quote

Hey RG,

It's easy to search the log for a single deployment.

You can use your browser's built-in search functions once you load the log in your browser.

But can you do the same across all logs for a given project?

I'm trying to find old logs that contained a particular error message to help explain why we changed one of our deployment procedures.

I don't see any way to do it in the web interface.

I can't find the log files on the file system, or in the RavenDB database.

Maybe I'm looking in the wrong place.

Do you know a way to do this?

Happy Christmas to you all!
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Fri Dec 20, 2013 4:18 pm    Post subject: Reply with quote

Hi Iain- the logs live on the agents, in this folder:

C:\ProgramData\Red Gate\DeploymentAgent\Applications\.Agent\Logs

What I can't quite find is where the GUID these files use as their names is stored... the 'Deployments' document in the RavenDB doesn't seem to obviously hold it.

I'll see if I can find that out next week.
Back to top
View user's profile Send private message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Fri Dec 20, 2013 5:13 pm    Post subject: Reply with quote

Thanks, James.

What about SQL Server deployment logs?

SQL Server deployment agents run on the RGDM host.

On my RGDM host the folder

C:\ProgramData\Red Gate
does not contain a folder called DeploymentAgent.

It contains just one folder called Licenses, which is empty.
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Mon Dec 23, 2013 1:50 pm    Post subject: Reply with quote

I've checked with the team, and the SQL logs never get as far as ending up on disk; they get streamed straight between the special SQL deployment agent and Deployment Manager itself.

For both Agent and SQL deployments, the logs are stored in the Raven DB as document attachments, however the Raven Studio viewer (http://localhost:10300) doesn't appear to offer any easy way to view these. You may be able to use the commandline tool to export them out though: http://ravendb.net/docs/server/administration/export-import
Back to top
View user's profile Send private message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Mon Jan 13, 2014 3:37 pm    Post subject: Reply with quote

Thanks, James.

Here's the Smuggler command I used:

Code:

Raven.Smuggler out http://localhost:10300/ dump.raven --operate-on-types=Attachments


It finished with this message:

Code:

Done with reading attachments, total: 1764


It created a file called dump.raven approximately 13MB in size.

I opened it in Notepad++.

The attachments appear to be serialized as JSON, but they don't look familiar to me.

I can't paste a complete example because it contains binary data.

Each attachment looks a bit like this (I've elided the data stream):

Code:

   {
      "Data": "H4sIAAAAAAAEAOy9B2AcSZYl[...]ubBEevVOAAA=",
      "Metadata": {
        "ContentType": "text/xml"
      },
      "Key": "attachments/tasks-34/output-log"
    }


How should I decode this?

As an aside, apparently you can't view attachments in Raven Studio because the developers "don't want to encourage their use." See issue RavenDB-909 on the Hibernating Rhinos issue tracker.
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
Mike Upton



Joined: 11 May 2011
Posts: 108
Location: Red Gate

PostPosted: Mon Jan 13, 2014 4:34 pm    Post subject: Reply with quote

Hi Iain,

The Data elements in those JSON objects look like they're base64 encoded. When we write the logs to the database, they're an XML string, so you should be able to get the XML back by reversing that. In C#, the code would be something like:
Code:
Text.Encoding.UTF8.GetString(Convert.FromBase64String("...base 64 string from data..."))

_________________
Mike Upton

Software Engineer
Red Gate Software Ltd.
Back to top
View user's profile Send private message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Mon Jan 13, 2014 7:53 pm    Post subject: Reply with quote

I was able to decode something, but it still makes no sense.

The value of the Data property is not pure base64. There is some binary stuff in there as well.

In Notepad++ it looks like this:



If you ignore the the sequences of "NUL NUL EOT ÿ û" and copy the value to a new buffer, you get this:

Quote:

H4sIAAAAAAAEAOy9B2AcSZYlJi9tynt/SvVK1+B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAzO2dvPfee++999577733ujudTif33/8/XGZkAWz2zkrayZ4hgKrIHz9+fB8/Ih7/Hu8WZXqZ101RLT/7aHe881GaL6fVrFhefPbRuj3fPvjo9zj6jZPH2bQtLov2OqX2y+bRu2b22Ufztl09unv36upqfHVvXNUXd/d2dnbv/t5fPH89neeL7CPbuLi58XaxbNpsOc0/SpfZIv/so6f5qqyuF/myPamWbV2VZV4fKxYfpdS2XTefffR6PZ3mTfNRWhBCOx8Rqmn6uKwujvZ2du9t7+5s39tNd/ce7e4+2t9PP9nZebSzk569ePZlmqav2+wiT3cfpU+rq2VZZRhzusqmb+njJj2vq0X6Yv153qZNXoNCv3HSh/kgCnPvUfrVqgexrdI2qy8I4CKbzotlHoN4fycK8d6j9Az0KcuspZnqvbj36H4XFUe/dFotVmXe5r8rTeRdEIepZKdU6P2+5PiIRkM8MtPW28s1DW3bvLB5hm49R19kb4FGlpZF06bVeUBN0/VtJua4LN278+wyTyd5vrQg8plPmj5xDE0MjPT12+tmmi2XeT1+/RMnz1OSnPG9hzvj+3s+qc7zfPYofZNni5MbufY9qBKZoRsRElQ+VjFsCaVp0V4/Otjd3b2LD4/X7fxutlrd5Xm8e7l79xm98Zrmupjm4+ZyevfjGJ0tak9Pn3z1uUMtt6Rq0quCiD/JafRVTZ8Xy0fp549+31f5LPs8a/Pf96W0OyGhyG/TxbNiGYx8K2vbfLFq010wyP07MRifdmFU6+VtJvM2rGV5w4hZh5XuGl4Ssev8GZXC2yoQlcE1t/1mJLA7upd1vspq4EHdS0cWo9tQpz+O8Xh8G8WnbGH6jOqwPv2+CjDszSyNQWmXqiQ0zezyajGbX+WT3d298Sy/HDfupbKaZuWj3Z17ZKq+tvhakijznczzKau1q3nezvP6FnyYFg3JTUqtbeMp5CWtlin9QTr+vYfDc9DB9KCDaW/yNqFItP0GUXnYQeV1LlI/na+Xb0XUdw/e/8W9r/viva/74v7XffH+133x06/74oOv++LB4ItOrKMvPvy6L+7ufO03h3nnpjeHmeemN4e556Y3h9nnpjeH+ce9GX9zmIFuenOYg2568zYs1LEMG3RRIxr6fF2W12pByPP4EA31oSa948HfxjqndweM3Os2X8FT7xJA4ziPEOogrOoKfWw39OL2p/u72e7Ow/3tnYPJbHs/f3Bv+yDPPt2+v/Pppw92zu/t7+azr2vr7u925ur0XT5dt/D9COWb8b1NYMOjN+5Ax2/v0+o9ESAGyRCJWjeBmGJ372Zy3J4gl1m5JhYgzp+5CO2SXKxsUsZDwi6IJ/kF+QLu7Y3vGL+cW/cs+c+6Jd8cmDox7buEkTePGalq3a7WFFtRRqC6ah7dQtV0CeYFkj0hGm90/nugvcEIdsXyvKoXLONw2iY5IfnoNpDSbYp2hefAuY+Eq16/fvqT3/3i6be/e/qESH9LOGdN+un+9qRo0y9fK5w39ToWXG1+W9XGo/RZVja3fV1DxnRNeQIdyevf5/Wb0y9u+f7J81dGLhX3dB9TsnNv9+F4b+/+bcGs6xrzMSvqfEqB5/Wj9OTR7/tdih2JaX7f5pr0weLebQn65esuSukXxbSumuq8TRVm+uJN+ilxz4NPd3YtEWCq0t1b9iIMFHS0N6b/dsa7O7cBofZlg5B7zC1ZAWsaw6iiWJIKoBjdvf77Pj39Se9Phvb7bhaVrtry0Pt42HR/HA0Ou7BUDYt9LwQwjeLrw1USvqHBi2IBmlYrp1mdp9llVpT4MyrOXUg0n9+jFAcyHKfLy6KullASL0gevp9+ln5M5IzlUzaBUe1gQbBlel8gr/Iyz5r8xXoxyWsG45Hoa8Kq2rxhUO8L4KmX4l3gx9cDo/6gJUyHCb4muJ8UOfxAGnm4HS9nPsxh0XzvPurqp0nHufGD84W2N4EaAPU0b6Z1sWoNru8L5k3WvD2b8ast/dps39t/XxDKXLMn1wyGEpC/v0lavi8oIcWHgXKs6knz1xBDB+dl1jRXVS1U+pb3fA1YX5GtXZrpx/hm9qtbQ1P1/OFoKaAAJ1gP4PUTayiL57A59a0BElPO1lOh9uufeM4wOFv8nnDUKDs5+UVlF5Y2uTXIniJjG+6LubXgm2GqXaOlrksCQC+7tHm9XprcnzNNNEN5vdkQKUhC8xVBoDyuYDleNc4h2fze8WpF5nU8rZbnBeV/62zZwLm9VZBCr3+1mhFtUlpUeJ23iHiauwRqSfqFxve6RVKZs5rawVYTS93HAJ9Uq2vjsdxqpt6lT7M2m5A+sY6Ok5B0Va4RH2wtqxbIlsUUVp7T7XNy4fWN2yIHYr83pU+YAmvyMs7OXt/ynac5AmA7oPOi3EyMDj9UTXtLNJWKz6uKs9bEAvT2FQnLPCcOFUvRsKs/ew82U6gvKgthKk5cACKlfsD3tFxzK8fQRzJbXqfCWutaAjLLwvInSHYrZg6gOnamHrAwEWFpdG3k5uZelBSv4Q5wSixAGu8/ei9v/PelVMT+Mm8Vgw19/yz07X/VV7Dj/F3+c4KWAPh9Z/l5ti5/biijKFQU4dXFLG9+XyRS7F//r0Fp9/+NOO39vxGpT/9fiNT7EeqHg9P/6+hE+Y3/t6HEWdT/tyFVrShN1vy/jlaUl5z9v05LAan/1wkfkLq3AamfM6T2/9+I1P3/NyL1/zrlCaQe/L8RqYP/tyHVXOz+/v+v1FUGsb3/tyL2/zqd9f9GM3i2bPMLgfeG0Pt/ZTwRw/H/deIQRfL/dbY8iuUHWocfApb/L/Uhe3gCyf/3cyaw/H8/ZwLL/9epcE2KiQb6ucyQbcRs74eK2nuh9un/O1FDMH8zYns/fMSM5vt/JXLil/6/FrXbiMHPEWr3/t+L2v77ovbDQ+3+/3tRu41e+zlC7cH/e1E7+H8nai7m/n81ev8v0W9D6P2/VMe9rznV5ds3c7dMPs+adJLnS1qlbVpZcG6r98PvNsQ4O0+vq3V6Va3LWVoWb/O09ZBoK8IhwIBWjNOsbPN6SVS7zFMsB4B8o3RV5lmTp80qnxbn1wzmMquLbFLm6cev8tnnWZu/FLhPi5pWoqv6+mXWzj++DZq3WtPfvKDfJfYJDQkge+AepR+/D5XJ5US3v6/rvT+ivUf3P+1xIfr64K6IiRdE+Tafjd+j12rdrtbto9u88Rsnv3HyVYP5XuRp+DxKX79++pPf/eLpt797+mR3d+93+42T03f5dM3i9LIqi+m1bfnkepU1zW+cvKyrad40NI3H9XRetMQH6zqnBsdfPP10/zdOvlvVPNGWRQTA+zI+T2rDk/qT9AsQEjz2xjsYEf5TXKmzlWstLPBevfVnZTymyfhqqXKDHtJLxcJH8nVbrVb49vUvKn9iXbX5STad58+rbEYMmdeXxTSXRtv6Fyiuv308/E669fonng98d+fjtCGA6XlG2g4sc/yhY320t5NO51n9aI+k7hPLGwHav7BsD/3/p+nv1shXvz/Y6jdO8MYnJJEUntOUny3PK5/Lvlzly9O6rupHNLRrilUWY4Ws3GT+PKmWbV2Rpqof9T65k37Px+n7I0PKk2qxyJaz03fTfAXONdg8W5fl9U+ss7I4L/IZ9382wwwQFH31GVNx9EUxraumOm/HTpeMFSzh5trrZ9TDz9XkY1arFkodbLAipTlbs5qHuj4nQlVXwCkXap9Ia+aYYbCpzmVK7A1ttCbrkH48/vib5K6fRe46Kasm/38Pe53AEr+oWo9tvh6DnS2LtiDwDU0yuSIXNK+/cXJS5xmrvCmmkf4OXKBXebMuYRa+93o9xci/n76A2h+ee7/p01yUJyA5DYKXU3475ddTed+QxwegH93Y5RtKWBAJztPLe+Pd8e7Og/HOKB2/OH2TPqvp3SsyIenlPnTWzr3dh+O9vfui8OFerY1WTgv4V0QVuIJddX0zs//Gyat8UV2i8ellvmyfVxdpU63r6SbUqRsDZkMPjBgA1+iBZu+8rhYsoI1w5/h2cK7Ig2yEtudgtkF4fdrA+fQdi+OW2q2YcUhXmFZlBj/XmLbfOHnKRqHnWf/GyU+evnry5etTMvIBHPjNVzX5AOnH8sJd60mr0/xxelW08/Q3TnoN7saXPj72OvtSwTuEevAN+FtCv81IQq9/wwC0oXbVeW1gGGGrQew3gYZisHyerVbkpeks/TzSBa/WEgFmaVtnyyabEpsbtubhs0w8MeqBhUXJlq7mCHOqc/6w845H2tuoEK+5saEblMctxd4GjoHs2xhu7BmB22sueh/Mlx47hmF8RHGEpLGKI8BgRJHjjGkGI1W02rijhQVe0ILB9cDZtv0J7Gmv770++X56Ms+WF7mz25Cf11+dnJy+fn2bMOhVTnEK/JsZz30kaNy5DZjj9CqfGPFFxoG1NJys82q9nI1oJuiv9OzsdSc7cVVQV+S2rfKaYmCKdsfp70NxO8lVum4kaNcYm3yH4+/mk9fQRpQi+WkKo1wgXmi4ny1bDZcXxALpF9mSwvIaqqx5W6z6/W8OLjWifpoTxTmekjD//YK27ldj99V4uV697WdQIvS9VZ7gZdW0T20AcxuwLyoLYiqZkBAGzWoN/5mmkBnzMVTwkbh6afr4LvFncVm01/pR54PgT++P/ycAAP//ubBEevVOAAA=


When I decode that, I don't see any XML. All I get is a stream of bytes, mostly unprintable.

I get similar behavior when I use Raven's HTTP interface to fetch the same attachment.

This PowerShell script fetches the same attachment and prints the raw response:

Code:

$Response = Invoke-WebRequest 'http://localhost:10300/static/attachments/tasks-34/output-log'
$Response.RawContent


The response looks like this:

Code:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Raven-Server-Build: 2666
ContentType: text/xml
Temp-Request-Time: 0
Date: Mon, 13 Jan 2014 18:43:44 GMT
Expires: Sat, 01 Jan 2000 00:00:00 GMT
ETag: 02000000-0000-0001-0000-00000000002B
Server: Microsoft-HTTPAPI/2.0

.?........??.`.I?%&/m?{.J?J??t?.?`.$??@.???????.iG#)?*??eVe]f.@??????{????{????;?N'????\fd.l??J???!???.?~|.?".?.?.ez??MQ-??hw??Q?/???X^|???=?>???8??????-.??:??????f??G??]
=?{???j|uo\?.w?vvv???_<.=????#?????v?l?l9??J??"????????.??=??m]?e^.+..???]7?}?z=??M?QZ.B;..?i???.??vv?m??l??Mw?.??>??O???y?????x?e?????"Ow.?O??eYe.s???o??&=??E?b?y??M^?B?
q???
s?Q???.???6?/.?"???e.?x.'
???? ?)?????????~..G?tZ-Ve???+M?].??d?T??????FC<2????5m???y?n=G_do?F??E???y@M??m&??,????2O'y?? ??O?>q M ?????f?-?y=~?.'?S?????;??{>???|?(}?g??.??=?.??...T>V1l   ?i?^?:?????
.????n?Z??y?{?{?.?????????rz??.?-jOO?|??C-??j??????i?UM?.?G???~?W?,?<k?????;!??o???b.?|+k?|?j?]0??;1.?vaT??m&?6?ey??Y???.^.???.???*.??5??f$?;??u??j?A?KG.??P????x|.??la???
?>??
0??,?Ai??$4???j1?_??????,?.7????f????{d????Z?(????)???y????.|?.?MJ?m?)?%??)?A:????s?????io?6?H??.Qy?A?u.R????oE?w.?????????????}???}????????????N??/>??/??|?7?y??7????7???
?7????7????..s??nzs??nz?6,?? .tQ#.?|]??jA???.??&?????:?w.???6_?S?.@?8?.? ??
}l7???????????????l{?.po? ?>??????.v???????k???v???]>]???.????M`??7?@?o???=. .?.?Z7??bw?fr?? ?Y?&. ???.??\?lR?C?.?'?.?.????.??[?,???%?.?:1????7?.?j???.[QF??j.?B?t   ?.?=!.o
t?{???.v????.,?p?&9!??6??m?v???????^?~????????{??H.K8gM?????h?/_+?7?:.\m~[????YV6?}]C?tMy..???????/n????WF..?t.S?so??xo??m???.?1+?|J?????????]?.?i~???????m   ???.J?.??????M
.f??M?)q??Owv-.`???[?" .t?7??v??;?.??e??{?-Y.k.???X?
?.????>=?I?O???n.??????x?t...??T?}/.0????U.????b.?V+?Y???eV??3*?]H4???..2.??????P./H.??~?~L???S6?Q?`A?ez_ ??2????z1?k.???k????aP? ????]???.???%L?   ?&??.9?@.y?./g>?a?|?>???
I???????7?..?4o?u?j?? ?M??=???-??l??._.?\?'? ?.???IZ?/(!???r??I??.C.?e?4WU-T???|X_??]????f??[CS???h)?.'X.??.k(???9??..S??S????x?08[??p?(;9?Ee.?6?5??"c.??????a?]???K.@/??y
?^???3M4Cy??.)HB?.A?<?`9^5?!????jE?u<???.?.?l????U?B?.??.mRZTx???x??.jI?????ER?????V.K??.?T?k???j???O?6??>?????tU?..l-?.???.V???sr????".b?7?O?.k?2??^????9.`;????L?.?TM{K4
????????.??.   ?<'..K????{.6S?/* a*N\."?~???\s+??G2[^??Z?Z.2???'Hv+f.?:v?.?0.aitm???^?.??.pJ,@.??z/o???T??2o.?}?,??.?W???]?s??.?}g?y?..n(?(T.???,o~_$R?_??Ai???8???.?O?_???.
???????D??????Y???!U?(M????V?????NK.???   .???.??3????????7"??:?   ?.??.????!?\??????U.????"??:???F3x?l? ?????.e<.???u?.E??u?<??.Z?..??/?!{x.???s&??.?g.????pM??.????m?l????{?
???;QC0.3b{?|????.%r????.?????.j??????????C???{Q??^?9B???{Q;?.'j.??.5z?/?oC???T???9???7s?L>??t??KZ?mZYpn????6?8;O??uzU??YZ.o????h+?!??V???l?zIT??S,.?|?tU?Y???*?.?? ?2??lR
???????Y??.?O??V????e??????Z?????%?       {?.?.?.???D????????G???q!??????.D?6?????j????????.'?q?U??^?i?<J_?~????????{?dww?w????w?t????*???m??z?5?o????i?44???t^??.?:?.?_<?t?7N
?[?<??E.??2>Oj????? ..<??;..?S\???k-,?^??ge<???j?r?.?K??G?u[?V???/*.b]??I6????lF ????4?F??.(??}<?N???'?.|w???!??yF?.,s??c}???N?Y?h???.?..???l.???????W????7N??'$?.????-?+?
??\??????G4?k?U.c???d?<??m]???.?>??~????#C??j?????w?|.?5?<[???O???8/?.?.6? ..}?.Sq?E1???:o?N??.,????g?????cV?.J.l?"?9[????>'BUW?).j?Hk??a???eJ?m?&??~<??????E?:)?&?..{??.?
?Z?m?.??-?? ?M2?".4??qrR?.??)???.\?Wy?.a.??z=???????.?{???\?' 9??S~;??Sy???.?.???.JX.   ???{????????(.?8}?>???+2!??>t????????????^??VN ?WD.??]u}3?????|Q]???e?l?W.iS???&??..
fC.?..???f???.,??p??vp???l???`?Ax}????.??????qHW?Ve.?????8y?F??Y???O??z???S2?..??W5?.????]?I???qzU???7Nz???>>?:?R?;?z??[B??HB?.?.??v?ym`.a?A?7??b?|??V???,?<?.??..fi[g?&?.
?.????L<1???E????.s?s????G???.????.??-??.????.n?.??k.z.??.;?a|Dq????#?`D???i.#U?????.^????????   ?i???>?~z2??.??????_????~}?0?UNq
??.?}$h?????*?.?E???4???j???h&?????u';qUPW?????b`?v???Cq;?U?n$h?.?|???????F?"?i
?\ ^h??-[?.?.?.????.??y[???o..5?~?.?9??0?????Wc??x?^??gP"??U??e??Om.s.?/* b*??..?j????.?1T???zi??.?gqY???Q???O???'..????Dz?N..


The content type says text/xml, but that sure doesn't look like XML!

Any ideas on how I should further decode this output?
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Mon Jan 13, 2014 8:14 pm    Post subject: Reply with quote

Hmm, running the command against my DB gives a file that's all binary with no json at all. Curious.

I'll dig a little more at this end...
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Mon Jan 13, 2014 8:46 pm    Post subject: Reply with quote

OK, I needed to open the dump file inside dump.raven in 7Zip first. Smile
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Mon Jan 13, 2014 9:14 pm    Post subject: Reply with quote

OK, some progress.
Like you, my unzipped file was full of this:


Code:
{
      "Data": "H4sIAAAAAAAEAIVSy27bMBC8B8g/LHio2wMp0bKetWzUcRoUqFujbtAedGEoWhEqkYJIWfbfl45sN0WD5rRY7HA4s7PT+b6uYCdaXSqZIkpcBEJylZeySFFntjhC89n11ZRxU+5KcwCLlzrZ6zxFj8Y0ieP0fU96j6i2cMauS52fq88b/ihqhi7g8nUwLqU2THKBQLJapGipelkplkPX5MwIsNoIDQi1Ci3QdDpFm45zoTWC0qpxkdUJMK1UMRu71MMuxZ4L1Ev8MKET+PTl41cAONNag2fm0VI0lTrUQhpYMckK0Z5+G8G2VTWMnolvRY4L+4pwVTsDAVfStKrSTn7iJkw3+7kwPMV07IU09L3If8Pq5n3RWakPNKAR8wMcbPMYT6I4wCyPQ+xRL4wiP/b88YAe6FN/Ejy1vCqtxJSOrq9eMri8Xdzf/W2QhpR6bhTBw8EIDUbBTZL9KKUVqrPvt6t1Zupm8SEeE1tfYJ1c1rZhuyPh2aG4xDJwrltVtKxeMsOyb3Z4Z0fZn7Wetpo9zdeM/7LdDbO5Z2Sg0dm/GeBL4qTW5X/V3b8e5FsawGrxDvRwM9uuqg7P7BB7487xdI71fOyz31lQzmgfAwAA",
      "Metadata": {
        "ContentType": "text/xml",
        "Content-Type": "application/json; charset=utf-8"
      },
      "Key": "attachments/tasks-415/output-log"
    },



I took the data value, and Base64 decoded it here, saving the output to a .bin file

I renamed the bin file to .7z and opened it up in 7zip, then extracted out the contents. I now have this:

Code:
<?xml version="1.0" encoding="utf-8"?>
<activity xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Download update 1.0.16.10" status="Success" id="0">
  <log>2013-01-30 13:57:14 INFO   Downloading update 'Deployment Manager 1.0.16' from 'http://www.red-gate.com/updatecontrols/download.aspx?etc=-1237175385&amp;guid=b1618a56-6fd9-4896-ad97-313788593525&amp;update=546&amp;client=1'
2013-01-30 13:57:14 DEBUG  Downloading 17113088 bytes to C:\Windows\TEMP\tmpBA92.tmp
2013-01-30 13:57:44 INFO   Saving downloaded update to C:\ProgramData\Red Gate\DeploymentManager\Data\PackageCache\.updates\Deployment Manager-1.0.16.10.msi
2013-01-30 13:57:44 INFO   Update 'Deployment Manager 1.0.16' (16 MB) successfully downloaded.
</log>
</activity>


Success!

So the steps seem to be:

- extract data
- unzip
- grab each data string
- base64 Decode
- unzip (again) the output

Hope that helps!
Back to top
View user's profile Send private message
Mike Upton



Joined: 11 May 2011
Posts: 108
Location: Red Gate

PostPosted: Tue Jan 14, 2014 9:05 am    Post subject: Reply with quote

Amazing sleuthing without using the source code James Smile

I was about to reply and tell you that, having looked at the source code for our log serialization, we GZip the XML string before it is saved as an attachment. RavenDB Smuggler then base64 encodes that to store it as a JSON text field in its dump file.

If you go straight to the HTTP API to download the attachment (http://hostname:10300/static/attachments/tasks-{foo}/output-log), you'll get back a binary stream that can be GZip decompressed to get the XML log.

You can also get a list of all the available attachments by reading the JSON array at http://hostname:10300/static
_________________
Mike Upton

Software Engineer
Red Gate Software Ltd.
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Tue Jan 14, 2014 9:19 am    Post subject: Reply with quote

Actually I did poke the source code and saw some references to gzip so figured it was worth a punt... Wink
Back to top
View user's profile Send private message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Tue Jan 14, 2014 7:54 pm    Post subject: Reply with quote

Thanks for your help!

I distilled your helpful advice into a simple PowerShell function to export the deployment logs.

I ran it on the RGDM host to dump XML versions of the deployment logs to a log directory.

Code:

PS C:\Users\admin_iain> Export-DeploymentLogs -TargetDirectory C:\Users\admin_iain\Desktop\Logs

PS C:\Users\admin_iain> Get-ChildItem -Path C:\Users\admin_iain\Desktop\Logs


    Directory: C:\Users\admin_iain\Desktop\Logs


Mode                LastWriteTime     Length Name                                                                                                                                         
----                -------------     ------ ----                                                                                                                                         
-a---        14/01/2014     18:43      61741 deployments-7966.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7967.xml                                                                                                                         
-a---        14/01/2014     18:43      57730 deployments-7968.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7969.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7970.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7971.xml                                                                                                                         
-a---        14/01/2014     18:43      50796 deployments-7972.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7973.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7974.xml                                                                                                                         
-a---        14/01/2014     18:43     137176 deployments-7975.xml                                                                                                                         
-a---        14/01/2014     18:43      50909 deployments-7976.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7977.xml                                                                                                                         
-a---        14/01/2014     18:43     136534 deployments-7978.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7979.xml                                                                                                                         
-a---        14/01/2014     18:43     136534 deployments-7980.xml                                                                                                                         
-a---        14/01/2014     18:43       6305 deployments-8001.xml                                                                                                                         



Now I can search the files to find what I need.

In case anyone else wants to do this, here's the code I used.

Code:

function Export-DeploymentLogs {
  param (
    [String] $TargetDirectory = '.'
  )

  $AttachmentList = Invoke-RestMethod -Uri 'http://localhost:10300/static?start=0&pagesize=999999999'

  $AttachmentList |
  Where {
    $_.Key -like '*deployment*'
  } |
  Foreach {
    $AttachmentName = $_.Key.Split('/')[1]

    Invoke-WebRequest -Uri "http://localhost:10300/static/$($_.Key)" -OutFile "$TargetDirectory\$AttachmentName.xml.gz"

    gzip.exe --decompress "$TargetDirectory\$AttachmentName.xml.gz"
  }
}


It requires the gzip utlity to be on the path. I use version 1.2.4. You can download it from http://www.gzip.org/.

RavenDB pages the links to attachments through the static resource. Starting at 0 with the pagesize set larger than the number of attachments in the database, the static endpoint returns a link to all the attachments in the database.

The key property contains the word 'deployment' if the attachment is a deployment log. The function filters on this to ignore other types of attachment.

The full key is a relative URL for the attachment. If you fetch it, you get a gzip-compressed XML file.

gzip.exe deletes the input file and writes the decompressed version with an .xml extension.

In the end, the target directory contains a bunch of XML files.
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
james.billings



Joined: 16 Jun 2010
Posts: 1116
Location: My desk.

PostPosted: Tue Jan 14, 2014 7:56 pm    Post subject: Reply with quote

Awesome, thanks for posting back your solution for everyone else to use if they need it!
Back to top
View user's profile Send private message
isme



Joined: 12 Jun 2012
Posts: 74
Location: Edinburgh

PostPosted: Thu Jan 16, 2014 2:14 pm    Post subject: Reply with quote

I added this script to a new Bitbucket repo called "RGDM Tools".

I'll copy the rest of my posted code there too.

Thanks again!
_________________
Iain Elder, Skyscanner
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic All times are GMT + 1 Hour
Page 1 of 1

 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group