I was asked to search several teams' code-bases recently based on a list of over 270 different search strings. We are a Windows OS organization and doing a search like that would take some time. By the way, has anyone noticed that the Windows explorer search tool is flaky? Even if I used a tool like this it would still take a while.
I have cygwin installed, and I know grep works well - when I search for something I know is in the file, I find it (unlike Windows sometimes). What I didn't know was that there is a -f option on grep that allows you to provide a file with all you search terms separated by newlines. An astute worker made me aware of this and I tried it out this morning. It worked nicely and saved me some time. Here's how I implemented it:
grep -r -i -f file_of_search_strings.txt * | grep -v .svn | grep -v Assert | grep -v .sql >> resultFile.txt
-r is a recursive search based on where I'm current sitting on the command line
-i is a case insensitive search
-f allows me to pass in my file of search strings
* tells grep to search every folder/file at this level. Used with -r it tells grep to search every single folder and file that is currently at my current cursor position, or a child of the folder I've cd'ed to.
The | grep -v .svn tells grep to ignore/not display files with *.svn in the results
and then >> allows me to pipe my results to a file for inspection.
You may have to run the dos2unix command on the text file with your list of search items to get grep to see the new lines in the file correctly.
Thursday, July 22, 2010
Wednesday, July 21, 2010
MS Sql Server Stored Proc Search Query
If you work in an org where there's a lot of business logic put into (MS Sql Server) stored procedures or you have an application that is stored proc intensive, you might find this code snipped helpful. It searches all the stored procs in the database for a specific string
select distinct o.name from syscomments c
inner join sysobjects o
on c.id = o.id
where o.type = 'p'
and c.text like '%string to search for%'
order by o.name
select distinct o.name from syscomments c
inner join sysobjects o
on c.id = o.id
where o.type = 'p'
and c.text like '%string to search for%'
order by o.name
Monday, July 12, 2010
.Net Self Serve Automated deploy
There are a couple of prerequisites that need to be in place in order for a self serve automated deployment system to work.
1. You should have a Nant, MsBuild, Visual Build, or a similar type of scripting file that is doing deploys for you already - see my previous posts on doing that here
2. Your folder structure for every project you want to deploy should be similar. To have a good Self-Serve Deploy tool, it needs to work on the assumption that directory structures are consistent for certain types of projects. This could be set up using your repository tags directory, or automatically copying deployment files to the required folder structure after the build is successful. A folder structure I've used in the past is b:\deployment\projectname\version\ My build file(s). Any other required folders and files reside in the version folder with a consistent folder structure.
3. You need either a properties file or a DB table to manage which tag/version has been deployed to which environment. We've used a symlink in the past to denote the current 'tip' build. This example will use a file. The file is structured like:
Once the deploy is finished to a particular environment, logic in the deployment file automatically updates the respective project in the appropriate version file.
Here's a bit of sample code for a client side asp page for a Self Serve app. It passes some values to a server side asp page (deploy.asp) so that page will know which version of which project to deploy into which environment. The onchange event of the env select control called an AJAX function that retrieves the currently deployed version of the project from the versions.xml file.
Then on my deploy.asp page, the request parameters are set to variables which are then used to dynamically build the path to the deploy file (in this case a visual build file called Deploy-<serverName>-<component>.bld) This could be a nant, msbuild, or ant file as well, you'd just have to change which executable you're using in the executeString. The command is then executed in a Windows Shell and if it has a return code of '0' it was successful. Successful or not, we do some very simple logging to show who tried to deploy what, where, and when.
Here's the code:
1. You should have a Nant, MsBuild, Visual Build, or a similar type of scripting file that is doing deploys for you already - see my previous posts on doing that here
2. Your folder structure for every project you want to deploy should be similar. To have a good Self-Serve Deploy tool, it needs to work on the assumption that directory structures are consistent for certain types of projects. This could be set up using your repository tags directory, or automatically copying deployment files to the required folder structure after the build is successful. A folder structure I've used in the past is b:\deployment\projectname\version\ My build file(s). Any other required folders and files reside in the version folder with a consistent folder structure.
3. You need either a properties file or a DB table to manage which tag/version has been deployed to which environment. We've used a symlink in the past to denote the current 'tip' build. This example will use a file. The file is structured like:
<versions>
<serverType>
<server name="blahboxType" version="1.2" />
<server name="fooboxType" version="1.0" />
</serverType>
<blahboxType>
<project name="MyProject" version="3.1.0.23" />
<project name="MyProject2" version="3.1.4.11" />
</blahboxType>
<fooboxType>
<project name="fooProj1" version="6.2.123"/>
<project name="fooProj2" version="6.3.92"/>
</fooboxType>
</versions>
Here's a bit of sample code for a client side asp page for a Self Serve app. It passes some values to a server side asp page (deploy.asp) so that page will know which version of which project to deploy into which environment. The onchange event of the env select control called an AJAX function that retrieves the currently deployed version of the project from the versions.xml file.
<form action="deploy.asp" method="get" name="MyProject">
<input name="project" type="hidden" value="MyProject" />
<input name="serverType" type="hidden" value="blahboxType" />
<select name="version">
<option value="">--Select--</option>
<option value="" & version & "">" & version & "</option>
</select>
<select id="MyProject" name="env" onchange="getVersions('MyProject',this.value)">
<option value="">--Select--</option>
<option value="Test1">Test 1</option>
<option value="Test2">Test 2</option>
<option value="SB">Sandbox</option>
<option value="STG">Staging</option>
<option value="PRD">Production</option>
</select>
<div id="MyProjecttarget" style="color: green;">
-</div>
<input type="submit" value="Deploy" />
</form>
Then on my deploy.asp page, the request parameters are set to variables which are then used to dynamically build the path to the deploy file (in this case a visual build file called Deploy-<serverName>-<component>.bld) This could be a nant, msbuild, or ant file as well, you'd just have to change which executable you're using in the executeString. The command is then executed in a Windows Shell and if it has a return code of '0' it was successful. Successful or not, we do some very simple logging to show who tried to deploy what, where, and when.
Here's the code:
' need a longer server timeout setting so our deploy can finish
Server.ScriptTimeout=1200
'instantiate some vars
env = ""
version = ""
project = ""
severType = ""
'set the vars
set version = Request.queryString("version")
set env = Request.queryString("env")
set project = Request.queryString("project")
set serverType = Request.queryString("serverType")
'gets the user
Set WSHNetwork = CreateObject("WScript.Network")
user = WSHNetwork.UserName
if version="" then
Response.Write("Please ensure you chose a version <br/>")
elseif env="" then
Response.Write("Please ensure you chose an environment")
elseif project="" then
Response.Write("error occured - project required")
elseif LCase(env)="PRD" then
Response.Write("We don't want to deploy to Production from this app")
else
Response.Write(version & " " & env & " " & project)
'set up a dynamic path here to the deployment file (nant or ant or visual build file) This string assumes the
'build file is taking a parameter called Env - the environment to deploy to
execString = "<pathToVisualBuildExecutable>\VisBuildCmd.exe /b " & Chr(34) & version & "\visual build\Deploy-" & serverType & "-" & component & ".bld" & Chr(34) & " Env=" & env
Response.Write("<br/>" & executeString)
set oFs = server.createobject("Scripting.FileSystemObject")
set oTextFile = oFs.OpenTextFile("<pathOfFileToLogTo>\deploylog.txt", 8, True)
Dim WshShell
Set WshShell = CreateObject("WScript.Shell")
proc = WshShell.Run(executeString,6,true)
if proc="0" then
response.write("<br/><br/><div style='color:green;font-size:16px;'>Deploy of build " & component & " " & version & " to " & env & " as " & user & " was successful")
dt = now()
logtext = dt & ": Deploy of build " & component & " " & version & " to " & env & " was successful" & vbCrLf
oTextFile.Write logtext
oTextFile.Close
else
response.write("<br/><br/><div style='color:red;font-size:16px;'>Deploy of build " + component + " " + version + " to " + env + " failed using user " + user)
dt = now()
logtext = dt & ": Deploy of build " & component & " " & version & " to " & env & " failed as user " & user & vbCrLf
oTextFile.Write logtext
oTextFile.Close
end if
response.write("<br/>Deploy times and results are logged at <pathOfFileToLogTo>\deploylog.txt")
set oTextFile = nothing
set oFS = nothing
set WshShell=nothing
end if
response.write("<br/><a href='javascript:history.back()'>Back to deploy page<a/>")
Saturday, July 10, 2010
Key Questions in Software Sustainment
Here are some of the questions I ask myself when a software system has stopped running properly...
The first big question is: What changed between when the application was running well, and the time the application stopped running well? To get to the bottom of that question, there are a number of other questions that can help point the way.
Question isolation - Is this problem isolated in any way? Is it only on a specific network, environment, or group of servers? Does it only happen for a specific group of users or a specific client? Is there a period of time it's isolated to? Is it isolated to a particular 'item' in your data? I've seen users try to make an application use a different browser version than the documented supported versions. Some times it takes a while to get to the bottom of simple issues like that. Also, we've run into situations where an organization will not have given users the rights on their machines to install third party active X components for their browser that the application they are trying to use requires.
Question data integrity - Whether you are looking at legacy data or no logic to manage special characters in your data, you need to break the problem down. With legacy data, is the problem isolated to a particular user or group of users, or a particular item? Or if it's an ETL function, does relational integrity from the first DB line up to the second DB correctly - are you missing or adding 'types' of data that the second DB either is or isn't expecting? Sometimes you need to pinpoint the exact row or time when the issue occurred to determine what the problem was.
Question continuity - Did something stop running or listening? A service, appPool, web site, 3rd party server? Are your cron jobs or scheduled tasks still there? Monitoring would quickly and easily answer this question for you.
Question communication - Are the lines of communication open to all of the dependencies that your application has? Has a network cable been severed by a backhoe down the street? (I've seen that before) Is there too much communication going on? Too many calls from one routine can take you system down. We had an issue like that with some javascript that called a data access function to a GIS server. As soon as one too many layers got added to the map, performance died as there were too many round trip calls to the server in one request. Internal systems that need to do identity verification or IP Geolocation are heavily dependent on external third parties to operate. These external vendors can in turn can be dependent on other external services. Know all your dependencies and have monitoring and SLA's in place for all of them or you could be sweating bullets.
Question dependencies - Here I'm thinking more in the context of internal dependencies - what internal 3rd party services are you dependent on? Databases, reporting tools, monitoring systems, document management systems, any internal system that your application depends on that you aren't responsible for fits into this category. Are they up and running? How do you know? Are they running properly?
Question the un-questionable - Is your hvac system working at your Co-Lo? As redundant as your service provider tries to be, there will always be something under the radar. Always. I've seen an external hvac system take down an entire enterprise. Many of you have likely seen a McAfee or a Norton patch take down an enterprise. How secure is your UPS (Uninterrupted Power Supply) management console? I've logged into one I've found by accident using username:admin password:password!
Question known changes - software enhancements and patches, file or db permissions, config changes, etc. I've seen issues where a changed file path will take down a critical Ftp routine, a 3rd party software patch for a document management system crippled a production system by removing key indexes in the database, and an (unfortunately un-automated) database refresh will have missing roles or users or the db will still be restricted mode. Doing anything manually can get you into trouble. A fat finger can push a wrong dll/library or mis-type an entry in a config file. Even fat fingering an automated deploy can get you into problems. Automated deploys still depend on data that is manually entered. We have pointer to app servers in our web.config files entered wrong and multiple entries in machine.config for a particular component.
Question security - Malicious changes are possible either internally or externally. Unfortunately, the question of security is a larger one everyday. And it has to be scrutinized at every level. Does your system have a firewall and IP Sec rules in place? Does you application provide you with an audit trail? How secure is your production data?
Resources at your disposal that can help in your investigation:
Log files - event logs, server logs, if you're logging to tables in the DB, don't forget to look there.
Users - it's like CSI - you need to get ALL the information you can about their problem. You cannot be afraid to ask.
Thread dumps - killing a hurting server and ensuring that it does a thread dump when it terminates can be very effective in your problem search
Networking tools - WireShark, telnet, ping, netstat - these are great for checking your communication.
Monitoring tools - Nagios, SCOM, GroundworkOpenSource
Books - Michael Nygard's Release It! and Luke Hohmann's Beyond Software Architecture.
The first big question is: What changed between when the application was running well, and the time the application stopped running well? To get to the bottom of that question, there are a number of other questions that can help point the way.
Question isolation - Is this problem isolated in any way? Is it only on a specific network, environment, or group of servers? Does it only happen for a specific group of users or a specific client? Is there a period of time it's isolated to? Is it isolated to a particular 'item' in your data? I've seen users try to make an application use a different browser version than the documented supported versions. Some times it takes a while to get to the bottom of simple issues like that. Also, we've run into situations where an organization will not have given users the rights on their machines to install third party active X components for their browser that the application they are trying to use requires.
Question data integrity - Whether you are looking at legacy data or no logic to manage special characters in your data, you need to break the problem down. With legacy data, is the problem isolated to a particular user or group of users, or a particular item? Or if it's an ETL function, does relational integrity from the first DB line up to the second DB correctly - are you missing or adding 'types' of data that the second DB either is or isn't expecting? Sometimes you need to pinpoint the exact row or time when the issue occurred to determine what the problem was.
Question continuity - Did something stop running or listening? A service, appPool, web site, 3rd party server? Are your cron jobs or scheduled tasks still there? Monitoring would quickly and easily answer this question for you.
Question communication - Are the lines of communication open to all of the dependencies that your application has? Has a network cable been severed by a backhoe down the street? (I've seen that before) Is there too much communication going on? Too many calls from one routine can take you system down. We had an issue like that with some javascript that called a data access function to a GIS server. As soon as one too many layers got added to the map, performance died as there were too many round trip calls to the server in one request. Internal systems that need to do identity verification or IP Geolocation are heavily dependent on external third parties to operate. These external vendors can in turn can be dependent on other external services. Know all your dependencies and have monitoring and SLA's in place for all of them or you could be sweating bullets.
Question dependencies - Here I'm thinking more in the context of internal dependencies - what internal 3rd party services are you dependent on? Databases, reporting tools, monitoring systems, document management systems, any internal system that your application depends on that you aren't responsible for fits into this category. Are they up and running? How do you know? Are they running properly?
Question the un-questionable - Is your hvac system working at your Co-Lo? As redundant as your service provider tries to be, there will always be something under the radar. Always. I've seen an external hvac system take down an entire enterprise. Many of you have likely seen a McAfee or a Norton patch take down an enterprise. How secure is your UPS (Uninterrupted Power Supply) management console? I've logged into one I've found by accident using username:admin password:password!
Question known changes - software enhancements and patches, file or db permissions, config changes, etc. I've seen issues where a changed file path will take down a critical Ftp routine, a 3rd party software patch for a document management system crippled a production system by removing key indexes in the database, and an (unfortunately un-automated) database refresh will have missing roles or users or the db will still be restricted mode. Doing anything manually can get you into trouble. A fat finger can push a wrong dll/library or mis-type an entry in a config file. Even fat fingering an automated deploy can get you into problems. Automated deploys still depend on data that is manually entered. We have pointer to app servers in our web.config files entered wrong and multiple entries in machine.config for a particular component.
Question security - Malicious changes are possible either internally or externally. Unfortunately, the question of security is a larger one everyday. And it has to be scrutinized at every level. Does your system have a firewall and IP Sec rules in place? Does you application provide you with an audit trail? How secure is your production data?
Resources at your disposal that can help in your investigation:
Log files - event logs, server logs, if you're logging to tables in the DB, don't forget to look there.
Users - it's like CSI - you need to get ALL the information you can about their problem. You cannot be afraid to ask.
Thread dumps - killing a hurting server and ensuring that it does a thread dump when it terminates can be very effective in your problem search
Networking tools - WireShark, telnet, ping, netstat - these are great for checking your communication.
Monitoring tools - Nagios, SCOM, GroundworkOpenSource
Books - Michael Nygard's Release It! and Luke Hohmann's Beyond Software Architecture.
More ideas
Business idea: Create a computer screen with removable sides so there's not space when you put it against another computer screen - they'll almost look like one screen instead of having the 1-2 inch separation between the two.
Business idea: If I could create a machine/robot that would automatically wash windows I think I could make a lot of money. Every window I can think of besides one has to be washed manually by hand. All the high rises downtown still have all their windows washed manually. The only place they use a different technique as far as I'm aware is with race cars where they use layers of thin plastic over the windshield. They rip off a layer every time the car is in the pit and instantly the driver has a clean windshield.
Business idea: A new service industry helping people/seniors navigate their way around the health maze we have created for ourselves. You need a Personal Medical Manager! Here's why: There are significant disconnects between specialists who analyze medical tests, the lap techs who perform/analyze the tests, and the doctors who originally asked for the tests. Then there's another layer of indirection at the pharmacy. How does an older person manage all their prescriptions and make sure they aren't 'colliding' with each other, or even just remember to take them all and follow all the directions? I've had my wife 2nd guess a doctor's prescription, ask the pharmacist about it, and have him adamantly agree with her that the prescription was wrong.
Political idea: Why are we still running democracy the way we are? The reason we elected people as 'representatives' long ago was so they could represent us in parliament (or the house, or congress) - or at least represent our vote. Lobbyists have done away with our representation in the vote. But why do we need a representative now anyway? Technology could allow us to instantly vote on every single issue before parliament either using a computer or a phone. We wouldn't need a representative to vote for us anymore. Don't get me wrong, I'm all for democracy! I just wonder if it can be done better now with technology and removing the 'middle man' and the lobbyists.
Business idea: If I could create a machine/robot that would automatically wash windows I think I could make a lot of money. Every window I can think of besides one has to be washed manually by hand. All the high rises downtown still have all their windows washed manually. The only place they use a different technique as far as I'm aware is with race cars where they use layers of thin plastic over the windshield. They rip off a layer every time the car is in the pit and instantly the driver has a clean windshield.
Business idea: A new service industry helping people/seniors navigate their way around the health maze we have created for ourselves. You need a Personal Medical Manager! Here's why: There are significant disconnects between specialists who analyze medical tests, the lap techs who perform/analyze the tests, and the doctors who originally asked for the tests. Then there's another layer of indirection at the pharmacy. How does an older person manage all their prescriptions and make sure they aren't 'colliding' with each other, or even just remember to take them all and follow all the directions? I've had my wife 2nd guess a doctor's prescription, ask the pharmacist about it, and have him adamantly agree with her that the prescription was wrong.
Political idea: Why are we still running democracy the way we are? The reason we elected people as 'representatives' long ago was so they could represent us in parliament (or the house, or congress) - or at least represent our vote. Lobbyists have done away with our representation in the vote. But why do we need a representative now anyway? Technology could allow us to instantly vote on every single issue before parliament either using a computer or a phone. We wouldn't need a representative to vote for us anymore. Don't get me wrong, I'm all for democracy! I just wonder if it can be done better now with technology and removing the 'middle man' and the lobbyists.
Subscribe to:
Posts (Atom)