blogs.xtivia.com

Rolling DB2 tablespace location change using HADR

2013-05-23T20:28:36Z

One of our Virtual-DBA clients has a small DB2 environment with version 9.7 on Linux and HADR connecting a primary and standby system. When the application team created the database, it was created using Automatic Storage on the /home filesystem. This will always happen when the default setting of DFTDBPATH in the DBM configuration file is set to the instance owner's home unless the CREATE DATABASE command specifies a different location.

So, they had all tablespaces in the database residing in /home and it was quickly growing. Here are the details of the project.

The Mission:

Move the database and all tablespace containers from /home to a new /db2data tablespace. Do this while the application is online and the database is available.

The Plan:

Use HADR and Redirected Restores to move the database while the application is online and have no data loss. First restore a copy of the Primary onto the Standby using a redirected restore to put the tablespace containers on the new filesystem. Then sync up with HADR and Takeover HADR on the Standby to swap roles. After the Primary has changed roles to Standby, stop HADR there. Take a new online backup from the new primary and restore it on the new Standby using another redirected restore to the new filesystem. Then start HADR again and let them sync. Once they are in PEER state you can takeover again on the orginal Primary. At this point both servers will have the database on the new filesystem.

Execution:

Server1 is the primary and Server2 is the standby.

1. Online backup of the database on Server1. This is a small database and it takes 2 minutes to backup.

=>db2 backup db ITIMDB online compress

2. Copy the backup image from Server 1 to Server 2

3. From Server2, generate a redirected restore script using the newly copied backup.

=>db2 restore db ITIMDB from /db2backups redirect generate script restore.ddl

4. Edit the script to restore the db using the ON and DBPATH ON paramaters set to the new filesystem

5. Stop HADR on the standby (Server2)

=>db2 deactivate db ITIMDB
=>db2 stop hadr on db ITIMDB

6. Drop the database on the standby (Server2)

=>db2 drop db ITIMDB

7. Restore the database by running the redirect restore script (restore.ddl) on Server2

=>db2 -tvf restore.ddl

8. Once the restore finishes, edit the HADR settings on Server2

=>db2 update db cfg for ITIMDB using HADR_LOCAL_HOST SERVER2 HADR_REMOTE_HOST SERVER1

9. Start HADR on the standby (Server2)

=>db2 start hadr on db ITIMDB as standby

10. Once they are in PEER state, Takeover the database on Server2 so the same steps can take place on Server1. Run this from Server2

=>db2 takeover HADR on db ITIMDB

11. Now take an online backup of the database on Server2

=>db2 backup db ITIMDB online compress

12. Copy the new backup from Server2 to Server1

13. From Server1, generate a redirected restore script using the fresh backup.

=>db2 restore db ITIMDB redirect generate script restore.ddl

14. Edit the script to restore the db using the ON and DBPATH ON paramaters set to the new filesystem.

15. Stop HADR on the standby (Server1)

=>db2 deactivate db ITIMDB
=>db2 stop hadr on db ITIMDB

16. Drop the database on the standby (Server1)

=>db2 drop db ITIMDB

17. Restore the database by running the redirect restore script (restore.ddl)

=>db2 -tvf restore.ddl

18. Once the restore finishes, edit the HADR settings

=>db2 update db cfg for ITIMDB using HADR_LOCAL_HOST server1 HADR_REMOTE_HOST server2

19. Start HADR on the standby (Server1)

=>db2 start hadr on db ITIMDB as standby

20. Once they are in PEER state, Takeover the database on Server1.

=>db2 takeover HADR on db ITIMDB

Now both Server1 and Server2 have their tablespace containers on the /db2data filesystem. This was done online with no data loss! Gotta love HADR.

We also had Alternate Client Reroute set on both the Primary and Standby so that applications would quickly know to connect the primary database server when we swapped roles with the TAKEOVER HADR command.

Keep in mind that if the application uses connection pooling, the app should be restarted after setting the Automatic Client Reroute settings.

Moving datafiles to a different disk/lun linux Oracle 10gR2

2013-05-13T22:08:03Z

We have a Virtual-DBA client that was rapidly running out of space for one of their databases in VMWare. To rectify the problem we decided to move a few tablespaces to a new LUN/disk presented to the VMWare OS.

We had a scheduled downtime window so we didn’t need to leave the database online to perform the move. If we had wanted to leave the database online we could have used RMAN to copy the files ahead of time, took the tablespace offline and then just recovered the tablespace.

Instead we took the database offline and then made a copy of the database files from one drive to the new drive. We then started the database in mount mode. After the database was up we used the below command in a sqlplus prompt to change the file location.

alter database rename file ‘/u01/oradata/datafile1.dbf’ to ‘/u02/oradata/datafile1.dbf’;

We used the above command to change all the datafiles associated with the tablespaces we wanted to move and then we simply opened the database with alter database open command.

We then verified that the datafiles were moved by selecting from the v$datafile view.

Once everything was verified we removed/deleted the old datafiles and then took a full backup of the database.

File libclntsh.so.11.1 still in use patching Oracle 11gR2

2013-05-23T20:31:52Z

This Saturday we were scheduled to patch eight oracle 11gR2 databases with the latest CPU/SPU set for one of our Virtual-DBA clients. The databases were running in a zone on Solaris 10. This is normally a pretty straight forward process, but this weekend we ran into an issue that we wanted to share.

We started by moving the patches to the server and unzipping them in a directory. We made a copy of the current $ORACLE_HOME by making a tar file out of it with the below command.

tar -cvf /u02/ora_home_before_patch.tar /u01/app/oracle/product/11.2.0/db_home1

We then gzipped it so it would take up less space on the drive. We then shutdown all 8 databases and the listeners that were running. We changed to the patch directory and attempted to apply the patch with the below command.

opatch napply -skip_subset -skip_duplicate

Opatch then threw an error complaing that the libclntsh.so.11.1 file was still in use and to make sure to stop all databases and listeners from the $ORACLE_HOME being patched. We checked all the oracle processes and no databases or listeners were started in the zone. We didn't have root so I couldn't use lsof if it was installed or pfiles. We got a Solaris admin involved and we couldn't find anything holding the file open.

In the end we had to end of rebooting the zone. After the zone was rebooted we were able to apply the patch without any issues and restart the databases and run our catbundle.sql cpu apply command in sqlplus.

SSIS 2012 Reporting Driver Errors

2013-05-08T17:05:31Z

Often times when implementing data integration projects - we come across a need to implement a row/key based lookup and based on the found/not-found condition update or insert a row.

I started implementing a simple data flow in SSIS. The idea was simple and has been implemented by quite a few folks and information is available all over the web. Yet when I built the simple workflow, I ran into an issue - the insert operation would fail with the following error - "[Insert Rows [23]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Connection Name" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed."

After trying several different options with package configurations (yes, they are helpful!) and storing sensitive information like password/s with a password - nothing seemed to work until I realized the driver for destination connection I was using. Everything with the connection string, test connection was working except the insert (weird).

I changed the driver from native client to MSFT OLE DB Driver and viola - everything worked as expected. Took me a couple of hours to figure this out. Hopefully it will save you some time...

Patching Oracle Applications R12

2013-05-08T17:13:53Z

So today we had a Virtual-DBA client ask us to help them patch their Oracle Financials R12 for a bug they were having a problem with. So I thought I would share with you the process we used to patch their server.

They had sent me the patch number that Oracle support had recommended, so we went to Metalink and downloaded the patch. The patch readme itself basically just says to patch the server. Sometimes these readmes have pre and post tasks that need to be accomplished. This one was just to install the patch and restart the services. Though Oracle had some additional instructions to clear the mid-tier cache and restart the server.

After the patch was downloaded I used an SFTP client like Filezilla or Winscp to transfer the patch to their server. The patch was then placed in a directory wd could easily find again. Something like /u01/patches. The file was then uncompressed and it created it's own sub directory.

This particular client had a refresh that runs and refreshes their QA environment so we made sure to disable that until they had time to test the patch after it was successfully applied.

We used adadmin and put the application server into maintenance mode. Which is as simple as typing adadmin on the command line and answering the series of questions about your particular environment. A lot of the defaults are correct until it asks you for your appsys user and system database user passwords.

Once the system is put into maintenance mode you can then change to your patch directory and apply your patch using adpatch inside the patch directory. You can use adpatch apply=n to just test the patch and see if there are going to be any errors and to also get a list of files the patch is going to change. We tested the patch with the adpatch apply=n and then went ahead and applied the patch.

After the patch successfully installed we then disabled maintenance mode by again running adadmin and then answering the questions again and selecting the disable maintenance mode opton.

We were then instructed by Oracle to clear the mid-tier cache. In 11i JSP pages were handled by apache jserv engine and in R12(our version) they are handled by OC4J. What that means is that if you just delete the $COMMON_TOP/_pages like you would in 11i. You will get a blank login page because AppsLocalLogin.jsp doesn’t get compiled by default. After you delete the _pages directory you can recompile by navigating to $JTF_TOP/admin/scripts and then running the below command:

perl ojspCompile.pl –compile –flush -p 20

We stopped the app services using adstpall.sh. We then cleared the cache and then rebooted the server as per Oracle’s instructions.

When the server came back online we started the mid-tier by using the start adstrtal.sh script. We then verified the application was up and running by logging into the web interface. We then tested the application to ensure the patch fixed the bug. It indeed did and that was the end of that.

A DB2 on Windows Upgrade Adventure

2013-04-25T18:05:06Z

Many of our Virtual-DBA clients have DB2 upgrade plans in the works. One of them wants to move from V9.1 to V10.1 on Windows. I'll share some of my experience with this project. Right now I've completed the set up of the 10.1 instance and restored a copy of their database there. They are now kicking the tires. Here are some more details:

System Background:

This a client of ours that has a busy, 250 GB, OLTP database using DB2 version 9.1 on Windows. We have set up an HADR Standby and also a disaster recovery server using log shipping. The application is a home-grown java application.

My Mission:

Move the production database from DB2 version 9.1 to a new version 10.1 DB2 instance and set up multiple HADR standby systems and Read On Standby.

My Plan:

Install DB2 v9.7 and v10.1 on a new Windows 2008 server.
Create 2 instances on the new server. One with version 9.7 and one with version 10.1
Take an offline backup of the production database and restore it onto the V9.7 instance.
Take an offline backup of the database on the V9.7 instance and restore that into the V10.1 instance.
Perform testing of the applications against the V10.1 database.
When a cutover date is assigned, Take the current 9.1 production instance offline. Backup the database and restore onto 9.7 and then onto 10.1 instance in the same way as before.
Then set up the two HADR standby servers from online backups of the 10.1 database.

My Progress and Lessons Learned.

When restoring a database between versions of DB2, you need to use an Offline backup of the database. This is because the transaction logs written during an Online backup aren't usable on the new version. So, because we could not take the production database offline, I had to restore an Online backup of the database onto a development server which also uses version 9.1. I could then take that database offline for a backup.

Once I got an Offline backup copied to the new hardware, I restored it into the V9.7 instance. The restore also migrates the database to the new version. After the restore completed I got an error:

SQL1243W

Dropping SYSTOOLS.DB2LOOK_INFO db2look operation table failed in a DB2 Version 8 database.

Explanation

The UPGRADE DATABASE command checks whether the DB2 Version 8 database has a db2look operation table called SYSTOOLS.DB2LOOK_INFO, and drops this table if it exists.

After looking through the diagnostic log I found that the upgrade was successfull so I dropped the DB2LOOK_INFO table and proceeded to test the database by connecting and selecting from a few tables. It looked good, so I took an Offline backup, and restored that into the 10.1 instance. That finished without error. All in all it was a pretty smooth process. I'll keep you updated when we go live!

Informix DRDA Connections

2013-04-19T21:29:24Z

One of our Virtual-DBA clients recently contacted me for assistance in setting up connections using the DRDA protocol. They are moving from Java to .NET for their development work, and when they tried to connect via the .NET drivers, they received the following error:

Unable to open socket to server:

ERROR [08001] [IBM] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS".

Location where the error was detected: "10.105.223.33". Communication function

detecting the error: "recv". Protocol specific error code(s): "*", "*", "0". SQLSTATE=08001

Some things to consider:

- is the port number correctly specified

50000 is the default for LUW servers

svcename dbm cfg parm for LUW servers

446 is the only port for iSeries servers, default for zSeries

- for a LUW server, is the TCPIP listener started

db2set DB2COMM=TCPIP, db2stop / db2start

- for an Informix server, ensure DRDA listener is started

and you are using DRDA port vs the SQLI port

I found that they were not set up to connect via DRDA. In order to do so, we had to add an entry to DBSERVERALIASES with a corresponding entry in the sqlhosts file for the DRDA protocol. The DBSERVERNAME for the instance cannot be used for DRDA. The protocol on the sqlhosts entry for the new alias was set to drsoctcp since their server uses sockets. Likewise, if they were using tlitcp, the entry would be drtlitcp. The new entry will need a corresponding entry with the DRDA port number in /etc/services.

Example:

demo_on onsoctcp on_hostname on_servername

demo_dr drsoctcp on_hostname dr_servername # DRDA protocol

2013 IIUG Informix Conference

2013-04-16T22:26:54Z

XTIVIA will be a Bronze sponsor at this year’s International Informix Users Group (IIUG) 2013 Informix Conference next week (April 21-24) in San Diego, California. This conference is the largest IBM Informix gathering in the world and is a technical conference for IBM Informix users. The IIUG Conference offers a unique opportunity to network with a large gathering of fellow Informix users while also taking advantage of numerous sessions showcasing the latest Informix technology. Please be sure to visit with Jack Dotson, Paul Anderson and myself at the XTIVIA table during the conference to see how we can help meet your Informix licensing and professional service needs.

Bad Application Architecture Limits Tuning Ability

2013-04-08T17:02:34Z

We have a client who needs to run monthly reports from their financial application. When we first started working for them, the monthly report took nearly a week to complete. We noticed a huge amount of logfile waits and did the classic tuning task of increasing the size and number of logfiles, while putting them on alternating disks. That reduced the processing time from several days to several hours, and kept them happy for about 5 years. However, over the years, the size of the database and the processing time have kept getting bigger and longer. It is now taking about 14 hours to complete the report. While that is much better than it was originally, it's becoming burdensome again.

This time, the statspack report needed a bit more analysis. The client is running Standard Edition without Diagnostic or Tuning Packs on a Windows 32bit OS. That rules out AWR, and using more than a couple Gigs of RAM. We kicked off an hourly statspack snapshot a week before the monthly run to ensure we had a good baseline, then picked a time in the middle of the run to compare to normal processing.

The first thing that jumped out at us was that they were doing MUCH more work in MUCH smaller transactions during the monthly report. The Load Profile section showed redo per second going from 72K to 1.5M, while the redo per transaction went from 800K to 3K. Likewise Block Changes jumped from 470/s to 8.4K/s, but plummeted from 5.2K/tran to 17/tran. Logical reads stayed approximately the same per second--13.5K to 16.6K, but per transaction dropped from 150K to 35. The only way to interpret those stats is that the application is performing row-by-row (Tom Kyte refers to this as slow-by-slow) processing. There's almost never a reason to do this, and is rightly regarded as a bug in programming. Unfortunately, it's also extremely common. Like the vast majority of tuning problems, there really isn't anything we as DBAs can do to fix it. The application needs to be fixed to allow bulk processing.

Working our way down the statspack reports, in the Top 5 Timed Events, we saw that the db file sequential read went from 5K to 295K, but the Buffer Hit Ratio had stayed in the 99%+ range. That indicates full table scans, which are sequential and bypass the Buffer Cache. If we could add RAM, it's possible we could pin those tables in SGA, but we'd really like to avoid full table scans if we can. Looking ahead to the Top SQL by reads, we saw that the top several queries all related to a table named xxx_TEMP_DETAIL. That's trouble right there. Temporary tables are almost always a bad idea in Oracle. It's pretty standard practice in MS SQL Server, so sometimes a programmer with a Microsoft background or a vendor that's trying to be "DBMS agnostic" will use the technique in Oracle. It's a bug. What's worse is that the table isn't actually a temporary table, so it incurrs much more overhead than if it were actually a temp table. Again, there isn't much we can do as DBAs to fix a poorly architected application. This code needs to do bulk processing and avoid temp tables. We can try to force better behavior by making it an actual temp table, pinning in RAM, or by making better indexes or turning it into an Index Only Table. Unfortunately, without knowing exactly what the application is doing, any of these could break the application (more than it already is) by changing side effects that the programmer is wrongly relying on.

Finally, we saw in the Top SQL sorted by CPU that the third most CPU intensive query is updating seq$. That's internal Oracle housekeeping. It means that Oracle spent more time updating sequences than executing all but two user statements. That is highly unusual, since sequences are extremely efficient. Looking at the sequence table, I see that all the application owned sequences are set to not cache. This is a very bad thing. It means that every row that relies on a sequence, also requires an update to the seq$ table. There were 1.6M unneeded updates. This is usually caused by programmers who do not understand sequences. They are not incremental row counters. Programmers who try to use them as if they were often turn off caching to try to force them to behave like row counters, but they cannot be used that way. Typically, this indicates a bug in the program. On the other hand, it is possible that some tool, or legacy code sets caching off for all tables, and the code is actually correct. If we can turn on caching, we will pave the way for much lower transaction load (with less logging) and parallel processing. Unfortunately, based on what we see, it appears the application is relying on sequences being incremental row counters, so turning on caching might break the application.

Tweet! Tweet! Need to Integrate Twitter feeds into Liferay?

2013-03-30T04:20:40Z

Twitter is now ubiquitous in the social landscape. More and more organizations are showing twitter feeds in their external facing sites. In a recent project, I was tasked with the integration of Twitter tweets into an Enterprise Portal. In this blog post, I am going to walk through the steps for installing and setting up the twitter portlet from the Liferay Marketplace. As you explore the twitter portlet, you will probably face the same limitations that I did during this project. Fear not, as I will also provide my solution to the limitations that I faced.

The requirement for this project was to have a twitter portlet that can show tweets from multiple tweeter profiles.

The instructions below are specifically for the following bundle liferay-portal-tomcat-6.1.10-ee-ga1 (tomcat 7.0.25) and the twitter portlet available on the Liferay Marketplace for EE version.

Let's deploy and try it out!

Copy the lkg (plug-in) file into your deploy folder of the Liferay bundle.
Start your Tomcat server and login in to your Liferay instance.
Select Add->Social->Twitter, add it to your public or private page.
Twitter Portlet will let you set up a Twitter account, this will go to Liferay component which is under Control Panel->My Account->Social Network. Here you can configure Twitter Account (Your Twitter Screen Name). For example, my Twitter screen name is RujuanXing.
Go back to the page you've added Twitter portlet. It'll say "Your activities will be shown in Activities Portlet".
Now you need to add Activities Portlet which is under Social category in your portal. This is where you will see the tweets of the user that you configured.

If you have not yet posted a blog or done any social activities on the portal, the Activities Portlet will display something like "No activities now".

So, how does this portlet work?

1. When you deploy twitter portlet plug-in, a table called twitter_feed is created. The table is used to store twitter screen names that you configure in Control Panel.

2. Twitter portlet schedules a job to fetch tweets from Twitter REST API. The job is configured in liferay-portlet.xml

3. When the user that you configured in the Control Panel tweets in Twitter, the Twitter Portlet will get the data and store the tweet in socialactivity table which is a default table in Liferay.

4. The Activities portlet grabs the data from the socialactivity table and displays it.

I will let you absorb the above information, and think about the constraints that you are facing with this particular offering of the Liferay Marketplace.

Some of these are obvious, some are not. Here are the items that I ran into:

The Twitter portlet available on Liferay Marketplace is using the Twitter REST API. Twitter has limited the REST API on the number of calls allowed. Unauthenticated user calls are permitted 150 requests per hour. Authenticated calls are permitted 350 requests per hour. For more details, please refer to https://dev.twitter.com/docs/rate-limiting/1.1. This becomes a key issue on a public facing site which could possibly have 1000s of users.
The REST API call is hard coded in the class. Whenever Twitter changes its API (which happens more often than you can imagine), you will need to change the code and compile and deploy. I faced this issue right at the outset. The twitter portlet that I was using from Marketplace, it uses legacy Twitter REST API to fetch tweets from Twitter.com. In March of 2012, Twitter announced their new API and in early October 2012, they turned off old endpoints. Since our project uses Liferay EE 6.1.10, I could not use the latest version of Twitter portlet which was targeted for Liferay EE 6.1.20. I did change the code and compile and redeploy. If only we could put the API in properties file, we will be spared the trouble of changing code, compile and deploy.
One more limitation that I observed, the marketplace twitter plug-in does not fetch the tweets when a user retweets other user’s tweets.
If you want to configure two or more twitter accounts in the twitter portlet, you are out of luck. I found this to be a serious limitation for my requirement.
All the tweets are shown in Activities portlet. Any social activity such as blogging will also show up on the Activities portlet effectively crowding out the tweets. There was no means of configuring a filter for just the tweets.

Because of these shortcomings, I decided to develop our own twitter-portlet for this particular project.

Below you will see the approach I took to solve the shortcomings I faced.

Twitter provides another API: Streaming API which is long live connection to Twitter. Whenever our subscribed twitter account tweets, Twitter will push the tweets to us. For more info, refer to https://dev.twitter.com/docs/streaming-apis. When you use REST API, tweets are stored at Twitter. With Streaming API, you need a place to store those tweets. In short, we need to design some tables in the database to store those tweets.

For this project, I designed three tables: organizationstatus, tweet and feed.

organizationstatus table: This table was used to store twitter screen name and organization id which is configured in Liferay.

feed table: This table was used to store twitter user id and organization id.

tweet table: This table was used to store tweets from the subscribed users on Twitter.

There are three specific applications: twitter-persistence, twitter standalone app and twitter portlet.

Twitter Persistence: This app is used to perform all database operations.

Twitter Standalone: This app is used to set up a long live connection to Twitter.com. It listens to a list of twitter accounts. When those accounts tweet on Twitter, those tweets will be pushed by Twitter.com to our connection. Our standalone app will store those tweets in tweet table. Because our listening list is dynamic, when user adds/updates/deletes organization's twitter account in Liferay, we need to change our listening list. In order to do that, we have a job to check organizationstatus table every 30 minutes (you can specify the time you want). When the job finds that the table is changed, it'll make the corresponding change into feed table including get the new twitter user id and then change the listening list.

Twitter Portlet: It has two functions: Primary function is to display tweets and the other is actually a hook which stores organization's twitter screen into organizationstatus table. Whenever a user logs into Liferay Portal, the portlet will get the User’s organization, and display the tweets. If there are no tweets with current organization, the portlet will display the organization's parent organization's tweets. When admin adds/updates/deletes organization's twitter account, we make a corresponding update to the organizationstatus table.

Second, how can we set up a long live connection to Twitter.com? Previously, I was thinking I need to start coding from scratch. But then I found twitter4j which is an unofficial Java Library for Twitter API. You can download jar files and demo here.

I was able to overcome the limitations that I faced while using the Liferay’s twitter portlet by essentially writing my own twitter portlet. With a clean design approach, and efficient coding standards, I was able to make the portlet do what I specifically started out to do.

Let me share a small demo with you. This demo is used to get the tweets which we specify in the filter. In this case, I am only listening to the tweets of RujuanXing and CNN.

Another question is how to get twitter userId based on twitter screen name. It's really easy to do that when we use twitter4j.

Here is a screenshot of the twitter portlet that I developed –

Hopefully this helps in getting you started in writing your own twitter portlet! If you have any question, need help or want to share your perspective, just comment at the bottom of this post or contact Xtivia, my employer!

How to shrink the SQL Server log file?

2013-04-04T17:34:59Z

SQL Server 2005/2008 : Error: 9002, Severity: 17, State: 2

Error: 9002, Severity: 17, State: 2

The transaction log for database 'mydatabase' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases

Very often clients come to us requesting assistance because the log file is full and the transactions won’t complete anymore. They try to fix this problem by expanding the size of the log file or adding more space to the disk drive where the log file is located. Although this gives them some breathing room, very soon they land in the same situation as before.

SQL Server logging mechanism is not that complicated but it’s best to leave the log file management to the experts (DBAs). I will try to explain the most common reason why the log file will get full.

Databases in SQL Server can be in either SIMPLE recovery mode or in FULL recovery mode. In SIMPLE recovery mode as soon as the transactions are committed the log space can be re-utilized. But SIMPLE mode does not give you the option of recovering the database using log backups.

If your database is in SIMPLE recovery mode and the log file is full then either the file size is small for the amount of active transactions or you have run out disk space.

But if your database is in FULL recovery mode and the transaction log file is full then follow these steps to fix the problem.

If the log file is configured with a preset max size then see if you can increase that for now.

Now lets look at why the file got full in the first place. First thing that you need to check is the log_reuse_wait_desc column in the sys.databases.

select name, recovery_model_desc, log_reuse_wait_desc from sys.databases

There are several reasons that could come up in this column and some of them are noted here.

NOTHING

CHECKPOINT

LOG_BACKUP

ACTIVE_BACKUP_OR_RESTORE

ACTIVE_TRANSACTION

DATABASE_MIRRORING

REPLICATION

DATABASE_SNAPSHOT_CREATION

LOG_SCAN

OTHER_TRANSIENT

If the database in question is TEMPDB then the process to resolve it would be different and also the reasons for which TEMPDB gets full are different. But let me discuss the most common reason why a user database’s log file gets full.

LOG_BACKUP

In most cases you will see the reason noted in 'log_reuse_wait_desc' is given as 'LOG_BACKUP'. This means that the database is in FULL recovery model and is waiting for a log backup to be taken.

If you have scheduled a regular log backup job then check its status and wait for it to finish before you shrink the log file. If you check the free space in the log file then you will indeed see a lot of unused space but you can not shrink it. Once the log backup completes you can shrink the file.

You can run the following TSQL to get the amount of free space vs used space in the log file.

select

[FileSizeMB] =

convert(numeric(10,2),round(a.size/128.,2)),

[UsedSpaceMB] =

convert(numeric(10,2),round(fileproperty( a.name,'SpaceUsed')/128.,2)) ,

[UnusedSpaceMB] =

convert(numeric(10,2),round((a.size-fileproperty( a.name,'SpaceUsed'))/128.,2)) ,

[DBFileName] = a.name

from

sysfiles a

But if the data file is not as big as the log file then instead of doing a log backup, I would do the following.

1) Change the recovery model to SIMPLE

2) Run a CHECKPOINT.

3) Shrink the log file

3) Change the recovery model to FULL

4) Take a full backup and subsequently schedule log backups.

Sometimes the above steps take a lot less time to complete than taking a log backup and then shrinking the file. But please keep in mind that when you do this you have essentially broken the log chain and will have to resync the database if it is configured for log shipping.

The question of whether to truncate the log or not is dependent on the DB size. If it is not too big then truncate it and take a full backup. Otherwise it is best to take log backups.

ACTIVE_TRANSACTION

Another prominent reason that I have seen is 'ACTIVE_TRANSACTION'. In this case, it would be best if you first add a new log file to the database or extend it. Then run the DBCC OPENTRAN on that database and check the open transactions. This should give you more information about the transaction that is consuming most of the log space and has not yet completed.

If the reason given is ACTIVE_BACKUP_OR_RESTORE then refer to my earlier post to find what is the expected time to finish the current backup or restore.

If the reason is related to either replication or mirroring then first check the status of replication or mirroring to ensure that they are up to speed and don't have any latency. This should help in reducing the log reuse wait time.

SQL Server backup or restore progress

2013-04-03T16:21:59Z

SQL Server 2005/2008: Find backup and restore status/progress

Ever wondered how long it would take to complete a backup or restore that is running in the background. I, for one, do not want to keep looking at the job or TSQL that is taking the backup or doing the restore.

I found this neat TSQL that will give you the progress report of the backup/restore and will also tell you how long it would take to complete it. In case you are running the backup with the VERIFY option then it does not tell you how long it would take to complete the VERIFY phase.

Run the following TSQL query to get the status

SELECT command,

s.text,

start_time,

percent_complete,

CAST(((DATEDIFF(s,start_time,GetDate()))/3600) as varchar) + ' hour(s), '

+ CAST((DATEDIFF(s,start_time,GetDate())%3600)/60 as varchar) + 'min, '

+ CAST((DATEDIFF(s,start_time,GetDate())%60) as varchar) + ' sec' as running_time,

CAST((estimated_completion_time/3600000) as varchar) + ' hour(s), '

+ CAST((estimated_completion_time %3600000)/60000 as varchar) + 'min, '

+ CAST((estimated_completion_time %60000)/1000 as varchar) + ' sec' as est_time_to_go,

dateadd(second,estimated_completion_time/1000, getdate()) as est_completion_time

FROM sys.dm_exec_requests r

CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) s

WHERE r.command in ('RESTORE DATABASE', 'BACKUP DATABASE', 'RESTORE LOG', 'BACKUP LOG')

Creating a simple Elastic Search cluster

2013-03-24T18:10:55Z

In this post (my second post in a series on Elastic Search and Liferay) we are going to see how to install and configure Elastic Search on our computer and perform simple queries. I am going to use 0.19.4 version in Windows, but the process is really similar with other versions / platforms.

Follow these steps:

Select and download a version of Elastic Search.
Unzip in two different locations (yes, we are going to create a simple cluster with two nodes):
- C:\Projects\ElasticSearch\elasticsearch-0.19.4-node-1
- C:\Projects\ElasticSearch\elasticsearch-0.19.4-node-2

Modify \node_name\config\elasticsearch.yml in both nodes:

Original	\elasticsearch-0.19.4-node-1\config\elasticsearch.yml	\elasticsearch-0.19.4-node-2\config\elasticsearch.yml
[...] ################################### Cluster ################################### # Cluster name identifies your cluster for auto-discovery. If you're running # multiple clusters on the same network, make sure you're using unique names. # # cluster.name: elasticsearch #################################### Node ##################################### # Node names are generated dynamically on startup, so you're relieved # from configuring them manually. You can tie this node to a specific name: # # node.name: "Franz Kafka" # Every node can be configured to allow or deny being eligible as the master, # and to allow or deny to store the data. # # Allow this node to be eligible as a master node (enabled by default): # # node.master: true # # Allow this node to store data (enabled by default): # # node.data: true [...] # Set a custom port for the node to node communication (9300 by default): # # transport.tcp.port: 9300 # Enable compression for all communication between nodes (disabled by default): # # transport.tcp.compress: true # Set a custom port to listen for HTTP traffic: # # http.port: 9200 [...]	[...] ################################### Cluster ################################### # Cluster name identifies your cluster for auto-discovery. If you're running # multiple clusters on the same network, make sure you're using unique names. # cluster.name: my_cluster_name #################################### Node ##################################### # Node names are generated dynamically on startup, so you're relieved # from configuring them manually. You can tie this node to a specific name: # node.name: node1 # Every node can be configured to allow or deny being eligible as the master, # and to allow or deny to store the data. # # Allow this node to be eligible as a master node (enabled by default): # node.master: true # # Allow this node to store data (enabled by default): # node.data: true [...] # Set a custom port for the node to node communication (9300 by default): # transport.tcp.port: 9300 # Enable compression for all communication between nodes (disabled by default): # # transport.tcp.compress: true # Set a custom port to listen for HTTP traffic: # http.port: 9200 [...]	[...] ################################### Cluster ################################### # Cluster name identifies your cluster for auto-discovery. If you're running # multiple clusters on the same network, make sure you're using unique names. # cluster.name: my_cluster_name #################################### Node ##################################### # Node names are generated dynamically on startup, so you're relieved # from configuring them manually. You can tie this node to a specific name: # node.name: node2 # Every node can be configured to allow or deny being eligible as the master, # and to allow or deny to store the data. # # Allow this node to be eligible as a master node (enabled by default): # node.master: false # # Allow this node to store data (enabled by default): # node.data: true [...] # Set a custom port for the node to node communication (9300 by default): # transport.tcp.port: 9301 # Enable compression for all communication between nodes (disabled by default): # # transport.tcp.compress: true # Set a custom port to listen for HTTP traffic: # http.port: 9201 [...]

Original

\elasticsearch-0.19.4-node-1\config\elasticsearch.yml

\elasticsearch-0.19.4-node-2\config\elasticsearch.yml

[...]

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

# cluster.name: elasticsearch

#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

# node.name: "Franz Kafka"

# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

# Allow this node to be eligible as a master node (enabled by default):

# node.master: true

# Allow this node to store data (enabled by default):

# node.data: true

[...]

# Set a custom port for the node to node communication (9300 by default):

# transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):

# transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:

# http.port: 9200

[...]

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

cluster.name: my_cluster_name

#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

node.name: node1

# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

# Allow this node to be eligible as a master node (enabled by default):

node.master: true

# Allow this node to store data (enabled by default):

node.data: true

[...]

# Set a custom port for the node to node communication (9300 by default):

transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):

# transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:

http.port: 9200

[...]

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

cluster.name: my_cluster_name

#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

node.name: node2

# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

# Allow this node to be eligible as a master node (enabled by default):

node.master: false

# Allow this node to store data (enabled by default):

node.data: true

[...]

# Set a custom port for the node to node communication (9300 by default):

transport.tcp.port: 9301

# Enable compression for all communication between nodes (disabled by default):

# transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:

http.port: 9201

[...]

It may be necessary to copy several jars into \lib folder. If they are not present, be sure to copy these jars in both nodes:
1. elasticsearch-0.19.4.jar
2. jna-3.3.0.jar
3. jts-1.12.jar
4. log4j-1.2.17.jar
5. lucene-analyzers-3.6.2.jar
6. lucene-core-3.6.2.jar
7. lucene-highlighter-3.6.2.jar
8. lucene-memory-3.6.2.jar
9. lucene-queries-3.6.2.jar
10. snappy-java-1.0.4.1.jar
11. spatial4j-0.3.jar
And if you want to use analyzers, may be also in \plugins\analysis-icu folder,
1. elasticsearch-analysis-icu-1.7.0.jar
2. icu4j-4.8.1.1.jar
3. lucene-icu-3.6.1.jar
Create a file named start_elasticsearch_cluster.bat like this:

start call "c:/Projects/ElasticSearch/elasticsearch-0.19.4-node-1/bin/elasticsearch.bat" PING -n 1 -w 15000 1.1.1.1>NUL start call "c:/Projects/ElasticSearch/elasticsearch-0.19.4-node-2/bin/elasticsearch.bat"

What does this file do? It starts the first node, it waits for 15 seconds (just pinging for 15 seconds to nowhere and redirecting the result to nowhere :), this is really useful for wait some time until doing something else), and then it starts the second node.
Go to http://localhost:9200. You will see a page like this:

{
"ok" : true,
"status" : 200,
"name" : "node1",
"version" : {
"number" : "0.19.4",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"
}

And right now, you are prepared to create an index, add a document to the index, and perform amazingly fast searches against your new Elastic Search Cluster!

If you have any question, need help or want to share your perspective, just comment at the bottom of this post or contact Xtivia, my employer!

Elastic Search: A distributed RESTful Search Engine

2013-03-24T15:26:44Z

Traditionally, building portals and/or web sites with "search that just works" has been hard but over the last few years, life has become easier. Various search tools such as Lucene and Solr have emerged, and platforms such as Liferay have integrated with them to provide the kind of search capabilities that I and my customers/users expect - fast, east to set up, easy to scale, and easy to find relevant results. Recently I had an opportunity to use elasticsearch instead of Solr or Lucene as the search engine on an enterprise scale portal implementation project with Liferay. This is the first blog post in a series that talks about elasticsearch and how you can integrate it with Liferay.

Elastic Search: A distributed RESTful Search Engine

That's how they define themselves on their website - www.elasticsearch.com. It was created as a scalable search solution with a common interface, JSON over HTTP, and the first version was released in February 2010.

These are the most important features:

Based on Apache Lucene, elasticsearch can be used to search all kinds of documents. Lucene's features are available through the JSON and Java API. It supports faceting that is really useful to use with advanced search systems such as Liferay's (a really fantastic explanation about that can be found in Ray Auge's blog), and percolating, which can be useful for notifying users if new documents match for registered queries.
Open Source, it is stored in GitHub and its latest version can be reached in its repository. There are a lot of clients and integrations available (clients for Groovy, Perl, Python, Java, Ruby, PHP or .Net, among others, and integrations with Jetty, Drupal, Spring or Liferay, among others).
It is distributed, which means that indices can be divided into shards and each shard can have zero or more replicas. Each node hosts one or more shards, and acts as a coordinator to delegate operations to the correct shard(s). It provides a scalable search solution, has near real-time search and support for multitenancy.

RESTful, it uses JSON models for representing data. With JSON it is simple to provide semi-structured data with complex entities as well as being programming language neutral with first-level parser support.

Of course, the first thing that comes to mind is, ok, "for Search", but what can Elastic Search do and how can I compare it to Solr, for example? I am not going to answer this second question because a lot of people has done so before me, but about what Elastic Search can do, I will stop and reply: a lot of things, easily and really fast!!

Explaining every functionality about Elastic Search would be really useful, but if someone has already done so, why do it again? While I have been working with Elastic Search, I found some additional resources really helpful. Here is a list of some of those useful resources:

Elastic Search Website
Elastic Search Guide
- Glossary of terms
- Index management
  - Mapping
  - Templates
- Search API
  - Sort
  - Highlighting
  - Facets
- And, of course, Java API
Elastic Search code
A lot of Elastic Search projects:
- A really useful example about how Elastic Search can work
- Spring factories for Elastic Search
- Search Schemer, a simple application which allows to convert index structure between Solr, Elastic Search and SenseiDB
- Jest, a Java HTTP Rest client for ElasticSearch
Curl, a command line tool for transferring data with URL syntax.
And, of course, Elastic Search Users, where you can share your experience and get help.

The three most useful Elastic Search vs Solr comparisions:

If you have any question, need help or want to share your impressions, just comment below on this blog post or contact Xtivia!

Preping Oracle Linux(OEL5 / OEL6) for an Oracle Database install using oracle-validated RPM

2013-03-08T16:58:55Z

Today we were helping a Virtual-DBA client prep an Oracle Linux 6 server to install Oracle Grid Control 12c. We were configuring the server to have the database on the local Grid Control server since it will be monitoring a small number of databases. We wanted to speed up the prep work for the Oracle database install so we were planning to use the oracle-validated RPM package. This will speed up the process by creating an oracle user, oinstall and dba groups, setting kernel parameters, sets hard and soft shell resource limits, downloads and installs the required packages for the Oracle database install to complete successfully. Below are the steps outlined to accomplish this task.

After the server software is installed we need to either register our server with the Unbreakable Linux Network(ULN) or the public repository. Unless you have bought support for the ULN you will be using the public repository.

To register your server with the ULN you will need an account with the correct permissions most likely root. You will then either type in uln_register for OEL6 or type in up2date --register for OEL5. You will then be prompted for your Oracle account information and your ULN CSI number. If you selected the options at the bottom of the screen you server will then upload information about the servers packages and configuration. After the upload you will be able to visit https://linux.oracle.com/ and manage the channel subscription for that server.

If you haven’t purchased ULN support you can get your updates from Oracle’s public repository. For OEL5 and OEL6 you cd to /etc/yum.repos.d and then type

OEL5:

wget http://public-yum.oracle.com/public-yum-el5.repo

OEL6:

wget http://public-yum.oracle.com/public-yum-el6.repo

Once you have downloaded the updated repository information you can then edit the file with your favorite text editor. To update your subscriptions simply edit the file by changing enabled=0 to enabled=1 below the channel you wish to subscribe.

After you have either registered with ULN or the public repository you can then type yum update to update your subscriptions. You can get a list of packages by typing yum list.

Finally to install an RPM from the repository of your choice you can type yum install <package_name>.

For the automated configuration of the server to prep for the Oracle Database install the command you are looking for is either:

OEL5:

yum install oracle-validated

OEL6:

yum install oracle-rdbms-server-11gR2-preinstall

(Note on licenses: You are allowed to install a copy of Oracle Database to house your repository for both Grid Control and a Recovery Catalog as long as you don’t use the database for any other purposes and the databases being monitored are licensed.)

Looping Construct for ETL - Simplified by Talend DI

2013-03-02T21:57:22Z

I am not a regular ETL developer but I do "dabble" in developing code from time to time. One requirement that I come across is looping (while, for, infinite) in a job and performing a task based on condition and then going to sleep for a period of time. This certainly can be accomplished through a scheduler as well but is often time easier to accomplish in the ETL job design.

I have had the opportunity to implement this construct using SSIS and Datastage. In my opinion, Datastage was the most complicated - took me a good 5 hours to get the code to do exactly what I wanted. SSIS was more cooperating - I was able to do the same in about 3 hours.

As I started working with Talend Open Studio earlier this week, I was amazed to see how simple it was to work with Talend.

As you open the Studio, you can find various orchestration steps very quickly - something the other two tools do not do. i.e. the IDE user interface is more intuitive for Talend

From here, it was drag and drop easy...

Step 1: Drag the while loop on the canvas - define the parameters.

Step 2: Create a job run step to be executed in each of the iterations

Step 3: Drag the "sleep" step and assign values.

I was done start to finish in about 20 minutes and the code generated in Java was pretty easy to read and understand. Very cool!

The Importance of a Security Audit

2013-02-27T14:01:54Z

Often I am asked about the importance of a security audit, with the majority of these questions being asked by representatives of smaller organizations. The questions are centered around if there is a need to perform a security audit when the organization has taken some measures to protect themselves on the network in one fashion or another. For example I was recently asked why a company should perform a security audit when the SQL Server they have is behind a firewall and the company is small enough that everyone knows all the staff.

I recommend security audits for all SQL Servers in my care. Let me explain why…

Security can be very complex with multiple layers, yet even the most complex systems can be as risk and sometimes even from their own complexity. If the data in the database is important enough to keep around it is important enough to secure. The responsibility is multiplied when the data contains sensitive information or Personally Identifiable Information (PII). As a database professional, I want to know that even if a layer of security is breached at points where security is intended to protect the database server that I still have a layer of protection. In other words, if for some reason the firewall were to be circumvented or compromised the database still has its own layer of protection. According to statistics from datalossdb.org (http://datalossdb.org/statistics) almost 40% of data breach incidents come from lost, stolen or fraud to obtain access to physical media such as a lost or stolen laptop, or lost or stolen backup tape, none of these methods protected by a firewall.

At one point during a Sarbanes Oxley audit I was asked why management of our company determined that the data should be easily accessible by IT and Development staff. I learned a great lesson when the auditor explained to me that security is not a matter of trust. Security also protects from legitimate mistakes when no malicious intent is involved. The statistics from datalossdb.org reflect that 38% of data loss incidents come from internal to the organization. 21% of all incidents are accidents from internal sources with 10% of all incidents as being malicious in nature from internal sources. Just over half of the incidents come from outside the organization, how does this reflect on completing a security audit? By completing a security audit and making appropriate changes a company can limit its risk by nearly 50%.

The scary part as a database administrator is the fact that as a whole we have only seen the decrease in the number of incidents once over the last 9 years, 2012 had the most incidents with a drastic increase over 2011. If the trend holds true, as a database professional I urge organizations complete audits on a regular basis.

Our Configuration and Performance Review for SQL Server can be focused on security. Contact us for more information through our web site or by sending an email to info@xtivia.com. Or you can reach out to me via email at cshaw@xtivia.com.

Excel Chart Portlet

2013-02-25T04:15:20Z

Microsoft Excel is the de facto standard spreadsheet tool used by businesses around the world and houses some of the most critical business information. However businesses often struggle to make this information available to employees and other stakeholders that would benefit from having access to this information.

An enterprise portal such as Liferay is a framework for integrating information, people and processes across organizational boundaries. It provides a secure unified access point in the form of a web-based user interface, and is designed to aggregate and personalize information through application-specific portlets. If we can take some of the business information housed in Excel documents, and surface that information graphically via portlets on the enterprise portal, then enterprises would be able to derive much greater value.

Many years back, Plumtree Portal was a leading portal product in the early 2000s until its acquisition by BEA Systems in 2005, and Oracle subsequently acquired BEA. Plumtree's former portal product became part of BEA AquaLogic Use Interaction Portal suite which has now been replaced by another Oracle product. One of Plumtree's innovations was providing the Plumtree Gadget Framework for Microsoft Excel, which empowered portal users to embed Microsoft Excel tables and charts in a personalized or community portal page. The Excel Gadget Framework helped reduce development costs by allowing customers to publish a vast range of new resources in the portal from every business unit and workgroup in the organization without any coding, including the data imported from applications such as Oracle, SAP and Siebel, as well as the objectives and performance metrics stored in spreadsheets on manager's computers.

In conversations with customers looking to migrate from Plumtree to Liferay, we found that many of these customers found the Plumtree Excel portlets a killer feature. So, we at Xtivia, decided to build the Xtivia Excel Chart Viewer portlet to address this need - you can upload relevant Excel documents to the Liferay document library using Liferay out-of-the-box functionality, and then use the Xtivia portlet to point to the Excel document in the Liferay document library, and configure the portlet to display a chart using the data in the specified Excel worksheet cell range. You can add multiple instances of this portlet to the Portal with each instance configured to point to different Excel files, different worksheets, and different cell ranges.

In this blog, I’m going to provide some usage information about the Xtivia Excel Chart Viewer portlet that was developed using Apache POI and some charting APIs. In the portlet, users can easily configure the data source using excel files uploaded in the Liferay document library, the data range inside the excel file and the chart type to be rendered.

The following steps describe how to add an Excel Chart Viewer portlet on a page and configure it:

1. Upload the Excel file that you want to generate charts from to the Liferay document library.

2. Add “Excel Chart” portlet on a page.

3. Choose to configure the portlet.

Charting API: Currently we have two chart libraries available which are Fusion Chart and Any Chart
Chart Type: For each chart library, we have 4 different chart types available which are Column 2D, Column 3D, Pie 2D and Pie 3D.
Worksheet Name, From Cell, To Cell: To specify the cell range of the data in the excel file. The available values of “From Cell” and “To Cell” are the cell id in the spreadsheet. For example, we can specify “$A$1” as the “From Cell” and “$D$8” as the “To Cell”.

4. Select the Excel file from the document library.

5. Save the settings and close the dialog box. The chart shows in the portlet.

Please note that this portlet will soon be available for download from the Liferay Marketplace free of charge.

We plan to offer other innovations in terms of surfacing Excel data via portlets including more charting options, more charting libraries, data grids, and more. Please do comment on this blog post with features that you would like to see added.

For more information about expert help with your Liferay implementation, check out http://www.xtivia.com/portal/liferay-portal, or contact us at info@xtivia.com.

SQL Server Key Performance Metrics – Page Life Expectancy

2013-02-23T13:54:47Z

One of the important performance metrics we monitor for our Virtual-DBA and consulting customers is Page Life Expectancy (PLE). We look for low values as well as values that are trending lower over time. A drop in PLE is a prime indicator of memory pressure.

PLE is how many seconds a page can be expected to reside in memory before being flushed to disk. Microsoft recommends that SQL PLE not fall below 300. In real life we often see PLE in the 2000+ range in healthy systems.

Maintenance activities can drop PLE substantially. Rebuilding indexes or doing backups can drop PLE. When a drop in PLE occurs often determines its importance. For instance a drop in PLE overnight may not be as important as a drop during business hours.

Fixing Page Life Expectancy

In order to fix PLE problems we need to examine the entire Server. We need to look at Available Mbytes of Memory as well as what functions the server performs. We don’t want to create a situation where SQL Server has to give back memory to the operating system since this can result in clearing the SQL Server cache with an associated temporary performance drop. For this, and other reasons we prefer to set Max Server Memory to a specific value rather than leave it unlimited.

I like to look at a long term perfmon trace that includes Memory:Available Mbytes as well as some other counters like Total and Target server memory and Memory Grants Pending from SQL Server Memory Manager. If you consistently have a large number of available Mbytes of memory available consider increasing SQL Server Max server memory.

It’s important to note that the Max Server Memory setting has changed in SQL 2012. Before it measured the amount of memory allocated to the buffer pool. Starting in SQL 2012 the Max server memory includes many more important memory consuming areas of SQL Server. In SQL 2008 and earlier we needed to estimate the memory needs of non-buffer areas of SQL Server in calculating an optimum Max Server memory.

Oracle Enterprise Manager Grid Control Agent Installation Linux

2013-02-21T13:55:36Z

We had a Virtual-DBA customer request that we install and configure a new Grid Control agent on a new server with existing databases. While there are several ways to do this, we wanted the customer to be able to perform this task by themselves in the future.

The method we chose to use was to deploy the agent from Enterprise Manager. The process of adding a new agent can be complex and difficult sometimes, but this procedure is fairly straight forward.

We logged into the server hosting Grid Control and ensured that the agent we would be deploying was available for the platform we were deploying. We were deploying to Linux so we navigated to $OMS_HOME/sysman/agent_download. Here you will see versions of the agent currently available on the server and under the versions there will be the platform specific agent files. So if you wanted to install a windows agent and it wasn't available on your Grid Control server you would download it from Oracle or Oracle Support and place it in this location.

Next we logged into both servers and made sure we were able to SSH to the target and from the target to the Grid Control server. If SSH isn't already setup between Oracle Management Server (OMS) and the target the deployment wizard will try to set it up by creating keys, but we have had issues with this in the past so we choose to test and set it up ourselves if there are any issues.

Our host was in a different domain so we edited our host files on both the target server and Grid Control server to add both fully qualified host names and IP address as our client isn't using DNS for their Linux servers. For safe measure we also added entries to the TNSNAMES.ORA file on the Grid Control server to ensure we could tnsping the databases that were to be added to Grid Control.

With that little bit of setup we were then ready to log into Enterprise Manager and begin the deployment by clicking on the deployment tab from the home page once logged in. We then selected 'Install Agent' on the next page. We did a fresh install so we selected 'Fresh Install'.

On the next page we selected the default Source Shiphome Directory as we verified in the beginning that the agent we wanted to install was indeed available via Grid Control server. We selected our agent and our platform versions from the dropdown boxes. We then provided our list of hosts that were to have the agent installed. We made sure to only put hostnames here and not IP addresses. We then entered in our host credentials and base installation directory that we wanted the agent to be installed on the target server. We left the default port as that’s what they were already using. With the information filled it we were able to then click next and let the automated process finish the agent install. After the agent install we were able to look and manage the new hosts and databases from our Enterprise Manager console.