Knife Data Bag from file error

I created a data bag and saved it to disk in json format.  I made some changes in the files and then tried to upload data bag from file: knife data bag from file c:\dev\…\data_bag_folder\* When I received this unhelpful error message: ERROR: Chef::Exceptions::ValidationFailed: Option data_bag’s value C:/Dev/…/data_bag_folder/bagItem.json does not match regular expression /^[\-[:alnum:]_]+$/ It has nothing to do with regular expressions.  If you look closely at the command, I failed to specify the name of the data bag.  It should have been: knife data bag from file dataBagName c:\dev\…\data_bag_folder\* Once I specified the data bag name, it worked just fine.

ssh connection refused mystery solved

I moved a vmware host into a different vlan and suddenly people could not connect to it from a special jump box that was created to access servers in that particular vlan.  I check a bunch of places that I could think of then asked my co-worker.

This is a centos 5.x linux server and we checked the following:

/etc/ssh/sshd_config
/etc/sysconfig/network-scripts-ifcfg-eth0
/etc/sysconfig/network
/etc/hosts
/etc/sysconfig/network-scripts/route-eth0/etc/hosts.allow

And finally found: /etc/hosts.deny which had a line that specified:

sshd : ALL EXCEPT 10.xxx.xx.x/255.xxx.xxx.xxx

We added an entry for the new subnet, saved and closed the file and voila.

Thanks to Rachael Stewart for finding this one.  It seems this is an older way of securing the server before they started using iptables.

Solr running in glassfish – 503 server shutting down error (centos, linux)

This one was caused by two items:

  1. Log4j files were incorrect
  2. Permissions on the core.properties file

Log4j Files.  There was a bug in my deployment script that copied two of the log4j files to the same name in the destination.  So, two files appeared missing and two others were the wrong size.  Once I fixed that, I started getting solr logs.

Permissions.  The solr logs showed that it was not able to access the core.properties file.  The file was there, owned by the user running glassfish with permissions 664.

The containing folder had permissions 664 also.  Changed the permissions on that folder to 764 and voila.  It started working.

 

Java JPS cannot see running java processes

This one took a lot of digging.

JPS is the java version of linux ps command.  It looks for running java processes.

We installed zookeeper and created a user for the process to run under.  Exhibitor (created and open-sourced by netflix) uses jps to find the zookeeper process.  We ran exhibitor under the same user credentials as zookeeper.

Looking in the exhibitor logs, I could see errors like “jps unable to find zookeeper instance.”  To try to research the issue I tried to run the jps command from the command line.  Using the normal ps command I could see the process but jps could not.  I would su as the new user and run jps.  I couldn’t see the zookeeper process with that user even though I could see with ps.

It turns out that java uses a temporary directory on linux in /tmp/hsperfdata_{username} and jps uses this temporary directory to find java processes.

For whatever reason, the directory had not been created so jps could not find the temporary files.  I’ve seen this has happened to other users, too.

There are two options.

Because exhibitor was running jps internally and I couldn’t change the call, I had to use the first option.  Assume I have created a user called “myAppUser.”  We also created a group like “MyAppGroup” and myAppUser is a member of “MyAppGroup.”

mkdir /tmp/hsperfdata_myAppUser
chown myAppUser.MyAppGroup /tmp/hsperfdata_myAppUser

Option 2

Start java with the following option: -Djava.io.tmpdir=/someDir
Then start jps with -J-Djava.io.tmpdir=/someDir (check jps instructions to verify the syntax)
Make sure your user has rights to /someDir.

chef_gem Illformed Argument “”

Developed my cookbook on the dev server and it works great.  Try it on the UAT server in the prod environment and get chef_gem error Illformed Argument “”.  Are you kidding me?!  The code hasn’t changed at all.

It turns out that these new machines don’t have internet access for security reasons.  So, the chef_gem resource cannot download the gem generating a completely useless error message.

But, the solution is simple, or so I thought.  Just download the gem, add it as a file in the recipe and use the cookbook_file resource to download it first.

But, as it turns out, the chef_gem resource runs during compile time.  the cookbook_file resource runs at converge time. (later than compile).

Solution, force the cookbook_file resource to run at compile time.

cookbook_file “abc-gem-1.2.3.gem” do
path “/tmp/abc-gem-1.2.3.gem”
mode “0764”
action :create
end

resource(:cookbook_file => ‘abc-gem-1.2.3.gem’).run_action(:create)

chef_gem ‘abc-gem’ do
source ‘/tmp0/abc-gem-1.2.3.gem’
end

voila!

Now the chef_gem package is pulling the gem from a local file downloaded by the recipe.

Note: the chef_gem package runs at compile time so that your recipe can use in the converge whatever it installs.

Subversion Commit Fails – directory is already locked

I issued the subversion command-line command from windows dos prompt:

svn commit {path} -m ‘automated server backup’

to which subversion responsed:

Commit Failed
Working copy {path} locked
{Path} is already locked
run ‘svn cleanup’ to rmove locks

I tried svn cleanup and it didn’t help.  After a lot of trial and error, I changed the single quotes to double quotes in the -m paramaters.  Voila.  Subversion was ready to do my bidding.

svn commit {path} -m “automated server backup”