Category Archive: Programming

Subcategories: C# / .NET  Mono  C++  JavaScript  PHP  SQL  PL/SQL  Web Services  WordPress  Plug-Ins 

Posts related to programming

Database Abstraction v0.8

  By Daniel | October 22, 2011 - 9:00 pm | C# / .NET, Database Abstraction, MySQL, PostgreSQL, SQL Server, SQLite Leave a comment

When we began developing C# web applications, we found ourselves in the position of determining what the best way of accessing the database is.  We evaluated several technologies…

  • NHibernate – May be very good, but it was overkill for what we were trying to do.
  • LINQ to SQL – This brings C#’s LINQ (Language-Integrated Query) to SQL databases.  You create database-aware classes and use LINQ to select from collections, which LINQ to SQL converts to database access.  This is a good abstraction, but it relies on SQL Server; as we typically deploy to PostgreSQL, this didn’t work.  (We also couldn’t get DBLinq, a database-agnostic implementation, to work.)
  • ADO.NET – This is the tried-and-true database access methodology, released as part of the initial release of the .NET framework.  The downside to this is that it encourages SQL in the code at the point of data retrieval; it does not provide a clean separation of data access from data processing.
  • EF Code First – This didn’t exist; it’s also very SQL Server-centric.  Not faulting Microsoft for that, especially since they release a free version now; but, as we deploy on Linux, until they release a Linux version, SQL Server is not an option.

With our PHP applications, we had written a database service that read queries from XML files.  Then, queries were accessed by name, with parameters passed via arrays.  The one thing that ADO.NET has that was useful was the fact that it is based on interfaces.  This means that if we wrote something that exposed, manipulated, and depended on IDataConnection (instead of SqlConnection, the SQL Server implementation of that interface), we could support any implementation of database.  The SqlDataReader implements IDataReader as well.  Our solution was becoming apparent.

Over time, we developed what is now the Database Abstraction project hosted on CodePlex.  On Thursday, we released the first public release (although the DLLs are in the repository, and are usually current at every commit).  If you are looking for a way to separate your data access from the rest of your code, or want a solution that’s database-agnostic, check it out.  It supports SQL Server, MySQL, PostgreSQL, SQLite, and ODBC connections *, using the data provider name to derive the proper connection to implement.  There is also a Mock implementation to support unit tests; this mock can provide data, providing a useful way to test methods.  Finally, there is a membership and role provider based on Database Abstraction; simply configure the connection string, create the database tables, and away you go! **

A pre-released version is already in production use in our PrayerTracker application, and others are being built around it.  If this sounds like something that could help your project, certainly feel free to check it out!

* Oracle is omitted from this list, as their DLL had redistribution restrictions; this meant that the source code repository, upon check-out, would have build errors.  There may be an Oracle implementation in the future (it would be trivial), but there is not one now.

** The membership and role providers are untested; they will be tested and tweaked by version 0.9.

Tagged , , , , ,

HCSB Verse of the Day (Plus) 3.0.1 – WordPress Plug-In

  By Daniel | September 3, 2011 - 10:30 pm | Plug-Ins 2 Comments

After a nearly four-year run at version 2, the HCSB Verse of the Day plug-in has been updated to version 3.  The latest version available is 3.0.1, which contains a quick fix that was found just after I had released version 3.  When you see that “.0.1,” just think, “Oh, this was written by a human!”

Major changes in this version include:

  • The addition of “(Plus)” to the name, as this version supports five different translations – in addition to the Holman Christian Standard Bible (HCSB), it now supports the English Standard Version (ESV), the New King James Version (NKJV), the New International Version (NIV), and the King James Version (KJV).
  • A new settings page, where you can select the version.
  • A widget, that will drop right in any widget zone.
  • The replacement of several function calls that have been deprecated in the WordPress API over the past 3 years and 8 months.
  • Formal dropping of support for PHP 4, following WordPress’s lead in 3.2.

While a lot of the code is different, if you only used the template tags, you should notice nothing different with this release.  You can download HCSB Verse of the Day (Plus) 3.0.1 at the WordPress Plug-In Directory, or upgrade on the WordPress Plug-In Administration Page in your blog.

Tagged , , , , , , , ,

Tech Blog 3.0 (aka “You win, PHP…”)

  By Daniel | August 24, 2011 - 10:12 pm | Apache, General Info, Mono, PHP 2 Comments

After a little over a year running on Tech Blog 2.0, you are now viewing version 3.0.  For this version, we’ve returned to WordPress from BlogEngine.  There are several issues that colluded to drive this change, most of which surrounded PHP and its crazy behavior.  (Geeky details follow – skip to the paragraph starting with “Bottom line:” if you don’t want the geek stuff.  I bolded it so it would be easy to spot.)

PHP’s recommended configuration is to run under Apache using the pre-fork multi-processing module (MPM).  The advantage to this is that Apache does not have to spin off another process to handle each request; it handles it in the same thread.  However, this means that each instance of the server must have all enabled modules loaded.  This means that each instance of the server (AKA “thread”) is very large, so the number of threads run is lower (typically 5-15 in a server the size we’re on).  Also, this means that each thread can only handle one request at a time; if you have 7 threads configured, each serving one of 7 requests, and an 8th request comes it, it has to wait for one to finish.  If the requests are served quickly, this may not be a problem; however, the avalanche of request that follow the typical front-page mention on mega-blogs can easily overwhelm it.

To fix this problem, there is another MPM, this one called worker.  In this scenario, there are spare thread waiting to fill requests, and these can spawn other threads to do further work if required.  So, the Apache threads would realize that a request needs to be handled by PHP, and pass it off to that process to be completed.  The Apache memory footprint is much smaller; it serves the images, scripts, and other static files, and passes off the requests that require heavy lifting.  PHP, then, has a (FastCGI) process where it receives these requests, processes them, and returns the response to the caller.  Because each of these threads only has to load the PHP requirements, they are smaller too, so you can have more threads processing at the same time; you just might survive that front-page mention!  (This is the same technique applied by LightTPD and Nginx, two other servers I tried at various times.)

It is in this scenario where PHP fails to live up to its expectations.  These PHP processes would simply stop responding, but the controller thinks they’re still there.  The end result to the user is a site that just sits and waits for output that will never come.  Eventually, they may receive a Gateway Timeout or Bad Gateway error.  The problem is worse on slower sites, but even popular sites seemed to fall victim to this from time to time.  This was also a problem whether PHP controlled its threads, or Apache controlled them.

The one thing that really perturbs me is instability.  If something is broken, I can fix it; if it works, I can fix it ’til it’s broke.  :)  But something that works sometimes, and other times doesn’t, simply won’t fly.  I was able to introduce some stability by restarting the server 4 times a day, but that’s a band-aid, not a long term solution.  I was tired of fighting.

Bottom line: the configuration required for a stable server is in opposition to a lean-and-mean configuration.  So, I installed the required Apache modules, and will continue to run my PHP-serving server at a configuration twice as large as it needs to be.  I’ll eventually move the Mono (.NET) processes to another machine, where the fast configuration won’t stability problems.

But, PHP isn’t all.  While I would still heartily recommend BlogEngine.NET to someone who was going to serve the blog from a Windows machine, but I had some issues getting upgrades to go smoothly under Mono.  It also is optimized for fast serving, at the expense of RAM.  At this point, that’s not the tradeoff we need.

Finally, with this update, the blog has received its first new theme.  It’s a clean, clear theme that should serve the content well.  Plus, the social media icons up in the corner are just too cool, IMO.  I’ve also applied tags to all posts except the “My Linux Adventure” series, and this theme displays them.  (Comments are not here now, but will be migrated shortly.)

So, there you have it.  Enjoy!

Tagged , , , , , , , , ,

40/40 Web Service

  By Daniel | September 19, 2010 - 7:55 pm | Web Services Leave a comment

The Ethics and Religious Liberty Commission of the Southern Baptist Convention is holding a “40/40 Prayer Vigil,” encouraging prayer through the end of October.  While some of the prayer is focused on the upcoming elections, the focus is on national revival.  They have produced a prayer guide, which details suggestions for prayer over the course of 40 days, beginning September 20th, and for 40 hours, beginning October 29th at 4pm.

We have created a web service to break this guide up into day and hour-sized chunks.  The service is at http://services.djs-consulting.com/FortyForty.asmx.  There are several ways to retrieve this information.

  • GetDay
    This gets one of the 40 days, by the day number.  (September 20th is 1, September 21st is 2, etc.)  The “day” parameter controls which day is returned.
  • GetHour
    This gets one of the 40 hours, by the hour number (10/29 4pm is 1, 10/29 5pm is 2, etc.)  The “hour” parameter controls which hour is returned.
  • GetDate
    This gets one of the 40 days, by the current date.  The “date” parameter controls which day is returned.  (The time portion may be given, but it is ignored.)
  • GetTime
    This gets one of the 40 hours, by the date/time.  The “time” parameter controls which hour is returned.
  • GetDayHTML, GetHourHTML, GetDateHTML, and GetTimeHTML
    This is the same as the above 4 calls, except what is returned is a formatted block of text that can be displayed on a web page.

In all cases, if the day/hour/date/time does not match a valid value for the vigil, a null is returned.

If you’re not interested in consuming the web service, but you’d like to see the suggested prayer each day, the Hoffmantown Prayer site is displaying the days and hours on Mountain Time.  This information is on the front page with no login required.

This web service will be discontinued at some point after December 31, 2010.

Tagged , , ,

Mono / FastCGI Startup Script

  By Daniel | September 3, 2010 - 7:28 pm | C# / .NET, Mono, Web Servers Leave a comment

We’ve begun running Mono on some DJS Consulting servers to enable us to support the .NET environment, in addition to the PHP environment most of our other applications use.  While Ubuntu has nice packages (and Badgerports even brings them up to the latest release), one thing that we were missing was a “conf.d”-type of configuration; my “/applications=” clause of the command was getting really, really long.  We decided to see if we could create something similar to Apache / Nginx’s sites-available/sites-enabled paradigm, and we have succeeded!

To begin, you’ll need to create the directories /etc/mono/fcgi/apps-available and /etc/mono/fcgi/apps-enabled.  These directories will hold files that will be used define applications.  The intent of these directories is to put the actual files in apps-available, then symlink the ones that are enabled from apps-enabled.  These files have no name restrictions, but do not put an extra newline character in them.  The script will concatenate the contents of that file to create the MONO_FCGI_APPLICATIONS environment variable, which tells the server what applications exist.  (The syntax is the same as that for the “/applications=” clause – [domain]:[URL path]:[filesystem path].)  Here’s how the site you’re reading now is configured (from the file djs-consulting.com.techblog.conf)…

techblog.djs-consulting.com:/:/path/to/install/base/for/this/site

Finally, what brings it all together is a shell script.  This should be named “monoserve” and placed in /etc/init.d.  (This borrows heavily from this script, which we used until we wrote this one.)  Note the group of variables surrounded by the “make changes here” notes – these are the values that are used in starting the server.  They are at the top so that you can easily modify this for your own needs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

#!/bin/bash
 
### BEGIN INIT INFO
# Provides:          monoserve.sh
# Required-Start:    $local_fs $syslog $remote_fs
# Required-Stop:     $local_fs $syslog $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start FastCGI Mono server with hosts
### END INIT INFO
 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mono
NAME=monoserver
DESC=monoserver
 
## Begin -- MAKE CHANGES HERE --
PROGRAM=fastcgi-mono-server2 # The program which will be started
ADDRESS=127.0.0.1            # The address on which the server will listen
PORT=9001                    # The port on which the server will listen
USER=www-data                # The user under which the process will run
GROUP=$USER                  # The group under which the process will run
## End   -- MAKE CHANGES HERE --
 
# Determine the environment
MONOSERVER=$(which $PROGRAM)
MONOSERVER_PID=""
FCGI_CONFIG_DIR=/etc/mono/fcgi/apps-enabled
 
# Start up the Mono server
start_up() {
    get_pid
    if [ -z "$MONOSERVER_PID" ]; then
        echo "Configured Applications"
        echo "-----------------------"
        # Construct the application list if the configuration directory exists
        if [ -d $FCGI_CONFIG_DIR ]; then
            MONO_FCGI_APPLICATIONS=""
            for file in $( ls $FCGI_CONFIG_DIR ); do
                if [ "$MONO_FCGI_APPLICATIONS" != "" ]; then
                    MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS,
                fi
                MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS`cat $FCGI_CONFIG_DIR/$file`
            done
            export MONO_FCGI_APPLICATIONS
            echo -e ${MONO_FCGI_APPLICATIONS//,/"\n"}
        else
            echo "None (config directory $FCGI_CONFIG_DIR not found)"
        fi
        echo
 
        # Start the server
        start-stop-daemon -S -c $USER:$GROUP -x $MONOSERVER -- /socket=tcp:$ADDRESS:$PORT &
        echo "Mono FastCGI Server $PROGRAM started as $USER on $ADDRESS:$PORT"
    else
        echo "Mono FastCGI Server is already running - PID $MONOSERVER_PID"
    fi
}
 
# Shut down the Mono server
shut_down() {
    get_pid
    if [ -n "$MONOSERVER_PID" ]; then
        kill $MONOSERVER_PID
        echo "Mono FastCGI Server stopped"
    else
        echo "Mono FastCGI Server is not running"
    fi
}
 
# Refresh the PID
get_pid() {
    MONOSERVER_PID=$(ps auxf | grep $PROGRAM.exe | grep -v grep | awk '{print $2}')
}
 
case "$1" in
    start)
        start_up
    ;;
    stop)
        shut_down
    ;;
    restart|force-reload)
        shut_down
        start_up
    ;;
    status)
        get_pid
        if [ -z "$MONOSERVER_PID" ]; then
            echo "Mono FastCGI Server is not running"
        else
            echo "Mono FastCGI Server is running - PID $MONOSERVER_PID"
        fi
    ;;
    *)
        echo "Usage: monoserve (start|stop|restart|force-reload|status)"
    ;;
esac
 
exit 0

This needs to be owned by root and be executable (“chmod +x monoserve”).  You can use “update-rc.d monoserve defaults” to set this to start at boot.

Tagged , , , ,

Tech Blog 2.0

  By Daniel | August 5, 2010 - 7:09 pm | C# / .NET, General Info 1 Comment

After three years on WordPress, the DJS Consulting Tech Blog has moved to BlogEngine.NET.  There are several reasons for this change, some technical and some not.

  • PHP’s Fast CGI processor has a problem where, if all of the processes are busy, the server will simply time out.  While this hasn’t afflicted my server as much as others, it has caused problems; when this problem occurred, none of the PHP sites were accessible.
  • Through experience with a very heavily-used site, I became less enamored of WordPress’s “read from the database every time” way of doing business.  I also found that various caching plug-ins for WordPress, on this particular site, did very little to ease the load.
  • Since I first looked at Mono (Linux’s implementation of the .NET framework), it has matured significantly.  It supports most of C# 4.0 already, which was released earlier this year.
  • BlogEngine.NET is a rapidly-maturing blog platform, and the project has a stated goal of 100% compatibility with Mono.  This is good, because you can mention Mono problems to the team, and you’re not dismissed because you’re running Linux.

As part of the move, the URL has changed; the new link is http://techblog.djs-consulting.com .  I have implemented redirection for each post, the category and category feed links, and the main blog feed and home page from the old URL, so you may not have even realized that you’re looking at the new site.  The DJS Consulting Software Repository remains at http://djs-consulting.com/linux/software.

I’m looking forward to this new setup!

Tagged , , , , ,

Oracle SQL Developer Debian Package

  By Daniel | October 29, 2008 - 7:17 am | MySQL, Oracle, PostgreSQL, SQL, SQL Server Leave a comment

Oracle SQL Developer is a Java-based tool that provides a graphical interface to a database. While it’s main focus is Oracle (of course), it can be hooked up, via JDBC, to many other databases, such as MySQL, PostgreSQL, and SQL Server. It’s similar to Toad, but is provided by Oracle at no cost.

Oracle provides SQL Developer in either an RPM, or a generic binary install. I like the ability to manage packages, but I’ve never had much luck at getting RPM to run on Ubuntu. I downloaded the RPM file, and, using alien, I converted the package to a .deb package (Debian package format) and installed it. It worked like a charm!

I haven’t tested it with gcj, but using Sun’s Java 6 update 7 from the Ubuntu repositories, it ran just fine. After you install the package, do a directory list on /usr/lib/jvm. You’re looking for the Sun JDK – if it’s installed, you’ll have a symlink java-6-sun that points to java-6-sun-1.6.0.07. Once you’ve determined the location of the JDK, run “sqldeveloper” from the command line – the program will prompt you for the path to your JDK. Enter it (probably “/usr/lib/jvm/java-6-sun”) and you’re good to go. (You have to install the package as root – but, for the rest of these steps, use your normal user, not root, as this puts settings in a .sqldeveloper directory off your home directory.) The package installs an icon in the “Programming” or “Development” group. Once you’ve told it where the JDK is, you can use this to launch it.

Download SQL Developer 1.5.1 Debian Package

Tagged , , , , ,